Speculative Decoding
Dual MI50, part 2: from 10 to 74 tokens/s, a 200k-token context, and two attention kernels
Two bugs made DFlash2 speculative decoding useless on AMD MI50. I fixed both, added custom HIP kernels that reach 74 tokens/s, got a 200k-token context for coding agents, and wrote two attention kernels that make that context fast. All of it on two 2018 GPUs at 90 W.
Tags:
vLLM, AMD MI50, ROCm, gfx906, Qwen, Speculative Decoding, Claude Code
Sami
Sami
11 min read