Dual MI50, part 3: DeepSeek-V4.1-Flash (552B) on two 2018 GPUs

Where we left off

Part 2 ended with Qwen3.8-27B running at 74 tokens/s and a 200k-token context on two AMD MI50 16 GB cards. This part is about a much bigger model: DeepSeek-V4.1-Flash-1017, 552 billion parameters, on the same two cards from 2018.

That number is not a typo. The model is a mixture of experts: 40 layers with 384 experts each, and every token uses only 6 of them per layer. Beside it sits a ~200-billion-parameter “Engram” n-gram table that the model reads a few rows of per token. Even squeezed to under 2 bits a weight, the experts alone are about 115 GB. My cards have 32 GB together.

A first attempt with the Xolotl engine (a fork of Strata) ran at 7 tokens/s. This week two things changed: Strata shipped its own DeepSeek engine, ported from Project Maya, and Maya itself kept improving. So I ported both to the MI50 and measured all three.

TL;DR

Both cards are capped at 75 W, passively cooled, and every run happens under a temperature watchdog.

The box

• 2× AMD MI50 / Radeon Pro VII 16 GB (gfx906), PCIe 3.0 x16 each, no peer-to-peer
• 12 cores of a Xeon E5-2680 v4 (2016, AVX2), 62.5 GiB of RAM, a 2 TB NVMe drive
• ROCm 7.14 (mixa3607’s gfx906 build), everything in Docker inside a Proxmox LXC

The cards hold about 16% of the experts. Everything else has to come from RAM or the SSD on every token, and that fact shapes everything below.

Three engines

Xolotl (my port from the previous round) keeps the hottest experts on the GPUs and copies the rest over PCIe from a pinned RAM arena. On PCIe 3.0 that copy is the ceiling: about 570 MB per token. The newest Xolotl was even slower on gfx906 (5.96 vs 7.07 tok/s) because it skips a prefetch step that helps on this platform.

Strata 0.2.4 added strata-dsv41, a separate DeepSeek engine ported from Project Maya. Its AMD build only targets RDNA cards (wave32). The CMake file explicitly excludes it from gfx906 builds.

Project Maya (maya-ds4) is the original. It already supports one wave64 card, the MI100 (gfx908): four of them ran DeepSeek at 52 tok/s, with 128 GB of VRAM between them. An open pull request adds the MI50 for Maya’s GLM engine, and it says plainly that DeepSeek was not tested. As far as I can tell, nobody had run Maya’s DeepSeek engine on gfx906 before.

Instead of copying experts to the GPU, Maya and Strata compute the experts that don’t fit in VRAM on the CPU (the “CPU lane”), reading them where they sit in the model files: from RAM through the page cache, or from the SSD.

Porting to gfx906: the 64-lane problem

These engines are written in CUDA style: a warp is 32 threads. gfx906 runs 64-lane wavefronts. Maya’s MI100 support solves this with a compatibility layer that maps one CUDA warp onto half a wavefront:

• shuffles run with width 32, so lanes 32–63 form their own logical warp;
• __ballot returns 64 bits, so each half shifts its own 32 bits into place;
• __syncwarp becomes wavefront-scope fences around a wave barrier;
• HIP’s own warp-sync builtins are switched off, because they would span all 64 lanes;
• mapped host memory is allocated coherent, because GPU and CPU spin on each other’s flags.

For the MI50 that meant registering gfx906 as a wave64 target and mapping CUDA’s __dp4a to the card’s v_dot4_i32_i8 instruction. Maya’s own kernel tests then passed on the MI50: the matrix kernels within 1e-7 of a double-precision reference, and the expert lanes for IQ1_S, IQ1_M and IQ2_XXS, the packed KV caches and the router bit for bit. The one miss is an IQ4_XS/Q4_K path whose result is correct but not bit-identical between two code paths, and no model here uses it.

For Strata I moved the same layer into its tree: 62 lines in four files, all behind a wave64-only switch.

Feeding it the model

I used ISTA-DASLab’s GSQ-RCO quants: IQ1_M (121 GB, mostly IQ1_S experts with Q5_K dense weights) and IQ2_XXS (138 GB), plus their 111 GB IQ4_NL Engram table, and Maya’s own Maya-XS for comparison. Two pieces were missing. ISTA’s files leave out the Engram’s hash parameters and the DSpark drafter’s geometry, so Maya’s packer couldn’t use them. I gave it an --engram-meta option, as Strata has, that takes those keys from Unsloth’s 5.8 MB metadata shard. For Maya-XS, a --partial-split option uses its model shards with ISTA’s Engram table, which saved a 161 GB download. Every file was SHA-256-checked against the pinned hashes.

Maya then lays the model out like this:

• GPU 0: layers 0–19, GPU 1: layers 20–39 and the output head. Tensor parallelism can’t split these 256-value blocks evenly, so it falls back to a layer split. Each card holds ~3 GB of dense weights and ~1,300 experts.
• CPU lane: the other ~12,800 experts (95 GB), on 10 threads (Maya’s own choice; 12 was slower).
• Maya also measures whether streaming experts over PCIe to the GPU beats the CPU. On this box it didn’t, so it turned streaming off by itself.

Results

Each arm starts with the model files dropped from the page cache, runs a 256-token warm-up, then three 128-token answers per prompt. The table shows medians in tokens/s.

Decoding is roughly a tie between Maya and Strata. Prompt reading decides it: Maya reads 1.7–5× faster, and coding agents read a lot more than they write. One Strata surprise: it sized its CPU lane from the host’s core count and started 28 threads on my 12 cores, which gave 1.8 tok/s until I passed --cpu-threads 10.

Quality. I measured perplexity on a fixed 3,000-token text I wrote myself (code and prose the models haven’t seen). This is a relative check, not a benchmark:

Pushing Xolotl’s cache-aware routing harder reached 8.97 tok/s (+27% over its default) but raised perplexity by 41% (51% over exact routing), so I dropped it. ISTA IQ1_M on Maya matches Xolotl’s default quality and is the fastest, so it’s my default. IQ2_XXS is better but slower and less stable here: 132 GB of experts are too much for 62 GiB of RAM.

Where the time goes

Maya can time each stage of a decoded token on the GPU (DS4_PROF=1):

The GPU side takes about 49 ms a token. The rest is spent waiting for experts on the CPU. When the page cache is cold, the wait is a disk wait.

The freezes, and a ten-line fix

In my first agent runs, decoding sometimes froze for 10–50 seconds in the middle of an answer and then carried on at full speed. The engine logged nothing during a freeze. What I found:

• Linux pressure stall info showed 12–20% of time with every task blocked on I/O.
• The page cache was full at the container’s 62.5 GiB limit.
• During a freeze, engine threads sat in folio_wait_bit_common, waiting for file pages of the mapped experts.
• A direct-I/O probe on the SSD never saw a read slower than 17 ms, so the drive was fine.

The cause was the Engram table. Each lookup reads one 144-byte row at a hashed offset: 48 per token, and hundreds of thousands for a long prompt. Maya does this with ordinary buffered reads, so every lookup brings whole pages, plus a readahead window around them, into the page cache. That junk pushes the CPU lane’s experts out of a RAM that is already full. Maya already reads the Engram unbuffered on Windows, but not on Linux.

The fix is to tell the kernel those reads are random: posix_fadvise(POSIX_FADV_RANDOM) on the Engram reader’s file handles, with DS4_EG_RANDOM=0 to turn it off.

Both retested tasks got 20–43% faster, from a colder start, with no freeze of 8 s or more. It isn’t a complete cure: the long-context task still had stalls of about 10–20 s, and a deliberately cold benchmark that switches prompts can still produce an occasional slow request. The 50 s freezes are gone.

Does MTP or DFlash2 help?

DeepSeek-V4.1 ships its own drafter, DSpark: three MTP blocks that draft a block of up to five tokens at once, starting from a noise token. It’s the same idea as DFlash, and Unsloth’s GGUF of it even names its architecture dflash. There is no separate DFlash2 drafter for V4.1. The DFlash drafters on Hugging Face are for the older V4-Flash, mostly in vLLM’s speculators format.

I measured DSpark on both engines:

• Xolotl uses it: 2.10 tokens per pass, already counted in the 7.07 tok/s.
• Maya with a 4.25 GB 2-bit DSpark: it kept 79–92% of first drafts and ~2.2 tokens per pass, yet decoded no faster (8.81 vs 10.10 tok/s in the same repeated-prompt test).

The reason is that this box is CPU-bound. Verifying two or three tokens touches the union of their experts, so the CPU lane does almost that much more work. And the drafter’s 4.25 GB of VRAM evicts about 600 experts. Strata’s engine reaches the same verdict on its own and refuses to load DSpark here (“its verify rounds read 3.3× a token’s bytes and gain nothing”). Drafting pays off when the experts live in VRAM: Strata reports 31 → 55–69 tok/s on four 32 GB cards.

Why not 20 tokens/s?

Per token there are ~49 ms of GPU work (at 75 W the cards run around 1 GHz) plus the CPU-lane wait, for ~95 GB of experts that don’t fit in 32 GB of VRAM or 62 GiB of RAM. What would move it: 32 GB cards, enough RAM to hold every expert, a higher power cap, or a faster CPU and PCIe 4.0. The software lever that remains is cache-aware routing, and as the table shows, that costs quality.

Safety notes

• The temperature watchdog kills any job at 80 °C. A 10-minute agent task took card 1 to 79 °C at 75 W, so long DeepSeek sessions run at the edge on passive cards.
• The host went down once during a cold IQ2_XXS run, at 41–54 °C, so not from heat. I couldn’t find the cause, and the same run later went through fine.

How I worked: me and Claude Code

Same rules as part 2: everything in Docker, nothing on the host changed without my OK, and every number from a logged run. I chose the goals and made the calls: which models to download (I grew the disk for this), power caps, and continuing after the reboot. The agent read three engines’ worth of code, ported two of them, ran the A/B benchmarks and tracked the freezes down to a readahead window. When the host went down and the power caps reset to 190 W, it stopped and asked before any GPU work.

Code

• Maya fork with the gfx906 port, the Engram fix and the pack options: github.com/0c33/project-maya, branch gfx906-deepseek
• Strata 0.2.4 wave64 patch and all scripts: in the same repo under docs/mi50/
• Models: ISTA-DASLab GSQ-RCO GGUF, Maya-XS

Thanks to the Project Maya, Strata and Xolotl authors for engines worth porting, to ISTA-DASLab and Unsloth for the quants, and to DeepSeek for releasing the model under MIT.