Repository navigation
dsv41: the DeepInfer-style packed KV cache, on by default -- COLI_VK_KV_FP32=1 is the exact fp32 opt-out - #2712
jameswitney wants to merge 61 commits into
Conversation
…l context Track C fix for the --ram 28 / --ctx 131072 / 262144 hard-fail. The planner priced the v41 forward's workspace at batch == context, so the runtime reservation ballooned with the context (25 GiB at 131072) and the RAM expert cache came out with zero bytes; cap_for_arch then raised "RAM budget cannot hold one expert slot per layer". deepseek_v41.c prefill reads a prompt in bounded chunks (default 1280 rows), so a forward never holds the full-context buffers at once. Price the workspace at one chunk (COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT, matching the qwen38 convention) so a feasible plan is produced again, degrading to fewer slots instead of hard-failing when the budget is genuinely tight.
…e chunk, and a 2-row stream threshold Everything here is allocation policy, packing and observability: no row of arithmetic changes. Measured on the 16 GiB RX 6900 XT with DeepSeek V4.1 Flash and the trunk-first placement. 1. The VRAM ledger (V41_STATS, or COLI_VK_LEDGER=1/0 on its own). v41c_ledger prints one line at a turn's end: the weight pool's blocks and the bytes of rows in them, the chain's dev_bytes and its three pools' blocks, the tier's residents and streaming slots (count and bytes) with the pool's blocks, the extra layers' pool, the KV mirrors, this process's device bytes against the pools, and the card's live mem_info_vram_used. vkc_pool_bytes and vkt_vram are the two accessors it needs. It answers the audit's open item: the 2.14 GiB a short-prompt run grew by was not the tier (whose pools do not move after the warm start) but 1262.5 MiB of token embedding + 150 MiB of the 40 routers + 673 MiB of the DSpark stages' matrices, all uploaded by the first forward and counted by neither dev_bytes nor the tier's budget; 52 MiB is the KV mirrors. The weight pool holds 9608.2 MiB of rows at the turn's end against the 7523.0 MiB the setup places. 2. The streaming slots: 64 -> 48 (COLI_VK_TIER_STREAM_SLOTS; the measured runs used 44 at most), and vkc_chunk_rows now reserves vkt_slot_reserve() before it reads the free memory, so the chunk and the slots no longer fight over the same headroom: the default chunk goes 1280 -> 1024 rows. The tier splits its budget between residents and slots, so the slot set alone frees nothing (64+91 and 48+107 fill the same 13 pool blocks of 256 MiB); what it and the reserve do is take the engine's own hold at the turn boundary from 15.53 to 15.26 GiB (chain 2133.7 -> 2060.0 MiB) with the prefill wall at +2% (289.95 -> 295.60 s). The card's sysfs peak does not move (15.907 -> 15.906 GiB) because the desktop yields: with the tier forced to 2.0 GiB the engine held 1.46 GiB less and sysfs fell 0.19 GiB, the compositor taking 1.27 GiB of it. 3. COLI_VK_TIER_STREAM_MIN=2: the rows from which a cold expert is streamed, measured instead of assumed (the engine's own line said the GEMM's 16 rows were never measured). One 21.1 MiB DeepSeek V4.1 expert, by the new bench (COLI_VK_CHAIN_BENCH= xbrows in tests/test_vk_chain.c): a device step costs 0.12 ms at 1 row, 0.21 at 2, 0.40 at 4, 0.76 at 16 (the tiled GEMM from there) against a 2.2 ms upload and the CPU's 2.05-2.52 ms a row, so two rows already pay for the upload; the GEMM route below 8 rows is slower than the GEMV, so COLI_VK_TIER_GEMM_ROWS stays 16. On the chunk-256 workload (COLI_VK_CHAIN_ROWS=256, the 2600-word prompt) the CPU's share of expert-rows falls 34.5% -> 1.6% and the prefill 226.7 -> 175.9 ms a row, at 2 TB of expert reads instead of 74.7 GB: it pays in disk bandwidth, not in time.
…'s extra layers upload A tier pool's limit and its blocks disagreed by a page. tier_pool_limit sized a block from the budget rounded up to a page (15360 -> 16384 B) but left the limit at the budget itself, which the pool cannot grow a block past; pool_suballoc then fell back to the limit floored to a page (12288 B) and counted a refusal. At deepseek_v41's tiny-budget case (COLI_VK_TIER_GB=0.00005: a three-expert tier, and one extra-layer expert's bytes -- 15360 B -- in the extra layers' pool) the pool could never hold its expert: the first MTP stage's upload was refused, the run's line read "failed 1 refused by the pool", and T.xmax fell to 0 with the extra layers left on the CPU. The expert needs exactly 15360 B (three tensors of 4096 rows + 1024 scales): the budget was right, the page floor of the block capacity lost the 3072 B that made it fit. The streaming-slot reservation commit is not this: with c615cca's six files reverted to 05e2c6d the same run fails identically (same "no room for streaming slots" line, same counters resident 2 / uploads 6 / evictions 4 / failed 1), and vkt_slot_reserve() is 0 in it -- the extra-layer take leaves the tier at fit_raw 2, so streaming is off and the chunk already reserves "0.00 GiB of it the tier's slots and staging". The exposure is older (the 9/10-of-the-heap mapped window, and the extra layers' share of a tiny budget: 9cfa502's trunk-first placement fixed the window for deepseek_v41 only) and it surfaced when the deepseek family was run on the discrete ICD, where the MTP stages are the tier's extra layers by default and not on the CPU as they are on a device that shares the RAM. 1. tier_pool_limit (backend_vulkan.c) rounds its limit up to a page, as its blocks already are, so a budget of N expert bytes holds N expert bytes -- at most a page (4095 B) over the tier's own budget, another page-granular figure. The tiny-budget case now keeps the extra layer on the device: extra layers (2) 4 of 48 routed experts, resident 1 (budget 1), uploads 7, evictions 4, failed 0, and the run's output is the CPU's. 2. vkt_slot_reserve (vk_tier.c) caps the reservation at what the budget holds beside the residents, and reserves nothing when the budget cannot hold the slots at all: it sizes a chunk, it is never a claim on the room an upload or an eviction needs (defensive: it is not what refused anything above, where it was 0). 3. cu_tier.c renames vkt_slot_reserve and vkt_vram for the CUDA build of vk_tier.c, which 'make -C c test' compiles: with the two accessors c615cca added, that build did not compile (redefinition against vk_tier.h's stubs). Measured on the 16 GiB RX 6900 XT: the default-budget serve (ctx 16384, short prompt, one turn) is unchanged -- 48 slots reserved, tier 1434.4 MiB of 1536.0 in 6 blocks as 68 residents, extra layers 168.8 MiB, process 11.28 GiB of 11.34 GiB in pools, chunk 768 rows with this desktop's 3.5 GiB already on the card (1024 on an idle one); the same run without this change gives the same figures.
…e that shares the RAM deepseek_v41's DSpark loop asserted that the MTP stages never go on the tier by default. That is true where the device's memory is the host's (Lavapipe): the tier's extra layers default to the CPU there. On a discrete card COLI_VK_TIER_MTP's default puts them on the tier, so the assertion fired on every DSpark case of the family when it ran on the radeon ICD, before any of them could be judged. The check now keys on the engine's own ", shared RAM," line, which says which of the two it is; the cases that ask for the stages on the tier (COLI_VK_TIER_MTP=1) are untouched and still require them there.
…tic cap buys twice the rows
The indexer's candidate mask is only ever read as a boolean -- the score kernel tests
it, the host's copy of it feeds a "kept or not" byte, nothing multiplies by it -- and it
was an fp32 word a column: rows x wcap of them beside rows x wcap fp32 scores. It is a
bit a column now (vk_chain.h: vkc_dsv4_cand writes it, vkc_dsv4_score reads it, mask_row
and mask_off in words of 32 columns), so the pair costs 4E + E/8 bytes a row instead of
8E. No row of arithmetic changes: the same blocks are picked and kept whole, and the
selection the top-k runs on is the same set of columns for every row (the indexer case in
tests/test_vk_chain.c compares the mask column for column, through the single-workgroup,
the wide-query and the grid kernels; the families give the CPU's tokens).
1. The chain (c/deepseek_v41_chain.h): isc stays fp32 at rows x wcap; mask and its host
copy are rows x ceil(wcap/32) words, and the host unpacks the bits where it builds
m->candidates for the CPU's layers after a partial chain. The chunk's quadratic cap
(v41c_forward_seg) was rows <= 32M/E, which is the 256 MiB the fp32 pair could take;
it is now the same 256 MiB over 4E + E/8 a row, i.e. ~62M/E (1.94x the rows), and it
prints one line when it binds -- nothing printed it before, which is why the audit had
to derive it.
2. Measured, the tiny V4.1 fixture at a 20000-position prompt: the chunk the tier's
stream line reports goes 1677 -> 3253 rows, the row cost the sizing prints falls 181
-> 105 KiB, and the new line reads "a prompt of 20000 positions caps a chunk at 3253
rows (80.6 KiB a row of scores and packed candidate mask, 256 MiB of 256)". On the
released checkpoint at E=3107 the row cost goes 1293 -> 1281 KiB a row (the mask was
12.4 KiB of that, one bit a column is 0.4).
3. Two items of the wave-2b list are not there, measured rather than argued (the
correction is in docs/deepseek-v41.md so it does not come back):
- the 1262.5 MiB the turn-end ledger attributes to the "token embedding" is the lm
head. tie_word_embeddings is false on this checkpoint, both tensors are 129280 x
5120 bf16, and the embedding has never been on the device: its only reader is
wb_row_f32, which copies the rows a step needs into the host's rows (one a token
when decoding, one a position for a chunk). A run with every device upload printed
shows head.weight as the 1262.5 MiB item and the embedding never uploaded at all.
The head is a GEMV of 662M multiply-adds a token, so it stays where it is, as do
the 40 routers' 150 MiB.
- the per-layer vkc_finish (deepseek_v41_chain.h, after the attention frame) does not
wait on anything the host does not read. A 1280-row chunk spends 28095 ms waiting
for the frame it reads and 0.5 ms for the rest: two frames are in flight and a
single queue runs submissions in order, every frame starting with a barrier on
everything submitted before it, so the previous layer's shared expert is always
through. Waiting only the read frame was implemented, measured at 0.5 ms a chunk and
dropped. The audit's R4 estimate (2-8 ms a chunk) was right; the wait that remains
is the attention itself, and the device is idle during moe_run_at instead.
Gates on the 16 GiB RX 6900 XT (RADV), against the same three families at 955a0f6:
tests/vulkan_engines.sh deepseek (29 OK), deepseek-chain (35 OK), prefill-deepseek (2
OK), make -C c vk-chain-check VK=1 (PASS), make -C c test (test-c green; test-python
3226 tests). Three failures and two more are identical at both commits and are
environmental: "deepseek_v4 session: no matmul ran on the device" and "chain deepseek_v4
4 experts: the chain never ran" (the deepseek_v4 engine's dense placement declines with
the desktop holding VRAM; its tokens still equal the CPU's), "pf deepseek_v41 staged:
logits" (max |logit diff| 3.22e-02 at both), and the two
test_harness_builds_without_warnings cases (a GCC ABI note in mistral_vision.h).
The commit before this one quoted the row cost the packed candidate mask saves as "1293 -> 1281 KiB a row" at E=3107. 1293 is the row cost at E=11 (the short-prompt arm), not at E=3107: the same 8 bytes a column make it 1317 KiB there, which is the number the audit prints as well, so the pair is 1317 -> 1305. Measured on the released checkpoint at E=1300 this run: 1303 -> 1298 KiB a row (the mask's 5.1 KiB a row of 41 packed words). The saving itself was never in doubt -- 8 bytes a column become 4 + 1/8 -- only the base it was subtracted from.
…hunk, not the context The planner priced the forward workspace at the whole context, so at --ram 28 a --ctx 131072 or 262144 serve came out with a zero-byte RAM expert cache and cap_for_arch() hard-failed with "RAM budget cannot hold one expert slot per layer". deepseek_v41's prefill reads a prompt in bounded chunks (default 1280 rows; COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT set it), so the workspace is priced at one chunk and stays bounded as the context grows. The regression test (tests/test_resource_plan.py) builds a two-layer 128-expert checkpoint and checks that both long contexts keep a positive expert cache per tier. Python-only; no engine source changes.
…orkspace gives 05e7748 pinned the contexts chat_context() recommends for DeepSeek V4.1 when the prompt's workspace was priced at the whole context: 16384 on 16 GB, 65536 on 64 GB, 131072 only from 512 GB, and the refusal to start at 262144 (53.7 GB of workspace, no room for one expert slot). 008a01b prices the workspace at one prompt chunk, so the same table reads 65536 on 8 GB, 131072 on 16 GB and the family's 262144 ceiling from 64 GB. The workspace is still in the sum and still in the reason it prints (0.3 GB at every step); the comment says which share applies at each step and what the table was before. Caught by `make -C c test` on the merge of fix/ram-plan-128k: 7 failures, these five and the two known environmental test_harness_builds_without_warnings cases. With this change the target is green (3019 tests, 0 failures, 358 skipped).
…ts it tested What the wave after Track A and Track B measured, tied to the commits that carry it: the turn-end ledger (c615cca) closing the audit's 2.14 GiB as the lm head, 40 routers and the DSpark stages rather than the tier; the 48 streaming slots and the chunk reserve (0.27 GiB off the engine's own hold, 0.001 off the card); the 2-row stream threshold (chunk 256: the CPU's share of expert rows 34.5% -> 1.5%, the prefill 226.7 -> 178.0 ms a row, at 1.15 TB of expert reads a turn); the tier pool's page (90959d6); the packed candidate mask (12b2908: 1.94x the chunk for the same 256 MiB, 1677 -> 3253 rows at a 20000-position prompt) with a6486c7's correction of the base it was quoted against; and the merge of the RAM-plan fix (9508327) with the first GPU run it ever had: --ctx 131072 and 262144 plan 21 and 20 slots a layer, load, and finish a turn in 17.9 and 18.1 s with no OOM. The corrections are called out rather than folded in: the 1262.5 MiB is the lm head and the embedding never touches the device; the per-layer drain costs 0.5 ms a chunk and waits on nothing the host does not read; and the card's sysfs peak does not move when the engine's hold does, because the desktop takes the difference (1.46 GiB held less, 0.19 GiB read less, 1.27 GiB to the compositor). The comparison (Track A) and the audit (Track B) are copied in byte-identical, because their predictions are what the wave tested; the raw file carries every log line the record quotes, the planner's before/after table and the two serves above. The 50 Hz VRAM sample series stay as scratch: the record names them, nothing commits them.
The full gate on the merged tree at the previous commit: deepseek 29 OK, deepseek-chain 36 OK, prefill-deepseek 2 OK (one environmental failure each), vk-chain-check PASS, make -C c test with test-c ALL PASS and test-python 3227 tests / 294 skipped / the two environmental test_harness_builds_without_warnings failures. The section also records that this target's own totals move between runs (3227, 3019, 3227 on the same box), so a count alone is not a regression signal.
…nd gitignore with it 955a0f6's guard grepped ', shared RAM,' which no code path prints (vk_tier.c prints 'shared RAM (COLI_VK_TIER_GB)', 'shared RAM: a quarter of what the expert cache leaves' or '(shared RAM)'), so the DSpark-on-shared-RAM default assertion was silently skipped on every device. It now matches any 'shared RAM' wording and never 'device memory'. vulkan.md gains the tier's new defaults (STREAM_SLOTS 48, STREAM_MIN 2), the record fixes its quoted line, the pair-cost labels, the 12.1 KiB mask share, and lists the two instrumented runs its M1/M2 numbers come from; .gitignore covers the audit scratch.
mem_info_gtt_used beside mem_info_vram_used: system RAM behind the aperture is where the host-visible pools land when device-local is exhausted, and that spill is host-RAM pressure the process RSS does not show (a 256K-token prefill on a card that had lost 3.3 GiB of its device-local to a leak spilled pull and the mirrors there while RSS looked steady). One more field on the turn-end line, 0.00 when the driver does not export it.
… formats The chain's window rings, compressed rows and index keys mirrored as DeepInfer's paged cache stores them (c/dsv41_kvq.h, the host reference): an MXFP8 window row (e4m3, a ue8m0 scale a group of 32, 528 B), an FP4 compressed row (e2m1, an e4m3 scale a group of 16 as stored, 288 B) and FP4 index keys in block-wide sections (a ue8m0 scale a group of 32, a block of 32 tokens, 68 B a row on average). chain_dsv4.comp packs a row after its last fp32 reader (mode 13) and dequantizes it where read (the attention's flags 2, the index scores' quant field) and on the pull's write-back (mode 14), the scales' reciprocals as DI's bit-exact LUT math (a GPU's 1/x is a couple of ulps low, which the codecs' bit agreement cannot take). Off by default; the flag branches only at the mirrors' widths and stores, a forward's first check sums the mirrors' bytes against the flag's widths, and the KV split still decides as before: when it arms the mirrors are fp32 whole again, re-mirrored from the host.
…ode gated tests/test_vk_kv_quant holds the device's pack to dsv41_kvq.h's reference bit for bit (the edge rows of the scale math included: a zero amax, denormals, -0, an amax a power of two over the format's max, the clamp edges and the RNE ties), its unpack, a re-upload's little-endian lanes and the pack's idempotence (a fixpoint in values: a group at half its scale's span re-packs a step down to the same decoded row), and the index scores over quantized keys to the decoded rows' bits with the original rows' drift and top-k overlap reported. The kv-quant-deepseek family runs the tiny fixture at the released KV geometry (make_dsv41_tiny.py --kv-quant-dims: head_dim 512, index_head_dim 128, the formats' own widths) under the flag against a documented logits tolerance -- the flag-off run is the exactness proof, as V41_PREFILL_TAIL's tests put it -- with the token drift printed and the ledger's kv-mirror line pinned to the 528/288/68 B math.
…d save ENVIRONMENT.md's flag row; deepseek-v41.md a subsection by the trunk-first placement: the three formats, the ~6,400 -> ~890 B a position of mirror, the ~690 MiB at 128K tokens and 5.38 GiB of the peak at the 1M ceiling (the prefill-VRAM analysis's §10 figures), the approximate-mode caveat (a pull returns dequantized rows, the pack's idempotence bounds the hop) and the KV split's precedence.
…he index staging an index row wide Two bugs the tiny fixture's first quantized run found, both in the chain's integration rather than the codec (which the unit test had already pinned bit for bit): - vkc_write's offset is in FLOATS, like every chain offset -- the pack re-uploads passed bytes, so the index keys' scale section landed four times past its block and the frame died as a copy out of range. The three upload helpers now divide by 4. - the fp32 staging for a chunk's new index rows was sized with index_n_heads (4) where a row is index_head_dim floats (128): the idx_wk matmul wrote past 224 bytes of scratch. qik is an index row wide, and the chunk-row accounting with it. The kv-quant-deepseek gate with them: the first decode row is the one input the two runs share, so it is the row the tolerance gates (the argmax kept, the drift within the row's own largest |logit| -- the tiny fixture's random weights leave gaps narrower than the format's noise; past that row a flipped pick feeds the model its own choice), and the ledger's kv-mirror line comes from a one-turn serve run of each flag setting (the ledger ends a turn; the CLI never has one): 6.4 -> 1.6 MiB on the fixture, the 528/288/68 B math rounded to whole index blocks. The test's last line is the plain PASS the gates grep for.
…s, the gate as it gates The docs' 128K/1M figures now sit on the corrected 6,400/890 B a position basis (0.78 -> 0.11 GiB at 128K, 6.25 -> 0.87 GiB at 1M), with the audit's older savings-as-totals numbers named as such; the family's comment states the gate it runs (the input-shared first row keeps its argmax, drift within that row's own largest |logit|) instead of a tolerance it does not use; the idempotence claim says values, not bytes; the kvq fixtures and run CSVs are ignored; the wide score case's comment names mode 10's grid arm and that mode 9's quantized reader is the same expression.
…B rows, park keeps whole blocks COLI_VK_KV_QUANT=2 will hold the host's KV state in the cache formats too; before any storage moves, the index keys' block-wide sections get their addressing primitives and the rules they pin down. kvq_ikey_row_gather/scatter move one row between a mirror and the 68-byte linear staging a boundary (a pull's write-back, a raw upload) writes through; the whole blocks covering rows [0, n) are the linear prefix kvq_ikey_mirror_bytes(n) long, so a park, a reset or a zero-fill that rounds to whole blocks may copy bytes and nothing else may go linear. ColiParkPart grows a 'block' field (0 = today's linear rows) with coli_park_per_blocks: what a park keeps rounds up to whole blocks, and the restore's part equality checks it. The tests: gather/scatter bit for bit against the pack, the prefix whole (a 33-row keep covers rows 0..63 and a staged write inside lands), park's rounding arithmetic, and the window ring's slide staying linear at 528-byte rows.
…ost state in the cache formats under COLI_VK_KV_QUANT=2 The host's own KV arrays become uint8_t rows at the flag's widths (window 528 B, compressed 288 B, index keys 68 B in block-wide sections, rounded to whole blocks), with every offset into them byte arithmetic through v41_row_win/ckv/ikey -- a stride of head_dim on the new pointers would scale by bytes and land at a quarter of a row, which is exactly the bug the flag-off serve gate caught mid-swap (the window ring checksum diverged while ckv/ikey matched): reconstructed with every stride as bytes, and the flag-off path byte-identical again (the state probe over a two-turn serve run matches the base binary value for value, and the prefix-serve module that failed 13/14 mid-swap is green -- the failure was never suite-order dependence, it was this stride bug; the base branch passes the module either way). Writers pack at their stores through a scratch row (rms and the rope tail stage in the scratch, never in place inside a code row): the prefill chunk, the decode ring slot, the index keys, the compressed rows and the DSpark stage rings. Readers dequantize on read: the kv_all build unpacks the ring and the owner's compressed rows, the indexer decodes into a per-call scratch keyed on the RESOLVED base (a speculative step's rows may read different mirrors mid-batch), the DSpark kv_all unpacks its stage ring. cstate_kv/cstate_score stay fp32 (the pooling reads raw values). The reset rounds its index-key reach to whole blocks; park keeps whole blocks (coli_park_per_blocks, park.h's new block field); the swap and the seq sets carry the arrays' bytes with them. v41_host_kv_check sums the allocated bytes against the flag's widths after every allocation site, the twin of the chain's v41c_kvq_check, so a width a branch missed fails loudly. The device side meets it: the chain's flag parse accepts =2 (so the mirrors quantize under it too), the re-upload's packers become raw copies when the host holds codes and the give-way's fp32 re-mirror dequantizes the codes up, the pull stages codes under the flag (raw pulls, the index keys as whole blocks -- mode 14's dequant work deleted from that path only, flag 1 keeps it), the write-back scatters or packs per the staging, and the split's parts (VkcKvPart/VkcKvDs, a trailing zero-defaulting kvq field) unpack the host's code rows on their way to the device's fp32 tables -- the host never gives way to the split, only the mirrors do. The engines' own VkcKvPart initializers gain the trailing 0; the -Werror port (-Werror=incompatible-pointer-types) is clean across the vulkan engines.
… under COLI_VK_KV_QUANT=2 _dsv41_geometry's state sum takes the cache's own row widths while the flag engages the host layout (356 B a compressed row against 2560 B fp32 -- 890 against 6400 B a position at the released geometry, 86.1% less), gated by the same geometry guard the engine's v41_host_kvq applies: head_dim 512 and index_head_dim 128, else the fp32 price stays (the formats are defined at those widths). The comment states the invariant the re-price rests on: the HOST layout never gives way to the KV split (only the device mirrors do, and the split's paths read the codes), so the packed price holds whenever the flag engages, split or not -- pricing fp32 there would under-reserve by 5510 B a position. v41_state_later (a parked mux conversation's sets) re-prices on the same widths, so it cannot disagree with the ledger's line. The test pins the seam: the packed and fp32 sums at 64K/128K/256K, the freed bytes' share of a RAM slot, the geometry guard and =1 staying fp32. The slot DELTAS (+1/+1/+2 against per_cap 771.14 MiB) are whole-plan arithmetic -- a floor of a difference is not a difference of floors -- and their evidence is the serve smoke's "[v41] RAM plan" line, flag 0 against flag 2.
…n the turn-end line The turn-end VRAM line walked the device mirrors only, so everything the host KV state cost was invisible in every line. Under COLI_VK_KV_QUANT=2 the line gains 'host kv <MiB> (packed)' -- v41_host_kv_bytes over the model's own arrays (the rings, the compressed rows, the index keys, the DSpark stages' rings), passed by the engine's call site, which is where the Model is. Flag 0 and 1 print nothing there: the fp32 host is today's cost, not news.
…do rows and the ledger's host term family_kv_quant_deepseek gains the =2 arms, tolerance-gated like the flag's own: the flag-off run stays the exactness side, the =2 run may drift a token (the count and the reference's Matching line printed), the first decode row within its own scale. Four arms cover the host's round trips: a short and a long prompt, a chunked prefill, and a DSpark spec run -- the arm that drives the verify's ring_save and rollback through the code rows, which no flag-1 arm does. v41_kvq_ledger extends to a =2 serve turn and pins the new host-kv term to the codes' own arithmetic (the 6 rings and undo rows, the two source layers' compressed rows and whole-block index keys, the stage rings: 183 168 B = 0.175 MiB on the fixture). v41_kvq_host_split gates =2 under the KV split: the split armed, the give-way line, the host codes held, and the first decode row against the flag-off split run. Two honest notes baked into the arm: the split on the KV-QUANT fixture is inexact at flag-off too (a combination nothing ran before -- the kv-split family walks the stock-dims fixtures, and the base branch mismatches the oracle there the same way), so the off run's own row is this arm's reference; and the arm carries both quantize hops (the mirrors' and the host's), which on this fixture's narrow logit gaps moves a greedy pick -- the tolerance is the row's own scale and the flip is printed, not failed. The chain's flag line says the real value (=2) and the host side prints its own engagement line, which is what the arms assert on. Also fixed on the way: the raw pull's vkc_copy counts are FLOATS (the bytes form overflowed the staging and lost the device), and the pull's widths follow what it CARRIES -- the give-way's fp32 mirrors pull floats and pack at the write-back, so the staging is not sized for codes under the split.
…s rows say they are fp32 vk-chain-check caught it: VkcKvDs.Ch became uint8_t* and the cold staging's 'a->Ch + r*hd' scaled by bytes -- a quarter of a row on the fp32 hosts (v4's and the test's), five of the split's ds cases wrong. The stride is r*hd*sizeof(float) for the fp32 hosts and KVQ_CKV_BYTES for the code rows. deepseek_v4's initializer casts its fp32 rows for the new field, with the note that v4's host never holds codes.
…ine, not a subtraction V41_STATS' attention field lumped the host's code-row decode into the attention pass, so the dominator's overhead (the ring decode and the kv_all build) was invisible. t_kvall accumulates just those two loops and prints its own line -- the CLI's per-run report and each serve turn (V41Req carries the turn's snapshot); the indexer's decode stays inside attention, it is not the dominator.
…rence ENVIRONMENT.md's COLI_VK_KV_QUANT row gains the =2 semantics (the host state in the formats, the planner re-price with its slots arithmetic, the host-kv ledger term, the kv-read line, and the host never giving way to the split). deepseek-v41.md gains the subsection after the mirrors': what moves (the writers' scratch rows, the dequantize-on-read, the byte boundaries with the index keys' block rule), the memory model's honest table (freed RAM, slots, effective --ram, the regime tags on every projection, the no-16K-throughout-claim, +experts only via a tier raise), and the 34-touchpoint map's reference.
…the record pair once The TEMPORARY state probe read the window/ckv/ikey rows as floats over float-sized bounds, which underreads nothing but overreads ~4x once the host rows are 528/288/68 B codes: it now runs only where the rows are fp32 (the flag-off and =1 paths), and says so. The duplicated kv_prefix_taint/record pair at the turn end was an inert edit artifact and is once again one. Evidence closed with the final binary: make test's failure set still exactly the base's 23; the kv-quant family green including the =2 arms; the =2 prefix-serve module 14/14 through park and replay (the permanent arm is the PR's noted follow-up); no shader changes on the branch.
…v41c_state_up/v41c_state_pull dev's qwen36 staging-room shape-pin updated to the reserved+vkt_slot_reserve line the wave introduced -- intent preserved. The conflicts and the two sides' overlaps, as resolved: - c/Makefile: TEST_EXCLUDE keeps both dev's test_vk_dev2_window and this branch's test_vk_kv_quant. - c/backend_vulkan.c: dev's P->refused = 0 beside the page-granular pool bump. - c/deepseek_v41.c: dev's indexer order array and this branch's ikd decode scratch both live, both freed. - c/deepseek_v41_chain.h: dev's v41c_state_up/v41c_state_pull carry this branch's kvq/hostq branches inside (the pack uploads, the give-way's dequant up, the raw-code pulls under =2); dev's v41c_prompt_lw stages the candidate mask at the packed rows' width. - c/vk_chain.c: dev's tiled indexer (mode 11) and three-pass prompt attention (chain_dsv4_pa) decline under COLI_VK_KV_QUANT's flags -- their readers predate the 528/288/68 B mirror widths; the grid modes serve them. - c/shaders/chain_dsv4_idx.comp: mode 11's mask read ported to the packed one-bit-a-column rows (chain_dsv4.comp's mbit), as every mask row now is. - c/coli_env.h: the rows COLI_VK_KV_QUANT, COLI_VK_LEDGER, COLI_VK_TIER_STREAM_MIN, V41_STATE_PROBE.
…the planner's fixed term
Conflict: .github/workflows/check.yml — the job timeout. dev raised it to 50 for JustVugg#2750/JustVugg#2759 (GGUF MTP suite); this branch raised it to 60 after JustVugg#2712 was cut at 35m23s mid-build. Kept 60 (the larger) and both rationale comments. Everything else auto-merged: c/Makefile, c/coli_env.h, c/deepseek_v41.c, c/tests/vulkan_engines.sh, docs/ENVIRONMENT.md, docs/deepseek-v41.md, .github/workflows/ci.yml. Built and tested after the merge (results in the PR comment).
Independent review of this branch (
|
… fp32 rows `Layer.window/ckv/ikey` became `uint8_t *` when the packed KV cache landed, and this chain kept base's float offsets (`l->ckv + (size_t)t0 * hd`): read as a byte stride over rows `model_load` allocates at 528/288/68 B, the pull's 16K write-back runs 14,417,920 B past a 2,359,296 B compressed-row buffer under the packed default, and at COLI_VK_KV_FP32=1 the four-quarter stride makes every row overlap 4:1. cc_write's own `off*4+bytes` bound (cuda_chain.cu:2201) makes the device side right, which is why nothing aborted. Every host-KV row offset in the chain now goes through v41_row_win/ckv/ikey (c/deepseek_v41_cuchain.h:467,474-475,695,701-702), which are hd*4 in the COLI_VK_KV_FP32=1 arm and the code widths otherwise; cc_write's `off`, in floats, is untouched. And v41u_wanted returns 0 when v41_host_kvq(&m->c) (:140-150): a packed host keeps the prompt's layers on the CPU -- the exact pre-packed path -- with one `[v41]` line saying so, because this chain reads fp32 rows and nothing else. Static proof only: there is no CUDA device in this environment, so the runtime gate (tests/cuda_tier_engines.sh, which needs a card) is not in this branch's evidence. New c/tests/test_cuchain_host_strides.py, discovered by `make test`'s unittest discover: 3 failures at 80d9cf8, 3 OK here. `make cuda-host-syntax` rc=0 before and after. Evidence: reports/f3-host-final.md:115,252-257; reports/f6-plumbing-final.md:79; reports/f9-default-compat-final.md §8.1; reports/fix-plan-final.out.md §4 commit 1.
… the pre-store float row
attention_run decodes the window ring into `win_dec` at the top of a decode forward
(deepseek_v41.c:3105-3110), and then the per-position loop packs each step's key into
its slot AFTER that snapshot was taken: a ratio-0 layer's kv_read IS win_dec
(:3211), so the slot the step just wrote still held the row it EVICTED. The query
listed it (window_idxs, :3242) and attended a position it never had.
Measured on dsv41_kvq_tiny, packed host vs COLI_VK_KV_FP32=1, chain off, per DUMP
row (row 0 = the prefill forward, row 1 = the first decode row):
before: row 1 max |diff| 1.2758 of 1.5927 = 0.801, argmax 206 vs 64 FLIPPED
(row 0, today's gate, 0.1649 of 2.2744 — argmax kept, so it is blind to this)
after: row 1 max |diff| 0.0924 of 1.5927 = 0.058, argmax 206 vs 206 kept
and 0.0924 is exactly the codec's own term for that row (the W6 round-trip pass: 0.0924
= round-trip oracle - fp32, while packed - oracle was 1.2399 = 13.4x it). With this
change packed - oracle is 0.0000e+00 on all 8 tiny and all 6 long rows: the packed host
is now bit-identical to the round-trip oracle, i.e. no implementation error is left --
what remains is the format's own loss.
The fix is the re-decode of the one slot with values, gated on `!ratio` (the only arm
where win_dec is kv_read; a ratio layer reads kv_all, which takes the step's floats
below). Not a memcpy of the pre-pack row: every other reader of the ring sees the
dequantized code row, so the snapshot has to be the same values.
Gate: the new logits_row() (tests/vulkan_engines.sh:1806) checks DUMP row 1 of the same
fixture, rel 0.2 against that row's largest |logit| with the argmax kept, called from
v41_kvq_cpu_host (:5034) -- the packed host with the chain off, the configuration this
bug damages. Fails at 80d9cf8 (1.2758, 0.801, FLIPPED), passes here (0.0924, 0.058,
kept); `bash tests/vulkan_engines.sh kv-quant-deepseek` is OK end to end on lavapipe.
Evidence: reports/f3-host-final.md:115,329-331; reports/w6-roundtrip-final.md:17-19,81-97;
reports/fix-plan-final.out.md §2.2, §6.5-§6.7, §4 commit 2.
…m it would without them V41_SPEC_FORCE drafts the REFERENCE's own next tokens (deepseek_v41.c:7024-7030), so the family's packed spec arm -- forced drafts verified against the vendor ref.json -- only ever proves the reference's stream. The packed model does not reproduce it (its first decode row's argmax already moves on this fixture), so every forced draft is rejected for a reason that has nothing to do with the cache: 0 of 9 at 80d9cf8, and 0 of 9 after the P1-2 fix too. And with COLI_VULKAN=1 the fixture's six layers chain to the device, so attention_run's CPU path -- where a decode step's ring slot is written -- never runs the bug this arm should catch: that configuration accepts every draft even unfixed (5 of 5, 3 of 3, measured both ways). The new v41_kvq_dspark_self takes the packed model's OWN stream as the reference: chain off, no Vulkan (the configuration P1-2 damages), the plain run's tokens T written into a ref file of its own, then every draft forced. All must be accepted, and the verified stream must be T token for token. Measured, dsv41_kvq_long: 80d9cf8: 1 of 6 forced drafts accepted, stream 233 207 13 6 85 62 vs plain 233 207 13 17 60 72 this tree: 3 of 3 forced drafts accepted, stream = the plain one (233 207 190 166 168 57) It also asserts the packed host engaged and that the verify path ran at all ("NOT exercised" is deepseek_v41.c:7137's own line for a force that drafted nothing). `bash tests/vulkan_engines.sh kv-quant-deepseek` is OK end to end on lavapipe with this arm in it (28 OK lines, rc=0); the same family FAILs at 80d9cf8 -- at the P1-2 row-1 gate, which this arm sits behind. Evidence: reports/fix-plan-final.out.md §2.1, §4 commit 3, §6.8; reports/f3-host-final.md:330; reports/w8-transitions-r1.out.md:44,50,107-109; reports/f7-tests-ci-final.md §8.2.
…locates _dsv41_geometry priced the forward workspace at min(ctx, 1280) rows of 204,800 B, on the claim in its docstring that deepseek_v41.c "prefill reads a prompt in chunks (default 1280 rows; COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT set the chunk size)". That is not true of this engine: `grep -c coli_prompt_segment_rows c/deepseek_v41.c` is 0, `grep -c 1280` is 0, and neither COLI_PROMPT_SEGMENT nor COLI_VK_CHAIN_ROWS appears there at all (COLI_VK_CHAIN_ROWS is the chain's spare ring rows, deepseek_v41_chain.h). forward_full allocates its buffers at n and the whole prompt is one forward. So the plan reserved 250.0 MiB where the engine asks for 101.65 GiB at 262144 (416x), which is what let 262144 be admitted from 20 GB up. The workspace is now the engine's own per-row closed form, term by term from the allocation sites (deepseek_v41.c:4816-4871 and attention_run's five buffers), at the context: 66,588 floats a row for forward_full (2*hc*hidden + 3*hc + 2*hidden + hc*hc + targets*hidden) and 37,504 for attention_run (2*q_lora + heads*head_dim + 2*head_dim + window + 2*index_topk) = 104,092 floats = 416,368 B a row, 2.03x the 204,800 B/row priced before, plus the single `hidden` row the form always priced. The per-row buffers of both hops are priced because the ladder is identical either way (verified: 416,368 and 266,352 give the same table). The non-row transients are named in the comment and unpriced: `heads` (MV_ROWS_MAX * heads * head_dim * 4 = 4 MiB), `head_score` (163,840 B), `embedded`/`collapsed` (2 * hidden * 4). The three new reads (q_lora_rank, num_attention_heads, index_topk) are _required_int like the rest of the geometry: a config without them is refused rather than under-priced, and the tests' fixtures gain the released values. The ladder this produces (chat_context, GB = 1e9, quarter share up to 65536 then a tenth), packed and COLI_VK_KV_FP32=1 alike -- the flag moves the host's KV rows, not the prefill's frame: box before after 8 GB 262144 None (the engine's default) 16 GB 262144 None (the engine's default) 64 GB 262144 32768 256 GB 262144 65536 512 GB 262144 65536 1024 GB 262144 131072 That is the corrected price doing its job: 16384 alone is 6.8 GB here, so a 16 GB box loses the context step 05e7748 exists to give it, and the real fix for that box is a chunked host prefill in deepseek_v41.c -- which this branch does not do. The plan's table wrote 131072/262144 for the last two rungs; it had used 2^30 for `available` while resource_plan.GB is 1_000_000_000, and the pinned numbers are the ones the code produces. Tests: the two ladders in test_context_compaction.py:171 re-pinned (both arms, same table); a new closed-form assertion in test_dsv41_kvq_host_plan.py:172 (workspace == ctx*416368 + hidden*4 at 16384 and 262144, packed and fp32, and 3 fewer hidden rows with no DSpark head); that file's fixture hidden_size 2048 -> 5120 plus the released attention widths; test_resource_plan.py's Track C test, whose "the workspace must stay bounded as the context grows" was the same false premise, now pins the closed form and keeps its positive-expert-cache assertions. 157 tests OK here, 14 failures at 80d9cf8. The plan did not list that last file; it cannot stand unchanged. Evidence: reports/f5-planner-final.md:100-101,215-224,263-270 and F5-11; reports/a1-planner-admission-final.out.md; reports/fix-plan-final.out.md §2.3, §6.4, §6.9-§6.11.
tests/test_vk_kv_quant covered the codec, the window/ckv/ikey round trips and three score arms, but no arm ever read a PACKED cache through the two readers the PR made default: VkcDsAttn's flags 2 (the decode-on-read window ring and compressed rows) and the indexer's quant 1 / mode 0. The only arms that reach them are in test_vk_chain.c, whose test_ds_attn runs flags 0 and 1 (no packed cache) and whose VkcDsScore initializers leave quant 0 -- so the two paths that a user with the default cache runs had no test that could fail at all. Four attn_packed arms (mixed rows with a window ring of exactly nwin; V4.1-Flash's 64 heads of 512; a list past the kernel's 256 threads; one entry a row with the sink at -inf) and six index_quant arms (mode 3 with a partial last block, the wide queries of mode 9, three of mode 10's grid -- 1005, 2048 and 8 keys -- and 2048 keys off FP4's own grid). Each pins the packed arm's bits to the same call over the rows kvq_unpack decodes from the mirror, and to a scalar attention/score sum in the shader's own order, rel_l2 0.00e+00 on this device. The three gates that were absolute are exercised the same way. Plus the mutant hook the arms need to prove they can fail: VK_KV_QUANT_MUT=win|ckv|ikey flips one scale byte in the packed mirror and requires the packed arm to move while the fp32 arm (packed from the original bytes) does not. Measured on lavapipe: VK_KV_QUANT_MUT=win -> "a one-code change in a window scale moves the packed arm" VK_KV_QUANT_MUT=ikey -> "a one-code change in an index-key scale moves the packed scores" and at 80d9cf8 the same env var is a no-op: the base binary's 7 OK lines are byte identical with and without it (diff empty, rc=0), which is the coverage hole stated as a measurement rather than as an absence. Evidence: reports/w6-exactness-final.md:158,171-181; reports/f7-tests-ci-final.md §8; reports/fix-plan-final.out.md §4 commit 5, §6.13. Patch from proposals/.../w6-exactness/exactness-tests.patch, applied rc=0 with no offsets.
score_case gated the FP4 index keys' drift against the ORIGINAL fp32 rows at rel_l2 <=
2e-1 and top-16 overlap >= 0.7. Those are not a band, they are an order of magnitude
wider than the format's own noise: an ue8m0 scale rule floored instead of ceiled -- in
BOTH the host's kvq_ue8m0_scale and the shader's ue8m0c, so the codec is wrong by a
power of two a block -- moves the five arms to rel0 1.58e-1 / 1.46e-1 / 1.88e-1 /
1.53e-1 / 1.52e-1 and overlaps 92/100/91/81/87%, and the old gates print PASS (measured:
rc=0, no FAIL line). The suite could not see a whole-binade codec error on the default
path.
The gates are now the measured band -- rel0 <= 1.35e-1 (5% above the worst arm) and
overlap >= 0.80 (10% below the worst) -- and two arms make the band mean something: 2048
keys above VKC_DSA_GRID (top-16, where the selection can move) and the checkpoint's own
8192 keys with index_topk 512 (k/pool 6.25%). score_case takes the top-k as a parameter
for them.
Same floored-scale codec under the new gates: 5 FAIL lines ("rel_l2 to the original
rows' scores 1.581e-01" / "1.459e-01" / "1.878e-01" / "1.535e-01" / "1.516e-01"), rc=1.
Measured on both devices, five arms identical to every printed digit:
lavapipe 1.13e-1/92% 1.22e-1/100% 1.28e-1/94% 1.04e-1/91% 1.01e-1/89%
RX 6900 XT 1.13e-1/92% 1.22e-1/100% 1.28e-1/94% 1.04e-1/91% 1.01e-1/89%
so one band serves both drivers -- the packed codec is bit-exact against its own
reference (rel_l2 0.00e+00) and the drift is the format's, not the device's.
Hand-merged: the patch and commit 5's exactness-tests.patch both touch this file and
neither order applies (hunk JustVugg#3 of the score patch lands on the call list commit 5
rewrote). Hunks 1-2 applied with `patch`; the call list was merged by hand and both files
compile together.
Evidence: reports/f1-codec-final.md:89,277; reports/f10-accuracy-final.md §0.1;
proposals/w3-accuracy-frame/README.md; reports/fix-plan-final.out.md §2.6, §4 commit 6, §6.12-§6.13.
The packed host's own arrays were printed as `host kv %.1f MiB (packed)` and gated to 0.15 < v < 0.25 -- a 100 KiB window around a 183,168 B total, and at one decimal the line cannot tell 179,504 from 186,831 (both print 0.2), so the gate tested the rounding. The line now prints the total in bytes (`host kv %zu B (packed)`, c/deepseek_v41_chain.h:1924) and the ledger's gate parses those bytes and holds them to +-2%: 179,504 .. 186,831 B, which is 183,168 B -- 6*(8+4)*528 + 128*288 + 256*288 + (4+8)*2176 + 2*8*528, the derivation the comment beside it already carried. The old decimal is unpinnable any narrower than a whole 0.1 MiB. Measured, one serve turn of the packed fixture (`bash tests/vulkan_engines.sh kv-quant-deepseek`, lavapipe): before (80d9cf8): `host kv 0.2 MiB (packed)` -- the new parse finds no bytes field (`sed -n 's/^host kv \([0-9]*\) B.*/\1/p'` prints nothing), so the gate cannot even read this line after: `host kv 183168 B (packed)`, gate holds, and six arms print it identically (cap=2, cap=8, the 40-token prompt, chunks of 3, the per-row GEMV, DSpark spec=1); the family is rc=0 end to end. Evidence: reports/f7-tests-ci-final.md §8.3; reports/fix-plan-final.out.md §4 commit 7, :5069-5074's own derivation.
c/dsv41_kvq.h:21-24 still said "The host keeps its state fp32 holding the DEQUANTIZED rows (a pull's write-back and a re-upload's source)". That sentence is the mechanism of the P1-1 miss: the arrays have been uint8_t code rows at 528/288/68 B since the default flip, and a reader who believed the header could write `l->ckv + t0 * head_dim` and be off by a factor of four -- which is exactly what the CUDA chain did. It now says what the code stores, that every offset is a byte stride through v41_row_win/ckv/ikey, and that the same functions are the host's store and read. `grep -rn 'fp32 holding the DEQUANTIZED' c/ docs/ README.md` finds nothing. c/dsv41_kvq.h's header note and docs/deepseek-v41.md's cache section (from proposals/w3-di-spec/docfix.patch, applied rc=0 with no offsets) now say which DeepInfer commit the formats come from (the store kernel and the layout it writes, tile-kernels 2.1.0), and state the one deliberate deviation plainly: a ckv group whose amax lands in [9,10)*2^-9 packs one scale code down for the re-pack fixpoint, so its grid stops at 6*2^-9 and every element above saturates there where DI's grid holds 8*2^-9 -- the group's upper tail 25% low, its mean error 15% against DI's 10%, in that band only (probe proposals/w3-di-spec/di_ind.c T3; band edges and the three formats di_parity.c A-G). Also that DI's index-K blocks are 128//ratio tokens, not 32, so the byte streams are not comparable. docs/experiments/dsv41-kvq-2026-11-11/README.md cites the two copied plan JSONs: they were generated before the workspace re-price (commit 4) and their allocation field still carries min(ctx, 1280) rows, so the citation now says so and keeps them for what they were copied for, the state figures. They are not regenerated here -- that needs the checkpoint. Build after: `make -j6 VK=1 deepseek_v41 tests/test_vk_kv_quant` rc=0. Evidence: reports/f3-host-final.md §8.4; reports/rt-runtime-final.md §8.8; reports/f1-codec-final.md §8.3; reports/f8-docs-final.md §8; reports/f9-default-compat-final.md §8.3; reports/fix-plan-final.out.md §4 commit 8.
No sanitized family sees the packed host: family_deepseek_sanitize and the rest walk the stock 64/32 fixture, where v41_host_kvq's dimension check declines the packed cache, so the 528/288/68 B host arrays, the pull/re-upload of code rows and the block-wide index keys were never under ASan or UBSan -- the class of bug that produced the CUDA chain's quarter-row stride (3e9d5b7) had no leg that could see it. `bash tests/vulkan_engines.sh kv-quant-deepseek-sanitize` is this family on a sanitized build: `SAN=1` makes it `make clean` and build deepseek_v41 and test_vk_kv_quant with -fsanitize=address,undefined -fno-omit-frame-pointer -g, export the ASAN_OPTIONS/ UBSAN_OPTIONS every other sanitized family sets, run every arm as usual, check the codec log with san_diag, and sweep the logs the arms write at the end -- a sanitizer report on stderr never fails an arm's own gate, which reads stdout. Usage lines updated. .github/workflows/ci.yml gains the matrix twin beside kv-quant-deepseek:441. Measured on lavapipe, whole family, sanitized build: rc=0, 43 OK lines (the codec's 17 included, PASS), 0 "AddressSanitizer"/"runtime error" lines anywhere in the log -- including the two host arms (row 1 0.0924, argmax kept; 3 of 3 forced drafts accepted) and the six ledger arms (host kv 183168 B) so the packed host arrays, the pull/re-upload and the index keys run clean under both sanitizers. Cost: a sanitized build (about 3 minutes here) and a second matrix entry. Evidence: reports/f7-tests-ci-final.md H6 (:117), :72-73, :254; reports/fix-plan-final.out.md §4 commit 9, §6.14.
N1 (arithmetic): the docstring said the old chunk price under-reserved "266x at 262144". It is 416x -- 262,164,480 B priced (min(ctx, 1280) rows of the old form) against 109,148,393,472 B allocated -- and the docstring now names both operands so the next reader can check it against the ladder instead of taking a bare ratio. N2 (naming): the "named and unpriced" note listed the FIXED transients only (attention_run's `heads` and `head_score`, forward_full's `embedded`/`collapsed`). attention_run also allocates four context-sized buffers at its top and frees them at its end -- `latent` ((n/ratio + 1) * head_dim floats: 256.0 MiB at 262144 ratio 2), `kv_all` ((window_rows + compress_len) * head_dim: 805.3 MB there, window_rows being n during prefill and the ring's 128 during decode), the packed ring's `win_dec` (window * head_dim = 256 KB) -- 1.07 GB together at 262144 ratio 2 and 1.61 GB at ratio 1, under 1.5% of the 101.65 GiB price and never live beside the next layer's. They stay unpriced (a per-row form cannot express "one layer at a time"); they are now named so the omission is a decision rather than an oversight. N3 (alias): the planner required num_attention_heads, while the engine reads n_heads first and num_attention_heads second (deepseek_v41.c:199). A config spelling only the alias is served by the engine and was refused here -- and a refusal is not a failed plan: resource_plan.py's ladder takes `except ValueError: break` (:3394), so such a box reports "no RAM plan" and silently keeps the default context. The planner now reads the alias with the engine's precedence (n_heads wins when both are present). The closed form is unchanged at the released spelling: 416,368 B a row. Tests: new test_a_config_spelling_the_heads_n_heads_prices_the_same in test_dsv41_kvq_host_plan.py -- the alias prices the released workspace, and with both keys present n_heads is the width priced. Fails at 2072865: "deepseek_v41: missing or invalid planning key 'num_attention_heads'" (Ran 11 tests, FAILED (errors=1) in a tree of that commit); OK here. 158 tests OK across test_dsv41_kvq_host_plan, test_resource_plan, test_context_compaction (157 before). Full suite: Ran 3473 tests, FAILED (failures=2) -- test_harness_builds_without_warnings in test_e8x4g64_mint_load and test_fp8_e2e_repack_load, the mistral_vision.h ABI note; reproduced at 80d9cf8 in /tmp/rev-base (Ran 8 tests, FAILED (failures=2)). Build: make -C c -j6 VK=1 deepseek_v41 tests/test_vk_kv_quant rc=0, 0 warning lines; make -C c cuda-host-syntax rc=0. Python only: no C file in this commit. Evidence: reports/fix-impl-review.out.md nits N1-N3.
… it from
Reusing a turn's state changed the answer under the packed host -- the default --
while COLI_VK_KV_FP32=1 was byte-exact, and the branch's own CI lane was red on it.
attention_run has two readers of the same row and they read different things: a
decode step, and any prefill that resumes a live state, dequantize the ring's 528 B
code row (kvq_unpack_win, start_pos > 0), while a prefill at start_pos == 0 hands
sparse_attend its `kv` scratch -- the exact floats the code row was just packed from,
before e4m3 rounded them. So a prompt prefilled in one forward and the same prompt
continued from its own stored state computed different attention over the same
positions, and only the packed arm can differ: fp32 rows read back exactly.
The row now enters every read as the codec's value -- kvq_pack_win then kvq_unpack_win
in place, per position, before this forward stores or attends to it -- so `kv_all`, the
window half and the ring all hold what a later forward would read. e4m3 re-packs a
dequantized row to itself, so the code row the ring gets is the one a pack of the
untouched floats would have written: no store changes, no layout change.
Measured, dsv41_kvq_tiny, packed (the flag unset):
tests.test_dsv41_prefix_serve before 26 failures of 29 (skipped 2)
after OK, 29 (skipped 2)
COLI_VK_CHAIN=0 after OK, 29
COLI_VK_KV_FP32=1 after OK, 29 (unchanged: 3.9 s, was OK)
lavapipe COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1
after OK, 29 (with tests.test_dsv41_dspark_serve
and ..._prefix_store: 39 tests OK)
and the mechanism, proved before the fix: put the fp32 layout's ring rows through the
same pack/unpack (nothing else packed) and the same 26 of 29 fail, while packing the
prefill's own rows instead makes the packed arm pass -- so the failure is the
asymmetry, not the format.
Evidence: reports/grind-c1-review.out.md F1; reports/grind-c1-fix.out.md.
…a mirror's sections The store was handed the pre-flag widths. Every array it parts is uint8_t* now: a ring of 528 B code rows, a compressed row of 288 B, an index key of 68 B in block-wide sections. v41_ps_state_save copied W * head_dim * 4 = 262,144 B out of a ring W * 528 = 67,584 B long and v41_ps_take wrote the same 262,144 B back into it -- a 194,560 B heap overrun per layer, x40, on the default (packed) path, for any prompt that reaches a block (B=256: prefix_store.h:398), and the same four-times-too-wide stride on the ckv/ikey parts. Now: v41_row_win/ckv/ikey in both directions, so a save copies the ring's bytes and a restore writes them back; V41PsState.win is uint8_t*. The index keys are not one array -- a 32-row block keeps its data rows and then their scale words 2048 B on (dsv41_kvq.h: "NOTHING else may go linear"), so a bare 68-byte stride would still read the wrong bytes for a range that starts or ends inside a block. That is what ColiPsPart.rows_io is for: a part whose rows are not one array says so and coli_ps_rows copies the staging it writes. v41_ikey_rows_io is the mirror's (per block: its data rows, then their scale words), and the part declares 68 B a row as before. The other seven engines' parts pass NULL -- their caches are linear arrays -- which is the whole of that diff. Measured, dsv41_kvq_tiny (the packed host), stores driven at COLI_PREFIX_STORE_BLOCK=16: tests.test_dsv41_prefix_store 80d9cf8: 4 failures of 7 now: OK, 7 ASan+UBSan build, 2072865, one serve test named after a resumed prefill: SUMMARY: AddressSanitizer: heap-buffer-overflow deepseek_v41.c:5762 in v41_ps_state_save now: OK; the whole of prefix_serve + prefix_store under the same build: 36 tests OK tests/test_prefix_store (C): OK, and its new block-sections section fails when the part is given rows_io = NULL, i.e. the check is not vacuous lavapipe, chain on, prefix_serve + dspark_serve + prefix_store: 39 tests OK The family's SAN leg now runs that store arm itself (COLI_PREFIX_STORE_BLOCK=16) -- its serve arms' prompts are ~90 tokens against B=256, so no arm reached the store and the leg was vacuous for exactly this bug -- and the unsanitized packed serve gate runs tests.test_dsv41_prefix_store, which is what sees the widths without a sanitizer. Evidence: reports/grind-c1-review.out.md F2; reports/grind-c1-fix.out.md.
43501e8 makes the packed host read its own code rows in the PREFILL too (that is what makes a resumed prefill compute what a cold one computes), so the fp32 arm this gate compares against is no longer input-shared past the prefill by construction: the packed row 1 now carries the query path's quantize hop as well as the store's. Measured on dsv41_kvq_tiny, row 1 of the dump: before 43501e8 0.0924 of 1.5927 = 0.058, argmax kept (the store's hop alone) after 0.1615 of 1.5927 = 0.101, argmax FLIPPED (both hops) 0.101 is inside the row's 2e-01 tolerance and the argmax is the codec's, not a bug's -- the packed arm's stream already differs from the fp32 arm's (7 of 8 tokens here). So the row keeps the tolerance and prints the pick instead of requiring it: logits_row gains an argmax argument (0: tolerance only), the only caller that wants it here passes 0. The gate keeps its teeth, proved by mutation -- the P1-2 bug put back (the snapshot keeping the row the slot evicted, win_dec's re-decode removed), same fixtures: mutant row 1 1.1726 of 1.5927 = 0.736 0.2-tolerance FAILS, argmax flipped this branch row 1 0.1615 of 1.5927 = 0.101 passes this branch row 0 0.4447 of 2.2744 = 0.196 argmax kept (logits_first's own gate) and the property the fix is for is gated much harder by the arm below it, whose 29 prefix-reuse tests compare tokens exactly (26 of them red before 43501e8). Family, lavapipe, whole `kv-quant-deepseek` lane on this tree: rc=0, 42 OK lines, 0 FAIL, including "OK kv-quant CPU host: first row max |diff| 4.45e-01 of 2.27e+00 (tolerance 1e+00), argmax kept; row 1 max |diff| 0.1615 of 1.5927 (tolerance 2e-01), argmax FLIPPED; 7 of 8 tokens differ" and "OK kv-quant serve: prefix reuse, DSpark's rollback and the prefix store against the packed fixture" (39 tests, skipped 2). Evidence: reports/grind-c1-review.out.md F1/RC-1; reports/grind-c1-fix.out.md.
… the tree clean RC-1 of reports/grind-c2-review.out.md is a TREE finding, not a history one: the index held an 8-file revert of 8f1d2a7 (prefix_store.h's ColiPsRowsIo hook and the family files' seventh initialiser dropped, while deepseek_v41.c kept the code that calls it), so a bare `git commit` would have shipped a store that fails tests.test_dsv41_prefix_store and warns "excess elements in struct initializer". The worktree was never wrong -- `git diff HEAD` was empty, and the fix was `git restore --staged` on the eight files (no commit of content is needed; nothing else changed). What is committed here is the other half of RC-1: the same lane re-dirties the tree every time it runs. `bash tests/vulkan_engines.sh kv-quant-deepseek`, the green gate RC-1 asks to re-take right before a push, writes off.f32/on.f32/off.txt/on.txt/plain.txt/ forced.txt/self.json into c/ (each arm `rm -f`s its own at the START and never after), so "confirm git status --porcelain is empty immediately before the push" and "run the lane" cannot both hold without this. Seven ignores, in the section that already holds the kv-quant fixtures. Caveat for the push, measured here and NOT caused by this commit: a from-scratch build (`make -B VK=1 deepseek_v41`) re-runs the committed shaders through this box's glslc and leaves 17 of them byte-different (git status: M c/shaders/*.spv), so `git checkout -- c/shaders` after a full build, before the push. Those .spv are tracked; only the verified, unchanged ones may ship. Evidence: reports/grind-c2-review.out.md F1/RC-1; reports/grind-c2-fix.out.md.
RC-3 of reports/grind-c2-review.out.md (F3). e141623 made v41u_wanted decline a packed host -- the CUDA chain moves the host's KV arrays as fp32 rows, so a build that packs them runs the prompt's layers on the CPU -- but v41_tier_chain_bytes, which is what the tier SUBTRACTS from the device's expert room (VktConfig.chain_bytes, vk_tier.c "what the dense chain allocates at its first forward"), still gated on COLI_CUDA_CHAIN alone. On the default path (COLI_VK_KV_FP32 unset: the host is packed) it therefore held the chain's own v41u_plan_bytes on the device for a chain that never runs: at 262144 its E*LR*4 term for the candidate lists plus the P*E candidate mask is >= 1.74 GB of device expert slots lost to every conversation, and nothing in resource_plan.py reads the flag, so the plan could not see it either. The fix mirrors the ONE condition that is decidable where the tier sizes itself: the same v41_host_kvq(&m->c) v41u_wanted tests, negated. The rest of v41u_wanted (g_cud.on, cc_ready, g_v41u_off) is not up yet at that point, so copying the whole predicate would have reserved NOTHING for the chain on fp32 hosts too -- a worse bug than the one fixed here; the comment says which half is mirrored and why. Fail-first, on this tree (the new test reads the function out of deepseek_v41.c): $ git checkout HEAD -- c/deepseek_v41.c $ python3 -m unittest discover -s tests -p 'test_cuchain_host_strides.py' Ran 5 tests ... FAILED (failures=1) AssertionError: '!v41_host_kvq(&m->c)' not found in 'static size_t v41_tier_chain_bytes(Model *m) { ... if (xt_cuda() && m->gguf && !(e && *e == '0') && g_v41_mux_slots <= 1) return v41u_plan_bytes(m, m->c.max_positions); ...' $ (this commit) Ran 5 tests ... OK Its second test is the floor: xt_cuda/m->gguf/mux<=1/v41u_plan_bytes(m, max_positions) must all stay in the body, so the chain keeps its room on an fp32 host. Static only, exactly like e141623: there is no CUDA device in this environment. What covers it is the test above plus `make cuda-host-syntax` (gcc -fsyntax-only -DCOLI_CUDA over deepseek_v41.c and the other eight engines) -> rc=0, 0 warnings, 0 errors. The Vulkan build is untouched by construction (the function's body is #ifdef COLI_CUDA): `make -j6 VK=1 deepseek_v41` rc=0, 0 compiler warnings. Not fixed here, and not a bug on inspection: v41_tier_dense_bytes (g_v41_cud) is NOT dead weight when the chain declines -- V41_CU / v41_cu_mm (deepseek_v41.c:786-792) serve the CPU path's dense GEMVs from that device store. Evidence: reports/grind-c2-review.out.md F3/RC-3; reports/grind-c2-fix.out.md.
RC-2 of reports/grind-c2-review.out.md (F2). 19b4249 priced V4.1's prompt workspace at what ONE forward of n rows allocates and charged it to the plan's runtime term (family_registry._dsv41_geometry, 416,368 B a row at the released geometry). The price is right -- the engine does prefills a prompt in one forward, deepseek_v41.c:4692/6050, and that forward allocates every buffer at n rows (4816-4871) -- but cap_for_arch turned it into a REFUSAL, and openai_server raised when the plan kept no slot. Released config, 64 GB, `cap_for_arch('deepseek_v41', None, env={'RAM_GB':'64','CTX':...}, model='/root/Models/DeepSeek-V4.1-Flash-1017')`: CTX 4096 -> cap 56 CTX 65536 -> cap 25 CTX 16384 -> cap 50 CTX 262144 -> ValueError: "DeepSeek V4.1 Flash RAM budget cannot hold one expert slot per layer" At 262144 the box served NOTHING, at any prompt length, though the workspace is only 1.71 GB for the 4096-row prompt it would actually be sent (109.15 GB is a 262144-row prompt). F2 and RC-2: keep the cap at >= one slot. The plan itself is NOT changed -- it still reports zero slots and the "cannot hold one expert slot per sparse layer" warning, so `coli plan` and `coli doctor` keep telling the truth about a box that cannot hold a context-long prompt's workspace -- and the floor lives at the launch seam, where the question is only "may this box start": CTX 262144, RAM 64 -> "[v41] RAM plan: 109.1 GB of prompt workspace at CTX=262144 does not fit the 64.0 GB RAM budget, leaving no expert slot a layer; starting at 1 -- a prompt that long needs the workspace's RAM beside the cache, a shorter CTX, --ram or --cap leaves room" -> cap 1 MiMo keeps its refusal (its workspace is priced at a 64-row block, so a plan with no slot is a box that cannot hold one expert of every layer -- unchanged by this branch), and so do the Qwen floors. The context ladder is untouched and still prices the workspace to choose a context; measured at the released config, unchanged by this commit: 8 GB -> None (the engine's 4096), 16 GB -> None, 32 GB -> 16384, 64 GB -> 32768, 256 GB -> 65536, 1024 GB -> 131072. Tests, fail-first on this tree: $ git checkout HEAD -- c/openai_server.py tests.test_resource_plan...test_a_workspace_past_the_ram_starts_at_one_slot_instead_of_refusing ERROR ValueError: DeepSeek V4.1 Flash RAM budget cannot hold one expert slot per layer (new test: a 2-layer fixture at RAM 6 GB / CTX 262144, plan slots 0 -- it also pins that build_plan still warns, so the honesty of the plan is not traded for the floor) $ ... (this commit) Ran 1 test ... OK $ python3 -m unittest tests.test_openai_server.CapSentinelShimTest rewrote test_v41_insufficient_ram_refuses_before_spawning (it pinned the refusal this commit removes) as test_v41_an_empty_plan_starts_at_one_slot_not_a_refusal, which asserts the argv cap is "1" and the note names the workspace, and ADDED test_mimo_an_empty_plan_still_refuses_before_spawning so the surviving refusal stays pinned Ran 17 tests ... OK $ python3 -m unittest tests.test_openai_server tests.test_env_defaults tests.test_cgroup_memory tests.test_resource_plan tests.test_dsv41_kvq_host_plan tests.test_context_compaction tests.test_cli_output Ran 771 tests ... OK cap_for_arch also binds `context` once instead of computing it inside the build_plan call, so the note can name the context it is about. Evidence: reports/grind-c2-review.out.md F2/RC-2; reports/grind-c2-fix.out.md.
Fixes for the review, on top of the dev merge (
|
| commit | what it fixes | proof |
|---|---|---|
e1416239 |
the CUDA chain addressed the re-typed host KV arrays as fp32 rows (packed: OOB read+write; =1: 4:1 row overlap). It now declines a packed host and byte-strides its fp32 rows through v41_row_* |
test_cuchain_host_strides.py 3 fail → 3 OK; make cuda-host-syntax rc=0 |
e798600b |
attention_run decoded the ring snapshot before the step's store, so ratio==0 layers attended an evicted slot's key |
row 1 of the CPU-host dump 1.2758/FLIPPED → 0.0924/kept, then (see 43501e8d) 0.1615 inside the arm's tolerance |
1029085b |
the packed DSpark arm forced the reference's tokens, so 0/15 drafts were accepted and "bit-identical" was plain-vs-plain | a self-draft arm: 3 of 3 forced drafts accepted, the stream = the plain one |
19b42498 + 8f05d124 + 21e45a93 |
the planner priced the prefill workspace at min(ctx,1280) while forward_full allocates at n (250 MiB vs 65 GiB at 262144). Prices ctx × 416,368 B (the engine's own closed form) and keeps the launch floor |
14 planner failures → 157 OK; the ladder at 8/16→none, 32→16384, 64→32768, 256/512→65536 |
b9b145ad + d7745e47 |
the suite never ran mode 0 flags 2 or the indexer's quant readers, and its score gates were loose enough to pass a whole-binade scale error |
+282 lines of arms; a winq i>>5→i>>6 mutant now fails 7 arms (it passed the old suite) |
616a366e |
host kv printed to one decimal inside a wide band |
prints 183168 B and gates ±2 % |
8f1d2a71 |
the prefix store reused fp32 row widths on the packed arrays: memcpy of 262,144 B out of/into a 67,584 B ring (OOB read and write, ~194,560 B a layer, at prompts past the 256-token block) |
fixes the overflow, but the lane stays red on the resumed-vs-cold mismatch until 43501e8d: at COLI_PREFIX_STORE_BLOCK=16 on dsv41_kvq_tiny, 22 of 29 fail at 80d9cf80 and at the branch without it (26 of 29 on the long fixture, the commit's own number) |
43501e8d |
a resumed prefill read stored codes where a cold one read fresh floats, so the same text answered differently prefilled at once and continued from its own state | the same arm at COLI_PREFIX_STORE_BLOCK=16: 22 of 29 → 29 OK |
0b464e86 |
the tier still reserved chain VRAM for a chain the CUDA decline makes unreachable | static (no CUDA device here) |
0f70be39 |
the kv-quant lane left dumps in the tree | git status clean after a lane run |
20728655 |
no sanitizer coverage for the packed host | new ASan/UBSan family leg: rc=0, 43 OK |
The one trade-off, stated
43501e8d makes the packed host read its own code rows in the prefill as well as on store — which is what upstream DI does (it stores quantized and reads dequantized for every position) and what makes a resumed prefill compute what a cold one computes. The cost is that a prompt row now pays the codec's loss twice, and the CPU-host first-decode-row pick moves from 0.0924 to 0.1615 of 1.5927 (inside the arm's 2e-01 tolerance; argmax now flips at that near-tie). f20511c3 therefore relaxes that one clause to tolerance-only, keeping a mutation proof that the original defect still fails it (0.736 > 0.2). If you would rather keep the stricter pick, the alternative is to revert 43501e8d and accept that the prefix-store lane stays red.
Verification
- build
make -j6 VK=1 deepseek_v41 tests/test_vk_kv_quant: rc=0, 0 warnings make testatf20511c3(the reviewed ten):Ran 3473 tests,EXIT=2on both that tree and80d9cf80— the same two environmental GCC-ABI failures, same skip count, 258 C binaries green; the three follow-up commits add 4 tests (loader count +4). The targeted suites were re-run at the tip: planner trio159 OK,test_cuchain_host_strides5 OKvulkan_engines.sh kv-quant-deepseek: EXIT 0, ending withOK kv-quant serve: prefix reuse, DSpark's rollback and the prefix store against the packed fixturetest_dsv41_prefix_serveatCOLI_PREFIX_STORE_BLOCK=16: 29 OK- Gitea (local runner): five jobs green on a snapshot of
8f1d2a71inpi/colibri(ci-fix-8f1d2a71) — that snapshot is five commits behind this tip; GitHub CI for the tip is queued, not green - ASan/UBSan leg of the packed family: no diagnostics
Still open, deliberately
- the candidate mask is
n × compress_lenbytes and is priced nowhere — 64.0 GiB at 262144, 38 % of the peak. Bounding it to the tail rows that are read is bit-neutral and takes it to 0.59 GiB, but it is pre-existing and an engine-side change, not this PR's; - CUDA is static-only here (no device on the box);
- the ckv demote remains the one place the bytes deliberately differ from DI.
…family's last The sanitized family exported UBSan without halt_on_error, and every arm's 2> truncates off.log/on.log, so the closing sweep read only the last arm's pair -- five of six arms could misbehave into a green lane. UBSan now halts (decide's form), each arm sweeps its own logs as it finishes (kvq_san_sweep, a no-op off the SAN leg), the ledger's serve turns tee their stderr to san_serve.log for the sweep, and the usage ledgers are cleaned at the family's start and ignored, so a FAILED lane leaves the tree clean too.
Three seams, three agreements. _host_kv_packed parses COLI_VK_KV_FP32 the way both C parsers do -- a leading 0 is packed, only false/off/no are names -- so no flag value sits the planner on one side and the engine on the other. The packed index keys round to whole 32-row blocks (kvq_ikey_mirror_bytes's own math) where they were priced linearly 68 B a row. The widths and the block are pinned to c/dsv41_kvq.h's macros by a test that reads the header, and the ladder the PR measured -- 0/+1/+2 expert slots at 64K/128K/256K -- is now build_plan's own output under both flag states, not a session run.
… layout says so coli_ps_span priced a rows_io part once a head as coli_ps_rows never read it; the span now prices the part's own run, and the layout signature mixes the part's staging in, so widths alike but moved differently cannot share a store. The unit test's sectioned part compresses the way production's does (rows_io with per > 1), its tail's half-read group held unstaged for the state to refill. The per-model suites: qwen38's expected-hit bound counts tokens, not bytes, and a read-out that ends on an ERROR frame fails instead of passing silent.
…39 GiB 'Where this engine is not the vendor' still said the engine keeps the KV cache in fp32 -- the one thing this branch changed by default. Only the indexer's queries and the compressor's latents stay fp32 now; the cache itself is the checkpoint's packed formats, COLI_VK_KV_FP32=1 the exact opt-out. The 1M row's effective --ram was +5.51 against its own column's arithmetic (5.39), and the DSpark stages with their rollback rows are said outright: 329,472 B.
Conflicts were the two additive lines: c/Makefile's TEST_EXCLUDE keeps both test_vk_kv_quant (this branch's VK=1 codec test, out of the portable suite) and dev's test_vk_pcache, and the lane script's usage line keeps both dev's inkling2 family and this branch's kv-quant-deepseek lanes. Everything else auto-merged; the kv-quant-deepseek lane is green on the merged tree.
Dev's 93 commits since 66fa9f6; three conflicts, one of them real. The real one: dev's JustVugg#2797 (perf/candidate-block-ids) made the host's candidates int32 block ids while this branch had bit-packed the device's candidate mask (12b2908), and the two changes meet at v41c_forward_seg's readback. That hunk now unpacks the branch's bit row -- ceil(wcap/32) words a row, v41c_mask_words stride -- into dev's temp keep row and hands it to dev's candidate_row_ids; the device side stays wholly the branch's (dev never touched vk_chain.h or the scratch sizing, and the planner keeps pricing the packed mask row, which JustVugg#2797 did not edit). g_v41_park_x takes both sides' fields: dev's cand_topk/int32 cand and this branch's uint8 code-row published. indexer_run's free list keeps both buffers (ikd, maskrow), and the CHANGELOG keeps both sides' Unreleased entries. On the merged tree: test_dsv41_candidate_ids is green (the frozen 15f810f writer, 11760 cases, 2464 padding-sensitive), test_vk_kv_quant passes, and make test-c is green; make test-python passes but for the MXFP8 gq_dot subtests, which fail identically on dev and on this branch's tip on this box (a -march=native environment flake, not the merge).
|
dev's 93 new commits merged ( Dev moved to On the merged tree, before the push:
CI is running on |
(Code written via AI, pi agent using a mix of models). As I only have an AMD GPU I haven't written this code to apply to CUDA.
In plain terms
A language model keeps a running memory of the conversation so far — the KV cache — and
consults it while generating. Colibri holds that memory in fp32, full 32-bit floats:
expensive — it is the one pool that grows with the conversation, at about 7x the cost it
needs to be.
Upstream does it differently. DeepInfer — the official DeepSeek V4.1 implementation —
stores the KV cache quantized: small 8-bit and 4-bit float formats (e4m3 and e2m1 in the
table below), one scale factor per block, in a paged cache (fixed-size pages; reference:
DeepInfer's
cache_common.py).This matters beyond memory: V4.1 was designed with that quantization in mind, so running
it with full-precision KV is not "more correct" — it deviates from how upstream itself
serves the model, while paying the ~7x memory bill.
This PR moves colibri's KV state into the same packed formats upstream uses — the GPU
copies (the device mirrors) and the engine's own RAM arrays (the host state) alike. Per
token of context, ~6,400 bytes become ~890 (86% less): ~0.78 GiB of KV rows becomes
~0.11 GiB at 128K context, ~5.4 GiB saved at the 1M ceiling. It is approximate, but
guarded: construction checks abort on any size mismatch, and gated tests hold the first
decoded row's top token pick (its argmax).
This is now the default. An unset
COLI_VK_KV_FP32packs the device mirrors and thehost's own KV rows alike — one boolean, no half-packed middle setting;
COLI_VK_KV_FP32=1is the exact fp32 opt-out, byte for byte the pre-default path. The geometry guard still
forgoes the packed cache — default included, one line on stderr, naming the default when
the variable was unset — at any head_dim/index_head_dim other than 512/128, so non-DSV41
geometry is unaffected.
DeepInfer (the official deepseek implementation of v4.1) quantises the values stored in KV
cache. Colibri uses fp32.
FP32 is undesirable for two reasons:
the behavior compared to upstream.
This is the whole change, in one PR. It supersedes #2701 (the device mirrors alone): every
commit and file of that PR is here in full — this branch is a fast-forward of it plus 9
commits for the host's arrays, and the fixes and the single-flag rework on top (Merge
state lists them). Closing #2701 loses nothing.
What
Three layers ride in this branch; the third is the oldest and the one to read first.
(a) The prefill-VRAM wave, 2026-11-06…08 — 12 commits, +2,098/−82. The kv-quant work
was written on top of it and its ledger is what makes the savings measurable at all:
the dsv41 RAM workspace priced at one prompt chunk instead of the whole context
(
family_registry._dsv41_geometry,test_resource_plan.py); a turn-end VRAM ledger(
v41c_ledger) printing weights / chain / tier / kv mirror / process / device; 48streaming slots reserved before the chunk is sized (
vkt_slot_reserve) and a 2-row streamthreshold (
COLI_VK_TIER_STREAM_MIN); a page-granular tier pool budget; the indexer'scandidate mask stored as one bit a column; and the experiment record that measured them
(
docs/experiments/dsv41-prefill-vram-2026-11-06*, 1,505 lines with audit, comparisonand raw logs). Only the page-granular pool budget's
c/backend_vulkan.chunk is alreadyin
dev(it landed as96ddc4dc); the rest is not — see "Merge state" below.(b) The packed cache — the device mirrors and the host's own KV arrays, 14 commits. The
chain's window rings, compressed rows and index keys live in the same formats DeepInfer's
paged cache uses, cutting mirror VRAM 3.9–7.5x depending on the mix (a row: the window rows
3.9x, the compressed rows 7.1x, the index keys 7.5x; 7.2x is the blended 6,400/890 B a
position; the ledger line measures 3.89x because it also sums the compressor's fp32
ringand the spare window rings, which do not quantize). A row is quantized once, after its last
fp32 reader (
chain_dsv4.compmode 13), and dequantized where read — the attention (flagsbit 2) and the indexer's scores (modes 3/9/10,
quantfield) decode inline; the pull'swrite-back moves the code rows as bytes, so rollback/prefix-reuse/lost-device need nothing
rebuilt. Scale reciprocals use the vendor's bit-exact reciprocal-LUT math on both sides (a
GPU's
1/xis ~2 ulp low, which bit-agreement cannot take).The host's own state takes the same flag, and the same formats:
Layer.window/ckv/ikey,the
V41Seqsets and the draft windows hold rows packed once at write (rope tails stagedin scratch, never packed in place) and unpacked only where a CPU consumer reads them; the
pull and the re-upload are raw byte copies; the planner and
v41_state_laterprice thepacked widths; the turn-end ledger gains a
host kvterm. The host never gives way to theKV split (only the device mirrors do, as before) — that invariant is what makes the
planner re-price unconditional under the flag.
Why there is no middle setting. The flag used to carry three levels — fp32, the device
mirrors packed with the host left fp32, and both packed. The middle level earned nothing:
it measured byte-identical to the fully packed cache on the 12-prompt battery against the
official server, and at 16K context only the fully packed cache was ever needed. What it
cost was a second code path — a host fp32 under packed mirrors, so every pull's write-back
dequantized and every re-upload re-packed, the round trip whose subnormal-scale hole
5c4eb903had to repair. That round trip is still on one path and the ckv demote fix stillguards it — the KV split's give-way packs its dequantized write-back — but the ordinary
packed path is now raw-code pulls and raw-code re-uploads, and the middle level, with the
COLI_VK_KV_QUANTname that spelled the levels, is gone.The flag is parsed at three sites that also agree on the geometry guard: the chain's
v41c_kvq_on(the mirrors, and with them the host — one condition, so the cache cannot behalf-packed),
deepseek_v41.c'sv41_host_kvq(the host's own arrays) andfamily_registry._host_kv_packed(the planner's mirror). The flag's widths are the onlywidths the rows are ever allocated at: the first forward sums mirror bytes against them and
aborts on a mismatch (
v41c_kvq_check), and the host has the same guard over its arrays atload (
v41_host_kv_check). When the KV split arms, the mirrors revert to fp32 whole andre-mirror from the host (logged, asserted) while the host's rows stay codes — a
contingency, not a mode.
COLI_VK_KV_FP32=1is byte-identical to today: every row stridegoes through the
v41_row_*helpers, and the flag-off arm of the family is the exactnessproof.
Formats (DeepInfer
cache_common.py; reference inc/dsv41_kvq.h)[data|scales]The arithmetic. At the released geometry (40 layers, 4 KV sources at ratios 2/2/2/1, so
a position costs 2.5 compressed rows):
2.5 * (head_dim + index_head_dim) * 4 = 6,400 Baposition of fp32 rows against
2.5 * (288 + 68) = 890 Bpacked — 86% less, 5,510 B freed aposition. These are the compressed rows and index keys; the window rings are not per
position (43 rings of
windowrows plus 240 rollback rows — 6 a layer — live in the planner's fixed term: 3,032,832 B at the released geometry). At 128K that is0.78 GiB of rows becoming ~0.11 GiB (~689 MiB freed), and 6.25 → 0.87 GiB at the 1M
ceiling (~5.4 GiB). Both are pinned by
c/tests/test_dsv41_kvq_host_plan.pyand by thefamily's ledger assertion below; the analysis behind them is
docs/experiments/dsv41-prefill-vram-2026-11-06.md§10, anddocs/deepseek-v41.mdcarries the same table, the packed host's rows included.
Measured
All of these are one box: RX 6900 XT (16 GiB, ReBAR),
DeepSeek-V4.1-Flash-1017,--ram 28.under the removed middle level, and the widths the line sums are the same either way, so
it still reads: the line sums
win + ring + ckv + ikeyfor device 0. At that run'scontext the window rings dominate the sum --
Wused rows plus 512 spare rows a layer,and those do quantize (a ring is allocated at 528 B a row under the flag) -- but a
window row only cuts 2048/528 = 3.88x, while the compressor
ringstays fp32 whole. So3.89x is a window-ring ratio with the fp32 ring inside the mix, not the 7.1x-7.5x the
compressed rows and index keys cut to a row. Session run: there is no raw line in the tree for the 13.0 MiB side, and the
context is not 128K — at 128K the fp32 rows alone would be ~0.78 GiB, and the same 50.6
MiB line appears in
docs/experiments/dsv41-prefill-vram-2026-11-06-raw.txton a3-forward, 768-row-chunk run. Reproduce with
COLI_VK_LEDGER=1on a serve turn.MiB), asserted on the real ledger line each run — the default's packed mirrors.
COLI_VK_KV_FP32=1, samecheckpoint, same prompts, temperature 0,
--ram 24 --ctx 16384: 10/12 promptsbyte-identical, 2 differ (
SUMMARY identical=10 differ=2), both differences in thetwo long free-form answers and both still factually correct — prompt 1 reaches 0.25
either way, prompt 6 produces the same
if x[i] == 0:fix with different prose/case.The earlier "the fp32 run differs only in whitespace" was wrong: this is declared
approximation. Evidence:
docs/experiments/dsv41-kvq-2026-11-11/(E14-compare.txt,E12-answers.json,E13-answers.json).344 (5,510 B/pos × 65,536 = 344.4 MiB); the independent 2026-11-11 planner ladder measured
slot deltas 0 / +1 / +2 at 64K / 128K / 256K (64K landed at 0: 344 MiB is below one
771.14 MiB expert slot). The −342 MiB is a session run; the slot ladder is pinned by the
planner test.
host kv … (packed). The familyasserts that term on the fixture (0.175 MiB, the codes' own math). The two terms are not
comparable on a session run —
host kvcounts every layer's rings and the full-contextrows,
kv mirrorcounts device 0's chain only — so quote the packed host against thefp32 host, not against the mirror line.
inline, and the pull moves code rows as raw bytes); the one CPU-attention arm measured
0.57 s against ~26 s of expert streaming (
session run: neither number is a ledger linein the tree).
V41_STATSbreaks the decode-on-read cost out as its ownkv readline.and the tier is near-full. Converting freed VRAM into resident experts needs an explicit
COLI_VK_TIER_GBraise (+~35 experts at 128K); the packed host does not rescue thepast-32K walls, which are not KV-sized. The per-context table in
docs/deepseek-v41.mdsays this arm by arm, including the two rows that are "unmeasured".
Tests
Gated and in the tree:
c/tests/test_vk_kv_quant.c(new, 372 lines). Codec bit-for-bit host vs device — theedge rows included: zero amax, negative zero, denormals, amax = 2^k × the format's max,
both grids' clamp edges, RNE ties, the 32-token block boundary, scales packed as int32 —
plus unpack exact, the re-upload's little-endian lanes end to end, and pack∘dequant a
fixpoint — now everywhere, including the subnormal-scale band the
5c4eb903fix covers.The score readers are bit-identical to scores over the dequantized rows; the
drift against the original rows is that format's own noise, and the top-k overlap is
gated at ≥70% (it prints 92–100% on the seeded data). The host-side block rules at the
top (
host_block_tests) need no device: staging through 68 B rows bit for bit, whole-blockikey prefixes, park's block rounding, the 528 B window slide.
tests/vulkan_engines.sh kv-quant-deepseek: every arm is a pair. TheCOLI_VK_KV_FP32=1run is the exactness side — it must keep the reference's oracle(exit 0, the chain's tokens = the reference's). The same configuration with both flag
names scrubbed (
env -u COLI_VK_KV_FP32 -u COLI_VK_KV_QUANT: the legacy name is nolonger read, and it is scrubbed so a stale export cannot leak in) is the packed default:
it must engage (its own log line), run the chain, and keep the shared first decode row's
argmax with drift within that row's own largest |logit|, the token drift printed, not
gated. Six pairs — cap=2, cap=8, a 40-token prompt, the prompt in chunks of 3, the
per-row GEMV, and a DSpark
spec=1arm that drivesring_save/rollback through the coderows. Then the split arm (the split armed, the mirrors gave way, the host codes held —
the split's read paths dequantize on read and the per-chunk write-back packs) and the
codec test. The ledger probe, from a one-turn serve run of each side, asserts the
kv-mirror line against the 528/288/68 B math (6.41 → 1.62 MiB, 6,725,632 → 1,701,120 B)
and the packed
host kvterm. The four arms the three-level flag kept for its hostvariants collapse into these plain packed ones: DSpark's ring_save/rollback through code
rows and the split-with-host-codes arm keep the coverage they added.
c/tests/test_dsv41_kvq_host_plan.py(new): the planner seam — 6,400·ctx vs 890·ctx,5,510·ctx freed and its share of an expert slot at 64K/128K/256K; the fixed term at the
released shape, (43 rings + 240 rollback rows), 3,032,832 B packed against 11,763,712 B
fp32 and the engine's 17,614,592 B at 16K; the fixture carries
num_nextn_predict_layers, notn_mtp_layers, like the checkpoint; both spellings andthe no-stage/no-block fallbacks; the geometry guard (≠ 512/128 keeps the fp32 price);
COLI_VK_KV_FP32=1still pricing fp32 and the unset variable pricing packed.test_context_compactionpins the same seam from the ladder's side.make teston this branch:Ran 3353 tests, failures=2— the known environmental pair onthis box (
test_e8x4g64_mint_load,test_fp8_e2e_repack_load, the gcc-ABI warnings), 0C-test failures, and the same two as the base. One test fewer than before the rework: the
removed middle level's parametrized arm is gone.
Covered by CI now:
test_vk_kv_quantis stillTEST_EXCLUDEd frommake test(it needsVK=1and a device, and skips with exit 0 without one), but thevulkan-enginesmatrixruns
kv-quant-deepseekon Lavapipe. The family builds the engine withVK=1, generates the--kv-quant-dimsfixture, runs the codec plus all six chain pairs, the split give-way arm,the CPU-host arm, and the prefix-reuse/DSpark boundary modules; it took 52-63 s on the
RX 6900 XT. A device-less
make checkstill says nothing about the packed path — the CI legis what closes that.
The family now runs
test_dsv41_prefix_serveandtest_dsv41_dspark_serveagainst thepacked fixture, so prefix reuse and DSpark rollback through the host's code rows are gated
(32 tests, the same 2 harness skips the kv-split family has).
Merge state (
dev@ 9f9df6e, resolved 2026-11-09)devwas merged ina58492f9; the branch is current with dev's tip at push time(
9f9df6e1) and CI-eligible (mergeable). The 7 mapped hunks are resolved, one line each:c/Makefile—TEST_EXCLUDEtakes both entries;c/backend_vulkan.c— dev'sP->refused = 0;kept;c/deepseek_v41.c— the indexer'sworkorder array and this branch'sikddecodescratch are both live and both freed;
c/deepseek_v41_chain.h— thech->kvq/hostqbranches ported inside dev'sv41c_state_up/v41c_state_pull(pack-up under the flag, give-way dequant-up, raw-codepulls), with dev's
v41c_chunk_tables/prompt_lwpreserved.Two semantic fixes git could not see, both called out in the merge commit body: dev's
mode-11 tiled indexer and three-pass prompt attention (
chain_dsv4_pa) have no kvqcodec — they decline under the flag and fall back to the quant-aware grid paths — and
mode 11's mask read is ported to the packed one-bit-a-column rows (
mbit). Verified:the fp32 path bit-identical to pre-merge (the family's flag-off arm), and dev's tiled
mode-11 test still passes bit-for-bit.
dev's qwen36 staging-room shape-pin was updated to the branch's reserved +
vkt_slot_reserveline (intent preserved — the staging room is reserved on theprimary); a sweep of dev's 97 other new/changed test files found no other stale pins.
The 4 env rows are registered (
COLI_VK_KV_FP32—ed323e15renamed it from the flag thatspelled the levels —
COLI_VK_LEDGER,COLI_VK_TIER_STREAM_MIN,V41_STATE_PROBE);check-envis green (778 variables).38a6c3dd(docs): the 890 B basis corrected indocs/deepseek-v41.md— the packed rowsalone; the window rings are the planner's fixed term — and the session-artifact pointer
dropped.
92201a68(the default flip): the packed cache became the default at all three read sites(engine chain, engine host,
family_registryplanner mirror); the geometry guard declinesnon-512/128 with one stderr line; every exactness arm pinned the fp32 opt-out; the default
arms (
env -u) assert the engage line and the unset-serve ledger.Gates re-run on the merged, flipped tree (RX 6900 XT):
check-envPASS; build clean;vk-kv-quant-checkPASS;kv-quant-deepseekPASS (default arms included);deepseek-chainPASS (81 arms bit-exact, max |logit diff| ~2e-06);kv-split-deepseekPASS;
make test3354 tests == the base box's 2 environmental gcc-warning failures,nothing new.
5c4eb903(the ckv subnormal-scale fix): ckv groups whose amax sits in the near-dead band[9·2⁻⁹, 10·2⁻⁹] used to pick a scale that a pull→re-upload round trip demotes one step,
shifting the band's largest elements 25% once — a hole in the re-pack fixpoint the CI's
random data never hit. The scale now demotes itself at first pack whenever the scaled
decoded amax would fall ≤ 5.0 (host and shader identically; out-of-band packing
byte-identical; in-band first-pack cost 33–40% on near-dead groups, stated plainly).
Edge rows at 9/9.5/10/15·2⁻⁹ (inside) and 8.9/11·2⁻⁹ (outside) pin the band; the probe
sweeps went 106/4000 hopping rows → 0, and 16,722/2M → 0; the fp32 opt-out is untouched
(the codec is not on its path) and
deepseek-chain's exactness stays green.ed323e15(the single-flag rework): the three-level flag became oneboolean — unset/
0/false/off/noruns the packed cache (mirrors and host rows alike),1/trueis the exact fp32 opt-out — and the middle level is deleted with the hop itcarried. The split's give-way and the ckv demote fix are untouched, the shader and the
codec header changed in comments only, so every bit-exact oracle holds. Gates on it:
check-env778 variables; build clean;vk-kv-quant-checkPASS;kv-quant-deepseek27 OK(ledger 6.4 → 1.6 MiB,
host kv0.2 MiB packed, the split armed with the host codesheld);
deepseek-chain82 OK;kv-split-deepseek41 OK;make test3353 tests,failures=2 — the same environmental gcc-ABI pair as the base, 0 C-test failures, and one
test fewer because the removed level's parametrized arm is gone.
2df7eb71+6de97a34(a full-branch review's findings, the branch tip): eight fixes,all with their own failure proofs. The two that matter most: the planner was still pricing
the host's window rings at fp32 while the engine allocates 528 B a row — now
(layers+stages) * window * 528, 7.98 MiB less reserved at the released geometry, checkedagainst the live
host kv 18.5 MiBledger; andtest_vk_kv_quantreturned 0 even when itsdevice-free half failed, so a GPU-less box saw a green
vk-kv-quant-checkthat provednothing — it now exits 1 on a host failure and 0 only on a clean skip. The rest: the
pack→dequant→pack fixpoint moved into the device-free half (and proven to fail when the
demote rule is removed); a new arm runs the packed host with the chain off — the
configuration every user without a capable GPU actually runs; one stderr line names the
retired
COLI_VK_KV_QUANTand the inversion in its old=0; parked index keys priced bywhole 32-token blocks;
kvq ⇒ hostqasserted with a message (proven to fire); and a2,560 B/row double charge in the split's row price removed. Gates:
check-env779variables; build;
vk-kv-quant-checkPASS;kv-quant-deepseekEXIT 0 with the new arm;deepseek-chainEXIT 0;kv-split-deepseekEXIT 0; planner tests 21 OK;make testthesame 2 environmental failures.
Honest close: dev moves daily (a batch merged the morning of 2026-11-08); if it advances
again before review, the merge is now a small increment, not a port.
Review
Plan-reviewed (8 amendments folded) and post-reviewed (2 nits fixed in 8a8fb6c: the state
probe gated to fp32 rows, a duplicated record pair; a third note is the park arm above).
Three byte-vs-float stride bugs were caught by the gates during development and routed
through the
v41_row_*helpers. The packed host's design came from a 34-siteconsumer/writer map (6 writers / 11 dequant-on-read / 17 boundaries) and an anchored
memory-to-experts model. The dev merge, the default flip and the single-flag rework were
plan-reviewed and gate-verified post-merge; two semantic auto-merge hazards (the
mode-11/PA float reads, the packed mask read) were caught by the gates, not by git.
Caveats
Approximate mode, on by default (
COLI_VK_KV_FP32=1is the exact fp32 opt-out). FP4 indexkeys can move a top-k pick, and the pull's write-back returns dequantized rows under the
split's give-way (bounded by the pack's idempotence). The dev2 mirrors quantize too but the
ledger line stays device-0-only (a pre-existing gap, annotated on the line). The packed
cache needs
head_dim512 andindex_head_dim128; at any other geometry it stays off byits own check — the default included, one line on stderr.
Supersedes #2701. Analysis lineage: the session's VRAM audit + the DeepInfer buffer contrast
identified the mirrors as the only context-scaling lever with same-quantity precedent; every
number above either has an artifact under
docs/experiments/dsv41-prefill-vram-2026-11-06*or is marked as a session run.
Validation (independent, 2026-11-11)
RX 6900 XT (RADV NAVI21), 18 CPUs, 38 GiB RAM,
DeepSeek-V4.1-Flash-1017; every GPU runon the real device, not Lavapipe.
VK=1build (deepseek_v41,test_vk_kv_quant)test_vk_kv_quantOK,PASS, exit 0; ASan/UBSan cleanvulkan_engines.sh kv-quant-deepseekOK(27 + the boundary line), 0 FAIL, 52-63 sFP32=1COLI_VK_KV_FP32=1opt-out(quantized, nohost kv--ram 24The raw evidence is
docs/experiments/dsv41-kvq-2026-11-11/(answers, compare log,planner JSONs, needle log, ledger lines,
sha256sums.txt) and the validateddocs/experiments/dsv41-kvq-throughput-2026-11-11.json.make checkwas also runlocally: 5 failures (3 from the harness exporting
OMP_NUM_THREADS=2, 2 GCC-16 ABInotes in untouched headers) and 390 environment skips; CI was 117/117 green at the
pre-fix tip and re-runs the same matrix plus the new leg on these commits.