Skip to content

dsv41: the DeepInfer-style packed KV cache, on by default -- COLI_VK_KV_FP32=1 is the exact fp32 opt-out - #2712

Open
jameswitney wants to merge 61 commits into
JustVugg:devfrom
jameswitney:r2b/kv-quant-host
Open

jameswitney wants to merge 61 commits into
JustVugg:devfrom
jameswitney:r2b/kv-quant-host

Conversation

@jameswitney

@jameswitney jameswitney commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

(Code written via AI, pi agent using a mix of models). As I only have an AMD GPU I haven't written this code to apply to CUDA.

Independent validation and fixes (2026-11-11, RX 6900 XT). A fresh-checkout run of the
branch measured the packed default against the fp32 opt-out on the released checkpoint:
10 of 12 prompts byte-identical, 2 differ (both answers still correct — prompt 1
reaches 0.25 either way; prompt 6 produces the same if x[i] == 0: fix). That is declared
approximation, not "only whitespace". It also found and fixed the planner's missing DSpark
stage rings + rollback rows (329,472 B at 16K), added the kv-quant-deepseek family to
CI (52-63 s), and put the planner/gate tests on the released config shape. Evidence:
docs/experiments/dsv41-kvq-2026-11-11/; commands and results in the Validation section.

In plain terms

A language model keeps a running memory of the conversation so far — the KV cache — and
consults it while generating. Colibri holds that memory in fp32, full 32-bit floats:
expensive — it is the one pool that grows with the conversation, at about 7x the cost it
needs to be.

Upstream does it differently. DeepInfer — the official DeepSeek V4.1 implementation —
stores the KV cache quantized: small 8-bit and 4-bit float formats (e4m3 and e2m1 in the
table below), one scale factor per block, in a paged cache (fixed-size pages; reference:
DeepInfer's cache_common.py).

This matters beyond memory: V4.1 was designed with that quantization in mind, so running
it with full-precision KV is not "more correct" — it deviates from how upstream itself
serves the model, while paying the ~7x memory bill.

This PR moves colibri's KV state into the same packed formats upstream uses — the GPU
copies (the device mirrors) and the engine's own RAM arrays (the host state) alike. Per
token of context, ~6,400 bytes become ~890 (86% less): ~0.78 GiB of KV rows becomes
~0.11 GiB at 128K context, ~5.4 GiB saved at the 1M ceiling. It is approximate, but
guarded: construction checks abort on any size mismatch, and gated tests hold the first
decoded row's top token pick (its argmax).

This is now the default. An unset COLI_VK_KV_FP32 packs the device mirrors and the
host's own KV rows alike — one boolean, no half-packed middle setting; COLI_VK_KV_FP32=1
is the exact fp32 opt-out, byte for byte the pre-default path. The geometry guard still
forgoes the packed cache — default included, one line on stderr, naming the default when
the variable was unset — at any head_dim/index_head_dim other than 512/128, so non-DSV41
geometry is unaffected.

DeepInfer (the official deepseek implementation of v4.1) quantises the values stored in KV
cache. Colibri uses fp32.

FP32 is undesirable for two reasons:

  1. It consumes more RAM/VRAM.
  2. Deepseek v4.1 was designed with quantization in mind. Not quantising it risks changing
    the behavior compared to upstream.

This is the whole change, in one PR. It supersedes #2701 (the device mirrors alone): every
commit and file of that PR is here in full — this branch is a fast-forward of it plus 9
commits for the host's arrays, and the fixes and the single-flag rework on top (Merge
state lists them). Closing #2701 loses nothing.

What

Three layers ride in this branch; the third is the oldest and the one to read first.

(a) The prefill-VRAM wave, 2026-11-06…08 — 12 commits, +2,098/−82. The kv-quant work
was written on top of it and its ledger is what makes the savings measurable at all:
the dsv41 RAM workspace priced at one prompt chunk instead of the whole context
(family_registry._dsv41_geometry, test_resource_plan.py); a turn-end VRAM ledger
(v41c_ledger) printing weights / chain / tier / kv mirror / process / device; 48
streaming slots reserved before the chunk is sized (vkt_slot_reserve) and a 2-row stream
threshold (COLI_VK_TIER_STREAM_MIN); a page-granular tier pool budget; the indexer's
candidate mask stored as one bit a column; and the experiment record that measured them
(docs/experiments/dsv41-prefill-vram-2026-11-06*, 1,505 lines with audit, comparison
and raw logs). Only the page-granular pool budget's c/backend_vulkan.c hunk is already
in dev (it landed as 96ddc4dc); the rest is not — see "Merge state" below.

(b) The packed cache — the device mirrors and the host's own KV arrays, 14 commits. The
chain's window rings, compressed rows and index keys live in the same formats DeepInfer's
paged cache uses, cutting mirror VRAM 3.9–7.5x depending on the mix (a row: the window rows
3.9x, the compressed rows 7.1x, the index keys 7.5x; 7.2x is the blended 6,400/890 B a
position; the ledger line measures 3.89x because it also sums the compressor's fp32 ring
and the spare window rings, which do not quantize). A row is quantized once, after its last
fp32 reader (chain_dsv4.comp mode 13), and dequantized where read — the attention (flags
bit 2) and the indexer's scores (modes 3/9/10, quant field) decode inline; the pull's
write-back moves the code rows as bytes, so rollback/prefix-reuse/lost-device need nothing
rebuilt. Scale reciprocals use the vendor's bit-exact reciprocal-LUT math on both sides (a
GPU's 1/x is ~2 ulp low, which bit-agreement cannot take).

The host's own state takes the same flag, and the same formats: Layer.window/ckv/ikey,
the V41Seq sets and the draft windows hold rows packed once at write (rope tails staged
in scratch, never packed in place) and unpacked only where a CPU consumer reads them; the
pull and the re-upload are raw byte copies; the planner and v41_state_later price the
packed widths; the turn-end ledger gains a host kv term. The host never gives way to the
KV split (only the device mirrors do, as before) — that invariant is what makes the
planner re-price unconditional under the flag.

Why there is no middle setting. The flag used to carry three levels — fp32, the device
mirrors packed with the host left fp32, and both packed. The middle level earned nothing:
it measured byte-identical to the fully packed cache on the 12-prompt battery against the
official server, and at 16K context only the fully packed cache was ever needed. What it
cost was a second code path — a host fp32 under packed mirrors, so every pull's write-back
dequantized and every re-upload re-packed, the round trip whose subnormal-scale hole
5c4eb903 had to repair. That round trip is still on one path and the ckv demote fix still
guards it — the KV split's give-way packs its dequantized write-back — but the ordinary
packed path is now raw-code pulls and raw-code re-uploads, and the middle level, with the
COLI_VK_KV_QUANT name that spelled the levels, is gone.

The flag is parsed at three sites that also agree on the geometry guard: the chain's
v41c_kvq_on (the mirrors, and with them the host — one condition, so the cache cannot be
half-packed), deepseek_v41.c's v41_host_kvq (the host's own arrays) and
family_registry._host_kv_packed (the planner's mirror). The flag's widths are the only
widths the rows are ever allocated at: the first forward sums mirror bytes against them and
aborts on a mismatch (v41c_kvq_check), and the host has the same guard over its arrays at
load (v41_host_kv_check). When the KV split arms, the mirrors revert to fp32 whole and
re-mirror from the host (logged, asserted) while the host's rows stay codes — a
contingency, not a mode. COLI_VK_KV_FP32=1 is byte-identical to today: every row stride
goes through the v41_row_* helpers, and the flag-off arm of the family is the exactness
proof.

Formats (DeepInfer cache_common.py; reference in c/dsv41_kvq.h)

mirror format bytes/row was
window ring e4m3 + ue8m0/32, [data|scales] 528 B 2048 B
compressed rows e2m1 + e4m3-sf/16 (scale as stored) 288 B 2048 B
index keys e2m1 + ue8m0/32, block-wide sections (32-token blocks, scales packed int32) 68 B avg (whole blocks) 512 B

The arithmetic. At the released geometry (40 layers, 4 KV sources at ratios 2/2/2/1, so
a position costs 2.5 compressed rows): 2.5 * (head_dim + index_head_dim) * 4 = 6,400 B a
position of fp32 rows against 2.5 * (288 + 68) = 890 B packed — 86% less, 5,510 B freed a
position. These are the compressed rows and index keys; the window rings are not per
position (43 rings of window rows plus 240 rollback rows — 6 a layer — live in the planner's fixed term: 3,032,832 B at the released geometry). At 128K that is
0.78 GiB of rows becoming ~0.11 GiB (~689 MiB freed), and 6.25 → 0.87 GiB at the 1M
ceiling (~5.4 GiB). Both are pinned by c/tests/test_dsv41_kvq_host_plan.py and by the
family's ledger assertion below; the analysis behind them is
docs/experiments/dsv41-prefill-vram-2026-11-06.md §10, and docs/deepseek-v41.md
carries the same table, the packed host's rows included.

Measured

All of these are one box: RX 6900 XT (16 GiB, ReBAR), DeepSeek-V4.1-Flash-1017, --ram 28.

  • Ledger, the mirrors' line: kv mirror 50.6 → 13.0 MiB on a serve turn — measured
    under the removed middle level, and the widths the line sums are the same either way, so
    it still reads: the line sums win + ring + ckv + ikey for device 0. At that run's
    context the window rings dominate the sum -- W used rows plus 512 spare rows a layer,
    and those do quantize (a ring is allocated at 528 B a row under the flag) -- but a
    window row only cuts 2048/528 = 3.88x, while the compressor ring stays fp32 whole. So
    3.89x is a window-ring ratio with the fp32 ring inside the mix, not the 7.1x-7.5x the
    compressed rows and index keys cut to a row. Session run: there is no raw line in the tree for the 13.0 MiB side, and the
    context is not 128K — at 128K the fp32 rows alone would be ~0.78 GiB, and the same 50.6
    MiB line appears in docs/experiments/dsv41-prefill-vram-2026-11-06-raw.txt on a
    3-forward, 768-row-chunk run.
    Reproduce with COLI_VK_LEDGER=1 on a serve turn.
  • Tiny fixture, the family's ledger assertion: 6,725,632 B → 1,701,120 B (6.41 → 1.62
    MiB), asserted on the real ledger line each run — the default's packed mirrors.
  • Quality (independent, 2026-11-11): packed default vs COLI_VK_KV_FP32=1, same
    checkpoint, same prompts, temperature 0, --ram 24 --ctx 16384: 10/12 prompts
    byte-identical, 2 differ
    (SUMMARY identical=10 differ=2), both differences in the
    two long free-form answers and both still factually correct — prompt 1 reaches 0.25
    either way, prompt 6 produces the same if x[i] == 0: fix with different prose/case.
    The earlier "the fp32 run differs only in whitespace" was wrong: this is declared
    approximation. Evidence: docs/experiments/dsv41-kvq-2026-11-11/ (E14-compare.txt,
    E12-answers.json, E13-answers.json).
  • RAM, the packed host: −342 MiB engine RSS at ctx 65536 against the planner model's
    344 (5,510 B/pos × 65,536 = 344.4 MiB); the independent 2026-11-11 planner ladder measured
    slot deltas 0 / +1 / +2 at 64K / 128K / 256K (64K landed at 0: 344 MiB is below one
    771.14 MiB expert slot). The −342 MiB is a session run; the slot ladder is pinned by the
    planner test.
  • Ledger, the host term: the turn-end line carries host kv … (packed). The family
    asserts that term on the fixture (0.175 MiB, the codes' own math). The two terms are not
    comparable on a session run — host kv counts every layer's rings and the full-context
    rows, kv mirror counts device 0's chain only — so quote the packed host against the
    fp32 host, not against the mirror line.
  • kv-read cost: 0 by construction on the chain path (the shaders decode the mirrors
    inline, and the pull moves code rows as raw bytes); the one CPU-attention arm measured
    0.57 s against ~26 s of expert streaming (session run: neither number is a ledger line
    in the tree)
    . V41_STATS breaks the decode-on-read cost out as its own kv read line.
  • Honest ceiling: no throughput claim in the measured 16K regime — prefill is disk-bound
    and the tier is near-full. Converting freed VRAM into resident experts needs an explicit
    COLI_VK_TIER_GB raise (+~35 experts at 128K); the packed host does not rescue the
    past-32K walls, which are not KV-sized. The per-context table in docs/deepseek-v41.md
    says this arm by arm, including the two rows that are "unmeasured".

Tests

Gated and in the tree:

  • c/tests/test_vk_kv_quant.c (new, 372 lines). Codec bit-for-bit host vs device — the
    edge rows included: zero amax, negative zero, denormals, amax = 2^k × the format's max,
    both grids' clamp edges, RNE ties, the 32-token block boundary, scales packed as int32 —
    plus unpack exact, the re-upload's little-endian lanes end to end, and pack∘dequant a
    fixpoint — now everywhere, including the subnormal-scale band the 5c4eb903 fix covers.
    The score readers are bit-identical to scores over the dequantized rows; the
    drift against the original rows is that format's own noise, and the top-k overlap is
    gated at ≥70% (it prints 92–100% on the seeded data). The host-side block rules at the
    top (host_block_tests) need no device: staging through 68 B rows bit for bit, whole-block
    ikey prefixes, park's block rounding, the 528 B window slide.
  • tests/vulkan_engines.sh kv-quant-deepseek: every arm is a pair. The
    COLI_VK_KV_FP32=1 run is the exactness side — it must keep the reference's oracle
    (exit 0, the chain's tokens = the reference's). The same configuration with both flag
    names scrubbed (env -u COLI_VK_KV_FP32 -u COLI_VK_KV_QUANT: the legacy name is no
    longer read, and it is scrubbed so a stale export cannot leak in) is the packed default:
    it must engage (its own log line), run the chain, and keep the shared first decode row's
    argmax with drift within that row's own largest |logit|, the token drift printed, not
    gated
    . Six pairs — cap=2, cap=8, a 40-token prompt, the prompt in chunks of 3, the
    per-row GEMV, and a DSpark spec=1 arm that drives ring_save/rollback through the code
    rows. Then the split arm (the split armed, the mirrors gave way, the host codes held —
    the split's read paths dequantize on read and the per-chunk write-back packs) and the
    codec test. The ledger probe, from a one-turn serve run of each side, asserts the
    kv-mirror line against the 528/288/68 B math (6.41 → 1.62 MiB, 6,725,632 → 1,701,120 B)
    and the packed host kv term. The four arms the three-level flag kept for its host
    variants collapse into these plain packed ones: DSpark's ring_save/rollback through code
    rows and the split-with-host-codes arm keep the coverage they added.
  • c/tests/test_dsv41_kvq_host_plan.py (new): the planner seam — 6,400·ctx vs 890·ctx,
    5,510·ctx freed and its share of an expert slot at 64K/128K/256K; the fixed term at the
    released shape, (43 rings + 240 rollback rows), 3,032,832 B packed against 11,763,712 B
    fp32 and the engine's 17,614,592 B at 16K; the fixture carries
    num_nextn_predict_layers, not n_mtp_layers, like the checkpoint; both spellings and
    the no-stage/no-block fallbacks; the geometry guard (≠ 512/128 keeps the fp32 price);
    COLI_VK_KV_FP32=1 still pricing fp32 and the unset variable pricing packed.
    test_context_compaction pins the same seam from the ladder's side.
  • make test on this branch: Ran 3353 tests, failures=2 — the known environmental pair on
    this box (test_e8x4g64_mint_load, test_fp8_e2e_repack_load, the gcc-ABI warnings), 0
    C-test failures, and the same two as the base. One test fewer than before the rework: the
    removed middle level's parametrized arm is gone.

Covered by CI now: test_vk_kv_quant is still TEST_EXCLUDEd from make test (it needs
VK=1 and a device, and skips with exit 0 without one), but the vulkan-engines matrix
runs kv-quant-deepseek on Lavapipe. The family builds the engine with VK=1, generates the
--kv-quant-dims fixture, runs the codec plus all six chain pairs, the split give-way arm,
the CPU-host arm, and the prefix-reuse/DSpark boundary modules; it took 52-63 s on the
RX 6900 XT. A device-less make check still says nothing about the packed path — the CI leg
is what closes that.

The family now runs test_dsv41_prefix_serve and test_dsv41_dspark_serve against the
packed fixture, so prefix reuse and DSpark rollback through the host's code rows are gated
(32 tests, the same 2 harness skips the kv-split family has).

Merge state (dev @ 9f9df6e, resolved 2026-11-09)

dev was merged in a58492f9; the branch is current with dev's tip at push time
(9f9df6e1) and CI-eligible (mergeable). The 7 mapped hunks are resolved, one line each:

  • c/Makefile — TEST_EXCLUDE takes both entries;
  • c/backend_vulkan.c — dev's P->refused = 0; kept;
  • c/deepseek_v41.c — the indexer's work order array and this branch's ikd decode
    scratch are both live and both freed;
  • c/deepseek_v41_chain.h — the ch->kvq/hostq branches ported inside dev's
    v41c_state_up/v41c_state_pull (pack-up under the flag, give-way dequant-up, raw-code
    pulls), with dev's v41c_chunk_tables/prompt_lw preserved.

Two semantic fixes git could not see, both called out in the merge commit body: dev's
mode-11 tiled indexer and three-pass prompt attention (chain_dsv4_pa) have no kvq
codec — they decline under the flag and fall back to the quant-aware grid paths — and
mode 11's mask read is ported to the packed one-bit-a-column rows (mbit). Verified:
the fp32 path bit-identical to pre-merge (the family's flag-off arm), and dev's tiled
mode-11 test still passes bit-for-bit.

dev's qwen36 staging-room shape-pin was updated to the branch's reserved +
vkt_slot_reserve line (intent preserved — the staging room is reserved on the
primary); a sweep of dev's 97 other new/changed test files found no other stale pins.

The 4 env rows are registered (COLI_VK_KV_FP32 — ed323e15 renamed it from the flag that
spelled the levels — COLI_VK_LEDGER, COLI_VK_TIER_STREAM_MIN, V41_STATE_PROBE);
check-env is green (778 variables).

38a6c3dd (docs): the 890 B basis corrected in docs/deepseek-v41.md — the packed rows
alone; the window rings are the planner's fixed term — and the session-artifact pointer
dropped.

92201a68 (the default flip): the packed cache became the default at all three read sites
(engine chain, engine host, family_registry planner mirror); the geometry guard declines
non-512/128 with one stderr line; every exactness arm pinned the fp32 opt-out; the default
arms (env -u) assert the engage line and the unset-serve ledger.

Gates re-run on the merged, flipped tree (RX 6900 XT): check-env PASS; build clean;
vk-kv-quant-check PASS; kv-quant-deepseek PASS (default arms included);
deepseek-chain PASS (81 arms bit-exact, max |logit diff| ~2e-06); kv-split-deepseek
PASS; make test 3354 tests == the base box's 2 environmental gcc-warning failures,
nothing new.

5c4eb903 (the ckv subnormal-scale fix): ckv groups whose amax sits in the near-dead band
[9·2⁻⁹, 10·2⁻⁹] used to pick a scale that a pull→re-upload round trip demotes one step,
shifting the band's largest elements 25% once — a hole in the re-pack fixpoint the CI's
random data never hit. The scale now demotes itself at first pack whenever the scaled
decoded amax would fall ≤ 5.0 (host and shader identically; out-of-band packing
byte-identical; in-band first-pack cost 33–40% on near-dead groups, stated plainly).
Edge rows at 9/9.5/10/15·2⁻⁹ (inside) and 8.9/11·2⁻⁹ (outside) pin the band; the probe
sweeps went 106/4000 hopping rows → 0, and 16,722/2M → 0; the fp32 opt-out is untouched
(the codec is not on its path) and deepseek-chain's exactness stays green.

ed323e15 (the single-flag rework): the three-level flag became one
boolean — unset/0/false/off/no runs the packed cache (mirrors and host rows alike),
1/true is the exact fp32 opt-out — and the middle level is deleted with the hop it
carried. The split's give-way and the ckv demote fix are untouched, the shader and the
codec header changed in comments only, so every bit-exact oracle holds. Gates on it:
check-env 778 variables; build clean; vk-kv-quant-check PASS; kv-quant-deepseek 27 OK
(ledger 6.4 → 1.6 MiB, host kv 0.2 MiB packed, the split armed with the host codes
held); deepseek-chain 82 OK; kv-split-deepseek 41 OK; make test 3353 tests,
failures=2 — the same environmental gcc-ABI pair as the base, 0 C-test failures, and one
test fewer because the removed level's parametrized arm is gone.

2df7eb71 + 6de97a34 (a full-branch review's findings, the branch tip): eight fixes,
all with their own failure proofs. The two that matter most: the planner was still pricing
the host's window rings at fp32 while the engine allocates 528 B a row — now
(layers+stages) * window * 528, 7.98 MiB less reserved at the released geometry, checked
against the live host kv 18.5 MiB ledger; and test_vk_kv_quant returned 0 even when its
device-free half failed, so a GPU-less box saw a green vk-kv-quant-check that proved
nothing — it now exits 1 on a host failure and 0 only on a clean skip. The rest: the
pack→dequant→pack fixpoint moved into the device-free half (and proven to fail when the
demote rule is removed); a new arm runs the packed host with the chain off — the
configuration every user without a capable GPU actually runs; one stderr line names the
retired COLI_VK_KV_QUANT and the inversion in its old =0; parked index keys priced by
whole 32-token blocks; kvq ⇒ hostq asserted with a message (proven to fire); and a
2,560 B/row double charge in the split's row price removed. Gates: check-env 779
variables; build; vk-kv-quant-check PASS; kv-quant-deepseek EXIT 0 with the new arm;
deepseek-chain EXIT 0; kv-split-deepseek EXIT 0; planner tests 21 OK; make test the
same 2 environmental failures.

Honest close: dev moves daily (a batch merged the morning of 2026-11-08); if it advances
again before review, the merge is now a small increment, not a port.

Review

Plan-reviewed (8 amendments folded) and post-reviewed (2 nits fixed in 8a8fb6c: the state
probe gated to fp32 rows, a duplicated record pair; a third note is the park arm above).
Three byte-vs-float stride bugs were caught by the gates during development and routed
through the v41_row_* helpers. The packed host's design came from a 34-site
consumer/writer map (6 writers / 11 dequant-on-read / 17 boundaries) and an anchored
memory-to-experts model. The dev merge, the default flip and the single-flag rework were
plan-reviewed and gate-verified post-merge; two semantic auto-merge hazards (the
mode-11/PA float reads, the packed mask read) were caught by the gates, not by git.

Caveats

Approximate mode, on by default (COLI_VK_KV_FP32=1 is the exact fp32 opt-out). FP4 index
keys can move a top-k pick, and the pull's write-back returns dequantized rows under the
split's give-way (bounded by the pack's idempotence). The dev2 mirrors quantize too but the
ledger line stays device-0-only (a pre-existing gap, annotated on the line). The packed
cache needs head_dim 512 and index_head_dim 128; at any other geometry it stays off by
its own check — the default included, one line on stderr.

Supersedes #2701. Analysis lineage: the session's VRAM audit + the DeepInfer buffer contrast
identified the mirrors as the only context-scaling lever with same-quantity precedent; every
number above either has an artifact under docs/experiments/dsv41-prefill-vram-2026-11-06*
or is marked as a session run.

Validation (independent, 2026-11-11)

RX 6900 XT (RADV NAVI21), 18 CPUs, 38 GiB RAM, DeepSeek-V4.1-Flash-1017; every GPU run
on the real device, not Lavapipe.

check result
clean VK=1 build (deepseek_v41, test_vk_kv_quant) exit 0, 0 warnings
test_vk_kv_quant 7 OK, PASS, exit 0; ASan/UBSan clean
vulkan_engines.sh kv-quant-deepseek exit 0, 28 OK (27 + the boundary line), 0 FAIL, 52-63 s
stock 64/32 geometry guard declines; first decode row byte-identical to FP32=1
COLI_VK_KV_FP32=1 opt-out 12/12 serve; no (quantized, no host kv
packed vs fp32, 12 prompts 10/12 identical, 2 differ (both correct)
16K needle, packed default pass, 16,046 prompt tokens, 978.7 s at --ram 24
planner vs engine at 16K 17,614,592 B host KV (planner = engine)
short-context decode throughput 0.3515 vs 0.3523 tok/s, no change (manifest validated)

The raw evidence is docs/experiments/dsv41-kvq-2026-11-11/ (answers, compare log,
planner JSONs, needle log, ledger lines, sha256sums.txt) and the validated
docs/experiments/dsv41-kvq-throughput-2026-11-11.json. make check was also run
locally: 5 failures (3 from the harness exporting OMP_NUM_THREADS=2, 2 GCC-16 ABI
notes in untouched headers) and 390 environment skips; CI was 117/117 green at the
pre-fix tip and re-runs the same matrix plus the new leg on these commits.

…l context

Track C fix for the --ram 28 / --ctx 131072 / 262144 hard-fail. The
planner priced the v41 forward's workspace at batch == context, so the
runtime reservation ballooned with the context (25 GiB at 131072) and the
RAM expert cache came out with zero bytes; cap_for_arch then raised
"RAM budget cannot hold one expert slot per layer". deepseek_v41.c
prefill reads a prompt in bounded chunks (default 1280 rows), so a forward
never holds the full-context buffers at once. Price the workspace at one
chunk (COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT, matching the qwen38
convention) so a feasible plan is produced again, degrading to fewer slots
instead of hard-failing when the budget is genuinely tight.
…e chunk, and a 2-row stream threshold

Everything here is allocation policy, packing and observability: no row of arithmetic
changes. Measured on the 16 GiB RX 6900 XT with DeepSeek V4.1 Flash and the trunk-first
placement.

1. The VRAM ledger (V41_STATS, or COLI_VK_LEDGER=1/0 on its own). v41c_ledger prints
   one line at a turn's end: the weight pool's blocks and the bytes of rows in them, the
   chain's dev_bytes and its three pools' blocks, the tier's residents and streaming
   slots (count and bytes) with the pool's blocks, the extra layers' pool, the KV
   mirrors, this process's device bytes against the pools, and the card's live
   mem_info_vram_used. vkc_pool_bytes and vkt_vram are the two accessors it needs.

   It answers the audit's open item: the 2.14 GiB a short-prompt run grew by was not the
   tier (whose pools do not move after the warm start) but 1262.5 MiB of token embedding
   + 150 MiB of the 40 routers + 673 MiB of the DSpark stages' matrices, all uploaded by
   the first forward and counted by neither dev_bytes nor the tier's budget; 52 MiB is
   the KV mirrors. The weight pool holds 9608.2 MiB of rows at the turn's end against the
   7523.0 MiB the setup places.

2. The streaming slots: 64 -> 48 (COLI_VK_TIER_STREAM_SLOTS; the measured runs used 44
   at most), and vkc_chunk_rows now reserves vkt_slot_reserve() before it reads the free
   memory, so the chunk and the slots no longer fight over the same headroom: the default
   chunk goes 1280 -> 1024 rows. The tier splits its budget between residents and slots,
   so the slot set alone frees nothing (64+91 and 48+107 fill the same 13 pool blocks of
   256 MiB); what it and the reserve do is take the engine's own hold at the turn
   boundary from 15.53 to 15.26 GiB (chain 2133.7 -> 2060.0 MiB) with the prefill wall at
   +2% (289.95 -> 295.60 s). The card's sysfs peak does not move (15.907 -> 15.906 GiB)
   because the desktop yields: with the tier forced to 2.0 GiB the engine held 1.46 GiB
   less and sysfs fell 0.19 GiB, the compositor taking 1.27 GiB of it.

3. COLI_VK_TIER_STREAM_MIN=2: the rows from which a cold expert is streamed, measured
   instead of assumed (the engine's own line said the GEMM's 16 rows were never
   measured). One 21.1 MiB DeepSeek V4.1 expert, by the new bench (COLI_VK_CHAIN_BENCH=
   xbrows in tests/test_vk_chain.c): a device step costs 0.12 ms at 1 row, 0.21 at 2,
   0.40 at 4, 0.76 at 16 (the tiled GEMM from there) against a 2.2 ms upload and the
   CPU's 2.05-2.52 ms a row, so two rows already pay for the upload; the GEMM route
   below 8 rows is slower than the GEMV, so COLI_VK_TIER_GEMM_ROWS stays 16. On the
   chunk-256 workload (COLI_VK_CHAIN_ROWS=256, the 2600-word prompt) the CPU's share of
   expert-rows falls 34.5% -> 1.6% and the prefill 226.7 -> 175.9 ms a row, at 2 TB of
   expert reads instead of 74.7 GB: it pays in disk bandwidth, not in time.
…'s extra layers upload

A tier pool's limit and its blocks disagreed by a page. tier_pool_limit sized a block from
the budget rounded up to a page (15360 -> 16384 B) but left the limit at the budget itself,
which the pool cannot grow a block past; pool_suballoc then fell back to the limit floored
to a page (12288 B) and counted a refusal. At deepseek_v41's tiny-budget case
(COLI_VK_TIER_GB=0.00005: a three-expert tier, and one extra-layer expert's bytes -- 15360
B -- in the extra layers' pool) the pool could never hold its expert: the first MTP stage's
upload was refused, the run's line read "failed 1 refused by the pool", and T.xmax fell to 0
with the extra layers left on the CPU. The expert needs exactly 15360 B (three tensors of
4096 rows + 1024 scales): the budget was right, the page floor of the block capacity lost
the 3072 B that made it fit.

The streaming-slot reservation commit is not this: with c615cca's six files reverted to
05e2c6d the same run fails identically (same "no room for streaming slots" line, same
counters resident 2 / uploads 6 / evictions 4 / failed 1), and vkt_slot_reserve() is 0 in
it -- the extra-layer take leaves the tier at fit_raw 2, so streaming is off and the chunk
already reserves "0.00 GiB of it the tier's slots and staging". The exposure is older (the
9/10-of-the-heap mapped window, and the extra layers' share of a tiny budget: 9cfa502's
trunk-first placement fixed the window for deepseek_v41 only) and it surfaced when the
deepseek family was run on the discrete ICD, where the MTP stages are the tier's extra
layers by default and not on the CPU as they are on a device that shares the RAM.

1. tier_pool_limit (backend_vulkan.c) rounds its limit up to a page, as its blocks already
   are, so a budget of N expert bytes holds N expert bytes -- at most a page (4095 B) over
   the tier's own budget, another page-granular figure. The tiny-budget case now keeps the
   extra layer on the device: extra layers (2) 4 of 48 routed experts, resident 1 (budget 1),
   uploads 7, evictions 4, failed 0, and the run's output is the CPU's.

2. vkt_slot_reserve (vk_tier.c) caps the reservation at what the budget holds beside the
   residents, and reserves nothing when the budget cannot hold the slots at all: it sizes a
   chunk, it is never a claim on the room an upload or an eviction needs (defensive: it is
   not what refused anything above, where it was 0).

3. cu_tier.c renames vkt_slot_reserve and vkt_vram for the CUDA build of vk_tier.c, which
   'make -C c test' compiles: with the two accessors c615cca added, that build did not
   compile (redefinition against vk_tier.h's stubs).

Measured on the 16 GiB RX 6900 XT: the default-budget serve (ctx 16384, short prompt, one
turn) is unchanged -- 48 slots reserved, tier 1434.4 MiB of 1536.0 in 6 blocks as 68
residents, extra layers 168.8 MiB, process 11.28 GiB of 11.34 GiB in pools, chunk 768 rows
with this desktop's 3.5 GiB already on the card (1024 on an idle one); the same run without
this change gives the same figures.
…e that shares the RAM

deepseek_v41's DSpark loop asserted that the MTP stages never go on the tier by default.
That is true where the device's memory is the host's (Lavapipe): the tier's extra layers
default to the CPU there. On a discrete card COLI_VK_TIER_MTP's default puts them on the
tier, so the assertion fired on every DSpark case of the family when it ran on the radeon
ICD, before any of them could be judged. The check now keys on the engine's own ",
shared RAM," line, which says which of the two it is; the cases that ask for the stages on
the tier (COLI_VK_TIER_MTP=1) are untouched and still require them there.
…tic cap buys twice the rows

The indexer's candidate mask is only ever read as a boolean -- the score kernel tests
it, the host's copy of it feeds a "kept or not" byte, nothing multiplies by it -- and it
was an fp32 word a column: rows x wcap of them beside rows x wcap fp32 scores. It is a
bit a column now (vk_chain.h: vkc_dsv4_cand writes it, vkc_dsv4_score reads it, mask_row
and mask_off in words of 32 columns), so the pair costs 4E + E/8 bytes a row instead of
8E. No row of arithmetic changes: the same blocks are picked and kept whole, and the
selection the top-k runs on is the same set of columns for every row (the indexer case in
tests/test_vk_chain.c compares the mask column for column, through the single-workgroup,
the wide-query and the grid kernels; the families give the CPU's tokens).

1. The chain (c/deepseek_v41_chain.h): isc stays fp32 at rows x wcap; mask and its host
   copy are rows x ceil(wcap/32) words, and the host unpacks the bits where it builds
   m->candidates for the CPU's layers after a partial chain. The chunk's quadratic cap
   (v41c_forward_seg) was rows <= 32M/E, which is the 256 MiB the fp32 pair could take;
   it is now the same 256 MiB over 4E + E/8 a row, i.e. ~62M/E (1.94x the rows), and it
   prints one line when it binds -- nothing printed it before, which is why the audit had
   to derive it.

2. Measured, the tiny V4.1 fixture at a 20000-position prompt: the chunk the tier's
   stream line reports goes 1677 -> 3253 rows, the row cost the sizing prints falls 181
   -> 105 KiB, and the new line reads "a prompt of 20000 positions caps a chunk at 3253
   rows (80.6 KiB a row of scores and packed candidate mask, 256 MiB of 256)". On the
   released checkpoint at E=3107 the row cost goes 1293 -> 1281 KiB a row (the mask was
   12.4 KiB of that, one bit a column is 0.4).

3. Two items of the wave-2b list are not there, measured rather than argued (the
   correction is in docs/deepseek-v41.md so it does not come back):
   - the 1262.5 MiB the turn-end ledger attributes to the "token embedding" is the lm
     head. tie_word_embeddings is false on this checkpoint, both tensors are 129280 x
     5120 bf16, and the embedding has never been on the device: its only reader is
     wb_row_f32, which copies the rows a step needs into the host's rows (one a token
     when decoding, one a position for a chunk). A run with every device upload printed
     shows head.weight as the 1262.5 MiB item and the embedding never uploaded at all.
     The head is a GEMV of 662M multiply-adds a token, so it stays where it is, as do
     the 40 routers' 150 MiB.
   - the per-layer vkc_finish (deepseek_v41_chain.h, after the attention frame) does not
     wait on anything the host does not read. A 1280-row chunk spends 28095 ms waiting
     for the frame it reads and 0.5 ms for the rest: two frames are in flight and a
     single queue runs submissions in order, every frame starting with a barrier on
     everything submitted before it, so the previous layer's shared expert is always
     through. Waiting only the read frame was implemented, measured at 0.5 ms a chunk and
     dropped. The audit's R4 estimate (2-8 ms a chunk) was right; the wait that remains
     is the attention itself, and the device is idle during moe_run_at instead.

Gates on the 16 GiB RX 6900 XT (RADV), against the same three families at 955a0f6:
tests/vulkan_engines.sh deepseek (29 OK), deepseek-chain (35 OK), prefill-deepseek (2
OK), make -C c vk-chain-check VK=1 (PASS), make -C c test (test-c green; test-python
3226 tests). Three failures and two more are identical at both commits and are
environmental: "deepseek_v4 session: no matmul ran on the device" and "chain deepseek_v4
4 experts: the chain never ran" (the deepseek_v4 engine's dense placement declines with
the desktop holding VRAM; its tokens still equal the CPU's), "pf deepseek_v41 staged:
logits" (max |logit diff| 3.22e-02 at both), and the two
test_harness_builds_without_warnings cases (a GCC ABI note in mistral_vision.h).
The commit before this one quoted the row cost the packed candidate mask saves as
"1293 -> 1281 KiB a row" at E=3107. 1293 is the row cost at E=11 (the short-prompt arm),
not at E=3107: the same 8 bytes a column make it 1317 KiB there, which is the number the
audit prints as well, so the pair is 1317 -> 1305. Measured on the released checkpoint at
E=1300 this run: 1303 -> 1298 KiB a row (the mask's 5.1 KiB a row of 41 packed words).
The saving itself was never in doubt -- 8 bytes a column become 4 + 1/8 -- only the base
it was subtracted from.
…hunk, not the context

The planner priced the forward workspace at the whole context, so at --ram 28 a
--ctx 131072 or 262144 serve came out with a zero-byte RAM expert cache and
cap_for_arch() hard-failed with "RAM budget cannot hold one expert slot per
layer". deepseek_v41's prefill reads a prompt in bounded chunks (default 1280
rows; COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT set it), so the workspace is
priced at one chunk and stays bounded as the context grows. The regression test
(tests/test_resource_plan.py) builds a two-layer 128-expert checkpoint and checks
that both long contexts keep a positive expert cache per tier.

Python-only; no engine source changes.
…orkspace gives

05e7748 pinned the contexts chat_context() recommends for DeepSeek V4.1 when the
prompt's workspace was priced at the whole context: 16384 on 16 GB, 65536 on 64 GB,
131072 only from 512 GB, and the refusal to start at 262144 (53.7 GB of workspace, no
room for one expert slot). 008a01b prices the workspace at one prompt chunk, so the
same table reads 65536 on 8 GB, 131072 on 16 GB and the family's 262144 ceiling from
64 GB. The workspace is still in the sum and still in the reason it prints (0.3 GB at
every step); the comment says which share applies at each step and what the table was
before.

Caught by `make -C c test` on the merge of fix/ram-plan-128k: 7 failures, these five
and the two known environmental test_harness_builds_without_warnings cases. With this
change the target is green (3019 tests, 0 failures, 358 skipped).
…ts it tested

What the wave after Track A and Track B measured, tied to the commits that carry it:
the turn-end ledger (c615cca) closing the audit's 2.14 GiB as the lm head, 40
routers and the DSpark stages rather than the tier; the 48 streaming slots and the
chunk reserve (0.27 GiB off the engine's own hold, 0.001 off the card); the 2-row
stream threshold (chunk 256: the CPU's share of expert rows 34.5% -> 1.5%, the
prefill 226.7 -> 178.0 ms a row, at 1.15 TB of expert reads a turn); the tier pool's
page (90959d6); the packed candidate mask (12b2908: 1.94x the chunk for the same
256 MiB, 1677 -> 3253 rows at a 20000-position prompt) with a6486c7's correction of
the base it was quoted against; and the merge of the RAM-plan fix (9508327) with the
first GPU run it ever had: --ctx 131072 and 262144 plan 21 and 20 slots a layer, load,
and finish a turn in 17.9 and 18.1 s with no OOM.

The corrections are called out rather than folded in: the 1262.5 MiB is the lm head
and the embedding never touches the device; the per-layer drain costs 0.5 ms a chunk
and waits on nothing the host does not read; and the card's sysfs peak does not move
when the engine's hold does, because the desktop takes the difference (1.46 GiB held
less, 0.19 GiB read less, 1.27 GiB to the compositor).

The comparison (Track A) and the audit (Track B) are copied in byte-identical, because
their predictions are what the wave tested; the raw file carries every log line the
record quotes, the planner's before/after table and the two serves above. The 50 Hz
VRAM sample series stay as scratch: the record names them, nothing commits them.
The full gate on the merged tree at the previous commit: deepseek 29 OK, deepseek-chain
36 OK, prefill-deepseek 2 OK (one environmental failure each), vk-chain-check PASS,
make -C c test with test-c ALL PASS and test-python 3227 tests / 294 skipped / the two
environmental test_harness_builds_without_warnings failures. The section also records
that this target's own totals move between runs (3227, 3019, 3227 on the same box), so a
count alone is not a regression signal.
…nd gitignore with it

955a0f6's guard grepped ', shared RAM,' which no code path prints (vk_tier.c
prints 'shared RAM (COLI_VK_TIER_GB)', 'shared RAM: a quarter of what the
expert cache leaves' or '(shared RAM)'), so the DSpark-on-shared-RAM default
assertion was silently skipped on every device. It now matches any 'shared RAM'
wording and never 'device memory'. vulkan.md gains the tier's new defaults
(STREAM_SLOTS 48, STREAM_MIN 2), the record fixes its quoted line, the
pair-cost labels, the 12.1 KiB mask share, and lists the two instrumented runs
its M1/M2 numbers come from; .gitignore covers the audit scratch.
mem_info_gtt_used beside mem_info_vram_used: system RAM behind the
aperture is where the host-visible pools land when device-local is
exhausted, and that spill is host-RAM pressure the process RSS does not
show (a 256K-token prefill on a card that had lost 3.3 GiB of its
device-local to a leak spilled pull and the mirrors there while RSS
looked steady). One more field on the turn-end line, 0.00 when the
driver does not export it.
… formats

The chain's window rings, compressed rows and index keys mirrored as
DeepInfer's paged cache stores them (c/dsv41_kvq.h, the host reference):
an MXFP8 window row (e4m3, a ue8m0 scale a group of 32, 528 B), an FP4
compressed row (e2m1, an e4m3 scale a group of 16 as stored, 288 B) and
FP4 index keys in block-wide sections (a ue8m0 scale a group of 32, a
block of 32 tokens, 68 B a row on average). chain_dsv4.comp packs a row
after its last fp32 reader (mode 13) and dequantizes it where read (the
attention's flags 2, the index scores' quant field) and on the pull's
write-back (mode 14), the scales' reciprocals as DI's bit-exact LUT
math (a GPU's 1/x is a couple of ulps low, which the codecs' bit
agreement cannot take). Off by default; the flag branches only at the
mirrors' widths and stores, a forward's first check sums the mirrors'
bytes against the flag's widths, and the KV split still decides as
before: when it arms the mirrors are fp32 whole again, re-mirrored from
the host.
…ode gated

tests/test_vk_kv_quant holds the device's pack to dsv41_kvq.h's reference
bit for bit (the edge rows of the scale math included: a zero amax,
denormals, -0, an amax a power of two over the format's max, the clamp
edges and the RNE ties), its unpack, a re-upload's little-endian lanes
and the pack's idempotence (a fixpoint in values: a group at half its
scale's span re-packs a step down to the same decoded row), and the
index scores over quantized keys to the decoded rows' bits with the
original rows' drift and top-k overlap reported. The kv-quant-deepseek
family runs the tiny fixture at the released KV geometry
(make_dsv41_tiny.py --kv-quant-dims: head_dim 512, index_head_dim 128,
the formats' own widths) under the flag against a documented logits
tolerance -- the flag-off run is the exactness proof, as
V41_PREFILL_TAIL's tests put it -- with the token drift printed and the
ledger's kv-mirror line pinned to the 528/288/68 B math.
…d save

ENVIRONMENT.md's flag row; deepseek-v41.md a subsection by the
trunk-first placement: the three formats, the ~6,400 -> ~890 B a
position of mirror, the ~690 MiB at 128K tokens and 5.38 GiB of the
peak at the 1M ceiling (the prefill-VRAM analysis's §10 figures), the
approximate-mode caveat (a pull returns dequantized rows, the pack's
idempotence bounds the hop) and the KV split's precedence.
…he index staging an index row wide

Two bugs the tiny fixture's first quantized run found, both in the
chain's integration rather than the codec (which the unit test had
already pinned bit for bit):

- vkc_write's offset is in FLOATS, like every chain offset -- the pack
  re-uploads passed bytes, so the index keys' scale section landed four
  times past its block and the frame died as a copy out of range. The
  three upload helpers now divide by 4.
- the fp32 staging for a chunk's new index rows was sized with
  index_n_heads (4) where a row is index_head_dim floats (128): the
  idx_wk matmul wrote past 224 bytes of scratch. qik is an index row
  wide, and the chunk-row accounting with it.

The kv-quant-deepseek gate with them: the first decode row is the one
input the two runs share, so it is the row the tolerance gates (the
argmax kept, the drift within the row's own largest |logit| -- the tiny
fixture's random weights leave gaps narrower than the format's noise;
past that row a flipped pick feeds the model its own choice), and the
ledger's kv-mirror line comes from a one-turn serve run of each flag
setting (the ledger ends a turn; the CLI never has one): 6.4 -> 1.6 MiB
on the fixture, the 528/288/68 B math rounded to whole index blocks.
The test's last line is the plain PASS the gates grep for.
…s, the gate as it gates

The docs' 128K/1M figures now sit on the corrected 6,400/890 B a position
basis (0.78 -> 0.11 GiB at 128K, 6.25 -> 0.87 GiB at 1M), with the audit's
older savings-as-totals numbers named as such; the family's comment states
the gate it runs (the input-shared first row keeps its argmax, drift within
that row's own largest |logit|) instead of a tolerance it does not use; the
idempotence claim says values, not bytes; the kvq fixtures and run CSVs are
ignored; the wide score case's comment names mode 10's grid arm and that
mode 9's quantized reader is the same expression.
…B rows, park keeps whole blocks

COLI_VK_KV_QUANT=2 will hold the host's KV state in the cache formats too; before any
storage moves, the index keys' block-wide sections get their addressing primitives and
the rules they pin down. kvq_ikey_row_gather/scatter move one row between a mirror and
the 68-byte linear staging a boundary (a pull's write-back, a raw upload) writes through;
the whole blocks covering rows [0, n) are the linear prefix kvq_ikey_mirror_bytes(n)
long, so a park, a reset or a zero-fill that rounds to whole blocks may copy bytes and
nothing else may go linear. ColiParkPart grows a 'block' field (0 = today's linear rows)
with coli_park_per_blocks: what a park keeps rounds up to whole blocks, and the restore's
part equality checks it. The tests: gather/scatter bit for bit against the pack, the
prefix whole (a 33-row keep covers rows 0..63 and a staged write inside lands), park's
rounding arithmetic, and the window ring's slide staying linear at 528-byte rows.
…ost state in the cache formats under COLI_VK_KV_QUANT=2

The host's own KV arrays become uint8_t rows at the flag's widths (window 528 B,
compressed 288 B, index keys 68 B in block-wide sections, rounded to whole blocks),
with every offset into them byte arithmetic through v41_row_win/ckv/ikey -- a stride
of head_dim on the new pointers would scale by bytes and land at a quarter of a row,
which is exactly the bug the flag-off serve gate caught mid-swap (the window ring
checksum diverged while ckv/ikey matched): reconstructed with every stride as bytes,
and the flag-off path byte-identical again (the state probe over a two-turn serve
run matches the base binary value for value, and the prefix-serve module that failed
13/14 mid-swap is green -- the failure was never suite-order dependence, it was this
stride bug; the base branch passes the module either way).

Writers pack at their stores through a scratch row (rms and the rope tail stage in
the scratch, never in place inside a code row): the prefill chunk, the decode ring
slot, the index keys, the compressed rows and the DSpark stage rings. Readers
dequantize on read: the kv_all build unpacks the ring and the owner's compressed
rows, the indexer decodes into a per-call scratch keyed on the RESOLVED base (a
speculative step's rows may read different mirrors mid-batch), the DSpark kv_all
unpacks its stage ring. cstate_kv/cstate_score stay fp32 (the pooling reads raw
values). The reset rounds its index-key reach to whole blocks; park keeps whole
blocks (coli_park_per_blocks, park.h's new block field); the swap and the seq sets
carry the arrays' bytes with them. v41_host_kv_check sums the allocated bytes
against the flag's widths after every allocation site, the twin of the chain's
v41c_kvq_check, so a width a branch missed fails loudly.

The device side meets it: the chain's flag parse accepts =2 (so the mirrors
quantize under it too), the re-upload's packers become raw copies when the host
holds codes and the give-way's fp32 re-mirror dequantizes the codes up, the pull
stages codes under the flag (raw pulls, the index keys as whole blocks -- mode 14's
dequant work deleted from that path only, flag 1 keeps it), the write-back scatters
or packs per the staging, and the split's parts (VkcKvPart/VkcKvDs, a trailing
zero-defaulting kvq field) unpack the host's code rows on their way to the device's
fp32 tables -- the host never gives way to the split, only the mirrors do.

The engines' own VkcKvPart initializers gain the trailing 0; the -Werror port
(-Werror=incompatible-pointer-types) is clean across the vulkan engines.
… under COLI_VK_KV_QUANT=2

_dsv41_geometry's state sum takes the cache's own row widths while the flag engages
the host layout (356 B a compressed row against 2560 B fp32 -- 890 against 6400 B a
position at the released geometry, 86.1% less), gated by the same geometry guard the
engine's v41_host_kvq applies: head_dim 512 and index_head_dim 128, else the fp32
price stays (the formats are defined at those widths). The comment states the
invariant the re-price rests on: the HOST layout never gives way to the KV split
(only the device mirrors do, and the split's paths read the codes), so the packed
price holds whenever the flag engages, split or not -- pricing fp32 there would
under-reserve by 5510 B a position. v41_state_later (a parked mux conversation's
sets) re-prices on the same widths, so it cannot disagree with the ledger's line.

The test pins the seam: the packed and fp32 sums at 64K/128K/256K, the freed bytes'
share of a RAM slot, the geometry guard and =1 staying fp32. The slot DELTAS
(+1/+1/+2 against per_cap 771.14 MiB) are whole-plan arithmetic -- a floor of a
difference is not a difference of floors -- and their evidence is the serve smoke's
"[v41] RAM plan" line, flag 0 against flag 2.
…n the turn-end line

The turn-end VRAM line walked the device mirrors only, so everything the host KV
state cost was invisible in every line. Under COLI_VK_KV_QUANT=2 the line gains
'host kv <MiB> (packed)' -- v41_host_kv_bytes over the model's own arrays (the
rings, the compressed rows, the index keys, the DSpark stages' rings), passed by
the engine's call site, which is where the Model is. Flag 0 and 1 print nothing
there: the fp32 host is today's cost, not news.
…do rows and the ledger's host term

family_kv_quant_deepseek gains the =2 arms, tolerance-gated like the flag's own:
the flag-off run stays the exactness side, the =2 run may drift a token (the count
and the reference's Matching line printed), the first decode row within its own
scale. Four arms cover the host's round trips: a short and a long prompt, a chunked
prefill, and a DSpark spec run -- the arm that drives the verify's ring_save and
rollback through the code rows, which no flag-1 arm does. v41_kvq_ledger extends to
a =2 serve turn and pins the new host-kv term to the codes' own arithmetic (the 6
rings and undo rows, the two source layers' compressed rows and whole-block index
keys, the stage rings: 183 168 B = 0.175 MiB on the fixture).

v41_kvq_host_split gates =2 under the KV split: the split armed, the give-way line,
the host codes held, and the first decode row against the flag-off split run. Two
honest notes baked into the arm: the split on the KV-QUANT fixture is inexact at
flag-off too (a combination nothing ran before -- the kv-split family walks the
stock-dims fixtures, and the base branch mismatches the oracle there the same way),
so the off run's own row is this arm's reference; and the arm carries both quantize
hops (the mirrors' and the host's), which on this fixture's narrow logit gaps moves
a greedy pick -- the tolerance is the row's own scale and the flip is printed, not
failed. The chain's flag line says the real value (=2) and the host side prints its
own engagement line, which is what the arms assert on.

Also fixed on the way: the raw pull's vkc_copy counts are FLOATS (the bytes form
overflowed the staging and lost the device), and the pull's widths follow what it
CARRIES -- the give-way's fp32 mirrors pull floats and pack at the write-back, so
the staging is not sized for codes under the split.
…s rows say they are fp32

vk-chain-check caught it: VkcKvDs.Ch became uint8_t* and the cold staging's
'a->Ch + r*hd' scaled by bytes -- a quarter of a row on the fp32 hosts (v4's and
the test's), five of the split's ds cases wrong. The stride is r*hd*sizeof(float)
for the fp32 hosts and KVQ_CKV_BYTES for the code rows. deepseek_v4's initializer
casts its fp32 rows for the new field, with the note that v4's host never holds
codes.
…ine, not a subtraction

V41_STATS' attention field lumped the host's code-row decode into the attention
pass, so the dominator's overhead (the ring decode and the kv_all build) was
invisible. t_kvall accumulates just those two loops and prints its own line -- the
CLI's per-run report and each serve turn (V41Req carries the turn's snapshot); the
indexer's decode stays inside attention, it is not the dominator.
…rence

ENVIRONMENT.md's COLI_VK_KV_QUANT row gains the =2 semantics (the host state in the
formats, the planner re-price with its slots arithmetic, the host-kv ledger term,
the kv-read line, and the host never giving way to the split). deepseek-v41.md
gains the subsection after the mirrors': what moves (the writers' scratch rows,
the dequantize-on-read, the byte boundaries with the index keys' block rule),
the memory model's honest table (freed RAM, slots, effective --ram, the regime
tags on every projection, the no-16K-throughout-claim, +experts only via a tier
raise), and the 34-touchpoint map's reference.
…the record pair once

The TEMPORARY state probe read the window/ckv/ikey rows as floats over
float-sized bounds, which underreads nothing but overreads ~4x once the
host rows are 528/288/68 B codes: it now runs only where the rows are
fp32 (the flag-off and =1 paths), and says so. The duplicated
kv_prefix_taint/record pair at the turn end was an inert edit artifact
and is once again one. Evidence closed with the final binary: make
test's failure set still exactly the base's 23; the kv-quant family
green including the =2 arms; the =2 prefix-serve module 14/14 through
park and replay (the permanent arm is the PR's noted follow-up); no
shader changes on the branch.
@jameswitney jameswitney changed the title dsv41: COLI_VK_KV_QUANT=2 -- the host KV state in the cache formats too (approximate, off by default) dsv41: COLI_VK_KV_QUANT — the device KV mirrors (=1) and the host KV state (=2) in the checkpoint's cache formats (approximate, off by default) Nov 8, 2026
…v41c_state_up/v41c_state_pull

dev's qwen36 staging-room shape-pin updated to the reserved+vkt_slot_reserve line the wave introduced -- intent preserved.

The conflicts and the two sides' overlaps, as resolved:
- c/Makefile: TEST_EXCLUDE keeps both dev's test_vk_dev2_window and this branch's test_vk_kv_quant.
- c/backend_vulkan.c: dev's P->refused = 0 beside the page-granular pool bump.
- c/deepseek_v41.c: dev's indexer order array and this branch's ikd decode scratch both live, both freed.
- c/deepseek_v41_chain.h: dev's v41c_state_up/v41c_state_pull carry this branch's kvq/hostq branches inside (the pack uploads, the give-way's dequant up, the raw-code pulls under =2); dev's v41c_prompt_lw stages the candidate mask at the packed rows' width.
- c/vk_chain.c: dev's tiled indexer (mode 11) and three-pass prompt attention (chain_dsv4_pa) decline under COLI_VK_KV_QUANT's flags -- their readers predate the 528/288/68 B mirror widths; the grid modes serve them.
- c/shaders/chain_dsv4_idx.comp: mode 11's mask read ported to the packed one-bit-a-column rows (chain_dsv4.comp's mbit), as every mask row now is.
- c/coli_env.h: the rows COLI_VK_KV_QUANT, COLI_VK_LEDGER, COLI_VK_TIER_STREAM_MIN, V41_STATE_PROBE.
Conflict: .github/workflows/check.yml — the job timeout. dev raised it to 50 for
JustVugg#2750/JustVugg#2759 (GGUF MTP suite); this branch raised it to 60 after JustVugg#2712 was cut at
35m23s mid-build. Kept 60 (the larger) and both rationale comments.

Everything else auto-merged: c/Makefile, c/coli_env.h, c/deepseek_v41.c,
c/tests/vulkan_engines.sh, docs/ENVIRONMENT.md, docs/deepseek-v41.md,
.github/workflows/ci.yml.

Built and tested after the merge (results in the PR comment).
@jameswitney

Copy link
Copy Markdown
Contributor Author

Independent review of this branch (96a6d3b8) — measurements and findings

We reviewed the branch against 78b459ce on the same class of machine the PR's evidence uses
(RX 6900 XT 16 GiB, DeepSeek-V4.1-Flash-1017), with adversarial review passes over all 57
changed files plus runtime lanes (builds, both kv-quant families, ASan, make test, the planner
ladder, real-checkpoint runs, and the real upstream service). Everything below is reproducible
from the artifacts; every number is from a log, not from the branch's prose.

What holds up (verified, not taken on trust)

  • The codec is a faithful DeepInfer implementation, measured on real rows — against DI's encoder; DI's service casts its inputs to bf16 first, so byte-parity with a running service is not attainable by construction. An independently
    written reference (from DI a679bb86: cache_common.py, asc/kv_quant.py,
    tile_kernels/torch/cast.py; no shared code) reproduces the branch's packed bytes on
    40,000/40,000 window rows, 13,395/13,395 index-key rows and 13,389/13,395 compressed rows —
    the six exceptions are the documented kvq_pack_ckv demote band. Element level: 0 differing
    e4m3 codes in 20.48 M and 0 differing e2m1 nibbles in 10.3 M. The device packer agrees with
    the host on all 56,771 captured rows. Index-key rank order vs the reference: 100 %.
  • The cache values are DI's, bit for bit. DI stores the dequantized value in bf16; every
    V4.1 cache format carries at most 6 significant bits, so bf16 holds them exactly — 0 of 2,359,296
    elements move
    . That is the strongest accuracy statement available for this change and the
    branch does not currently make it.
  • Accuracy on the released checkpoint is good. Paired teacher-forced prefill comparison (a
    prefill reads the unquantized window scratch, so this cannot see decode-path defects), 2,459
    positions, packed vs COLI_VK_KV_FP32=1: mean ΔNLL −0.0016 nats [−0.0099, +0.0067], mean
    |ΔNLL| 0.102, argmax agreement 94.7 %, and at the 130 flipped positions fp32's own
    top1−top2 gap has median 0.105 nats (vs 1.462 where the argmax was kept). The 16K needle is
    pass/fail identical on both arms (16,046 tokens; packed 842.7 / 908.6 s, fp32 886.7 s —
    the wall difference is disk noise; the device-wait delta is ~5 ms per prompt row).
  • Against the real upstream service (deepseek-v4.1-flash through our gateway, temperature 0,
    thinking off, two upstream runs — the colibri arms are the PR's own single E12/E13 runs): upstream is deterministic 12/12; exact match on the 12-prompt
    battery is 9/12 for both arms, and mean normalised edit distance to upstream is
    packed 0.039 vs fp32 0.059 — the packed arm is not the outlier. With 9/12 identical for both arms, at most two prompts are
    informative and two of them hit the 110-token cap — this is a sanity check, not a measurement of
    quality.
  • The fp32 opt-out holds. Fixtures: cmp-identical. Same-session real-checkpoint runs:
    median |Δlogp| 1e-6, max 3.6e-3, zero positions above 1e-2 (head vs head: byte-identical).
    A same-build placement change costs 1.4e-2 median / 0.72 max — three orders more than the
    build does. (An earlier pair that looked like a regression was cross-session spread: base
    differs from itself by 2.9e-2 median.)
  • Memory arithmetic is exact (17,614,592 B @16k, 5,510 B/pos, planner == engine to the byte), the
    geometry guard declines byte-exactly, the committed .spv regenerate from the .comp, and the
    whole-block rules hold at caps 33/97/1000/4097 with a mutation caught.

Findings, in severity order

P1 — the CUDA DSV41 chain was not ported, and fp32 alone does not fix it.
deepseek_v41.c:1319 re-typed Layer.window/ckv/ikey to uint8_t * with 528/288/68 B rows;
deepseek_v41_cuchain.h (not in the diff, zero kvq references) still uses `l->ckv + (size_t)t0

  • hdwithnphdsizeof(float) (:457, :464-465, :685, :691-692). Packed: at 16K the pull writes **14,417,920 B past a 2,359,296 B** buffer (*computed* from the chain's own expressions — f1_spans.c; there is no CUDA device on this machine, so this is a static proof plus arithmetic, and the 4:1 fp32 overlap is reproduced in a harness, not on a card); fp32: no overflow but rows overlap 4:1 (base 204,800 B at t0=100 vs head 51,200 B). Reachable on COLI_CUDA=1 + a V4.1 GGUF (COLI_CUDA_CHAINdefaults on). Fix needs the six strides throughv41_row_*` and a fallback, plus the tier/planner
    predicate; patch and a mock-backend test are available.

P1 — the packed host decodes the window ring before writing the current row (ratio==0).
deepseek_v41.c:3095-3100 fills win_dec before the loop's store at :3227; kv_read stays the
old snapshot (:3211). Measured with a round-trip oracle (fp32 plumbing with each format's pack→unpack at its stores),
the first decode row departs from what the codec says it stores by 1.2399 max|Δlogit| —
13.4× the codec's own 0.0924 on that row (8.1× its read-side 0.1615); the total packed-vs-fp32
gap on that row is 1.2758. On the long fixture the same pair gives 0.8223; its 2.3997 packed-vs-fp32
figure is not input-shared and is not a loss, so no ratio is quoted from it. On the fixture this is 3 of the 7 greedy steps the default loses against the torch
oracle (probe takes it 1/8 → 4/8), and it is 96 % of the prefix-reuse gap (1.293 → 0.0463).
spec_attention already does it in the correct order (:4429 then :4459).

P1 — the planner prices the prefill workspace at min(context, 1280) while the engine allocates
at n.
forward_full allocates n*hc*dim, … for the whole fresh prompt (:4809-4859) and
serve passes n_prompt - reuse in one call (:5808; coli_prompt_segment_rows is never called).
At 262144 that is 250 MiB priced against 65.0 GiB allocated (266×), plus an unpriced 64 GiB
candidate mask. Flag-independent, and it moves the admission ladder: 262144 from 20 GB up (base:
16384/32768/65536). The "53.7 GB removed" is a deleted price, not a removed allocation.

P1 — speculative decoding is not token-exact under the packed cache, and the shipped test cannot
see it.
The DSpark arm forces the vendor reference's tokens into the draft, which the packed
model reproduces 1/8 / 0/6 of the time, so every draft is rejected and "bit-identical" is
plain-vs-plain (0 of 15 forced drafts accepted). With the draft source changed to the packed
model's own greedy tokens, V41_SPEC_FORCE=1 accepts 2/9 and differs: 5 of 8 rows, first at
position 14, max 3.026. With the P1-2 probe applied the arm accepts 5/5 and emits the plain tokens,
residue ≤1.1e-2 — so most of this is P1-2 plus the codec's ring read, not a separate commit/rollback
defect. One confirmation run with the ordering fixed would settle it. Reproduced on the CPU path, so tier routing is excluded.

P2, selected (all with file:line and logs):

  • the packed default silently disables chain_dsv4_pa and the tiled indexer: +61.7 s of device
    wait at 8K (4.4×/forward, ~5 ms per prompt row), while docs/deepseek-v41.md's 16K table says "~0" / "nothing moves" and
    docs/ENVIRONMENT.md still lists both kernels as on. Bit-exact, so time only — and a prototype that restores it is bit-identical to mode 0
    (17/17 shapes) and to the fp32 pa kernel (0/524,288) at 4.9-6.3× mode 0's speed (and at parity with the fp32 pa pass it replaces) on lavapipe
    (not shippable yet:
    its barriers are Workgroup-scope over a global buffer);
  • the shipped end-to-end logits gate is a 100 % tolerance (1.0 × max|logit| on one row), and
    the unit test's absolute gates are loose enough that a whole-binade scale error on both sides still
    passes (rel0 0.158-0.188 against ≤ 0.2); a winq i>>5→i>>6 mutant leaves the suite PASS while
    an ikey mutant fails 3 arms. The read paths themselves are exact — mode 0 flags 2 and indexer modes 3/9/10 are bit-identical
    to the host codec and to a scalar reference (patch with those tests available);
  • the split's arming test still prices the mirrors at fp32 (cap*hd*4 = 5,120 B/pos vs 890), so
    the packed default buys 0-1 doubling steps; the documented "640K vs 160K" is arithmetically
    impossible (890 B × 640K = 556 MiB, below the ~1 GiB reserve it must beat);
  • the bundled tier wave changes fp32 outputs for every MoE engine (chunk 1536→1024, floor 16→2)
    and asserts GEMV≡GEMM bit-equality at vk_tier.c:1036-1039 with no test covering it;
  • the host e4m3 encoder is a 255-code linear scan: 470 µs/row vs 1.14 µs for the shader's
    bit-math encoder (411×, identical bytes);
  • housekeeping: the ledger probe hard-codes the tiny fixture (12 identical serves), host kv is
    printed at %.1f and gated on a wide band, plan-16384-*.json in the evidence dir are stale by 329,472 B
    (17,285,120 vs 17,614,592), battery.py/needle.py/compare.py are not in the tree, and
    the body's check-env/test counts do not reproduce at this tip (check-env 790 against the body's
    778/779; make test 3,435 here against base 3,418, i.e. +17, not "one fewer"; counts are
    box-dependent). Its "0 warnings" line is true of the two targets it names — but a plain VK=1
    build prints 2 (glm_chain.h:612, on the new kvq field) and the nine engine targets 11 in total.

On the ckv demote

Under "match DI" as the governing criterion the demote is the one place colibri deliberately differs
from upstream (stored sf 0x01 vs 0x02 in amax ∈ [9,10)·2^-9). Measured on real rows it is
small — 6 of 13,395 rows, ≤1.2e-3 row rel_l2, invisible to the indexer — so this is a fidelity and
documentation decision, not a defect: if DI parity is the goal, drop it on the first pack and keep
it only on the give-way re-pack.

Recommended before merge

  1. CUDA: stride fix + fp32 fallback (+ tier/planner predicate), with the mock-backend test.
  2. The decode ordering in attention_run (pack, then decode, as spec_attention does).
  3. Price the workspace at the forward's n (or chunk the forward), and re-pin the admission ladder.
  4. Decide the demote; document the band.
  5. Add the exactness tests for flags=2 / quant=1 and a DI-frame gate (golden bytes + a dNLL band
    with decode steps — a prefill-only read-out cannot see the window format or the ordering bug).
  6. Re-price the split at 288/68 B; regenerate the plan artifacts; fix the ledger probe and the
    %.1f gate; either split the tier wave out or land it with the old defaults plus one
    default-valued streaming arm per engine.

We can attach the harnesses, patches and raw logs. Nothing here was written into this branch by the
review; the measurements are read-only against 96a6d3b8 / 78b459ce.

… fp32 rows

`Layer.window/ckv/ikey` became `uint8_t *` when the packed KV cache landed, and
this chain kept base's float offsets (`l->ckv + (size_t)t0 * hd`): read as a
byte stride over rows `model_load` allocates at 528/288/68 B, the pull's 16K
write-back runs 14,417,920 B past a 2,359,296 B compressed-row buffer under the
packed default, and at COLI_VK_KV_FP32=1 the four-quarter stride makes every
row overlap 4:1. cc_write's own `off*4+bytes` bound (cuda_chain.cu:2201) makes
the device side right, which is why nothing aborted.

Every host-KV row offset in the chain now goes through v41_row_win/ckv/ikey
(c/deepseek_v41_cuchain.h:467,474-475,695,701-702), which are hd*4 in the
COLI_VK_KV_FP32=1 arm and the code widths otherwise; cc_write's `off`, in
floats, is untouched. And v41u_wanted returns 0 when v41_host_kvq(&m->c)
(:140-150): a packed host keeps the prompt's layers on the CPU -- the exact
pre-packed path -- with one `[v41]` line saying so, because this chain reads
fp32 rows and nothing else.

Static proof only: there is no CUDA device in this environment, so the runtime
gate (tests/cuda_tier_engines.sh, which needs a card) is not in this branch's
evidence. New c/tests/test_cuchain_host_strides.py, discovered by `make test`'s
unittest discover: 3 failures at 80d9cf8, 3 OK here. `make cuda-host-syntax`
rc=0 before and after.

Evidence: reports/f3-host-final.md:115,252-257; reports/f6-plumbing-final.md:79;
reports/f9-default-compat-final.md §8.1; reports/fix-plan-final.out.md §4 commit 1.
… the pre-store float row

attention_run decodes the window ring into `win_dec` at the top of a decode forward
(deepseek_v41.c:3105-3110), and then the per-position loop packs each step's key into
its slot AFTER that snapshot was taken: a ratio-0 layer's kv_read IS win_dec
(:3211), so the slot the step just wrote still held the row it EVICTED. The query
listed it (window_idxs, :3242) and attended a position it never had.

Measured on dsv41_kvq_tiny, packed host vs COLI_VK_KV_FP32=1, chain off, per DUMP
row (row 0 = the prefill forward, row 1 = the first decode row):
  before: row 1 max |diff| 1.2758 of 1.5927 = 0.801, argmax 206 vs 64 FLIPPED
          (row 0, today's gate, 0.1649 of 2.2744 — argmax kept, so it is blind to this)
  after:  row 1 max |diff| 0.0924 of 1.5927 = 0.058, argmax 206 vs 206 kept
and 0.0924 is exactly the codec's own term for that row (the W6 round-trip pass: 0.0924
= round-trip oracle - fp32, while packed - oracle was 1.2399 = 13.4x it). With this
change packed - oracle is 0.0000e+00 on all 8 tiny and all 6 long rows: the packed host
is now bit-identical to the round-trip oracle, i.e. no implementation error is left --
what remains is the format's own loss.

The fix is the re-decode of the one slot with values, gated on `!ratio` (the only arm
where win_dec is kv_read; a ratio layer reads kv_all, which takes the step's floats
below). Not a memcpy of the pre-pack row: every other reader of the ring sees the
dequantized code row, so the snapshot has to be the same values.

Gate: the new logits_row() (tests/vulkan_engines.sh:1806) checks DUMP row 1 of the same
fixture, rel 0.2 against that row's largest |logit| with the argmax kept, called from
v41_kvq_cpu_host (:5034) -- the packed host with the chain off, the configuration this
bug damages. Fails at 80d9cf8 (1.2758, 0.801, FLIPPED), passes here (0.0924, 0.058,
kept); `bash tests/vulkan_engines.sh kv-quant-deepseek` is OK end to end on lavapipe.

Evidence: reports/f3-host-final.md:115,329-331; reports/w6-roundtrip-final.md:17-19,81-97;
reports/fix-plan-final.out.md §2.2, §6.5-§6.7, §4 commit 2.
…m it would without them

V41_SPEC_FORCE drafts the REFERENCE's own next tokens (deepseek_v41.c:7024-7030), so
the family's packed spec arm -- forced drafts verified against the vendor ref.json --
only ever proves the reference's stream. The packed model does not reproduce it (its
first decode row's argmax already moves on this fixture), so every forced draft is
rejected for a reason that has nothing to do with the cache: 0 of 9 at 80d9cf8, and 0
of 9 after the P1-2 fix too. And with COLI_VULKAN=1 the fixture's six layers chain to
the device, so attention_run's CPU path -- where a decode step's ring slot is written --
never runs the bug this arm should catch: that configuration accepts every draft even
unfixed (5 of 5, 3 of 3, measured both ways).

The new v41_kvq_dspark_self takes the packed model's OWN stream as the reference: chain
off, no Vulkan (the configuration P1-2 damages), the plain run's tokens T written into a
ref file of its own, then every draft forced. All must be accepted, and the verified
stream must be T token for token. Measured, dsv41_kvq_long:
  80d9cf8:  1 of 6 forced drafts accepted, stream 233 207 13 6 85 62 vs plain 233 207 13 17 60 72
  this tree: 3 of 3 forced drafts accepted, stream = the plain one (233 207 190 166 168 57)
It also asserts the packed host engaged and that the verify path ran at all ("NOT
exercised" is deepseek_v41.c:7137's own line for a force that drafted nothing).

`bash tests/vulkan_engines.sh kv-quant-deepseek` is OK end to end on lavapipe with this
arm in it (28 OK lines, rc=0); the same family FAILs at 80d9cf8 -- at the P1-2 row-1
gate, which this arm sits behind.

Evidence: reports/fix-plan-final.out.md §2.1, §4 commit 3, §6.8; reports/f3-host-final.md:330;
reports/w8-transitions-r1.out.md:44,50,107-109; reports/f7-tests-ci-final.md §8.2.
…locates

_dsv41_geometry priced the forward workspace at min(ctx, 1280) rows of 204,800 B, on
the claim in its docstring that deepseek_v41.c "prefill reads a prompt in chunks
(default 1280 rows; COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT set the chunk size)". That
is not true of this engine: `grep -c coli_prompt_segment_rows c/deepseek_v41.c` is 0,
`grep -c 1280` is 0, and neither COLI_PROMPT_SEGMENT nor COLI_VK_CHAIN_ROWS appears
there at all (COLI_VK_CHAIN_ROWS is the chain's spare ring rows, deepseek_v41_chain.h).
forward_full allocates its buffers at n and the whole prompt is one forward. So the plan
reserved 250.0 MiB where the engine asks for 101.65 GiB at 262144 (416x), which is what
let 262144 be admitted from 20 GB up.

The workspace is now the engine's own per-row closed form, term by term from the
allocation sites (deepseek_v41.c:4816-4871 and attention_run's five buffers), at the
context: 66,588 floats a row for forward_full (2*hc*hidden + 3*hc + 2*hidden + hc*hc +
targets*hidden) and 37,504 for attention_run (2*q_lora + heads*head_dim + 2*head_dim +
window + 2*index_topk) = 104,092 floats = 416,368 B a row, 2.03x the 204,800 B/row
priced before, plus the single `hidden` row the form always priced. The per-row buffers
of both hops are priced because the ladder is identical either way (verified: 416,368
and 266,352 give the same table). The non-row transients are named in the comment and
unpriced: `heads` (MV_ROWS_MAX * heads * head_dim * 4 = 4 MiB), `head_score` (163,840 B),
`embedded`/`collapsed` (2 * hidden * 4).

The three new reads (q_lora_rank, num_attention_heads, index_topk) are _required_int like
the rest of the geometry: a config without them is refused rather than under-priced, and
the tests' fixtures gain the released values.

The ladder this produces (chat_context, GB = 1e9, quarter share up to 65536 then a
tenth), packed and COLI_VK_KV_FP32=1 alike -- the flag moves the host's KV rows, not the
prefill's frame:

  box      before    after
  8 GB     262144    None (the engine's default)
  16 GB    262144    None (the engine's default)
  64 GB    262144    32768
  256 GB   262144    65536
  512 GB   262144    65536
  1024 GB  262144    131072

That is the corrected price doing its job: 16384 alone is 6.8 GB here, so a 16 GB box
loses the context step 05e7748 exists to give it, and the real fix for that box is a
chunked host prefill in deepseek_v41.c -- which this branch does not do. The plan's
table wrote 131072/262144 for the last two rungs; it had used 2^30 for `available` while
resource_plan.GB is 1_000_000_000, and the pinned numbers are the ones the code produces.

Tests: the two ladders in test_context_compaction.py:171 re-pinned (both arms, same
table); a new closed-form assertion in test_dsv41_kvq_host_plan.py:172 (workspace ==
ctx*416368 + hidden*4 at 16384 and 262144, packed and fp32, and 3 fewer hidden rows with
no DSpark head); that file's fixture hidden_size 2048 -> 5120 plus the released
attention widths; test_resource_plan.py's Track C test, whose "the workspace must stay
bounded as the context grows" was the same false premise, now pins the closed form and
keeps its positive-expert-cache assertions. 157 tests OK here, 14 failures at 80d9cf8.
The plan did not list that last file; it cannot stand unchanged.

Evidence: reports/f5-planner-final.md:100-101,215-224,263-270 and F5-11;
reports/a1-planner-admission-final.out.md; reports/fix-plan-final.out.md §2.3, §6.4, §6.9-§6.11.
tests/test_vk_kv_quant covered the codec, the window/ckv/ikey round trips and three score
arms, but no arm ever read a PACKED cache through the two readers the PR made default:
VkcDsAttn's flags 2 (the decode-on-read window ring and compressed rows) and the
indexer's quant 1 / mode 0. The only arms that reach them are in test_vk_chain.c, whose
test_ds_attn runs flags 0 and 1 (no packed cache) and whose VkcDsScore initializers leave
quant 0 -- so the two paths that a user with the default cache runs had no test that
could fail at all.

Four attn_packed arms (mixed rows with a window ring of exactly nwin; V4.1-Flash's 64
heads of 512; a list past the kernel's 256 threads; one entry a row with the sink at
-inf) and six index_quant arms (mode 3 with a partial last block, the wide queries of
mode 9, three of mode 10's grid -- 1005, 2048 and 8 keys -- and 2048 keys off FP4's own
grid). Each pins the packed arm's bits to the same call over the rows kvq_unpack decodes
from the mirror, and to a scalar attention/score sum in the shader's own order, rel_l2
0.00e+00 on this device. The three gates that were absolute are exercised the same way.

Plus the mutant hook the arms need to prove they can fail: VK_KV_QUANT_MUT=win|ckv|ikey
flips one scale byte in the packed mirror and requires the packed arm to move while the
fp32 arm (packed from the original bytes) does not. Measured on lavapipe:
  VK_KV_QUANT_MUT=win  -> "a one-code change in a window scale moves the packed arm"
  VK_KV_QUANT_MUT=ikey -> "a one-code change in an index-key scale moves the packed scores"
and at 80d9cf8 the same env var is a no-op: the base binary's 7 OK lines are byte
identical with and without it (diff empty, rc=0), which is the coverage hole stated as a
measurement rather than as an absence.

Evidence: reports/w6-exactness-final.md:158,171-181; reports/f7-tests-ci-final.md §8;
reports/fix-plan-final.out.md §4 commit 5, §6.13. Patch from
proposals/.../w6-exactness/exactness-tests.patch, applied rc=0 with no offsets.
score_case gated the FP4 index keys' drift against the ORIGINAL fp32 rows at rel_l2 <=
2e-1 and top-16 overlap >= 0.7. Those are not a band, they are an order of magnitude
wider than the format's own noise: an ue8m0 scale rule floored instead of ceiled -- in
BOTH the host's kvq_ue8m0_scale and the shader's ue8m0c, so the codec is wrong by a
power of two a block -- moves the five arms to rel0 1.58e-1 / 1.46e-1 / 1.88e-1 /
1.53e-1 / 1.52e-1 and overlaps 92/100/91/81/87%, and the old gates print PASS (measured:
rc=0, no FAIL line). The suite could not see a whole-binade codec error on the default
path.

The gates are now the measured band -- rel0 <= 1.35e-1 (5% above the worst arm) and
overlap >= 0.80 (10% below the worst) -- and two arms make the band mean something: 2048
keys above VKC_DSA_GRID (top-16, where the selection can move) and the checkpoint's own
8192 keys with index_topk 512 (k/pool 6.25%). score_case takes the top-k as a parameter
for them.

Same floored-scale codec under the new gates: 5 FAIL lines ("rel_l2 to the original
rows' scores 1.581e-01" / "1.459e-01" / "1.878e-01" / "1.535e-01" / "1.516e-01"), rc=1.
Measured on both devices, five arms identical to every printed digit:
  lavapipe     1.13e-1/92%  1.22e-1/100%  1.28e-1/94%  1.04e-1/91%  1.01e-1/89%
  RX 6900 XT   1.13e-1/92%  1.22e-1/100%  1.28e-1/94%  1.04e-1/91%  1.01e-1/89%
so one band serves both drivers -- the packed codec is bit-exact against its own
reference (rel_l2 0.00e+00) and the drift is the format's, not the device's.

Hand-merged: the patch and commit 5's exactness-tests.patch both touch this file and
neither order applies (hunk JustVugg#3 of the score patch lands on the call list commit 5
rewrote). Hunks 1-2 applied with `patch`; the call list was merged by hand and both files
compile together.

Evidence: reports/f1-codec-final.md:89,277; reports/f10-accuracy-final.md §0.1;
proposals/w3-accuracy-frame/README.md; reports/fix-plan-final.out.md §2.6, §4 commit 6, §6.12-§6.13.
The packed host's own arrays were printed as `host kv %.1f MiB (packed)` and gated to
0.15 < v < 0.25 -- a 100 KiB window around a 183,168 B total, and at one decimal the line
cannot tell 179,504 from 186,831 (both print 0.2), so the gate tested the rounding.

The line now prints the total in bytes (`host kv %zu B (packed)`,
c/deepseek_v41_chain.h:1924) and the ledger's gate parses those bytes and holds them to
+-2%: 179,504 .. 186,831 B, which is 183,168 B -- 6*(8+4)*528 + 128*288 + 256*288 +
(4+8)*2176 + 2*8*528, the derivation the comment beside it already carried. The old
decimal is unpinnable any narrower than a whole 0.1 MiB.

Measured, one serve turn of the packed fixture (`bash tests/vulkan_engines.sh
kv-quant-deepseek`, lavapipe):
  before (80d9cf8): `host kv 0.2 MiB (packed)` -- the new parse finds no bytes field
                     (`sed -n 's/^host kv \([0-9]*\) B.*/\1/p'` prints nothing), so the
                     gate cannot even read this line
  after:             `host kv 183168 B (packed)`, gate holds, and six arms print it
                     identically (cap=2, cap=8, the 40-token prompt, chunks of 3, the
                     per-row GEMV, DSpark spec=1); the family is rc=0 end to end.

Evidence: reports/f7-tests-ci-final.md §8.3; reports/fix-plan-final.out.md §4 commit 7,
:5069-5074's own derivation.
c/dsv41_kvq.h:21-24 still said "The host keeps its state fp32 holding the DEQUANTIZED
rows (a pull's write-back and a re-upload's source)". That sentence is the mechanism of
the P1-1 miss: the arrays have been uint8_t code rows at 528/288/68 B since the default
flip, and a reader who believed the header could write `l->ckv + t0 * head_dim` and be
off by a factor of four -- which is exactly what the CUDA chain did. It now says what the
code stores, that every offset is a byte stride through v41_row_win/ckv/ikey, and that
the same functions are the host's store and read. `grep -rn 'fp32 holding the
DEQUANTIZED' c/ docs/ README.md` finds nothing.

c/dsv41_kvq.h's header note and docs/deepseek-v41.md's cache section (from
proposals/w3-di-spec/docfix.patch, applied rc=0 with no offsets) now say which DeepInfer
commit the formats come from (the store kernel and the layout it writes, tile-kernels
2.1.0), and state the one deliberate deviation plainly: a ckv group whose amax lands in
[9,10)*2^-9 packs one scale code down for the re-pack fixpoint, so its grid stops at
6*2^-9 and every element above saturates there where DI's grid holds 8*2^-9 -- the
group's upper tail 25% low, its mean error 15% against DI's 10%, in that band only
(probe proposals/w3-di-spec/di_ind.c T3; band edges and the three formats di_parity.c
A-G). Also that DI's index-K blocks are 128//ratio tokens, not 32, so the byte streams
are not comparable.

docs/experiments/dsv41-kvq-2026-11-11/README.md cites the two copied plan JSONs: they
were generated before the workspace re-price (commit 4) and their allocation field still
carries min(ctx, 1280) rows, so the citation now says so and keeps them for what they
were copied for, the state figures. They are not regenerated here -- that needs the
checkpoint.

Build after: `make -j6 VK=1 deepseek_v41 tests/test_vk_kv_quant` rc=0.

Evidence: reports/f3-host-final.md §8.4; reports/rt-runtime-final.md §8.8;
reports/f1-codec-final.md §8.3; reports/f8-docs-final.md §8; reports/f9-default-compat-final.md §8.3;
reports/fix-plan-final.out.md §4 commit 8.
No sanitized family sees the packed host: family_deepseek_sanitize and the rest walk the
stock 64/32 fixture, where v41_host_kvq's dimension check declines the packed cache, so
the 528/288/68 B host arrays, the pull/re-upload of code rows and the block-wide index
keys were never under ASan or UBSan -- the class of bug that produced the CUDA chain's
quarter-row stride (3e9d5b7) had no leg that could see it.

`bash tests/vulkan_engines.sh kv-quant-deepseek-sanitize` is this family on a sanitized
build: `SAN=1` makes it `make clean` and build deepseek_v41 and test_vk_kv_quant with
-fsanitize=address,undefined -fno-omit-frame-pointer -g, export the ASAN_OPTIONS/
UBSAN_OPTIONS every other sanitized family sets, run every arm as usual, check the codec
log with san_diag, and sweep the logs the arms write at the end -- a sanitizer report on
stderr never fails an arm's own gate, which reads stdout. Usage lines updated.
.github/workflows/ci.yml gains the matrix twin beside kv-quant-deepseek:441.

Measured on lavapipe, whole family, sanitized build:
  rc=0, 43 OK lines (the codec's 17 included, PASS), 0 "AddressSanitizer"/"runtime error"
  lines anywhere in the log -- including the two host arms (row 1 0.0924, argmax kept;
  3 of 3 forced drafts accepted) and the six ledger arms (host kv 183168 B)
so the packed host arrays, the pull/re-upload and the index keys run clean under both
sanitizers. Cost: a sanitized build (about 3 minutes here) and a second matrix entry.

Evidence: reports/f7-tests-ci-final.md H6 (:117), :72-73, :254; reports/fix-plan-final.out.md
§4 commit 9, §6.14.
N1 (arithmetic): the docstring said the old chunk price under-reserved "266x at
262144". It is 416x -- 262,164,480 B priced (min(ctx, 1280) rows of the old form)
against 109,148,393,472 B allocated -- and the docstring now names both operands so
the next reader can check it against the ladder instead of taking a bare ratio.

N2 (naming): the "named and unpriced" note listed the FIXED transients only
(attention_run's `heads` and `head_score`, forward_full's `embedded`/`collapsed`).
attention_run also allocates four context-sized buffers at its top and frees them at
its end -- `latent` ((n/ratio + 1) * head_dim floats: 256.0 MiB at 262144 ratio 2),
`kv_all` ((window_rows + compress_len) * head_dim: 805.3 MB there, window_rows being
n during prefill and the ring's 128 during decode), the packed ring's `win_dec`
(window * head_dim = 256 KB) -- 1.07 GB together at 262144 ratio 2 and 1.61 GB at
ratio 1, under 1.5% of the 101.65 GiB price and never live beside the next layer's.
They stay unpriced (a per-row form cannot express "one layer at a time"); they are
now named so the omission is a decision rather than an oversight.

N3 (alias): the planner required num_attention_heads, while the engine reads n_heads
first and num_attention_heads second (deepseek_v41.c:199). A config spelling only the
alias is served by the engine and was refused here -- and a refusal is not a failed
plan: resource_plan.py's ladder takes `except ValueError: break` (:3394), so such a
box reports "no RAM plan" and silently keeps the default context. The planner now
reads the alias with the engine's precedence (n_heads wins when both are present).
The closed form is unchanged at the released spelling: 416,368 B a row.

Tests: new test_a_config_spelling_the_heads_n_heads_prices_the_same in
test_dsv41_kvq_host_plan.py -- the alias prices the released workspace, and with
both keys present n_heads is the width priced. Fails at 2072865: "deepseek_v41:
missing or invalid planning key 'num_attention_heads'" (Ran 11 tests, FAILED
(errors=1) in a tree of that commit); OK here. 158 tests OK across
test_dsv41_kvq_host_plan, test_resource_plan, test_context_compaction (157 before).
Full suite: Ran 3473 tests, FAILED (failures=2) -- test_harness_builds_without_warnings
in test_e8x4g64_mint_load and test_fp8_e2e_repack_load, the mistral_vision.h ABI note;
reproduced at 80d9cf8 in /tmp/rev-base (Ran 8 tests, FAILED (failures=2)).

Build: make -C c -j6 VK=1 deepseek_v41 tests/test_vk_kv_quant rc=0, 0 warning lines;
make -C c cuda-host-syntax rc=0. Python only: no C file in this commit.

Evidence: reports/fix-impl-review.out.md nits N1-N3.
… it from

Reusing a turn's state changed the answer under the packed host -- the default --
while COLI_VK_KV_FP32=1 was byte-exact, and the branch's own CI lane was red on it.
attention_run has two readers of the same row and they read different things: a
decode step, and any prefill that resumes a live state, dequantize the ring's 528 B
code row (kvq_unpack_win, start_pos > 0), while a prefill at start_pos == 0 hands
sparse_attend its `kv` scratch -- the exact floats the code row was just packed from,
before e4m3 rounded them. So a prompt prefilled in one forward and the same prompt
continued from its own stored state computed different attention over the same
positions, and only the packed arm can differ: fp32 rows read back exactly.

The row now enters every read as the codec's value -- kvq_pack_win then kvq_unpack_win
in place, per position, before this forward stores or attends to it -- so `kv_all`, the
window half and the ring all hold what a later forward would read. e4m3 re-packs a
dequantized row to itself, so the code row the ring gets is the one a pack of the
untouched floats would have written: no store changes, no layout change.

Measured, dsv41_kvq_tiny, packed (the flag unset):
  tests.test_dsv41_prefix_serve   before 26 failures of 29 (skipped 2)
                                  after  OK, 29 (skipped 2)
  COLI_VK_CHAIN=0                 after  OK, 29
  COLI_VK_KV_FP32=1               after  OK, 29   (unchanged: 3.9 s, was OK)
  lavapipe COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1
                                  after  OK, 29   (with tests.test_dsv41_dspark_serve
                                         and ..._prefix_store: 39 tests OK)
and the mechanism, proved before the fix: put the fp32 layout's ring rows through the
same pack/unpack (nothing else packed) and the same 26 of 29 fail, while packing the
prefill's own rows instead makes the packed arm pass -- so the failure is the
asymmetry, not the format.

Evidence: reports/grind-c1-review.out.md F1; reports/grind-c1-fix.out.md.
…a mirror's sections

The store was handed the pre-flag widths. Every array it parts is uint8_t* now: a ring of
528 B code rows, a compressed row of 288 B, an index key of 68 B in block-wide sections.
v41_ps_state_save copied W * head_dim * 4 = 262,144 B out of a ring W * 528 = 67,584 B
long and v41_ps_take wrote the same 262,144 B back into it -- a 194,560 B heap overrun per
layer, x40, on the default (packed) path, for any prompt that reaches a block (B=256:
prefix_store.h:398), and the same four-times-too-wide stride on the ckv/ikey parts.

Now: v41_row_win/ckv/ikey in both directions, so a save copies the ring's bytes and a
restore writes them back; V41PsState.win is uint8_t*.

The index keys are not one array -- a 32-row block keeps its data rows and then their
scale words 2048 B on (dsv41_kvq.h: "NOTHING else may go linear"), so a bare 68-byte
stride would still read the wrong bytes for a range that starts or ends inside a block.
That is what ColiPsPart.rows_io is for: a part whose rows are not one array says so and
coli_ps_rows copies the staging it writes. v41_ikey_rows_io is the mirror's (per block:
its data rows, then their scale words), and the part declares 68 B a row as before. The
other seven engines' parts pass NULL -- their caches are linear arrays -- which is the
whole of that diff.

Measured, dsv41_kvq_tiny (the packed host), stores driven at COLI_PREFIX_STORE_BLOCK=16:

  tests.test_dsv41_prefix_store        80d9cf8: 4 failures of 7    now: OK, 7
  ASan+UBSan build, 2072865, one serve test named after a resumed prefill:
      SUMMARY: AddressSanitizer: heap-buffer-overflow deepseek_v41.c:5762 in v41_ps_state_save
      now: OK; the whole of prefix_serve + prefix_store under the same build: 36 tests OK
  tests/test_prefix_store (C): OK, and its new block-sections section fails when the part
      is given rows_io = NULL, i.e. the check is not vacuous
  lavapipe, chain on, prefix_serve + dspark_serve + prefix_store: 39 tests OK

The family's SAN leg now runs that store arm itself (COLI_PREFIX_STORE_BLOCK=16) -- its
serve arms' prompts are ~90 tokens against B=256, so no arm reached the store and the leg
was vacuous for exactly this bug -- and the unsanitized packed serve gate runs
tests.test_dsv41_prefix_store, which is what sees the widths without a sanitizer.

Evidence: reports/grind-c1-review.out.md F2; reports/grind-c1-fix.out.md.
43501e8 makes the packed host read its own code rows in the PREFILL too (that is what
makes a resumed prefill compute what a cold one computes), so the fp32 arm this gate
compares against is no longer input-shared past the prefill by construction: the packed
row 1 now carries the query path's quantize hop as well as the store's. Measured on
dsv41_kvq_tiny, row 1 of the dump:

  before 43501e8   0.0924 of 1.5927 = 0.058, argmax kept      (the store's hop alone)
  after             0.1615 of 1.5927 = 0.101, argmax FLIPPED   (both hops)

0.101 is inside the row's 2e-01 tolerance and the argmax is the codec's, not a bug's --
the packed arm's stream already differs from the fp32 arm's (7 of 8 tokens here). So the
row keeps the tolerance and prints the pick instead of requiring it: logits_row gains an
argmax argument (0: tolerance only), the only caller that wants it here passes 0.

The gate keeps its teeth, proved by mutation -- the P1-2 bug put back (the snapshot keeping
the row the slot evicted, win_dec's re-decode removed), same fixtures:

  mutant            row 1  1.1726 of 1.5927 = 0.736     0.2-tolerance FAILS, argmax flipped
  this branch       row 1  0.1615 of 1.5927 = 0.101     passes
  this branch       row 0  0.4447 of 2.2744 = 0.196     argmax kept (logits_first's own gate)

and the property the fix is for is gated much harder by the arm below it, whose 29
prefix-reuse tests compare tokens exactly (26 of them red before 43501e8).

Family, lavapipe, whole `kv-quant-deepseek` lane on this tree: rc=0, 42 OK lines, 0 FAIL,
including "OK kv-quant CPU host: first row max |diff| 4.45e-01 of 2.27e+00 (tolerance 1e+00),
argmax kept; row 1 max |diff| 0.1615 of 1.5927 (tolerance 2e-01), argmax FLIPPED; 7 of 8
tokens differ" and "OK kv-quant serve: prefix reuse, DSpark's rollback and the prefix store
against the packed fixture" (39 tests, skipped 2).

Evidence: reports/grind-c1-review.out.md F1/RC-1; reports/grind-c1-fix.out.md.
… the tree clean

RC-1 of reports/grind-c2-review.out.md is a TREE finding, not a history one: the index
held an 8-file revert of 8f1d2a7 (prefix_store.h's ColiPsRowsIo hook and the family
files' seventh initialiser dropped, while deepseek_v41.c kept the code that calls it), so
a bare `git commit` would have shipped a store that fails
tests.test_dsv41_prefix_store and warns "excess elements in struct initializer".  The
worktree was never wrong -- `git diff HEAD` was empty, and the fix was `git restore
--staged` on the eight files (no commit of content is needed; nothing else changed).

What is committed here is the other half of RC-1: the same lane re-dirties the tree every
time it runs.  `bash tests/vulkan_engines.sh kv-quant-deepseek`, the green gate RC-1 asks
to re-take right before a push, writes off.f32/on.f32/off.txt/on.txt/plain.txt/
forced.txt/self.json into c/ (each arm `rm -f`s its own at the START and never after), so
"confirm git status --porcelain is empty immediately before the push" and "run the lane"
cannot both hold without this.  Seven ignores, in the section that already holds the
kv-quant fixtures.

Caveat for the push, measured here and NOT caused by this commit: a from-scratch build
(`make -B VK=1 deepseek_v41`) re-runs the committed shaders through this box's glslc and
leaves 17 of them byte-different (git status: M c/shaders/*.spv), so `git checkout --
c/shaders` after a full build, before the push.  Those .spv are tracked; only the
verified, unchanged ones may ship.

Evidence: reports/grind-c2-review.out.md F1/RC-1; reports/grind-c2-fix.out.md.
RC-3 of reports/grind-c2-review.out.md (F3). e141623 made v41u_wanted decline a packed
host -- the CUDA chain moves the host's KV arrays as fp32 rows, so a build that packs them
runs the prompt's layers on the CPU -- but v41_tier_chain_bytes, which is what the tier
SUBTRACTS from the device's expert room (VktConfig.chain_bytes, vk_tier.c "what the dense
chain allocates at its first forward"), still gated on COLI_CUDA_CHAIN alone. On the
default path (COLI_VK_KV_FP32 unset: the host is packed) it therefore held the chain's own
v41u_plan_bytes on the device for a chain that never runs: at 262144 its E*LR*4 term for
the candidate lists plus the P*E candidate mask is >= 1.74 GB of device expert slots lost
to every conversation, and nothing in resource_plan.py reads the flag, so the plan could
not see it either.

The fix mirrors the ONE condition that is decidable where the tier sizes itself: the same
v41_host_kvq(&m->c) v41u_wanted tests, negated. The rest of v41u_wanted (g_cud.on,
cc_ready, g_v41u_off) is not up yet at that point, so copying the whole predicate would
have reserved NOTHING for the chain on fp32 hosts too -- a worse bug than the one fixed
here; the comment says which half is mirrored and why.

Fail-first, on this tree (the new test reads the function out of deepseek_v41.c):
  $ git checkout HEAD -- c/deepseek_v41.c
  $ python3 -m unittest discover -s tests -p 'test_cuchain_host_strides.py'
    Ran 5 tests ... FAILED (failures=1)
    AssertionError: '!v41_host_kvq(&m->c)' not found in 'static size_t
      v41_tier_chain_bytes(Model *m) { ... if (xt_cuda() && m->gguf && !(e && *e == '0')
      && g_v41_mux_slots <= 1) return v41u_plan_bytes(m, m->c.max_positions); ...'
  $ (this commit)                                          Ran 5 tests ... OK
Its second test is the floor: xt_cuda/m->gguf/mux<=1/v41u_plan_bytes(m, max_positions)
must all stay in the body, so the chain keeps its room on an fp32 host.

Static only, exactly like e141623: there is no CUDA device in this environment. What
covers it is the test above plus `make cuda-host-syntax` (gcc -fsyntax-only -DCOLI_CUDA
over deepseek_v41.c and the other eight engines) -> rc=0, 0 warnings, 0 errors. The
Vulkan build is untouched by construction (the function's body is #ifdef COLI_CUDA):
`make -j6 VK=1 deepseek_v41` rc=0, 0 compiler warnings.

Not fixed here, and not a bug on inspection: v41_tier_dense_bytes (g_v41_cud) is NOT dead
weight when the chain declines -- V41_CU / v41_cu_mm (deepseek_v41.c:786-792) serve the
CPU path's dense GEMVs from that device store.

Evidence: reports/grind-c2-review.out.md F3/RC-3; reports/grind-c2-fix.out.md.
RC-2 of reports/grind-c2-review.out.md (F2). 19b4249 priced V4.1's prompt workspace
at what ONE forward of n rows allocates and charged it to the plan's runtime term
(family_registry._dsv41_geometry, 416,368 B a row at the released geometry). The price is
right -- the engine does prefills a prompt in one forward, deepseek_v41.c:4692/6050, and
that forward allocates every buffer at n rows (4816-4871) -- but cap_for_arch turned it
into a REFUSAL, and openai_server raised when the plan kept no slot. Released config,
64 GB, `cap_for_arch('deepseek_v41', None, env={'RAM_GB':'64','CTX':...},
model='/root/Models/DeepSeek-V4.1-Flash-1017')`:

  CTX  4096 -> cap 56      CTX 65536 -> cap 25
  CTX 16384 -> cap 50      CTX 262144 -> ValueError: "DeepSeek V4.1 Flash RAM budget
                                         cannot hold one expert slot per layer"

At 262144 the box served NOTHING, at any prompt length, though the workspace is only
1.71 GB for the 4096-row prompt it would actually be sent (109.15 GB is a 262144-row
prompt). F2 and RC-2: keep the cap at >= one slot. The plan itself is NOT changed -- it
still reports zero slots and the "cannot hold one expert slot per sparse layer" warning,
so `coli plan` and `coli doctor` keep telling the truth about a box that cannot hold a
context-long prompt's workspace -- and the floor lives at the launch seam, where the
question is only "may this box start":

  CTX 262144, RAM 64 -> "[v41] RAM plan: 109.1 GB of prompt workspace at CTX=262144 does
  not fit the 64.0 GB RAM budget, leaving no expert slot a layer; starting at 1 -- a
  prompt that long needs the workspace's RAM beside the cache, a shorter CTX, --ram or
  --cap leaves room"   -> cap 1

MiMo keeps its refusal (its workspace is priced at a 64-row block, so a plan with no slot
is a box that cannot hold one expert of every layer -- unchanged by this branch), and so
do the Qwen floors. The context ladder is untouched and still prices the workspace to
choose a context; measured at the released config, unchanged by this commit:
  8 GB -> None (the engine's 4096), 16 GB -> None, 32 GB -> 16384, 64 GB -> 32768,
  256 GB -> 65536, 1024 GB -> 131072.

Tests, fail-first on this tree:
  $ git checkout HEAD -- c/openai_server.py
    tests.test_resource_plan...test_a_workspace_past_the_ram_starts_at_one_slot_instead_of_refusing
    ERROR ValueError: DeepSeek V4.1 Flash RAM budget cannot hold one expert slot per layer
    (new test: a 2-layer fixture at RAM 6 GB / CTX 262144, plan slots 0 -- it also pins
     that build_plan still warns, so the honesty of the plan is not traded for the floor)
  $ ... (this commit)                                      Ran 1 test ... OK
  $ python3 -m unittest tests.test_openai_server.CapSentinelShimTest
    rewrote test_v41_insufficient_ram_refuses_before_spawning (it pinned the refusal this
    commit removes) as test_v41_an_empty_plan_starts_at_one_slot_not_a_refusal, which
    asserts the argv cap is "1" and the note names the workspace, and ADDED
    test_mimo_an_empty_plan_still_refuses_before_spawning so the surviving refusal stays
    pinned                                                        Ran 17 tests ... OK
  $ python3 -m unittest tests.test_openai_server tests.test_env_defaults
      tests.test_cgroup_memory tests.test_resource_plan tests.test_dsv41_kvq_host_plan
      tests.test_context_compaction tests.test_cli_output          Ran 771 tests ... OK

cap_for_arch also binds `context` once instead of computing it inside the build_plan call,
so the note can name the context it is about.

Evidence: reports/grind-c2-review.out.md F2/RC-2; reports/grind-c2-fix.out.md.
@jameswitney

jameswitney commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Fixes for the review, on top of the dev merge (21e45a93)

Sixteen commits on top of the merge in 80d9cf80. Each fix carries the command that fails at
80d9cf80 and passes here (three commits are docs, tree hygiene and a static CUDA check, and say
so in their own row). The branch was reviewed commit-by-commit before it was pushed.

Defects fixed

commit what it fixes proof
e1416239 the CUDA chain addressed the re-typed host KV arrays as fp32 rows (packed: OOB read+write; =1: 4:1 row overlap). It now declines a packed host and byte-strides its fp32 rows through v41_row_* test_cuchain_host_strides.py 3 fail → 3 OK; make cuda-host-syntax rc=0
e798600b attention_run decoded the ring snapshot before the step's store, so ratio==0 layers attended an evicted slot's key row 1 of the CPU-host dump 1.2758/FLIPPED → 0.0924/kept, then (see 43501e8d) 0.1615 inside the arm's tolerance
1029085b the packed DSpark arm forced the reference's tokens, so 0/15 drafts were accepted and "bit-identical" was plain-vs-plain a self-draft arm: 3 of 3 forced drafts accepted, the stream = the plain one
19b42498 + 8f05d124 + 21e45a93 the planner priced the prefill workspace at min(ctx,1280) while forward_full allocates at n (250 MiB vs 65 GiB at 262144). Prices ctx × 416,368 B (the engine's own closed form) and keeps the launch floor 14 planner failures → 157 OK; the ladder at 8/16→none, 32→16384, 64→32768, 256/512→65536
b9b145ad + d7745e47 the suite never ran mode 0 flags 2 or the indexer's quant readers, and its score gates were loose enough to pass a whole-binade scale error +282 lines of arms; a winq i>>5→i>>6 mutant now fails 7 arms (it passed the old suite)
616a366e host kv printed to one decimal inside a wide band prints 183168 B and gates ±2 %
8f1d2a71 the prefix store reused fp32 row widths on the packed arrays: memcpy of 262,144 B out of/into a 67,584 B ring (OOB read and write, ~194,560 B a layer, at prompts past the 256-token block) fixes the overflow, but the lane stays red on the resumed-vs-cold mismatch until 43501e8d: at COLI_PREFIX_STORE_BLOCK=16 on dsv41_kvq_tiny, 22 of 29 fail at 80d9cf80 and at the branch without it (26 of 29 on the long fixture, the commit's own number)
43501e8d a resumed prefill read stored codes where a cold one read fresh floats, so the same text answered differently prefilled at once and continued from its own state the same arm at COLI_PREFIX_STORE_BLOCK=16: 22 of 29 → 29 OK
0b464e86 the tier still reserved chain VRAM for a chain the CUDA decline makes unreachable static (no CUDA device here)
0f70be39 the kv-quant lane left dumps in the tree git status clean after a lane run
20728655 no sanitizer coverage for the packed host new ASan/UBSan family leg: rc=0, 43 OK

The one trade-off, stated

43501e8d makes the packed host read its own code rows in the prefill as well as on store — which is what upstream DI does (it stores quantized and reads dequantized for every position) and what makes a resumed prefill compute what a cold one computes. The cost is that a prompt row now pays the codec's loss twice, and the CPU-host first-decode-row pick moves from 0.0924 to 0.1615 of 1.5927 (inside the arm's 2e-01 tolerance; argmax now flips at that near-tie). f20511c3 therefore relaxes that one clause to tolerance-only, keeping a mutation proof that the original defect still fails it (0.736 > 0.2). If you would rather keep the stricter pick, the alternative is to revert 43501e8d and accept that the prefix-store lane stays red.

Verification

  • build make -j6 VK=1 deepseek_v41 tests/test_vk_kv_quant: rc=0, 0 warnings
  • make test at f20511c3 (the reviewed ten): Ran 3473 tests, EXIT=2 on both that tree and 80d9cf80 — the same two environmental GCC-ABI failures, same skip count, 258 C binaries green; the three follow-up commits add 4 tests (loader count +4). The targeted suites were re-run at the tip: planner trio 159 OK, test_cuchain_host_strides 5 OK
  • vulkan_engines.sh kv-quant-deepseek: EXIT 0, ending with OK kv-quant serve: prefix reuse, DSpark's rollback and the prefix store against the packed fixture
  • test_dsv41_prefix_serve at COLI_PREFIX_STORE_BLOCK=16: 29 OK
  • Gitea (local runner): five jobs green on a snapshot of 8f1d2a71 in pi/colibri (ci-fix-8f1d2a71) — that snapshot is five commits behind this tip; GitHub CI for the tip is queued, not green
  • ASan/UBSan leg of the packed family: no diagnostics

Still open, deliberately

  • the candidate mask is n × compress_len bytes and is priced nowhere — 64.0 GiB at 262144, 38 % of the peak. Bounding it to the tail rows that are read is bit-neutral and takes it to 0.59 GiB, but it is pre-existing and an engine-side change, not this PR's;
  • CUDA is static-only here (no device on the box);
  • the ckv demote remains the one place the bytes deliberately differ from DI.

…family's last

The sanitized family exported UBSan without halt_on_error, and every arm's 2>
truncates off.log/on.log, so the closing sweep read only the last arm's pair --
five of six arms could misbehave into a green lane. UBSan now halts (decide's
form), each arm sweeps its own logs as it finishes (kvq_san_sweep, a no-op off
the SAN leg), the ledger's serve turns tee their stderr to san_serve.log for the
sweep, and the usage ledgers are cleaned at the family's start and ignored, so a
FAILED lane leaves the tree clean too.
Three seams, three agreements. _host_kv_packed parses COLI_VK_KV_FP32 the way
both C parsers do -- a leading 0 is packed, only false/off/no are names -- so no
flag value sits the planner on one side and the engine on the other. The packed
index keys round to whole 32-row blocks (kvq_ikey_mirror_bytes's own math) where
they were priced linearly 68 B a row. The widths and the block are pinned to
c/dsv41_kvq.h's macros by a test that reads the header, and the ladder the PR
measured -- 0/+1/+2 expert slots at 64K/128K/256K -- is now build_plan's own
output under both flag states, not a session run.
… layout says so

coli_ps_span priced a rows_io part once a head as coli_ps_rows never read it;
the span now prices the part's own run, and the layout signature mixes the part's
staging in, so widths alike but moved differently cannot share a store. The unit
test's sectioned part compresses the way production's does (rows_io with
per > 1), its tail's half-read group held unstaged for the state to refill. The
per-model suites: qwen38's expected-hit bound counts tokens, not bytes, and a
read-out that ends on an ERROR frame fails instead of passing silent.
…39 GiB

'Where this engine is not the vendor' still said the engine keeps the KV cache
in fp32 -- the one thing this branch changed by default. Only the indexer's
queries and the compressor's latents stay fp32 now; the cache itself is the
checkpoint's packed formats, COLI_VK_KV_FP32=1 the exact opt-out. The 1M row's
effective --ram was +5.51 against its own column's arithmetic (5.39), and the
DSpark stages with their rollback rows are said outright: 329,472 B.
Conflicts were the two additive lines: c/Makefile's TEST_EXCLUDE keeps both
test_vk_kv_quant (this branch's VK=1 codec test, out of the portable suite) and
dev's test_vk_pcache, and the lane script's usage line keeps both dev's inkling2
family and this branch's kv-quant-deepseek lanes. Everything else auto-merged;
the kv-quant-deepseek lane is green on the merged tree.
@jameswitney
jameswitney marked this pull request as ready for review November 15, 2026 09:34
Dev's 93 commits since 66fa9f6; three conflicts, one of them real.

The real one: dev's JustVugg#2797 (perf/candidate-block-ids) made the host's
candidates int32 block ids while this branch had bit-packed the device's
candidate mask (12b2908), and the two changes meet at v41c_forward_seg's
readback. That hunk now unpacks the branch's bit row -- ceil(wcap/32) words a
row, v41c_mask_words stride -- into dev's temp keep row and hands it to dev's
candidate_row_ids; the device side stays wholly the branch's (dev never
touched vk_chain.h or the scratch sizing, and the planner keeps pricing the
packed mask row, which JustVugg#2797 did not edit). g_v41_park_x takes both sides'
fields: dev's cand_topk/int32 cand and this branch's uint8 code-row published.
indexer_run's free list keeps both buffers (ikd, maskrow), and the CHANGELOG
keeps both sides' Unreleased entries.

On the merged tree: test_dsv41_candidate_ids is green (the frozen 15f810f
writer, 11760 cases, 2464 padding-sensitive), test_vk_kv_quant passes, and
make test-c is green; make test-python passes but for the MXFP8 gq_dot
subtests, which fail identically on dev and on this branch's tip on this box
(a -march=native environment flake, not the merge).
@jameswitney

Copy link
Copy Markdown
Contributor Author

dev's 93 new commits merged (9e43e6c7) — three conflicts, one real

Dev moved to 78be384e (Qwen3.8's MXFP4 expert sidecar, Inkling-2's MTP drafter, and — the one that mattered here — #2797, the candidate blocks as int32 ids), and the PR had gone dirty again. All three conflicts resolve at one seam: #2797 made the host's candidates block ids while this branch had bit-packed the device's candidate mask (12b2908a), and the two meet at v41c_forward_seg's readback. That hunk now unpacks this branch's bit row — ⌈wcap/32⌉ words a row, v41c_mask_words stride — into #2797's temp keep row and hands it to its candidate_row_ids; the device side stays wholly this branch's (dev never touched vk_chain.h, the scratch sizing, or the planner's mask pricing, which keeps ⌈wcap/32⌉ words a row and its test_resource_plan pin). g_v41_park_x takes both sides' fields — dev's cand_topk/int32_t *cand beside this branch's uint8_t code-row published — indexer_run frees both buffers (ikd and maskrow), and the CHANGELOG keeps both sides' Unreleased entries (with #2797's "the device masks stay dense" clause corrected to name this PR's packing).

On the merged tree, before the push:

check result
make test-c (portable suite, incl. dev's test_dsv41_candidate_ids gate) green — the frozen 15f810f0 writer, 11,760 cases, 2,464 padding-sensitive, repaints on [0,w)
make test-python green but for test_gq_dot's three MXFP8 subtests, which fail byte-identically on dev's own 78be384e and on this branch's 2390095d on the same box (a -march=native environment flake, not the merge; CI's lanes are the authority)
make VK=1 green; chain_dsv4.spv regenerates byte-identical to the committed one (the .comp is unchanged since 2390095d)
test_resource_plan + test_dsv41_kvq_host_plan 151 tests OK
vulkan_engines.sh kv-quant-deepseek (+ -sanitize) 39 tests OK both — no sanitizer diagnostic in any arm's log, the prefix store's included

CI is running on 9e43e6c7 now; the kv-quant-deepseek legs are the ones that cover the merged seam on a real device.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant