Skip to content

dsv41: COLI_VK_KV_QUANT -- the device KV mirrors in the checkpoint's cache formats (approximate, off by default) - #2701

Closed
jameswitney wants to merge 19 commits into
JustVugg:devfrom
jameswitney:r2/kv-quant-mirrors
Closed

jameswitney wants to merge 19 commits into
JustVugg:devfrom
jameswitney:r2/kv-quant-mirrors

Conversation

@jameswitney

@jameswitney jameswitney commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

DeepInfer (the official deepseek implementation of v4.1) quantises the values stored in KV cache. Colibri uses fp32.

FP32 is undesirable for two reasons:

  1. It consumes more RAM/VRAM
  2. Deepseek v4.1 was designed with quantization in mind. Not quantising it risks changing the behavior compared to upstream.

The following are changed:

  1. win[] wndow host ring - MXFP8
  2. xkv[] compressed KV rows - FP4 + 8-bit scale per 16 values
  3. ikey[] index keys - FP4 + 8-bit scale per 32 values

Stacks on #2637 (trunk-first) and carries the prefill-VRAM wave it builds on (ledger, 48 streaming slots + budget ordering, 2-row stream threshold, page-granular pool limits, 1-bit candidate mask, chunk-priced RAM planner).

What

The chain's window rings, compressed rows and index keys can live in the same formats DeepInfer's paged cache uses, cutting mirror VRAM ~4-7x (geometry-dependent). A row is quantized once, after its last fp32 reader (chain_dsv4.comp mode 13), and dequantized where read -- the attention (flags bit 2) and the indexer's scores (modes 3/9/10, quant field) decode inline; the pull's write-back dequantizes (mode 14), so the host's state stays fp32 holding dequantized rows and rollback/prefix-reuse/lost-device need nothing rebuilt. Scale reciprocals use the vendor's bit-exact reciprocal-LUT math on both sides (a GPU's 1/x is ~2 ulp low, which bit-agreement cannot take).

The flag branches only at the mirrors' widths and stores; the first forward sums mirror bytes against the flag's widths and aborts on a mismatch; the KV split decides exactly as before and, when it arms, the mirrors revert to fp32 whole and re-mirror from the host (logged, asserted). Flag-off is bit-identical: every existing oracle stays green.

Formats (DeepInfer cache_common.py)

mirror format bytes/row was
window ring e4m3 + ue8m0/32, [data|scales] 528 B 2048 B
compressed rows e2m1 + e4m3-sf/16 (scale as stored) 288 B 2048 B
index keys e2m1 + ue8m0/32, block-wide sections (32-token blocks, scales packed int32) 68 B avg (whole blocks) 512 B

~6,400 -> ~890 B a position: 0.78 -> ~0.11 GiB of mirrors at 128K tokens (~690 MiB saved), 6.25 -> ~0.87 GiB at the 1M ceiling (~5.4 GiB) -- the analysis in docs/experiments/dsv41-prefill-vram-2026-11-06.md and the docs note in this PR.

Measured

  • Real model (V4.1-Flash, RX 6900 XT, ctx 131072, one serve turn, the turn-end ledger line): kv mirror 50.6 -> 13.0 MiB (3.89x, the width math), process pools unchanged, chain on device, no lost device.
  • Tiny fixture (released KV geometry): 6.4 -> 1.6 MiB, the pinned 6,725,632 -> 1,701,120 B arithmetic.

Tests

  • NEW c/tests/test_vk_kv_quant.c: codec bit-for-bit host vs device (edge rows: zero amax, negative zero, denormals, amax=2^k, the e4m3/e2m1 clamp edges, RNE ties, the 32-token block boundary, scales packed as int32), unpack exact, the re-upload little-endian lanes pinned end to end, pack o dequant a fixpoint in values, the score readers vs decoded keys with top-k overlap 92-100%.
  • NEW vulkan_engines.sh family kv-quant-deepseek: flag-off is the exactness proof (the bit-exact families keep it byte for byte); under the flag the input-shared first decode row keeps its argmax (drift within that row's own largest |logit|), token drift printed not gated, and the ledger's kv-mirror line asserted against the 528/288/68 B math.
  • All existing gates green with the flag off (deepseek, deepseek-chain, prefill-deepseek, kv-split-deepseek, make test, vk-chain-check).

Caveats

Approximate mode, off by default (COLI_VK_KV_QUANT): FP4 index keys can move a top-k pick, and the pull's write-back returns dequantized rows (bounded by the pack's idempotence). The planner is not re-priced under the flag; the dev2 mirrors quantize too but the ledger line remains device-0-only (pre-existing gap, annotated).

Analysis lineage: the session's VRAM audit + the DeepInfer buffer contrast identified the mirrors as the only context-scaling lever with same-quantity precedent; every number above traces to docs/experiments/dsv41-prefill-vram-2026-11-06*.

pi and others added 19 commits November 6, 2026 14:03
The dense chain fit now runs before the mapped-weights window is
allocated, so the trunk sees the device's real free memory instead of
the ~0.5 GiB left after a 9/10-heap window ask. The window is then
asked for what remains beside the trunk less COLI_VK_FREE_RESERVE_GB
(1 GiB, floor+warn at <1 GiB ask; COLI_VK_WINDOW_ASK_MB still overrides),
the tier sizes into it with the host pool off by default beside the
trunk (COLI_VK_TIER_HOST=1 / _GB re-enable), and dense host copies
stay given back. Other engines, the CPU-only path and the integrated/
Lavapipe defaults are unchanged; explicit COLI_VK_STAGED=1 still forces
staging. Opt-in is deepseek_v41-only (incl. its GGUF path).

A bounded (<=3 s) wait covers RADV reporting the probe's freed blocks
back through VK_EXT_memory_budget before the tier reads it.

Measured on RX 6900 XT 16 GiB: 40 of 40 trunk layers on the device
(7.62 GiB), window 5 GiB, tier 3.28 GiB / 83 resident experts, no GTT
host-pool mirror, no new swap.
Baseline origin/dev vs the trunk-first commit, one build apart, same
serve parameters and the same warm start: decode 0.485 -> 0.786 tok/s
median (+62%) on a single-NVMe ReBAR card, the tier's 11.5 GiB host
pool on -> off with GTT at ~0.27 GiB, and 40 of 40 trunk layers on
the device beside it. Harness committed beside the record.
…l context

Track C fix for the --ram 28 / --ctx 131072 / 262144 hard-fail. The
planner priced the v41 forward's workspace at batch == context, so the
runtime reservation ballooned with the context (25 GiB at 131072) and the
RAM expert cache came out with zero bytes; cap_for_arch then raised
"RAM budget cannot hold one expert slot per layer". deepseek_v41.c
prefill reads a prompt in bounded chunks (default 1280 rows), so a forward
never holds the full-context buffers at once. Price the workspace at one
chunk (COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT, matching the qwen38
convention) so a feasible plan is produced again, degrading to fewer slots
instead of hard-failing when the budget is genuinely tight.
…e chunk, and a 2-row stream threshold

Everything here is allocation policy, packing and observability: no row of arithmetic
changes. Measured on the 16 GiB RX 6900 XT with DeepSeek V4.1 Flash and the trunk-first
placement.

1. The VRAM ledger (V41_STATS, or COLI_VK_LEDGER=1/0 on its own). v41c_ledger prints
   one line at a turn's end: the weight pool's blocks and the bytes of rows in them, the
   chain's dev_bytes and its three pools' blocks, the tier's residents and streaming
   slots (count and bytes) with the pool's blocks, the extra layers' pool, the KV
   mirrors, this process's device bytes against the pools, and the card's live
   mem_info_vram_used. vkc_pool_bytes and vkt_vram are the two accessors it needs.

   It answers the audit's open item: the 2.14 GiB a short-prompt run grew by was not the
   tier (whose pools do not move after the warm start) but 1262.5 MiB of token embedding
   + 150 MiB of the 40 routers + 673 MiB of the DSpark stages' matrices, all uploaded by
   the first forward and counted by neither dev_bytes nor the tier's budget; 52 MiB is
   the KV mirrors. The weight pool holds 9608.2 MiB of rows at the turn's end against the
   7523.0 MiB the setup places.

2. The streaming slots: 64 -> 48 (COLI_VK_TIER_STREAM_SLOTS; the measured runs used 44
   at most), and vkc_chunk_rows now reserves vkt_slot_reserve() before it reads the free
   memory, so the chunk and the slots no longer fight over the same headroom: the default
   chunk goes 1280 -> 1024 rows. The tier splits its budget between residents and slots,
   so the slot set alone frees nothing (64+91 and 48+107 fill the same 13 pool blocks of
   256 MiB); what it and the reserve do is take the engine's own hold at the turn
   boundary from 15.53 to 15.26 GiB (chain 2133.7 -> 2060.0 MiB) with the prefill wall at
   +2% (289.95 -> 295.60 s). The card's sysfs peak does not move (15.907 -> 15.906 GiB)
   because the desktop yields: with the tier forced to 2.0 GiB the engine held 1.46 GiB
   less and sysfs fell 0.19 GiB, the compositor taking 1.27 GiB of it.

3. COLI_VK_TIER_STREAM_MIN=2: the rows from which a cold expert is streamed, measured
   instead of assumed (the engine's own line said the GEMM's 16 rows were never
   measured). One 21.1 MiB DeepSeek V4.1 expert, by the new bench (COLI_VK_CHAIN_BENCH=
   xbrows in tests/test_vk_chain.c): a device step costs 0.12 ms at 1 row, 0.21 at 2,
   0.40 at 4, 0.76 at 16 (the tiled GEMM from there) against a 2.2 ms upload and the
   CPU's 2.05-2.52 ms a row, so two rows already pay for the upload; the GEMM route
   below 8 rows is slower than the GEMV, so COLI_VK_TIER_GEMM_ROWS stays 16. On the
   chunk-256 workload (COLI_VK_CHAIN_ROWS=256, the 2600-word prompt) the CPU's share of
   expert-rows falls 34.5% -> 1.6% and the prefill 226.7 -> 175.9 ms a row, at 2 TB of
   expert reads instead of 74.7 GB: it pays in disk bandwidth, not in time.
…'s extra layers upload

A tier pool's limit and its blocks disagreed by a page. tier_pool_limit sized a block from
the budget rounded up to a page (15360 -> 16384 B) but left the limit at the budget itself,
which the pool cannot grow a block past; pool_suballoc then fell back to the limit floored
to a page (12288 B) and counted a refusal. At deepseek_v41's tiny-budget case
(COLI_VK_TIER_GB=0.00005: a three-expert tier, and one extra-layer expert's bytes -- 15360
B -- in the extra layers' pool) the pool could never hold its expert: the first MTP stage's
upload was refused, the run's line read "failed 1 refused by the pool", and T.xmax fell to 0
with the extra layers left on the CPU. The expert needs exactly 15360 B (three tensors of
4096 rows + 1024 scales): the budget was right, the page floor of the block capacity lost
the 3072 B that made it fit.

The streaming-slot reservation commit is not this: with c615cca's six files reverted to
05e2c6d the same run fails identically (same "no room for streaming slots" line, same
counters resident 2 / uploads 6 / evictions 4 / failed 1), and vkt_slot_reserve() is 0 in
it -- the extra-layer take leaves the tier at fit_raw 2, so streaming is off and the chunk
already reserves "0.00 GiB of it the tier's slots and staging". The exposure is older (the
9/10-of-the-heap mapped window, and the extra layers' share of a tiny budget: 9cfa502's
trunk-first placement fixed the window for deepseek_v41 only) and it surfaced when the
deepseek family was run on the discrete ICD, where the MTP stages are the tier's extra
layers by default and not on the CPU as they are on a device that shares the RAM.

1. tier_pool_limit (backend_vulkan.c) rounds its limit up to a page, as its blocks already
   are, so a budget of N expert bytes holds N expert bytes -- at most a page (4095 B) over
   the tier's own budget, another page-granular figure. The tiny-budget case now keeps the
   extra layer on the device: extra layers (2) 4 of 48 routed experts, resident 1 (budget 1),
   uploads 7, evictions 4, failed 0, and the run's output is the CPU's.

2. vkt_slot_reserve (vk_tier.c) caps the reservation at what the budget holds beside the
   residents, and reserves nothing when the budget cannot hold the slots at all: it sizes a
   chunk, it is never a claim on the room an upload or an eviction needs (defensive: it is
   not what refused anything above, where it was 0).

3. cu_tier.c renames vkt_slot_reserve and vkt_vram for the CUDA build of vk_tier.c, which
   'make -C c test' compiles: with the two accessors c615cca added, that build did not
   compile (redefinition against vk_tier.h's stubs).

Measured on the 16 GiB RX 6900 XT: the default-budget serve (ctx 16384, short prompt, one
turn) is unchanged -- 48 slots reserved, tier 1434.4 MiB of 1536.0 in 6 blocks as 68
residents, extra layers 168.8 MiB, process 11.28 GiB of 11.34 GiB in pools, chunk 768 rows
with this desktop's 3.5 GiB already on the card (1024 on an idle one); the same run without
this change gives the same figures.
…e that shares the RAM

deepseek_v41's DSpark loop asserted that the MTP stages never go on the tier by default.
That is true where the device's memory is the host's (Lavapipe): the tier's extra layers
default to the CPU there. On a discrete card COLI_VK_TIER_MTP's default puts them on the
tier, so the assertion fired on every DSpark case of the family when it ran on the radeon
ICD, before any of them could be judged. The check now keys on the engine's own ",
shared RAM," line, which says which of the two it is; the cases that ask for the stages on
the tier (COLI_VK_TIER_MTP=1) are untouched and still require them there.
…tic cap buys twice the rows

The indexer's candidate mask is only ever read as a boolean -- the score kernel tests
it, the host's copy of it feeds a "kept or not" byte, nothing multiplies by it -- and it
was an fp32 word a column: rows x wcap of them beside rows x wcap fp32 scores. It is a
bit a column now (vk_chain.h: vkc_dsv4_cand writes it, vkc_dsv4_score reads it, mask_row
and mask_off in words of 32 columns), so the pair costs 4E + E/8 bytes a row instead of
8E. No row of arithmetic changes: the same blocks are picked and kept whole, and the
selection the top-k runs on is the same set of columns for every row (the indexer case in
tests/test_vk_chain.c compares the mask column for column, through the single-workgroup,
the wide-query and the grid kernels; the families give the CPU's tokens).

1. The chain (c/deepseek_v41_chain.h): isc stays fp32 at rows x wcap; mask and its host
   copy are rows x ceil(wcap/32) words, and the host unpacks the bits where it builds
   m->candidates for the CPU's layers after a partial chain. The chunk's quadratic cap
   (v41c_forward_seg) was rows <= 32M/E, which is the 256 MiB the fp32 pair could take;
   it is now the same 256 MiB over 4E + E/8 a row, i.e. ~62M/E (1.94x the rows), and it
   prints one line when it binds -- nothing printed it before, which is why the audit had
   to derive it.

2. Measured, the tiny V4.1 fixture at a 20000-position prompt: the chunk the tier's
   stream line reports goes 1677 -> 3253 rows, the row cost the sizing prints falls 181
   -> 105 KiB, and the new line reads "a prompt of 20000 positions caps a chunk at 3253
   rows (80.6 KiB a row of scores and packed candidate mask, 256 MiB of 256)". On the
   released checkpoint at E=3107 the row cost goes 1293 -> 1281 KiB a row (the mask was
   12.4 KiB of that, one bit a column is 0.4).

3. Two items of the wave-2b list are not there, measured rather than argued (the
   correction is in docs/deepseek-v41.md so it does not come back):
   - the 1262.5 MiB the turn-end ledger attributes to the "token embedding" is the lm
     head. tie_word_embeddings is false on this checkpoint, both tensors are 129280 x
     5120 bf16, and the embedding has never been on the device: its only reader is
     wb_row_f32, which copies the rows a step needs into the host's rows (one a token
     when decoding, one a position for a chunk). A run with every device upload printed
     shows head.weight as the 1262.5 MiB item and the embedding never uploaded at all.
     The head is a GEMV of 662M multiply-adds a token, so it stays where it is, as do
     the 40 routers' 150 MiB.
   - the per-layer vkc_finish (deepseek_v41_chain.h, after the attention frame) does not
     wait on anything the host does not read. A 1280-row chunk spends 28095 ms waiting
     for the frame it reads and 0.5 ms for the rest: two frames are in flight and a
     single queue runs submissions in order, every frame starting with a barrier on
     everything submitted before it, so the previous layer's shared expert is always
     through. Waiting only the read frame was implemented, measured at 0.5 ms a chunk and
     dropped. The audit's R4 estimate (2-8 ms a chunk) was right; the wait that remains
     is the attention itself, and the device is idle during moe_run_at instead.

Gates on the 16 GiB RX 6900 XT (RADV), against the same three families at 955a0f6:
tests/vulkan_engines.sh deepseek (29 OK), deepseek-chain (35 OK), prefill-deepseek (2
OK), make -C c vk-chain-check VK=1 (PASS), make -C c test (test-c green; test-python
3226 tests). Three failures and two more are identical at both commits and are
environmental: "deepseek_v4 session: no matmul ran on the device" and "chain deepseek_v4
4 experts: the chain never ran" (the deepseek_v4 engine's dense placement declines with
the desktop holding VRAM; its tokens still equal the CPU's), "pf deepseek_v41 staged:
logits" (max |logit diff| 3.22e-02 at both), and the two
test_harness_builds_without_warnings cases (a GCC ABI note in mistral_vision.h).
The commit before this one quoted the row cost the packed candidate mask saves as
"1293 -> 1281 KiB a row" at E=3107. 1293 is the row cost at E=11 (the short-prompt arm),
not at E=3107: the same 8 bytes a column make it 1317 KiB there, which is the number the
audit prints as well, so the pair is 1317 -> 1305. Measured on the released checkpoint at
E=1300 this run: 1303 -> 1298 KiB a row (the mask's 5.1 KiB a row of 41 packed words).
The saving itself was never in doubt -- 8 bytes a column become 4 + 1/8 -- only the base
it was subtracted from.
…hunk, not the context

The planner priced the forward workspace at the whole context, so at --ram 28 a
--ctx 131072 or 262144 serve came out with a zero-byte RAM expert cache and
cap_for_arch() hard-failed with "RAM budget cannot hold one expert slot per
layer". deepseek_v41's prefill reads a prompt in bounded chunks (default 1280
rows; COLI_VK_CHAIN_ROWS / COLI_PROMPT_SEGMENT set it), so the workspace is
priced at one chunk and stays bounded as the context grows. The regression test
(tests/test_resource_plan.py) builds a two-layer 128-expert checkpoint and checks
that both long contexts keep a positive expert cache per tier.

Python-only; no engine source changes.
…orkspace gives

05e7748 pinned the contexts chat_context() recommends for DeepSeek V4.1 when the
prompt's workspace was priced at the whole context: 16384 on 16 GB, 65536 on 64 GB,
131072 only from 512 GB, and the refusal to start at 262144 (53.7 GB of workspace, no
room for one expert slot). 008a01b prices the workspace at one prompt chunk, so the
same table reads 65536 on 8 GB, 131072 on 16 GB and the family's 262144 ceiling from
64 GB. The workspace is still in the sum and still in the reason it prints (0.3 GB at
every step); the comment says which share applies at each step and what the table was
before.

Caught by `make -C c test` on the merge of fix/ram-plan-128k: 7 failures, these five
and the two known environmental test_harness_builds_without_warnings cases. With this
change the target is green (3019 tests, 0 failures, 358 skipped).
…ts it tested

What the wave after Track A and Track B measured, tied to the commits that carry it:
the turn-end ledger (c615cca) closing the audit's 2.14 GiB as the lm head, 40
routers and the DSpark stages rather than the tier; the 48 streaming slots and the
chunk reserve (0.27 GiB off the engine's own hold, 0.001 off the card); the 2-row
stream threshold (chunk 256: the CPU's share of expert rows 34.5% -> 1.5%, the
prefill 226.7 -> 178.0 ms a row, at 1.15 TB of expert reads a turn); the tier pool's
page (90959d6); the packed candidate mask (12b2908: 1.94x the chunk for the same
256 MiB, 1677 -> 3253 rows at a 20000-position prompt) with a6486c7's correction of
the base it was quoted against; and the merge of the RAM-plan fix (9508327) with the
first GPU run it ever had: --ctx 131072 and 262144 plan 21 and 20 slots a layer, load,
and finish a turn in 17.9 and 18.1 s with no OOM.

The corrections are called out rather than folded in: the 1262.5 MiB is the lm head
and the embedding never touches the device; the per-layer drain costs 0.5 ms a chunk
and waits on nothing the host does not read; and the card's sysfs peak does not move
when the engine's hold does, because the desktop takes the difference (1.46 GiB held
less, 0.19 GiB read less, 1.27 GiB to the compositor).

The comparison (Track A) and the audit (Track B) are copied in byte-identical, because
their predictions are what the wave tested; the raw file carries every log line the
record quotes, the planner's before/after table and the two serves above. The 50 Hz
VRAM sample series stay as scratch: the record names them, nothing commits them.
The full gate on the merged tree at the previous commit: deepseek 29 OK, deepseek-chain
36 OK, prefill-deepseek 2 OK (one environmental failure each), vk-chain-check PASS,
make -C c test with test-c ALL PASS and test-python 3227 tests / 294 skipped / the two
environmental test_harness_builds_without_warnings failures. The section also records
that this target's own totals move between runs (3227, 3019, 3227 on the same box), so a
count alone is not a regression signal.
…nd gitignore with it

955a0f6's guard grepped ', shared RAM,' which no code path prints (vk_tier.c
prints 'shared RAM (COLI_VK_TIER_GB)', 'shared RAM: a quarter of what the
expert cache leaves' or '(shared RAM)'), so the DSpark-on-shared-RAM default
assertion was silently skipped on every device. It now matches any 'shared RAM'
wording and never 'device memory'. vulkan.md gains the tier's new defaults
(STREAM_SLOTS 48, STREAM_MIN 2), the record fixes its quoted line, the
pair-cost labels, the 12.1 KiB mask share, and lists the two instrumented runs
its M1/M2 numbers come from; .gitignore covers the audit scratch.
mem_info_gtt_used beside mem_info_vram_used: system RAM behind the
aperture is where the host-visible pools land when device-local is
exhausted, and that spill is host-RAM pressure the process RSS does not
show (a 256K-token prefill on a card that had lost 3.3 GiB of its
device-local to a leak spilled pull and the mirrors there while RSS
looked steady). One more field on the turn-end line, 0.00 when the
driver does not export it.
… formats

The chain's window rings, compressed rows and index keys mirrored as
DeepInfer's paged cache stores them (c/dsv41_kvq.h, the host reference):
an MXFP8 window row (e4m3, a ue8m0 scale a group of 32, 528 B), an FP4
compressed row (e2m1, an e4m3 scale a group of 16 as stored, 288 B) and
FP4 index keys in block-wide sections (a ue8m0 scale a group of 32, a
block of 32 tokens, 68 B a row on average). chain_dsv4.comp packs a row
after its last fp32 reader (mode 13) and dequantizes it where read (the
attention's flags 2, the index scores' quant field) and on the pull's
write-back (mode 14), the scales' reciprocals as DI's bit-exact LUT
math (a GPU's 1/x is a couple of ulps low, which the codecs' bit
agreement cannot take). Off by default; the flag branches only at the
mirrors' widths and stores, a forward's first check sums the mirrors'
bytes against the flag's widths, and the KV split still decides as
before: when it arms the mirrors are fp32 whole again, re-mirrored from
the host.
…ode gated

tests/test_vk_kv_quant holds the device's pack to dsv41_kvq.h's reference
bit for bit (the edge rows of the scale math included: a zero amax,
denormals, -0, an amax a power of two over the format's max, the clamp
edges and the RNE ties), its unpack, a re-upload's little-endian lanes
and the pack's idempotence (a fixpoint in values: a group at half its
scale's span re-packs a step down to the same decoded row), and the
index scores over quantized keys to the decoded rows' bits with the
original rows' drift and top-k overlap reported. The kv-quant-deepseek
family runs the tiny fixture at the released KV geometry
(make_dsv41_tiny.py --kv-quant-dims: head_dim 512, index_head_dim 128,
the formats' own widths) under the flag against a documented logits
tolerance -- the flag-off run is the exactness proof, as
V41_PREFILL_TAIL's tests put it -- with the token drift printed and the
ledger's kv-mirror line pinned to the 528/288/68 B math.
…d save

ENVIRONMENT.md's flag row; deepseek-v41.md a subsection by the
trunk-first placement: the three formats, the ~6,400 -> ~890 B a
position of mirror, the ~690 MiB at 128K tokens and 5.38 GiB of the
peak at the 1M ceiling (the prefill-VRAM analysis's §10 figures), the
approximate-mode caveat (a pull returns dequantized rows, the pack's
idempotence bounds the hop) and the KV split's precedence.
…he index staging an index row wide

Two bugs the tiny fixture's first quantized run found, both in the
chain's integration rather than the codec (which the unit test had
already pinned bit for bit):

- vkc_write's offset is in FLOATS, like every chain offset -- the pack
  re-uploads passed bytes, so the index keys' scale section landed four
  times past its block and the frame died as a copy out of range. The
  three upload helpers now divide by 4.
- the fp32 staging for a chunk's new index rows was sized with
  index_n_heads (4) where a row is index_head_dim floats (128): the
  idx_wk matmul wrote past 224 bytes of scratch. qik is an index row
  wide, and the chunk-row accounting with it.

The kv-quant-deepseek gate with them: the first decode row is the one
input the two runs share, so it is the row the tolerance gates (the
argmax kept, the drift within the row's own largest |logit| -- the tiny
fixture's random weights leave gaps narrower than the format's noise;
past that row a flipped pick feeds the model its own choice), and the
ledger's kv-mirror line comes from a one-turn serve run of each flag
setting (the ledger ends a turn; the CLI never has one): 6.4 -> 1.6 MiB
on the fixture, the 528/288/68 B math rounded to whole index blocks.
The test's last line is the plain PASS the gates grep for.
…s, the gate as it gates

The docs' 128K/1M figures now sit on the corrected 6,400/890 B a position
basis (0.78 -> 0.11 GiB at 128K, 6.25 -> 0.87 GiB at 1M), with the audit's
older savings-as-totals numbers named as such; the family's comment states
the gate it runs (the input-shared first row keeps its argmax, drift within
that row's own largest |logit|) instead of a tolerance it does not use; the
idempotence claim says values, not bytes; the kvq fixtures and run CSVs are
ignored; the wide score case's comment names mode 10's grid arm and that
mode 9's quantized reader is the same expression.
@jameswitney

Copy link
Copy Markdown
Contributor Author

Superseded by #2712, which is this branch in full plus the 9 commits that put the host's
own KV arrays in the same formats (COLI_VK_KV_QUANT=2).

Nothing is lost: git merge-base --is-ancestor f572efda 8a8fb6cb holds, so #2712 carries
every commit and every file here. The whole design — the formats, the codec, the family,
the docs — is reviewed there in one piece, with =1 and =2 described as the two levels
of one flag. One correction folded into that body: #2637's trunk-first work is already in
dev (it landed as 86eae5aa), so this no longer "stacks on" anything unmerged.

Closing in favour of #2712.

@jameswitney jameswitney closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant