Repository navigation
Batch 2026-11-15-c: DeepSeek V4.1's candidate ids and engram staging, Qwen3.8's MXFP4 experts, Inkling-2's O_DIRECT reads (6 changes) - #2853
Merged
Conversation
WHAT. The candidate source layer published what it keeps as a mask, `rows x reach` bytes, and every layer after it read one bit of that mask. At compress_ratio 1 the reach is the position, so the allocation was n^2: 64 GiB at 262144 tokens, 38 % of the engine's 166.6 GiB peak. It now publishes the blocks themselves -- rows x candidate_top_k_blocks int32 ids in the source's own coordinates (DI's block_candidate_idxs, cross_layer.py:296), -1 in every slot past the blocks a row keeps. `candidate_width` stays as the source's reach: the row is painted into a compress_len scratch at `w = min(compress_len, candidate_width)` -- exactly the guard the reader's score loop already applied -- and the readers paint the ids back into that scratch, so no reader sees a buffer it did not see before. WHY bit-identical. The mask was the union of the chosen blocks' ranges clipped at the reach, so "block b kept iff any of its positions is set" followed by repainting is the identity, independent of how the blocks were chosen; candidate_blocks() is unchanged except a comment, so the selection stays in one place. Only positions past `w` may differ (a block straddling a chain chunk's reach), where indexer_run has already set score[j] = -INFINITY, so keep[] is the same bytes everywhere a reader looks. MEMORY. rows x topk x 4 instead of rows x reach bytes, independent of the prompt: n = 262144, topk = 2048: 2.00 GiB instead of 64.00 GiB (1.2 % of the peak, from 38.4) n = 65536: 0.50 GiB instead of 4.00 GiB n = 16384: 0.125 GiB instead of 0.25 GiB (the mask does not bind below 16K) Measured on the 40-token fixture (an instrumented build, not committed): indexer_run [CAND] alloc rows=40 topk=2 bytes=320 where 15f810f prints rows=40 width=40 bytes=1600, and a decode step rows=1 topk=2 bytes=8 where 15f810f reallocs rows=1 width=41, 42, 43, 44, 45; the chain's candidate_output allocator prints the same pair (320 then 8) where 15f810f prints 1600 then 41..45, and the CPU consumer logged the ids it read back (2 and 4 positions set on w=40). v41_candidate_bytes is one sizing function, used by both allocators and by the park. EVIDENCE. `git archive 15f810f` in a separate tree, built with the same flags; tests/test_dsv41_prefill_tail's DUMP recipe (pretail 200 rows, tail 15 rows) and the 40-token CLI fixture, `cmp` on all five artifacts: identical (dump sha 63ee58b1d4e6f79b both paths, "Matching tokens: 6/6"). Vulkan/lavapipe partial chain (COLI_VK_CHAIN_LAYERS=4, the source on the device and layer 4 on the CPU, which is the only path through chain.h's candidate_output): tokens identical to the frozen tree and to the CPU run, and the 8 chain tests of test_dsv41_prefill_tail compare their 24 KiB logit dumps byte for byte. The gate is the next commit: 11760 shapes against a frozen copy of the pre-change writer, ASan+UBSan clean. NOT HERE. The device masks the chain paints (ch->cmd/mask/isc/lw_mask) are still dense, and the allocation is still the prompt's rows rather than the tail's: allocating the tail's 2414 rows takes the 2 GiB to 19.8 MB and is bit-neutral, and uint16 ids would halve it. Both are separate changes, named in CHANGELOG.md.
…writer The candidate source layer now publishes block ids instead of a mask, and the mask was the union of the chosen blocks' ranges, so "read the blocks out of the mask, paint them back" must be the identity. This gate holds that identity on 11760 shapes (14 widths x 5 block sizes x 7 topk x 6 reach x 4 score rows), and it holds it against a frozen copy of the pre-change writer: ref_candidate_blocks() is 15f810f's candidate_blocks(), verbatim, signature and body, with only the name changed. That copy is the point. The selection has to stay in one place, so the live writer is still candidate_blocks(); a gate that compared the live writer with the live reconstruction would compare the engine with itself and would not see a selection change at all -- flipping the tie-break > to >= passes such a harness on all 8820 cases and differs from the frozen text on 2649 of 11760 here. Asserted per case: 1. candidate_blocks() == ref_candidate_blocks(), byte for byte; 2. candidate_row_ids() on the frozen row leaves every one of the `topk` slots either -1 or a block in [0, blocks) -- the array is poisoned with 0x7f first, so an unwritten slot cannot pass, and 2464 of the cases have padding whose value would mask the row differently if it were stale; 3. candidate_keep_row() repaints the frozen row on [0, w) for every reach compress_len in 0..width, w = min(compress_len, candidate_width) -- the bound the reader's own score loop applies (an id past it writes nothing: the drafted shape, paint to the width from a compress_len scratch, is a heap overflow); 4. the repo's pinned tie vector, tests/test_dsv41_ref_topk.py:53: width 8, block 2, topk 2, all scores 0 -> 1,1,0,0,0,0,1,1; 5. v41_candidate_bytes: 262144 rows x 2048 slots = 2147483648 B, linear in rows, 4 B a row at topk 0, and no width parameter to get wrong. The Makefile rule is what puts it in `make test-c` / `check` / CI (TEST_RULES is derived from the rules). It is green on 15f810f itself plus only the three helpers: the identity holds with the pre-change writer, before any caller depends on it. ASan+UBSan clean (make ... EXTRA_CFLAGS/EXTRA_LDFLAGS with the -O1 -fsanitize=... pair test-asan uses); 0.2 s.
…fig's v41_candidate_bytes() took the Cfg and re-derived the slot count from candidate_topk_blocks. Both allocators do store that same value in candidate_topk, so it was equal at every site today -- but v41_park then sized the copy of a live array as v41_candidate_bytes(&m->c, candidate_rows), i.e. from the config of the moment rather than from the topk the array was allocated with. The two are the same value only for the life of a Model, which is exactly the kind of equality that stops holding when the next site stores a different topk, and its failure mode is a copy of the wrong size. So the helper takes the count: v41_candidate_bytes(int topk, int rows). Every caller hands it the topk it means -- the writer the one it is about to store, the park path m->candidate_topk, chain.h its own `topk` -- and no call site can size a copy by a different rule than the allocation it copies. One spelling of the size, and v41_dup()'s byte count is now the live array's. The gate's candidate_bytes() drops the Cfg too and asserts it directly (262144 x 2048 = 2 GiB, linear in rows, 4 B a row at topk 0) plus the property the change is for: the byte count follows the caller's topk. No behaviour change: the four sites pass the value the config produced.
…ssuming it candidate_width is not the ids' size any more, it is the reach the source layer painted its row to, and it is the reader's paint bound: w = min(compress_len, candidate_width). The source refreshes it on every row set, before any layer after it reads, which is why v41_unpark restores the ids (and their topk and rows) but not the width. That ordering is what makes the restored array safe, and it was stated in a comment at the two allocators and at the unpark, not checked anywhere: a reader that ran without a reach would take w = 0, candidate_keep_row() would clear the row, and every position would go to -INFINITY -- a whole forward masked away, silently, well past the point where the first token could show it. The reader now checks the one thing its arithmetic needs: ids to read and a row to mask, and no reach at all is that order broken, not a narrow row. It is one comparison per layer, cannot fire on compress_len 0 (nothing to mask), and its exit is the same shape as the rest of the engine's fatal paths. The unpark's comment now points at the check. Reachable, and proved so on a scratch copy: with the source's m->candidate_width = compress_len replaced by = 0, the mutated engine exits at this check on the first forward that has a reader, where the pre-change code ran on and answered; the unmutated tree is byte-identical to 15f810f on the logit dump (63ee58b1d4e6f79b, tail and no-tail) and 58/58 of the dsv41 python suite, chain runs included, never reaches it.
Three comments and the CHANGELOG describe m->candidates, which is a row of block ids now, and went on calling it a mask after the switch: the chain's statement of what the CPU's layers read from a chain layer (deepseek_v41_chain.h, "the candidate mask (m->candidates)"), and, in deepseek_v41.c, the list of what v41_take_tail carries, what v41_park keeps across requests, and what a prompt's rows past a prefix checkpoint republish before the layers after them read. The device's own handoff buffers (ch->cmd, ch->lw_mask) are still masks and keep the word; the host's array is not one. CHANGELOG: the 2 GiB is the prompt's rows, exactly as the mask was, and v41_take_tail leaves 19.8 MB of them -- the entry says so now instead of leaving "2 GiB at 262144" and "2 GiB to 19.8 MB" a paragraph apart, which read as if the change had already stopped allocating the prompt's rows. It has not; that is the separate change the entry names. Comments and prose only: no code, no behaviour, the gate's and the logit dump's numbers below are the post-change rebuild's.
…rose the premise Review fixes for the candidate-block-ids change (PR #2797), behaviour-neutral for every config the engine accepts: - gate: extract the reader's paint bound as candidate_paint_width() and assert its table; add the clipped-row case the chain's conversion produces (a block set on [8,12) only, so `hit =` for `hit |=` drops it); assert [w, width) keeps its poison, so an over-paint is a CHECK rather than a heap abort; CHECK(cases == 11760) and CHECK(padding > 0) so the printed figures are pinned. - comments: say what makes the handoff exact (the ids' reach wc, the reader's bound, the equal-ratio refusal in v41c_unsupported) instead of "block-aligned" and "the identity". - CHANGELOG: the rebuild-went-with-the-reach sentence, the 40-row fixture, the live-versus-resident 19.8 MB, candidate_topk_blocks spelled as the key is, and the unverifiable 38 % dropped; the speculative step's resize back to one row named beside its growth. - candidate_keep_row: bound the id by (w - 1) / block, not by its product, which wraps on a 32-bit long; and refuse an id at a block < 1, so the new division is defined on its own line. - CHANGELOG, second round: the ids "do not follow the reach" -- they resize when the row count changes, which with DSpark is twice a round, so the old clause read as "never reallocates". docs/ENVIRONMENT.md's LAYERWISE row names the device's candidate mask again (lw_mask, one byte a position): it is not host prose, so the file leaves the diff.
… as an experts-mxfp4/ sidecar AMD's Quark MXFP4 checkpoint's routed experts are copied byte for byte into <model>/experts-mxfp4/, the int4 sidecar's layout (layer-NNN.safetensors plus index.json, resumable through .part files): one record per expert, the E8M0 scales of gate, down and up, then their e2m1 nibbles, padded to the next 4096 bytes by a U8 record_pad tensor so the file has no hole. 2,611,200 bytes on the release, 2,613,248 with the pad. Dtypes, shapes and the E8M0 NaN are checked, and four experts a layer are compared with the FP8 release's (cosine, --min-cos) so a nibble order, scale axis or name mix-up is refused. make_qwen38_tiny.py --mxfp4-experts rounds the fixture's experts to MXFP4 (OCP rounding), writes them under quark/ with Quark's names, writes the FP8 shards with power-of-two block scales so they hold the same values exactly, converts quark/, and emits a float32 ref.json (a BF16 one is 0.9965 cosine from the engine on this fixture). make qwen38-tiny-mxfp4-generate builds it and checks every record against the Quark bytes and every value against the FP8 shards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y COLI_DISK_WEIGHTS qwen38 read its expert sidecar (experts-int4g64/ or the low-bit one in use) from one drive only: COLI_MODEL_MIRROR and COLI_DISK_WEIGHTS were registered for colibri, the V4 engines and glm53. Each mirror directory's copy of the sidecar is now added to the sidecar's shards with st_mirror_add (a file is taken only when its size and safetensors header equal the primary's, so a partial or stale copy leaves those files on the primary), and every record is read with st_read_range_rep from the replica a hash of its (layer, expert) picks, as V4.1 hashes its experts, in the shares COLI_DISK_WEIGHTS sets (equal when unset; no probe; a list that is not one positive weight per drive is refused). The same expert always comes from the same drive and every copy holds the same bytes, so tokens and logits do not move. The Vulkan tier's direct read and the prefetch advice go to the same replica; mapped experts (COLI_MAP_EXPERTS=1) and the snapshot's FP8 experts stay on the primary. The startup prints the split, the run's end the reads each drive served. make qwen38-tiny-mirror-check (tests/qwen38_mirror_check.py, in the qwen38 CI job): the int4 fixture with one and two copies in temporary directories, at caps 1 and 2, parallel and serial reads: tokens and last logits the bits of the run without a mirror, every replica serving reads; a copy with one differing header byte and an empty directory skipped; five bad COLI_DISK_WEIGHTS refused; no mirror opened without the sidecar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…), computed by matmul_mxfp4 Q38_EXPERT_LOWBIT=mxfp4 attaches <snap>/experts-mxfp4/, the sidecar tools/convert_qwen38_experts_mxfp4.py copies AMD Quark's MXFP4 experts into, through the low-bit sidecars' attach: its index.json is checked for the format, codes, scales, group size and the model's geometry, every record for its six U8 tensors back to back (gate, down and up scales, then their e2m1 codes), and a missing or incomplete sidecar is refused. A record is one read of 2,611,200 bytes on the release (4,915,200 for the FP8 expert); its matrices are a new weight kind, Q38_WEIGHT_MXFP4 (the nibbles in data, the ue8m0 bytes in x2q), multiplied by quant.h's matmul_mxfp4, whose row result does not depend on the batch, so decode and the prompt batch give the same bits. The grouped decode has no row kernel for it and takes the per-expert matmuls. The CUDA tier and the Vulkan tier say they do not take MXFP4 yet and keep the experts on the CPU; the trunk, shared expert and MTP head stay FP8. COLI_MODEL_MIRROR copies of experts-mxfp4/ serve it as they serve the int4 sidecar. tests/test_qwen38_mxfp4_experts.c (make test-c): the matmul against a dequantized double reference on the release's shapes, the tiny fixture's and short last groups, 1 to 9 rows, each batch row the bits of that row alone; the release record 2,611,200 bytes (2,613,248 on its stride) in the documented order. make qwen38-tiny-mxfp4-check (qwen38 CI job): every expert the single and batch paths load holds the sidecar's bytes and every value is the fixture's FP8 value to the bit; ref.json reproduced in 30 configurations (single, batched, prompt batch, serial prompt, mapped; caps 1, 2, 4; trunk on the CPU or not) with identical last logits; the FP8 run's last logits within cosine 0.99999 (1.00000000 on the fixture); refusals each in its own process; the mirror test's MXFP4 leg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…threads engram_row reads a row the cache does not hold with one pread and waits for it. V41_ENGRAM_ADVISE asks the OS for a block's rows first, but only where POSIX_FADV_WILLNEED exists (not on Windows) and not for a decode step, so there a prompt's rows still came from the disk one at a time. With V41_ENGRAM_STAGE=1 (default off), engram_advise_rows hashes the block's positions, takes the distinct rows the cache does not hold, reads them on V41_ENGRAM_STAGE_THREADS readers (default 8) and puts them in the cache; the gather that follows finds them there. It runs for a decode step too, and at the three places that already advise (engram_run, the Vulkan and the CUDA chains). A slot holds the same dequantized row whoever read it, so the gather's bits do not change; only which rows the LRU keeps can. At most half the cache is read at once; the gather reads any row it does not find itself, as before. The readers go through st_pread_full, so on Windows they take the shard's overlapped descriptor. engram_row is split into engram_find (the probe) and engram_fetch (the read and dequantization), the same behaviour. The engram stats lines count the staged reads as rows read. tests/test_dsv41_engram_stage: one 420-row table and its sidecar, no model. Staged in whole, in blocks of 32, one position at a time and on one reader, the gather gives the unstaged bits, finds every row, and each missing row is read once; a 16-row cache stages at most 8 rows a block and still gives the same bits; V41_ENGRAM_STAGE=1 makes engram_advise_rows stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he extraction's truncation fallback model_reset_rows and v41_ps_take kept the candidate tuple of the conversation they were dropping: its readers key on the pointer, not a row count, so the stale rows could not be consulted (the source layer rewrites every row before any reader, in every forward), but the last conversation's allocation -- 2 GiB at 262144 rows -- stayed live, unlike the shared top-k list they already dropped. Both now free it, which is also the honest accounting for the rows a reset keeps apart: park saved them before the reset runs. candidate_row_ids' truncation branch (a row with more set blocks than slots) is unreachable in the engine -- both writers paint at most topk blocks, and test_vk_chain holds the device's mask to the CPU's slot-for-slot -- so the gate now pins the fallback as a chosen answer rather than an accident: the first topk blocks in block order, every slot still written.
…e other prompt-scaled half a reset keeps
ink_fill_slots read each piece through the page cache and dropped it after (POSIX_FADV_DONTNEED). Now each piece's 4 KiB-aligned span goes with O_DIRECT into a buffer of the reader's and the piece is copied out (a container's slices start wherever its header ends). The buffers are a pool kept across rounds: a round's readers are a new team of the fill thread's each time. Measured on two NVMe in RAID0 with the engine's pieces of 2 MiB: 2.3 ms an expert against 3.0 in rounds of three, 8.2 GB/s against 6.0 deeper. Inkling-2-Small (int4 g64, the CUDA build, 48 tokens on two prompts, alternated): the decode 1.96-2.01 -> 2.30-2.33 tok/s, the prompt 12.8-13.1 -> 11.0-11.2 s, and the experts' compute 18.4 -> 16.6 s of a run (the page cache's copies took CPU and memory bandwidth). The tiny oracles match 24/24 either way. INK_FILL_DIRECT=0 reads as before; a file system that refuses O_DIRECT falls back to it.
deepseek_v41: the candidate blocks as ids, not a dense n² mask (64 GiB -> 2 GiB at 262144)
deepseek_v41: a block's engram rows read on reader threads (V41_ENGRAM_STAGE, off by default) c/Makefile: #2797's test_dsv41_candidate_ids rule and this test_dsv41_engram_stage rule, both kept.
tools: convert_qwen38_experts_mxfp4.py writes AMD Quark MXFP4 experts as an experts-mxfp4/ sidecar
qwen38: an expert sidecar read from COLI_MODEL_MIRROR copies, split by COLI_DISK_WEIGHTS
qwen38: routed experts from an MXFP4 sidecar (Q38_EXPERT_LOWBIT=mxfp4), computed by matmul_mxfp4
inkling: read the missed experts with O_DIRECT (INK_FILL_DIRECT)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six changes on dev after #2848, each merged with its own commits. One conflict, in
c/Makefile: #2797's and #2818's new test rules at the same place, both kept.What is in it
DeepSeek V4.1
V41_ENGRAM_STAGE=1(off by default) reads a block's missing engram rows on reader threads before the gather, on the CPU and both chains. Unset, the code path is dev's.Qwen3.8 with AMD Quark's MXFP4 experts
tools/convert_qwen38_experts_mxfp4.pycopies a Quark checkpoint's routed experts, byte for byte, into anexperts-mxfp4/sidecar next to the FP8 release, four experts a layer checked against the FP8 snapshot.COLI_MODEL_MIRRORcopies, in the sharesCOLI_DISK_WEIGHTSsets, as colibri and the DeepSeek engines read theirs.Q38_EXPERT_LOWBIT=mxfp4runs the routed experts from that sidecar on the CPU withmatmul_mxfp4. Unset, nothing changes.Inkling-2
perf/ink2-disk(ours)INK_FILL_DIRECT, on;=0reads as before, and a file system that refuses O_DIRECT falls back to it).By @jameswitney (#2797) and @KairaCode (#2818, #2822, #2824, #2825).
Tests
On ds (AMD Ryzen 7 PRO 8700GE, AVX-512) and on an RTX PRO 4000 Blackwell (sm_120, 24 GB), dev 1a06018 against this branch:
make check: Ran 3589 tests, OK (skipped=467).make test-asan: OK. The tree with this batch, batches D and E and windows: qwen38.exe in the CUDA package; setup says why an NVIDIA card stays unused (#2846) #2847 merged together: Ran 3629 tests, OK (skipped=502).dsv41-tiny-check,dsv41-cluster-parity,qwen38-tiny-checkandinkling2-oraclejobs: 24 of 28 pass, among them the MXFP4 sidecar step ofqwen38-tiny-check. The four Inkling-2 steps that build their reference with transformers' Inkling-2 (the tiny oracle, the converter, the vision tower, the drafter) did not run: neither machine has a transformers with it. CI runs them.COLI_CUDA=1), a 3036-token prompt then 128 greedy tokens, dev, this branch and batch D alternated twice: prompt 1196 to 1208 tok/s on dev and 1198 to 1203 here, decode 65.50 to 65.53 and 65.07 to 65.56, the same text, peak RSS 39.2 GB on both. On the CPU the same runs give 29.5 to 29.7 tok/s of prompt on all three, decode 10.53 and 10.74 on dev, 10.75 and 10.77 here, the same text.Q38_EXPERT_LOWBITwas unset: these check that nothing else moved.