Skip to content

Batch 2026-11-15-c: DeepSeek V4.1's candidate ids and engram staging, Qwen3.8's MXFP4 experts, Inkling-2's O_DIRECT reads (6 changes) - #2853

Merged
JustVugg merged 20 commits into
devfrom
batch/2026-11-15-c
Nov 15, 2026
Merged

JustVugg merged 20 commits into
devfrom
batch/2026-11-15-c

Conversation

@JustVugg

Copy link
Copy Markdown
Owner

Six changes on dev after #2848, each merged with its own commits. One conflict, in c/Makefile: #2797's and #2818's new test rules at the same place, both kept.

What is in it

DeepSeek V4.1

PR What it does
#2797 The indexer's candidate blocks are kept as block ids per row instead of an n x n byte mask: 64 GiB become 2 GiB at 262144 positions. A reader rebuilds its row's mask from the ids and masks its scores as before.
#2818 V41_ENGRAM_STAGE=1 (off by default) reads a block's missing engram rows on reader threads before the gather, on the CPU and both chains. Unset, the code path is dev's.

Qwen3.8 with AMD Quark's MXFP4 experts

PR What it does
#2822 tools/convert_qwen38_experts_mxfp4.py copies a Quark checkpoint's routed experts, byte for byte, into an experts-mxfp4/ sidecar next to the FP8 release, four experts a layer checked against the FP8 snapshot.
#2824 The expert sidecar is read from COLI_MODEL_MIRROR copies, in the shares COLI_DISK_WEIGHTS sets, as colibri and the DeepSeek engines read theirs.
#2825 Q38_EXPERT_LOWBIT=mxfp4 runs the routed experts from that sidecar on the CPU with matmul_mxfp4. Unset, nothing changes.

Inkling-2

Branch What it does
perf/ink2-disk (ours) The missed experts are read with O_DIRECT into a pool of aligned buffers instead of through the page cache (INK_FILL_DIRECT, on; =0 reads as before, and a file system that refuses O_DIRECT falls back to it).

By @jameswitney (#2797) and @KairaCode (#2818, #2822, #2824, #2825).

Tests

On ds (AMD Ryzen 7 PRO 8700GE, AVX-512) and on an RTX PRO 4000 Blackwell (sm_120, 24 GB), dev 1a06018 against this branch:

jameswitney and others added 20 commits November 13, 2026 10:32
WHAT. The candidate source layer published what it keeps as a mask, `rows x reach`
bytes, and every layer after it read one bit of that mask. At compress_ratio 1 the
reach is the position, so the allocation was n^2: 64 GiB at 262144 tokens, 38 % of the
engine's 166.6 GiB peak. It now publishes the blocks themselves -- rows x
candidate_top_k_blocks int32 ids in the source's own coordinates (DI's
block_candidate_idxs, cross_layer.py:296), -1 in every slot past the blocks a row keeps.
`candidate_width` stays as the source's reach: the row is painted into a compress_len
scratch at `w = min(compress_len, candidate_width)` -- exactly the guard the reader's
score loop already applied -- and the readers paint the ids back into that scratch, so
no reader sees a buffer it did not see before.

WHY bit-identical. The mask was the union of the chosen blocks' ranges clipped at the
reach, so "block b kept iff any of its positions is set" followed by repainting is the
identity, independent of how the blocks were chosen; candidate_blocks() is unchanged
except a comment, so the selection stays in one place. Only positions past `w` may
differ (a block straddling a chain chunk's reach), where indexer_run has already set
score[j] = -INFINITY, so keep[] is the same bytes everywhere a reader looks.

MEMORY. rows x topk x 4 instead of rows x reach bytes, independent of the prompt:
  n = 262144, topk = 2048: 2.00 GiB instead of 64.00 GiB (1.2 % of the peak, from 38.4)
  n =  65536:               0.50 GiB instead of  4.00 GiB
  n =  16384:               0.125 GiB instead of 0.25 GiB (the mask does not bind below 16K)
Measured on the 40-token fixture (an instrumented build, not committed):
indexer_run [CAND] alloc rows=40 topk=2 bytes=320 where 15f810f prints rows=40
width=40 bytes=1600, and a decode step rows=1 topk=2 bytes=8 where 15f810f reallocs
rows=1 width=41, 42, 43, 44, 45; the chain's candidate_output allocator prints the same
pair (320 then 8) where 15f810f prints 1600 then 41..45, and the CPU consumer logged
the ids it read back (2 and 4 positions set on w=40). v41_candidate_bytes is one sizing
function, used by both allocators and by the park.

EVIDENCE. `git archive 15f810f` in a separate tree, built with the same flags;
tests/test_dsv41_prefill_tail's DUMP recipe (pretail 200 rows, tail 15 rows) and the
40-token CLI fixture, `cmp` on all five artifacts: identical (dump sha
63ee58b1d4e6f79b both paths, "Matching tokens: 6/6"). Vulkan/lavapipe partial chain
(COLI_VK_CHAIN_LAYERS=4, the source on the device and layer 4 on the CPU, which is the
only path through chain.h's candidate_output): tokens identical to the frozen tree and
to the CPU run, and the 8 chain tests of test_dsv41_prefill_tail compare their 24 KiB
logit dumps byte for byte. The gate is the next commit: 11760 shapes against a frozen
copy of the pre-change writer, ASan+UBSan clean.

NOT HERE. The device masks the chain paints (ch->cmd/mask/isc/lw_mask) are still dense,
and the allocation is still the prompt's rows rather than the tail's: allocating the
tail's 2414 rows takes the 2 GiB to 19.8 MB and is bit-neutral, and uint16 ids would
halve it. Both are separate changes, named in CHANGELOG.md.
…writer

The candidate source layer now publishes block ids instead of a mask, and the mask was
the union of the chosen blocks' ranges, so "read the blocks out of the mask, paint them
back" must be the identity. This gate holds that identity on 11760 shapes (14 widths x
5 block sizes x 7 topk x 6 reach x 4 score rows), and it holds it against a frozen copy
of the pre-change writer: ref_candidate_blocks() is 15f810f's candidate_blocks(),
verbatim, signature and body, with only the name changed. That copy is the point. The
selection has to stay in one place, so the live writer is still candidate_blocks(); a
gate that compared the live writer with the live reconstruction would compare the engine
with itself and would not see a selection change at all -- flipping the tie-break
> to >= passes such a harness on all 8820 cases and differs from the frozen text on 2649
of 11760 here.

Asserted per case:
  1. candidate_blocks() == ref_candidate_blocks(), byte for byte;
  2. candidate_row_ids() on the frozen row leaves every one of the `topk` slots either
     -1 or a block in [0, blocks) -- the array is poisoned with 0x7f first, so an
     unwritten slot cannot pass, and 2464 of the cases have padding whose value would
     mask the row differently if it were stale;
  3. candidate_keep_row() repaints the frozen row on [0, w) for every reach
     compress_len in 0..width, w = min(compress_len, candidate_width) -- the bound the
     reader's own score loop applies (an id past it writes nothing: the drafted shape,
     paint to the width from a compress_len scratch, is a heap overflow);
  4. the repo's pinned tie vector, tests/test_dsv41_ref_topk.py:53: width 8, block 2,
     topk 2, all scores 0 -> 1,1,0,0,0,0,1,1;
  5. v41_candidate_bytes: 262144 rows x 2048 slots = 2147483648 B, linear in rows, 4 B a
     row at topk 0, and no width parameter to get wrong.

The Makefile rule is what puts it in `make test-c` / `check` / CI (TEST_RULES is derived
from the rules). It is green on 15f810f itself plus only the three helpers: the
identity holds with the pre-change writer, before any caller depends on it. ASan+UBSan
clean (make ... EXTRA_CFLAGS/EXTRA_LDFLAGS with the -O1 -fsanitize=... pair test-asan
uses); 0.2 s.
…fig's

v41_candidate_bytes() took the Cfg and re-derived the slot count from
candidate_topk_blocks. Both allocators do store that same value in
candidate_topk, so it was equal at every site today -- but v41_park then
sized the copy of a live array as v41_candidate_bytes(&m->c, candidate_rows),
i.e. from the config of the moment rather than from the topk the array was
allocated with. The two are the same value only for the life of a Model, which
is exactly the kind of equality that stops holding when the next site stores a
different topk, and its failure mode is a copy of the wrong size.

So the helper takes the count: v41_candidate_bytes(int topk, int rows). Every
caller hands it the topk it means -- the writer the one it is about to store,
the park path m->candidate_topk, chain.h its own `topk` -- and no call site
can size a copy by a different rule than the allocation it copies. One
spelling of the size, and v41_dup()'s byte count is now the live array's.

The gate's candidate_bytes() drops the Cfg too and asserts it directly
(262144 x 2048 = 2 GiB, linear in rows, 4 B a row at topk 0) plus the property
the change is for: the byte count follows the caller's topk.

No behaviour change: the four sites pass the value the config produced.
…ssuming it

candidate_width is not the ids' size any more, it is the reach the source layer
painted its row to, and it is the reader's paint bound: w = min(compress_len,
candidate_width). The source refreshes it on every row set, before any layer
after it reads, which is why v41_unpark restores the ids (and their topk and
rows) but not the width. That ordering is what makes the restored array safe,
and it was stated in a comment at the two allocators and at the unpark, not
checked anywhere: a reader that ran without a reach would take w = 0,
candidate_keep_row() would clear the row, and every position would go to
-INFINITY -- a whole forward masked away, silently, well past the point where
the first token could show it.

The reader now checks the one thing its arithmetic needs: ids to read and a row
to mask, and no reach at all is that order broken, not a narrow row. It is one
comparison per layer, cannot fire on compress_len 0 (nothing to mask), and its
exit is the same shape as the rest of the engine's fatal paths. The unpark's
comment now points at the check.

Reachable, and proved so on a scratch copy: with the source's
m->candidate_width = compress_len replaced by = 0, the mutated engine exits at
this check on the first forward that has a reader, where the pre-change code
ran on and answered; the unmutated tree is byte-identical to 15f810f on the
logit dump (63ee58b1d4e6f79b, tail and no-tail) and 58/58 of the dsv41 python
suite, chain runs included, never reaches it.
Three comments and the CHANGELOG describe m->candidates, which is a row of block
ids now, and went on calling it a mask after the switch: the chain's statement of
what the CPU's layers read from a chain layer (deepseek_v41_chain.h, "the
candidate mask (m->candidates)"), and, in deepseek_v41.c, the list of what
v41_take_tail carries, what v41_park keeps across requests, and what a prompt's
rows past a prefix checkpoint republish before the layers after them read. The
device's own handoff buffers (ch->cmd, ch->lw_mask) are still masks and keep the
word; the host's array is not one.

CHANGELOG: the 2 GiB is the prompt's rows, exactly as the mask was, and
v41_take_tail leaves 19.8 MB of them -- the entry says so now instead of leaving
"2 GiB at 262144" and "2 GiB to 19.8 MB" a paragraph apart, which read as if the
change had already stopped allocating the prompt's rows. It has not; that is the
separate change the entry names.

Comments and prose only: no code, no behaviour, the gate's and the logit dump's
numbers below are the post-change rebuild's.
…rose the premise

Review fixes for the candidate-block-ids change (PR #2797), behaviour-neutral for
every config the engine accepts:

- gate: extract the reader's paint bound as candidate_paint_width() and assert its
  table; add the clipped-row case the chain's conversion produces (a block set on
  [8,12) only, so `hit =` for `hit |=` drops it); assert [w, width) keeps its
  poison, so an over-paint is a CHECK rather than a heap abort; CHECK(cases ==
  11760) and CHECK(padding > 0) so the printed figures are pinned.
- comments: say what makes the handoff exact (the ids' reach wc, the reader's
  bound, the equal-ratio refusal in v41c_unsupported) instead of "block-aligned"
  and "the identity".
- CHANGELOG: the rebuild-went-with-the-reach sentence, the 40-row fixture, the
  live-versus-resident 19.8 MB, candidate_topk_blocks spelled as the key is, and
  the unverifiable 38 % dropped; the speculative step's resize back to one row
  named beside its growth.
- candidate_keep_row: bound the id by (w - 1) / block, not by its product, which
  wraps on a 32-bit long; and refuse an id at a block < 1, so the new division is
  defined on its own line.
- CHANGELOG, second round: the ids "do not follow the reach" -- they resize when
  the row count changes, which with DSpark is twice a round, so the old clause
  read as "never reallocates". docs/ENVIRONMENT.md's LAYERWISE row names the
  device's candidate mask again (lw_mask, one byte a position): it is not host
  prose, so the file leaves the diff.
… as an experts-mxfp4/ sidecar

AMD's Quark MXFP4 checkpoint's routed experts are copied byte for byte into
<model>/experts-mxfp4/, the int4 sidecar's layout (layer-NNN.safetensors plus
index.json, resumable through .part files): one record per expert, the E8M0
scales of gate, down and up, then their e2m1 nibbles, padded to the next 4096
bytes by a U8 record_pad tensor so the file has no hole. 2,611,200 bytes on
the release, 2,613,248 with the pad. Dtypes, shapes and the E8M0 NaN are
checked, and four experts a layer are compared with the FP8 release's
(cosine, --min-cos) so a nibble order, scale axis or name mix-up is refused.

make_qwen38_tiny.py --mxfp4-experts rounds the fixture's experts to MXFP4
(OCP rounding), writes them under quark/ with Quark's names, writes the FP8
shards with power-of-two block scales so they hold the same values exactly,
converts quark/, and emits a float32 ref.json (a BF16 one is 0.9965 cosine
from the engine on this fixture). make qwen38-tiny-mxfp4-generate builds it
and checks every record against the Quark bytes and every value against the
FP8 shards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y COLI_DISK_WEIGHTS

qwen38 read its expert sidecar (experts-int4g64/ or the low-bit one in use)
from one drive only: COLI_MODEL_MIRROR and COLI_DISK_WEIGHTS were registered
for colibri, the V4 engines and glm53. Each mirror directory's copy of the
sidecar is now added to the sidecar's shards with st_mirror_add (a file is
taken only when its size and safetensors header equal the primary's, so a
partial or stale copy leaves those files on the primary), and every record is
read with st_read_range_rep from the replica a hash of its (layer, expert)
picks, as V4.1 hashes its experts, in the shares COLI_DISK_WEIGHTS sets (equal
when unset; no probe; a list that is not one positive weight per drive is
refused). The same expert always comes from the same drive and every copy
holds the same bytes, so tokens and logits do not move. The Vulkan tier's
direct read and the prefetch advice go to the same replica; mapped experts
(COLI_MAP_EXPERTS=1) and the snapshot's FP8 experts stay on the primary. The
startup prints the split, the run's end the reads each drive served.

make qwen38-tiny-mirror-check (tests/qwen38_mirror_check.py, in the qwen38 CI
job): the int4 fixture with one and two copies in temporary directories, at
caps 1 and 2, parallel and serial reads: tokens and last logits the bits of
the run without a mirror, every replica serving reads; a copy with one
differing header byte and an empty directory skipped; five bad
COLI_DISK_WEIGHTS refused; no mirror opened without the sidecar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…), computed by matmul_mxfp4

Q38_EXPERT_LOWBIT=mxfp4 attaches <snap>/experts-mxfp4/, the sidecar
tools/convert_qwen38_experts_mxfp4.py copies AMD Quark's MXFP4 experts into,
through the low-bit sidecars' attach: its index.json is checked for the
format, codes, scales, group size and the model's geometry, every record for
its six U8 tensors back to back (gate, down and up scales, then their e2m1
codes), and a missing or incomplete sidecar is refused. A record is one read
of 2,611,200 bytes on the release (4,915,200 for the FP8 expert); its
matrices are a new weight kind, Q38_WEIGHT_MXFP4 (the nibbles in data, the
ue8m0 bytes in x2q), multiplied by quant.h's matmul_mxfp4, whose row result
does not depend on the batch, so decode and the prompt batch give the same
bits. The grouped decode has no row kernel for it and takes the per-expert
matmuls. The CUDA tier and the Vulkan tier say they do not take MXFP4 yet and
keep the experts on the CPU; the trunk, shared expert and MTP head stay FP8.
COLI_MODEL_MIRROR copies of experts-mxfp4/ serve it as they serve the int4
sidecar.

tests/test_qwen38_mxfp4_experts.c (make test-c): the matmul against a
dequantized double reference on the release's shapes, the tiny fixture's and
short last groups, 1 to 9 rows, each batch row the bits of that row alone;
the release record 2,611,200 bytes (2,613,248 on its stride) in the
documented order. make qwen38-tiny-mxfp4-check (qwen38 CI job): every expert
the single and batch paths load holds the sidecar's bytes and every value is
the fixture's FP8 value to the bit; ref.json reproduced in 30 configurations
(single, batched, prompt batch, serial prompt, mapped; caps 1, 2, 4; trunk on
the CPU or not) with identical last logits; the FP8 run's last logits within
cosine 0.99999 (1.00000000 on the fixture); refusals each in its own process;
the mirror test's MXFP4 leg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…threads

engram_row reads a row the cache does not hold with one pread and waits for it.
V41_ENGRAM_ADVISE asks the OS for a block's rows first, but only where
POSIX_FADV_WILLNEED exists (not on Windows) and not for a decode step, so there
a prompt's rows still came from the disk one at a time.

With V41_ENGRAM_STAGE=1 (default off), engram_advise_rows hashes the block's
positions, takes the distinct rows the cache does not hold, reads them on
V41_ENGRAM_STAGE_THREADS readers (default 8) and puts them in the cache; the
gather that follows finds them there. It runs for a decode step too, and at the
three places that already advise (engram_run, the Vulkan and the CUDA chains).
A slot holds the same dequantized row whoever read it, so the gather's bits do
not change; only which rows the LRU keeps can. At most half the cache is read
at once; the gather reads any row it does not find itself, as before. The
readers go through st_pread_full, so on Windows they take the shard's
overlapped descriptor.

engram_row is split into engram_find (the probe) and engram_fetch (the read and
dequantization), the same behaviour. The engram stats lines count the staged
reads as rows read.

tests/test_dsv41_engram_stage: one 420-row table and its sidecar, no model.
Staged in whole, in blocks of 32, one position at a time and on one reader,
the gather gives the unstaged bits, finds every row, and each missing row is
read once; a 16-row cache stages at most 8 rows a block and still gives the
same bits; V41_ENGRAM_STAGE=1 makes engram_advise_rows stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he extraction's truncation fallback

model_reset_rows and v41_ps_take kept the candidate tuple of the conversation
they were dropping: its readers key on the pointer, not a row count, so the
stale rows could not be consulted (the source layer rewrites every row before
any reader, in every forward), but the last conversation's allocation -- 2 GiB
at 262144 rows -- stayed live, unlike the shared top-k list they already
dropped. Both now free it, which is also the honest accounting for the rows a
reset keeps apart: park saved them before the reset runs.

candidate_row_ids' truncation branch (a row with more set blocks than slots)
is unreachable in the engine -- both writers paint at most topk blocks, and
test_vk_chain holds the device's mask to the CPU's slot-for-slot -- so the
gate now pins the fallback as a chosen answer rather than an accident: the
first topk blocks in block order, every slot still written.
ink_fill_slots read each piece through the page cache and dropped it after
(POSIX_FADV_DONTNEED). Now each piece's 4 KiB-aligned span goes with O_DIRECT into
a buffer of the reader's and the piece is copied out (a container's slices start
wherever its header ends). The buffers are a pool kept across rounds: a round's
readers are a new team of the fill thread's each time.

Measured on two NVMe in RAID0 with the engine's pieces of 2 MiB: 2.3 ms an expert
against 3.0 in rounds of three, 8.2 GB/s against 6.0 deeper. Inkling-2-Small (int4
g64, the CUDA build, 48 tokens on two prompts, alternated): the decode 1.96-2.01 ->
2.30-2.33 tok/s, the prompt 12.8-13.1 -> 11.0-11.2 s, and the experts' compute 18.4
-> 16.6 s of a run (the page cache's copies took CPU and memory bandwidth). The
tiny oracles match 24/24 either way. INK_FILL_DIRECT=0 reads as before; a file
system that refuses O_DIRECT falls back to it.
deepseek_v41: the candidate blocks as ids, not a dense n² mask (64 GiB -> 2 GiB at 262144)
deepseek_v41: a block's engram rows read on reader threads (V41_ENGRAM_STAGE, off by default)

c/Makefile: #2797's test_dsv41_candidate_ids rule and this test_dsv41_engram_stage rule, both kept.
tools: convert_qwen38_experts_mxfp4.py writes AMD Quark MXFP4 experts as an experts-mxfp4/ sidecar
qwen38: an expert sidecar read from COLI_MODEL_MIRROR copies, split by COLI_DISK_WEIGHTS
qwen38: routed experts from an MXFP4 sidecar (Q38_EXPERT_LOWBIT=mxfp4), computed by matmul_mxfp4
inkling: read the missed experts with O_DIRECT (INK_FILL_DIRECT)
@JustVugg
JustVugg merged commit 567ce93 into dev Nov 15, 2026
119 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants