Skip to content

Batch 2026-11-13-d: nineteen PRs (DeepSeek V4 decode pieces, the Vulkan tier's rooms per size, CUDA prompt attention, FreeBSD, translations) - #2820

Merged
JustVugg merged 50 commits into
devfrom
batch/2026-11-13-d
Nov 14, 2026
Merged

JustVugg merged 50 commits into
devfrom
batch/2026-11-13-d

Conversation

@JustVugg

Copy link
Copy Markdown
Owner

Nineteen PRs on dev after #2809, each merged with its own commits. Four merges needed a hand (below), and one commit of ours sizes the Vulkan tier's prompt staging so that #2789 costs no prompt speed.

What is in it

DeepSeek V4

PR What it does
#2796 A direct read of an expert leaves the record where its window landed and the views point there, instead of moving 12.6 MB after every read (V4_SLAB_SHIFT, on; 0 = the move as before).
#2778 A decode token's attention layer in one device round trip (V4_LAYER_FUSED, opt-in), with a verifier beside the usual path.
#2782 A decode mirror as one weight run and one scale run, filled with two copies (V4_MIRROR_BLOCK, opt-in).
#2784 The decode mirrors may use the prompt bank's VRAM; a bank releases the oldest mirrors when it is created (V4_BANK_MAKE_ROOM, opt-in).
#2788 A decode step's mirror uploads on an uploader thread, the expert group ordered after them by an event (V4_UPLOAD_THREAD, opt-in).
#2603 A native HIP build of the V4 tier (make deepseek-v4 HIP=1), and --gpu / --vram reaching it.
#2800 The Windows deepseek_v4.exe built with clang and LLVM's libomp, whose barriers spin (#2286), with a link probe for MSYS2's llvm-openmp 22.1.7.

Qwen3.8, Qwen3.6 and the tiers

PR What it does
#2789 The Vulkan tier gives every form of expert a room of its own size, not the biggest minority form's.
#2434 The CUDA chain's prompt attention (cc_attn_fk) stages K and V with 16-byte loads, the scalar loop's bits.
#2781 The prefix store's checkpoints on disk across engine runs (Q38_PREFIX_DISK=1, opt-in).
#2762 Under COLI_TIMERS=1, each served request's prefill and decode banks and where its experts came from.

Platforms, release and web

PR What it does
#2790 FreeBSD: memory total and available from sysctl, in the engine, the plan and setup.
#2798 FreeBSD: the hot-thread re-exec finds the executable through kern.proc.pathname, without procfs.
#2799 FreeBSD: O_NOFOLLOW on a link reports ELOOP, as on the other platforms.
#2802 The test suite runs on FreeBSD (gmake, -pthread, -lm, <signal.h>, a pinned CPU in the prebuilt tests).
#2714 A linux-aarch64 release archive (plain armv8-a, CPU only), and setup downloads it on ARM64 Linux.
#2742 The web tool's page fetch connects to the addresses its public-address check resolved.
#2740 docs: building the CUDA DLL from a conda-layout CUDA tree on Windows.
#2811 docs: Spanish, French, German and Russian READMEs, and the other translations synced with README.md.

By @dcutugno (#2778, #2782, #2784, #2788, #2796), @KairaCode (#2762, #2781, #2800, #2811), @risner (#2790, #2798, #2799, #2802), @CoTi1938 (#2434), @kflirik (#2603), @wenmiI (#2789), @ZacharyZcR (#2714), @divya0795 (#2742) and @tylnexttime (#2740).

Merges resolved by hand

With V4_MIRROR_BLOCK=1 and V4_UPLOAD_THREAD=1 together, the two-run refill still drains the compute stream on the uploader thread, as #2782 wrote it: the copies are done before the leases go, and that combination does not get #2788's enqueue-only copies.

Ours: the prompt staging under #2789

#2789 sizes the budget's count from the rooms' average. The staging of a prompt chunk's host pool experts took its share of a layer's experts at that average too, while each layer's experts keep their own form's size. On Qwen3.8-Flash-Next UD-IQ2_XS (five sizes) with COLI_VK_TIER_GB=6 on an RTX PRO 4000 the staging shrank from 1019 to 840 MiB, more of a prompt's experts were read from host memory, and the prompt lost 2.7 %. The staging is now sized per expert as before #2789; with one or two sizes nothing changes.

Qwen3.8 UD-IQ2_XS, Vulkan, 6 GiB tier, three runs a side alternated dev #2789 alone with the staging fix
experts on the card 2496 3144 3035
prompt staging 1019 MiB 840 MiB 1019 MiB
3036-token prompt (tok/s) 262.5 / 262.8 255.4 265.5
decode (tok/s) 23.38 / 23.04 23.87 24.33

(dev's two numbers are from the two sessions, each against its batch arm.)

Tests

ds (Ryzen, 16 threads)

  • make check: every C test and 3424 Python tests pass.
  • The qwen38 tiny oracles and fake-device checks CI runs (qwen38-tiny-check, -fp8-, -tier-engine-, -dn-gpu-, -cuda-chain-, -mtp-): 8 of 8 tokens in all 36 runs.
  • The prefix store tests on the tiny fixtures, qwen38: keep the prefix store's checkpoints on disk across engine runs (opt-in, Q38_PREFIX_DISK=1) #2781's disk tests among them: 24 pass.
  • test_v4_gpu_placement with the hand-merged fakes.
  • deepseek-v4-tiny-check (token-exact against the reference, prefix reuse, read-only), and the oracle again with V4_SLAB_SHIFT=0 and with V4_LAYER_FUSED=1.

RTX PRO 4000 Blackwell (sm_120, CUDA 13.1), i5-13500

model, backend runs a side prompt (tok/s), dev -> branch decode (tok/s), dev -> branch text
Qwen3.8 IQ2_XS, CUDA 3 30.0 -> 30.1 10.67 -> 10.65 the same
Qwen3.6-35B UD-Q4_K_M, CUDA 2 71.4 -> 72.3 13.67 -> 13.73 the same
the same, COLI_CUDA_CHAIN_FLASH2=0 2 72.2 -> 72.4 13.96 -> 13.97 the same
Qwen3.6-35B UD-Q4_K_M, Vulkan, 6 GiB tier 3 82.9 -> 82.7 13.67 -> 13.59 the same
Qwen3.8 UD-IQ2_XS, Vulkan, 6 GiB tier 3 262.8 -> 265.5 23.04 -> 24.33 varies run to run on dev too

A first session of two runs a side gave the same, except one branch run of Qwen3.8 on CUDA at 9.75 tok/s (10.58 to 10.77 in the other nine); the three-a-side session above was run because of it.

Not run here: DeepSeek V4 on a model (none on our machines), so #2796's and the opt-in V4 paths' speed is the authors' measurement; HIP (#2603); FreeBSD (#2790, #2798, #2799, #2802); the Windows builds (#2800, CI's Windows jobs); the linux-aarch64 archive (the release workflow runs on this PR).

ZacharyZcR and others added 30 commits November 9, 2026 07:31
A row in the release matrix on ubuntu-24.04-arm: every engine, the TUI and
the desktop window built for plain armv8-a, so the archive starts on a
Cortex-A72 too, CPU only, with the same verify steps and glibc floor as
linux-x86_64. release_asset_suffix() names it, so start-here.sh downloads
it on aarch64 Linux instead of stopping with "no prebuilt release".

Closes #2684
cc_attn_fk, a prompt chunk's attention on the tensor cores, stages each
block of 64 keys of K and V into shared memory as f16 with a loop that
reads one element a load: two bytes a load from the chain's float16
cache. Since cc_attn_f2 takes prompt chunks from sm_75, cc_attn_fk runs
them on sm_70 (and with COLI_CUDA_CHAIN_FLASH2=0).

At head dim 256 a thread now starts 8 loads of 16 bytes before it stores
any (cc_fa_stage16): eight halves of a float16 cache copied as they are,
or four floats of a float32 cache rounded as the one-element loop rounds
them, so the bits are the loop's. Groups of 16 loads spilled on sm_70,
where the kernel already holds 230 registers (255 registers and 392
bytes of stack with a float16 cache, and the prompt 4.5 % slower than
the loop at 8192 tokens); groups of 8 take 242 registers and no stack.
The staging needs K and V 16-byte aligned, as the chain's mirrors are:
cc_attn_t checks and keeps the loop otherwise, as head dim 64 and 128
and cc_attn_w2 (windows, sinks, the ring) do.

A new test, prompt_rows_16byte_staging, runs 2043 rows after 6144
positions with Qwen3.6's heads (the last block of keys is partial) from
K and V aligned and one element further on (the loop), from a float32
cache of values that are not float16 numbers and from a float16 cache,
and wants the same bits from both; four rows within 1e-4 of the
reference. It runs where cc_attn_fk takes a prompt chunk and says so
elsewhere.

Qwen3.6-35B-A3B int4-g64 on a Tesla V100-SXM2-32GB (every expert on the
card), a prompt from an empty state, 4 + 4 runs in ABBA order: 2048
tokens 2442.37 -> 2595.14 tok/s (+6.3 %), 4096 2341.59 -> 2643.43
(+12.9 %), 8192 2218.47 -> 3014.55 (+35.9 %), every run faster, with
the same fingerprint (a hash of the logits after the prompt and of the
first word) and nll at every length. Under nsys, cc_attn_fk takes 31.3 ms
of a 2048-token prompt instead of 83.9.
Measurements of the previous commit's change against dev bf0b0ea on one
Tesla V100-SXM2-32GB (sm_70, CUDA 12.8, a Vast.ai container with about
181 GiB and 15 CPUs), Qwen3.6-35B-A3B int4-g64, every expert on the
card. prompt.md and manifest.json: prompts of 2048, 4096 and 8192 tokens
in one process, each from an empty state, a warm-up and 4 + 4 runs in
ABBA order (the manifest's "ram" is the host's; its "every logit" is the
bench's fingerprint: the logits after the prompt and of the first word).
correctness.txt: the CUDA chain test of both versions (three failures on
sm_70 that dev has too, outside this change), the two counter-tests that
make the new test fail, registers and stack of cc_attn_fk on every arch,
the first port with groups of 16 loads, the kernel each version runs
(nsys), and tests/cuda_tier_engines.sh.

The code was measured as a local commit, a37779ed, whose c/ tree is the
previous commit's, 61e4fa3 (git rev-parse HEAD~1:c on this commit); the
previous commit adds only the docs paragraph.
page() checked a name's addresses, then urllib resolved the name again
to connect. A name that answers a public address first and 127.0.0.1
next (DNS rebinding) passed the check and the page read a service on
this machine; a redirect to such a name did the same. The connection
now goes to an address from the lookup that was checked. The name
stays in the Host header and the TLS handshake, and a proxy, which
resolves the name itself, is reached as before.
Section 3 assumes the CUDA Toolkit installer. A conda-layout tree works too
(no admin rights needed): its Library/bin on PATH for nvcc and cudart, and
CUDA_HOME pointing at Library.

Co-Authored-By: Claude <noreply@anthropic.com>
… banks

A served request reported one timer bank ("request"), prompt and decode
together. The prefill/decode split existed only for the process total, which
prints at exit, and a serve process stopped by its gateway never gets there.

Under COLI_TIMERS=1 both serve paths (serve_one and the KV_SLOTS mux) now
also print "request prefill" and "request decode" banks, cut where the
request's prompt has been read (q38_serve_start), and one line with where
the request's routed experts came from: RAM cache hits and disk reads, and
with the CUDA tier on its VRAM hit rate, CPU-computed experts, uploads, swaps
and q_skips (qt_counters, new, under the tier's mutex). Without COLI_TIMERS
the output is unchanged.

Tests: test_qwen38_metrics (the line with and without the tier, overflow),
test_qwen36_tier_replan (qt_counters on the fake backend).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p (V4_LAYER_FUSED, opt-in)

A decode token's attention layer is several device round trips with host
arithmetic between them. Under WDDM each trip costs more than the kernels
it launches: decprof's attention is 84 ms per token on an RTX 5080 at a
3.3k context.

V4_LAYER_FUSED=1 runs the layer as one round trip
(dsv4_cuda_attn_decode_fused): the token's six projections from one upload
of the hidden state; the query's norm and RoPE; the key row's norm, RoPE
and FP8 quantize-dequantize, entered into the device KV ring; on a token
that emits a compressed row, the row pooled from a device mirror of the
compressor state; the indexer's query, scores over resident keys and a
bitonic top-k with the host's tie order; the cached sparse attention over
the selected rows, inverse RoPE and the grouped output projection. The
sequence is a CUDA graph per layer. A token the fused layer does not take
falls back to the chained forms it is built from (V4_WO_CHAIN,
V4_DECODE_PROJ6), and from those to the existing paths.

Every step keeps the CPU reference's order of operations, so the results
are the CPU's bit for bit. V4_LAYER_FUSED_VERIFY=1 runs the fused layer
beside the usual path and counts differing words per call; DSV4_WO_VERIFY=1
does the same for the chained tail.

With the switch unset nothing changes. The new entry points are not called,
and the kernels the existing entry points would otherwise pick up (the warp
matvecs, the cached sparse attention's second form, the coalesced indexer
reads) default to off; each sub-switch follows V4_LAYER_FUSED unless set.
coli_v4_indexer_select_batch, the expert pipeline and the prompt's batch
paths are not touched.

With it on, the packed compressor copies and the resident indexer mirrors
(769 MiB on V4 Flash) default to off: the fused layer reads the plain bf16
mirrors and its own resident indexer weights.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d trip (V4_LAYER_FUSED)

The manifest and the raw logs: three interleaved runs per arm (the switch unset against 1), and the two verifier runs (V4_LAYER_FUSED_VERIFY, DSV4_WO_VERIFY), 1,505 calls each without a differing word. The measured figures in docs/deepseek-v4.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…copies (V4_MIRROR_BLOCK, opt-in)

A mirror in the per-device cache is three FP4 tensors, each with its own
weight and scale allocation, so an upload or an in-place recycle is six
cudaMemcpyAsync. The host record already keeps the three scale tensors
together and the three weight tensors together.

V4_MIRROR_BLOCK=1 gives a mirror one weight run and one scale run on the
device, laid out as the record has them (dsv4_cuda_mirror_alloc), and fills
it with two copies (dsv4_cuda_mirror_refill). The entry is still three
tensors to everything that reads it: the expert group, the peer pool, the
resident route. gate owns the two allocations, up and down point into them.
The cache, its growth guard, its LRU and its recycle in place are unchanged.

An expert whose host tensors are not two contiguous runs, or whose weights
would not start on a 16-byte offset, takes the three-tensor mirror as
before, in the same cache; a cache can hold both kinds. A DLL without the
new exports keeps the three-tensor mirrors.

V4_MIRROR_BLOCK_VERIFY=1 reads every fill back and compares it with the
host bytes.

tests/test_v4_gpu_placement.c covers: the switch unset, a block made and
filled from the record's runs in the file's tensor order, a hit, a
three-tensor victim replaced by a block, a block victim refilled in place,
a scattered record in a block victim's place, a refused refill, the
verifier's counts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…runs

The prefix store lives in RAM only, so the first request after every
engine start reads its whole system prompt again, even when the model
and the prompt are the same as before the restart.

With Q38_PREFIX_DISK=1 (default 0) the checkpoints a prompt keeps where
conversations part from it (where its system prompt ends, where its
messages start; not a turn's end) also go to Q38_PREFIX_DIR (default
<SNAP>/.coli_ckpt, as deepseek_v4's) as q38_<identity>_<ids hash>.bin:
the ids, the DeltaNet/PLE/MTP state and every position's K/V/index rows,
with a SHA-256 trailer. They are copied from the store after DONE and
written on a thread of their own (temp + rename), so decode never waits
on the disk. A request the RAM store does not take as far takes the
deepest matching file: size and hash checked before a row is written, a
bad file deleted.

The identity covers the format, row layout and block, state geometry,
the weights (SNAP's safetensors and config, a GGUF's files, the int4 or
low-bit expert sidecar's files, by name, size and time), the binary, the
loaded backend DLL, the Vulkan shaders, the knobs that pick the
prefill's kernels, the K/V row precision (Q38_KV) and the build.
Q38_PREFIX_DISK_MB (default 4096) caps the directory, least recently
used first; every file a request's prompt starts with gets that
request's age and the deeper go first at one age, so a conversation's
system-prompt file goes last.

Tests: tests/test_qwen38_prefix_disk.py (restart reuse with a fresh
read's bytes and logits, plain and MTP; another model or block size
never loads; corrupt or truncated files dropped; the cap keeps the
system prompt's file; off by default), run in CI next to
test_qwen38_prefix_store. The variables are in coli_env.h and
docs/ENVIRONMENT.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restored state's PLE history and pending MTP tokens index the
embeddings. The restore now refuses (and deletes) a file whose ids fall
outside the vocabulary, as the segment adapter does for its PLE history,
before any row is written into the slot.

The other-identity test now checks the first identity's own file by
name, with its bytes and mtime: the other runs write a file at the same
position into the same directory, so looking a position up could not
fail. It also checks that each of them wrote its own file under another
name.

ci.yml: the line continuation before tests.test_qwen38_prefix_disk is
back, as in the steps around it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e environment reference, and the experiment record

docs/deepseek-v4.md: what the switch changes, which experts keep three
tensors, the exit line, the verifier, and the measurement on a 16 GB card.

docs/experiments/v4-mirror-block-2026-11-12: the manifest and the raw logs.
Three runs per arm at DSV4_CUDA_VRAM_RESERVE_MB=600 (440 mirrors): 21.42 s
against 21.05 s for 35 tokens, +1.7 % tok/s. Three runs per arm at the
default reserve (32 mirrors on this card): 22.84 s against 22.91 s, no gain.
One verifier run: 5,618 fills read back, no difference. The text is the same
in all thirteen runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… bank releases the oldest when it is created (V4_BANK_MAKE_ROOM, opt-in)

With COLI_CUDA_MOE_BATCH=1 the decode mirror cache keeps the prompt's
expert bank free at all times (two banks with COLI_CUDA_MOE_DOUBLE=1), so
the next long prompt finds its room. On a 16 GB card that reserve is
8,329 MiB and leaves decode 32 mirrors: every routed expert of every token
is then an upload.

V4_BANK_MAKE_ROOM=1 turns the order around. The cache keeps only the
600 MiB headroom free and grows into the rest. Where a prompt creates a
bank (coli_v4_gpu_moe_batch_union, both banks), v4_gpu_mirrors_make_room
releases the least recently used mirrors of that device's caches until the
bank and the headroom are free, then lets the growth guard probe the card
again. A prompt too short for the bank creates none and releases nothing.

Not touched: a resident table (experts_per_layer), a cache on another
device (a layer placed elsewhere, the peer pool), unified memory. An
explicit DSV4_CUDA_VRAM_RESERVE_MB still decides the reserve; the release
at bank creation then runs only if the switch is on.

tests/test_v4_gpu_placement.c: the mock's free VRAM follows its live
tensors; the reserve with and without the switch, a full cache released
down to the bank's room in LRU order, nothing released when the bank
already fits, a resident table and another device's cache left alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nment reference, and the experiment record

docs/experiments/v4-bank-make-room-2026-11-12: the manifest and the raw
logs. Three runs per arm, one 3,278-token prompt and 36 tokens: 22.83 s
against 21.39 s for the 35 tokens after the first, +6.7 % tok/s, the same
text. Three such prompts in a row on one serve engine, one run per arm: with
the switch on, the second and third prompt each release 528 mirrors, both
banks are created, nothing fails to allocate, and every turn prints the text
it prints with the switch unset.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rs spin (#2286)

MinGW's libgomp is configured "mingw32 posix": its barriers wait on a
semaphore with no spin loop, so OMP_WAIT_POLICY=active and GOMP_SPINCOUNT,
which the launcher sets for deepseek_v4, do nothing on Windows, and each of
a token's many small parallel regions pays a thread wake-up. On an
i7-7700HQ (4 threads) an OpenMP barrier costs 22 us with libgomp whatever
those variables say, 0.3 us with LLVM's libomp.

- c/Makefile: clang's MinGW triple (x86_64-w64-windows-gnu) is Windows too,
  so `make deepseek-v4 CC=clang` reaches the V4 build; the static OpenMP
  archive check looks for libomp.a when the compiler is clang.
- c/Makefile.deepseek-v4: with CC=clang on Windows, libomp.a is linked in
  (-static as before, no new DLL), no -pthread (clang's driver links
  winpthreads; the flag only warns), LTO off unless asked for (GNU ld has no
  LLVM plugin in MSYS2's clang package). The gcc line is unchanged.
- c/deepseek_v4.c: drop v4_moe_bank_hash_layer, written and never read
  (clang's -Wunused-but-set-global), so the clang build has 0 warnings.
- c/coli, c/openai_server.py: KMP_BLOCKTIME=200 beside GOMP_SPINCOUNT for
  deepseek_v4. With OMP_WAIT_POLICY=active alone libomp's team spins
  forever: a served clang engine burned 3 cores while idle (#341).
- release.yml, ci.yml: the Windows deepseek_v4.exe is built with CC=clang;
  the OpenMP package is installed under either of its MSYS2 names; the
  release checks the packaged engine is a libomp build importing no
  MinGW DLL.
- docs and CHANGELOG.

Tiny V4 fixture on the i7-7700HQ, 7 interleaved rounds, same tokens in
every run: decode 500 -> 1547 tok/s, a 72-token prompt 67 -> 20 ms.
test_deepseek_v4_tiny/_prefix/_brio pass with both builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e expert group ordered after them by an event (V4_UPLOAD_THREAD, opt-in)

A decode step attaches each expert it read from the store before it starts
that lane's next read. A recycled mirror is the three tensors' copies and a
wait for them on the compute stream, on the thread that feeds the lanes, so
every upload delays a read.

V4_UPLOAD_THREAD=1 hands the view to one uploader thread and starts the next
read at once. The thread runs the same tiered attach as the step
(v4_gpu_expert_attach_tiered: the same caches, growth guard, LRU and recycle
in place), one view at a time. Its refills copy on the device's second
stream and return without waiting (dsv4_cuda_tensor_refill_fp4 sync 3).
Once the step's reads are in it joins the thread, records an event behind
the copies on the second stream and makes the compute stream wait for it
(dsv4_cuda_upload_fence), so the expert group runs after the copies on the
device and the host waits for no transfer. Before the step releases its
leases it waits for the second stream.

V4_UPLOAD_FENCE=0 keeps the thread and has it drain the second stream after
each mirror (sync 2), with no event. A backend without
dsv4_cuda_upload_fence (an older DLL) is asked once; the uploader then
copies on the compute stream and drains it, as the inline attach does. A
new mirror, while the cache still grows, is a synchronous upload in every
mode. The hybrid split (DSV4_HYBRID) keeps its own uploads.

The step makes no backend call between its first hand-over and the join: a
view the uploader refuses is attached inline only after a join. The backend
is entered by one thread at a time, as before.

Unset, nothing changes: the refill's two new sync values are only passed by
the uploader, and the fence is only called after a hand-over.

tests/test_v4_gpu_placement.c covers: the switch, a view of a layer no
cache owns, three views handed over and their copies enqueued by the
uploader with sync 3, one event per card at the join, the wait at the
settle, a hit, V4_UPLOAD_FENCE=0, an event that fails, a backend without
the entry, and the inline attach unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…the environment reference, and the experiment record

docs/deepseek-v4.md: what the switch changes, where the copies go in each
mode, the exit line, and the measurement on a 16 GB card.

docs/experiments/v4-upload-thread-2026-11-12: the manifest and the raw logs.
Three interleaved runs per arm, one 3,278-token prompt and 36 tokens. At
DSV4_CUDA_VRAM_RESERVE_MB=600: 21.49 s against 20.34 s for the 35 tokens
after the first, +5.6 % tok/s; with V4_UPLOAD_FENCE=0, 20.56 s. At the
default reserve: 22.98 s against 21.62 s, +6.3 % tok/s. The text is the same
in all fifteen runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
With formats by layer the main layers fell in two classes of rooms: the
form most layers have at its own size, every other form in the room of
the biggest of them. A file may give most of its layers other forms than
the most common one: Flint 40B-A2.5B UD-Q4_K_XL has Q5_K experts on 16
of its 49 layers and six other forms on the 33 others, all of those in
one room, 2.3 MiB an expert on average for 1.8 of tensors.

- One class of rooms for each size the forms' tensors take (lcls, ncls;
  forms of the same bytes share a class), class 0 still the most common
  form's, so no tensor sits in a bigger form's room. Two sizes are held
  as before.
- A newcomer displaces a resident of its own class first, then one of a
  class whose rooms hold it, the tightest first (fall, pick_victim): a
  smaller form's expert in a bigger form's room, never the other way.
  A candidate list per class (vcls_of: room class c at c + 1).
- The budget counts the rooms on average over the layers and streaming
  slots take the biggest room of all, as before.
- test_vk_tier: rooms, three layers in three forms of three sizes. A
  budget of the experts' own bytes holds them all (29 of 30 before), the
  smallest form's experts take the rooms the bigger forms' give up, and
  a bigger form's never a smaller one's; no upload refused.

Radeon RX 9060 XT (16 GB), 32 GB of RAM, COLI_VK_STAGED=1
COLI_VK_TIER_HOST=0, each server on a copy of one history, three
interleaved runs, medians: 5263 -> 6717 experts in 12.11 GiB, a
5899-token prompt's disk loads 16050 -> 8562, prompt 109.8 -> 167.5
tok/s, decode 8.2 -> 11.5 tok/s
(docs/experiments/vk-tier-rooms-per-size-2026-11-12).

Co-Authored-By: Claude <noreply@anthropic.com>
#2286)

The launcher's KMP_BLOCKTIME=200 default for deepseek_v4 is now set on
Windows only, and only when the resulting OMP_WAIT_POLICY is active.
libomp gives passive its blocktime of 0 only while KMP_BLOCKTIME is
unset, so the default turned a user's passive into 200 ms of spinning
after every region. macOS's V4 also links libomp (Homebrew), where the
change was not measured (#707), and Linux builds V4 with libgomp. coli's
GLM path, which sets the same active policy on Windows, gets the same
line: a clang colibri.exe now builds with the Windows flags.

ci.yml's Windows job builds the default gcc deepseek-v4 again, and the
release's XDNA=1 VK=1 CC=clang one after it, with the Vulkan headers and
shaderc installed, so the clang compile of the Vulkan units and the
LTO-less link are built on every PR. The Makefile's missing libomp.a
warning points to docs/deepseek-v4.md. Docs and CHANGELOG: same tokens
on the tiny fixture's oracle, the issue's 1.5x came from an LTO build,
the Windows CUDA tier now spins (not measured).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…: both sets of rows in the environment table

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…SLAB_SHIFT, on; 0 = the move as before)

A direct read of an expert lands a 4 KiB-aligned window of the file at the
slab's first byte, and the record starts some bytes into that window. The
reader then moved the record to the slab's first byte (a contiguous record),
or the weights to the end of the scale run (scales and weights apart in the
file): a memmove of the whole record, 12.6 MB for a V4 Flash expert, after
every read, on the thread that read it.

The slot now remembers where the record sits (data_shift for the scales,
weight_shift for the weights) and fill_tensor_view adds it. Nothing is moved.

- A contiguous record: one window, both shifts are its leading pad.
- Scales and weights apart: the weights' window lands on the first page
  boundary past the scale run, the scales are read buffered to the slab's
  first byte as before.
- A per-matrix or GGUF record, a buffered read, a direct read that fell back:
  both shifts are 0, as before. The reader resets them on every read.

A slab holds 16 KiB past its record instead of 8 (COLI_V4_EXPERT_SLAB_SLACK):
the scale run rounded up to a page, the window's pad and its round-up. The
private read's caller (the Vulkan and CUDA expert tier) sizes its buffer with
the same constant.

V4_SLAB_SHIFT=0 restores the move. The bytes the views show are the same
either way; only their address in the slab differs.

tests/test_deepseek_v4.c: test_expert_store_slab_shift reads the fixture's
expert with scales and weights apart and two contiguous ones through the
direct path, then two of them again from RAM, with the switch unset and with
0. Every byte of every tensor is compared, and the weights' address says
which path ran (their file offset modulo a page when kept, right after the
scales when moved).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e experiment record

docs/experiments/v4-slab-shift-2026-11-13: the manifest and the raw logs.
Three interleaved runs per arm, one 3,278-token prompt and 36 tokens, the
switch at 0 against 1. At the default mirror reserve: 22.15 s against 20.34 s
for the 35 tokens after the first, +8.9 % tok/s, and 65.1 s against 58.5 s to
the first token. At DSV4_CUDA_VRAM_RESERVE_MB=600: 20.75 s against 18.81 s,
+10.3 % tok/s. The text is the same in all twelve runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The hot-team re-exec on FreeBSD ran `/proc/curproc/file`, which exists
only when procfs is mounted; a default install does not mount it. The
exec failed and the engine ran without the hot team, printing "re-exec
failed, running untuned".

coli_self_exe_path (`omp_tune.h`) gets the path from the kernel with the
`kern.proc.pathname` sysctl, which works without procfs, and both re-exec
sites use it, keeping `/proc/curproc/file` as a fallback. The CPU mask is
deliberately not reset before the exec as it is on Linux, so a cpuset
the user set is kept. Other platforms are unchanged.
coli_own_fopen opens with O_NOFOLLOW and documents ELOOP for a name
that is a symlink. FreeBSD's open(2) returns EMLINK in that case,
not POSIX's ELOOP (see its STANDARDS section). So on FreeBSD
test_own_file failed, and coli_own_fopen_tmp (and its copy in
route_trace.h), which checks for ELOOP before replacing a link left
under the temp name, never replaced it: saving the state file failed
instead. FreeBSD now maps EMLINK from that open to ELOOP. Other
platforms are unchanged.
Four fixes, each a test that assumed Linux or a GNU toolchain:

- Eleven tests ran `make`, which on the BSDs is BSD make and cannot
  read this GNU Makefile. They now use gmake there, and skip if it is
  missing; Linux and macOS still run make.
- The e8x4g64 and fp8 repack harnesses mirror the Makefile's flags but
  lacked -pthread, which the Makefile has passed on the BSDs since
  #219. Without it pthread_create does not link on FreeBSD.
- test_tok_load_validation now links with -lm, as the Makefile links
  every engine: json.h calls isfinite(), which on FreeBSD is a call
  to __isfinite in libm.
- test_dsv41_cluster_protocol calls signal() without <signal.h>, which
  Linux and macOS pull in through <sys/wait.h> and FreeBSD does not.

Also, check_prebuilt ran setup as if on Linux or Windows but used the
real detect_cpu() of the test machine; where that reports no AVX2 (an
older x86, or FreeBSD) the download tests failed. They now pin
detect_cpu to a processor with AVX2, as the no-AVX2 test pins it to
one without.
backend_cuda_dsv4.cu builds with hipcc through backend_gpu_compat.h (name
mappings) and links into the engine like CUDA=1: make deepseek-v4 HIP=1
HIP_ARCH=gfx1102. The cooperative fused-MoE kernel is CUDA-only and compiled
out. It needs ROCm 6.3+ for the OCP fp8 type; the fp8 names live in
backend_cuda_dsv4_quant.cuh so the ROCm 6.2 HIP syntax check of
backend_cuda.cu does not see them.

The native engine read none of the GPU variables coli exports for V4. On the
HIP build it now reads the names the other engines take on HIP too
(GPU_BACKENDS.md): COLI_CUDA=0 (--gpu none), COLI_GPUS (--gpu N) and
CUDA_EXPERT_GB (--vram N). --vram is the VRAM the engine may use: the expert
mirrors grow until the card has that much less free than at open. The hold is
a floor for the mirror guard's reserve, so the engine's own headroom stays,
and an explicit DSV4_CUDA_VRAM_RESERVE_MB is a floor too. Below 10 GiB the
dense/attention mirrors stay in RAM and the VRAM goes to the mirrors; a long
prompt is prefilled on the CPU. With the mirror live, rows16 repacking turns
itself off.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
libomp.a defines omp_get_num_devices() only as a COFF weak symbol, and
GNU ld leaves it unresolved with MSYS2's llvm-openmp 22.1.7 or binutils
2.46, which is what GitHub's windows runner installs: `make deepseek-v4
CC=clang` failed at the link in #2805's CI, on libomp's own kmp_alloc
and kmp_ftn_extra. Only llvm-openmp 23.x with binutils 2.47 links it.

Makefile.deepseek-v4 now links one call to it the same way; only when
that fails on this symbol does backend_loader_dsv4.c define it,
returning 0, what libomp returns when no offload runtime is linked.
Where it links, nothing is added, so there is no second definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ced with dev

New README.es.md, README.fr.md, README.de.md and README.ru.md, translated
from the current README.md: code blocks, commands, flags, env vars, links,
anchors and table structure unchanged, prose and headings translated, with
the English-slug <a id> anchors the in-page links need (as README.it.md does).

The six existing translations (it, ja, pt-BR, zh-CN, zh-TW, id) gain what
README.md added since they were last synced: Kimi K3.1 Pro in the intro and
the model table, the Qwen-Image-2.1 / Turbo GGUF row, "these models" in the
GGUF intro, the joules sentence on the Profiling page and the "documents in"
bullet (/file PATH). Also a few small fixes where a translation had drifted
from the English (an extra "speculation" in the docs table's Tuning row, an
added "new" in the DeltaNet bullet, a missing troubleshooting row in id).

Every language switcher lists the four new languages. Korean is left to
the open PR #2803.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`compat_meminfo_gb` checks for macOS and Windows; everything else read
`/proc/meminfo`, which FreeBSD does not have unless `linprocfs` is
mounted. On FreeBSD both numbers came back 0, so every engine fell back
to its smallest expert cache, and `tests/test_compat_mem` failed.

Compiling on FreeBSD now reads `hw.physmem` for the total and free +
inactive pages (`vm.stats.vm.*`) for available, the counterpart of
Linux's MemAvailable and of the macOS branch. The laundry queue and
ZFS's ARC are left out, as MemAvailable leaves out memory that cannot
be reclaimed quickly. `hw_probe` in `telemetry.h` had the same gap and
now uses the same function.  Linux, macOS and Windows are otherwise
unchanged.
`resource_plan.memory_available` and `setup_hw.detect_memory` read
`/proc/meminfo`, which FreeBSD does not have unless `linprocfs` is mounted,
so on FreeBSD the planner had no memory figure (None) and setup knew
neither the total nor what is available. Both now read the kernel's
counters with sysctl: `hw.physmem` for the total, and free + inactive
pages for available, the definition `compat_meminfo_gb` uses in C.
`setup_hw` reuses resource_plan's parser. Other platforms are unchanged.
q38_ps_save conflicts with #2780's COLI_PREFIX_LOG lines: it keeps them and returns whether the checkpoint was kept, as #2781 needs.
test_v4_gpu_placement.c: #2782's test_mirror_block and this test_bank_make_room were added at the same place; both are kept.
#2782 (V4_MIRROR_BLOCK) and this both add backend entries, environment rows, a docs section and a placement test at the same places: both kept. The fake refill counts #2782's refills and this one's syncs and threads.
backend_cuda_dsv4.cu's Dev takes #2788's upload fence event and this one's buffers; coli_env.h keeps both sets of rows in order.
#2789 gives every form of expert a room of its own size, and the budget's count from the rooms' average. The staging of a prompt chunk's host pool experts took its share of a layer's experts at that average too, while a layer's experts keep their own form's size: on Qwen3.8-Flash-Next UD-IQ2_XS (five sizes) with COLI_VK_TIER_GB=6 on an RTX PRO 4000 it shrank from 1019 to 840 MiB, more of a prompt's experts were read from host memory, and the 3036-token prompt went from 262.5 to 255.4 tok/s (three runs a side, decode 23.38 -> 23.87).

The staging is now sized per expert as it was before (a layer of a form other than most layers' at those forms' biggest tensors), and the budget gives its experts up for those bytes. With one or two sizes that is the rooms' average, so nothing changes there.
@JustVugg JustVugg mentioned this pull request Nov 13, 2026
2 tasks done
@JustVugg
JustVugg merged commit b1e4747 into dev Nov 14, 2026
123 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants