Skip to content

Latest commit

 

History

History
952 lines (806 loc) · 54.8 KB

qwen38.md

File metadata and controls

952 lines (806 loc) · 54.8 KB

Qwen3.8-Flash-Next on colibri

c/qwen38.c runs the language model in Qwen/Qwen3.8-Flash-Next-FP8 directly from the official safetensors shards. No conversion or second copy of the weights is required. The engine supports text and images through the checkpoint's vision encoder; see Vision below. The checkpoint's MTP layer drafts tokens for speculative decoding, on by default (Q38_MTP=0 turns it off); see Speculative decoding with the MTP head.

The upstream language model has 125B ordinary parameters with 6B activated, plus a 51B hashed n-gram embedding. It has 48 layers arranged as 12 repetitions of three Gated DeltaNet layers and one Qwen Sparse Attention layer. Every layer uses a four-branch gated residual and a 512-expert top-10 MoE plus one shared expert.

Download and run

Pin the checkpoint revision so a later upstream update cannot silently change the local tensor contract:

hf download Qwen/Qwen3.8-Flash-Next-FP8 \
  --revision bcd9f01ddc9cff2316eb84281bebcd5b058bddce \
  --local-dir ~/Models/Qwen3.8-Flash-Next-FP8

make -C c qwen38
COLI_MODEL=~/Models/Qwen3.8-Flash-Next-FP8 ./c/coli chat

coli serve and coli web use the same gateway path. Qwen3.8 thinks by default; reasoning_effort accepts low, medium, high, and xhigh, and enable_thinking: false emits the model's official empty thinking prefix. Audio and grammar constraints are rejected explicitly. Images are supported; see Vision below.

Tool calling works. Qwen3.8 declares and emits calls in an XML-ish form of its own rather than the JSON block GLM uses, so it has its own renderer and its own parser:

<tool_call>
<function=NAME>
<parameter=KEY>
VALUE
</parameter>
</function>
</tool_call>

Both sides are transcribed from chat_template.jinja rather than paraphrased, because the declaration is what teaches the model the syntax it must emit: a preamble it has never seen is a different prompt. The whole rendering is pinned byte for byte against the official template (tests/test_qwen38_chat_template.py, 27 cases across the three reasoning levels).

One asymmetry worth knowing: the template writes a string argument unquoted, so a value's original type is not recoverable from the text alone. The parser reads the declared schema and restores numbers and booleans from it, and leaves anything the schema did not describe as a string rather than guessing.

The official FP8 repository is about 185.5 GB in decimal units (roughly 173 GiB). The default CPU engine keeps resident BF16 matrices in their native two-byte representation and selected experts in native block-FP8. Activations are FP32; BF16 matrices use FP32 accumulation, while native FP8 uses FP32 dot products within each 128-column block and FP64 accumulation across the scaled blocks. The bounded per-layer cache therefore spends about one quarter of the previous memory per FP8 expert. Context state grows by about 54 KiB per token.

What stays on disk

The 51B-parameter PLE table is never materialized in RAM. For each token the engine hashes its bigram and trigram history, reads sixteen 160-byte FP8 rows, and applies the checkpoint's scalar PLE scale. Routed experts are likewise read on demand: the official per-expert E4M3 gate, up, and down bytes and their 128 x 128 floating-point weight_scale_inv blocks remain native in an LRU whose capacity is the first engine positional argument. The canonical FP8 matmul decodes values during accumulation rather than materializing three FP32 matrices per slot. The launcher defaults to one expert slot per layer.

Prompt execution is expert-major in bounded chunks: it routes up to 32 rows, groups their assignments by expert, and consumes cache-sized parallel load groups when the complete demand set is larger than the configured LRU. Shared expert and DeltaNet projections are batched over the same bounded window; DeltaNet convolution and recurrent updates remain token-causal. The private chunk workspace is capped at 64 MiB independently of prompt length.

The serve path owns one hybrid prefix slot. When the next prompt begins with the exact cached token sequence, QSA K/V/index rows are reused in place and the saved DeltaNet/PLE recurrent state is restored before evaluating only the extension. An identical prompt also reuses its saved final logits. Mismatched or shorter prompts reset every hybrid component rather than guessing at cache identity. Decode and cancellation mutate only live state, not the published prompt snapshot. The complete configured QSA context bank is allocated once before the engine publishes READY; serve never grows it alongside a still-live old bank, so the planner's single context-bank reservation is also the peak.

QSA caches the two K/V heads and the indexer's raw key. Complete four-token blocks are pooled and scored, the best 512 blocks are retained, and a causal tail of up to three tokens is appended. The main 24-head attention then operates only on those original tokens. The native model limit is 262,144 tokens; Q38_MAXT defaults to 8,192 and may raise the server limit up to that native ceiling when the required RAM is available. coli serve --ctx N (and coli chat --ctx N) reach the engine as Q38_MAXT=N, not as CTX: that is the variable to look for in the engine process's environment. max_tokens is a ceiling: a budget the prompt leaves no room for is clamped to the room left (one [serve] max_tokens ... clamped line on stderr), and only a prompt that does not fit is refused.

Memory and speed

The 185.5 GB on disk is not a RAM requirement. What must be resident is the dense set: the DeltaNet and QSA projections, norms, gated-residual mixers, embedding, shared experts and LM head. In native BF16 that is 9.2 GiB (9.9 GB), and the engine prints it at load. Everything else is sized by a knob or by the prompt:

resident weights (native BF16) 9.2 GiB, fixed
routed-expert cache 4.7 MiB per slot per layer over 48 layers: cap 16 is 3.5 GiB, cap 32 is 7.0 GiB, cap 64 is 14.1 GiB (2.6 MiB with the int4-g64 sidecar: cap 64 is 7.9 GiB)
FP8 scale bank 28 MiB, fixed; every expert's block scales stay resident so a miss is one FP8 read
context state 54 KiB per token, allocated for the whole Q38_MAXT ceiling before READY: 432 MiB at the 8,192 default
recurrent and PLE state, prefix snapshot, cached logits 226 MiB, fixed
prompt and decode workspace at most 1.1 GiB peak, independent of context length
PLE n-gram table (51B parameters) 0; sixteen 160-byte row reads per token
routed experts on disk 120.8 GB, streamed

coli plan --model <dir> --ram <GB> --ctx 8192 --gpu none prints this accounting for a budget and chooses the expert cap from it.

Measured on the official checkpoint on an Intel Core i9-14900K (24 physical cores, OpenMP 24), 61 GiB RAM, Samsung 990 EVO on ext4, GCC 15.2, Linux 7.0, default 8,192 context, native FP8 and BF16; the engine reports RSS in binary units. Peak RSS for an eleven-token prompt plus one generated token was 12.9 GiB at cap 16, 16.5 GiB at cap 32 and 21.5 GiB at cap 64, with TTFT within 0.6 s across the three. Cap 32 is the short-request knee on this disk; it fits a 24 GB machine, and cap 64 wants 32 GB. A 16 GB machine is below the floor once the cache, context bank and workspace are added.

A full tools/datapoint.py campaign at 8563799 at cap 32: one persistent SERVE=1 engine, page cache evicted before load, greedy decoding, 128 completion tokens per request; one cold request, one warm-identical repeat of it, then four different prompts in fixed rotation as the primary workload. Engine load 11.2 s; cold and buffered iobench 2.26 and 2.55 GB/s:

phase prompt tok request s TTFT s decode tok/s hit RSS
cold 31 129.5 20.6 1.17 58.8% 16.6 GiB
warm-identical (exact prefix reuse; upper bound) 31 107.7 0.01 1.18 63.8% 16.6 GiB
rotating prompts, median of four (primary) 35 to 40 140.2 23.8 1.09 53.8% 16.6 GiB

Of the 140 s median request, 96 s was synchronous expert disk service, 17 s expert matmul, 14 s attention and 2.6 s LM head. One decode token routes ten of 512 experts in each of 48 layers, 4.7 MiB each: 2.2 GiB of expert weights when nothing is cached and roughly half that at the cap-32 hit rate, so at this cache size the engine spends about two thirds of every request waiting on the disk, and the planner labels cold expert reads as the expected bottleneck.

A long context

The model reads up to 262,144 positions (--ctx 262144). Twelve of its 48 layers keep K and V (two heads of 256) and an index key of 128 per position: about 55 KB a position in float32, 14.5 GB at the full context. What a long context costs is what it takes from the experts' caches, so:

  • K and V in float16 (Q38_KV, on when serving): the CUDA chain's mirrors take 2 bytes a value instead of 4. At 262,144 positions the chain keeps 12.86 GiB instead of 19.23, and the card's expert budget goes from 0.65 GB to 7.03 GB.
  • The RAM plan prices one prompt segment, not the context: serve reads a prompt in segments (8,192 rows with a dense chain, 2,048 without; COLI_PROMPT_SEGMENT), so a forward's row buffers hold one segment. The plan used to price the whole context as one forward: 38 GB at 262,144 positions, and no expert cache in RAM (coli serve started at 10 slots a layer).

Qwen3.8-Flash-Next IQ2_XS on an RTX PRO 4000 (24 GB) with 62 GB of RAM, served on the CUDA chain, a 452-token prompt and 128 tokens:

context prompt decode RAM cache, slots a layer card's expert budget
65,536, before 188 tok/s 48.4 tok/s 512 12.5 GB
131,072, before 132 tok/s 43.6 tok/s 337 8.6 GB
262,144, before 15.6 tok/s 9.6 tok/s 10 0.65 GB
262,144, the plan's segment 84 tok/s 21.9 tok/s 490 0.65 GB
262,144, and K/V in float16 149 tok/s 45.7 tok/s 490 7.03 GB

An agent's turns

A model with recurrent layers continues only from a state it kept: the live state (where the last answer ended) or the prompt-end snapshot. A turn whose prompt leaves both earlier reads the whole conversation again. Hermes Agent (a 14,785-token prompt with its tools) did so at every turn, for two reasons, both answered in the engine:

  • The answer comes back without its reasoning. The model wrote <think> and a line break; the next prompt renders that answer as <think>, two line breaks and </think>. The snapshot, taken after <think>, missed by one token (14,809 of 14,810). It is now taken where the answer starts, past <|im_start|>assistant\n, when only the answer's opening follows it.
  • A short request between two turns (Hermes asks for the conversation's title after the first) took the only state there is. The conversation is now parked for it (COLI_PARK): its tokens, its recurrent state and the K/V rows the request may write over, put back when the request ends.

Qwen3.8-Flash-Next IQ2_XS on the CUDA chain, Hermes's second turn (14,810 tokens): 86 s before, 4.9 s after (14,783 tokens reused, 27 read). COLI_PREFIX_LOG=1 says, for a prompt that reuses nothing, how far it follows the live state and the snapshot and the text where it stops.

The other engines that serve one conversation park it the same way (park.h): qwen36 (Qwen3.6, Flint, Muse Spark, Qwen3-Coder, LFM3), olmoe, colibri.c (GLM-5.2, Mistral-Large-4), deepseek_v4, deepseek_v41, kimi_k3, mimo and inkling. With the same agent's requests (its tools, a title request after the first turn), the second turn after the title: Qwen3.6-35B on the CPU 51.6 s before, 3.3 s after; OLMoE-1B-7B on the CPU 39.9 s before, 6.5 s after, every answer identical to the same conversation without the title. OLMoE, attention alone, also resumes where a prompt leaves the held state (an answer sent back without its reasoning, a changed turn) instead of reading it all.

Routed experts as int4-g64

Optional. Decode waits on routed experts: a miss is a disk read, and how many experts fit in the cache decides how often that happens. tools/convert_qwen38_experts_int4.py rewrites them as int4 with one f32 scale per 64 inputs into <model>/experts-int4g64/, next to the FP8 shards, which it only reads; qwen38 picks the directory up by itself.

python3 c/tools/convert_qwen38_experts_int4.py --model ~/Models/Qwen3.8-Flash-Next-FP8 --plan
python3 c/tools/convert_qwen38_experts_int4.py --model ~/Models/Qwen3.8-Flash-Next-FP8
FP8 (release) int4-g64 sidecar
bytes per expert 4,915,200 (scales in the 28 MiB resident bank) 2,764,800: codes 2,457,600, scales 307,200
experts per GiB of cache 218 388
cap per GiB of cache, 48 layers 4.5 8.1
all routed experts on disk 120.8 GB 68.0 GB, in addition to the FP8 shards

The values are the ones the engine computes from the release (e4m3 byte times its block's weight_scale_inv), quantized as tools/convert_qwen36.py --ebits 4 --gs 64 does: per 64 inputs, scale = absmax / 7, code = round half to even of w / scale, clamped to [-8, 7]. The codes are stored as v+8 in expert_ffn.h's planar layout and multiplied by its f32 kernel, activations staying f32 as on the FP8 path. On synthetic Gaussian weights at the release's geometry the relative L2 error is 10.9% per matrix; the converter prints the mean and the worst per layer on the real weights, where it measured 11-12% per layer and 17.5% for the worst single matrix.

Measured on the release (Ryzen 7 PRO 8700GE, 16 threads, 61 GiB, NVMe; the same binary with Q38_EXPERT_INT4=0 and =1, OMP_NUM_THREADS=8). Perplexity, teacher-forced over four 504-token chunks at cap 96:

chunk FP8 int4-g64
0 4.23 4.20
1 11.64 11.74
2 10.94 11.04
3 16.04 16.97

On average +0.017 nats per token. Three chunks move by under 1%; chunk 3 moves by 5.8%. Greedy answers stay on the same reasoning with a few words changed.

Decode, 100 tokens of one prompt, the model's files dropped from the page cache before every run, load average under 2.5:

cap FP8 int4-g64
32 1.91 tok/s, 14.4 GB RSS, 45.8% hits 2.87 tok/s, 11.6 GB, 46.4%
64 2.27 tok/s, 21.4 GB, 59.1% 3.25 tok/s, 15.2 GB, 59.4%
96 2.56 tok/s, 28.5 GB, 68.0% 3.55 tok/s, 19.2 GB, 67.8%
170 3.99 tok/s, 28.0 GB, 79.2%

At the same cap int4 is 1.4-1.5x faster, because each miss reads 56% of the bytes and the routed-expert compute halves (55 to 31 ms per decode forward at cap 96). At the same RAM, FP8 at cap 96 (28.5 GB) against int4 at cap 170 (28.0 GB), it is 1.56x faster: the RAM that held 96 experts per layer holds 170.

The converter streams: one expert per worker in flight, so peak RAM is a few hundred MB for the writer (records queue there while the disk catches up; at most one layer, 1.4 GB) plus 0.11 GB per worker, whatever the model size. On a 12-thread laptop CPU (i7-1355U) and a synthetic snapshot of the release's expert geometry a layer is 22 s of compute with 12 workers; the disk sets the rest. It is resumable: a layer is written to a .part file and renamed when whole, and index.json lists the layers done, so a second run continues where the first stopped. A run against a snapshot whose config.json or expert shards changed is refused.

Layout: one safetensors file per layer (layer-NNN.safetensors) and index.json. Every expert is one record, so a miss is one read:

gate codes U8 [I, H/2] | up codes U8 [I, H/2] | down codes U8 [H, I/2] |
gate scales F32 [I, H/64] | up scales F32 [I, H/64] | down scales F32 [H, I/64]

named like the release's tensors (...experts.E.gate_proj.weight, and the same name plus .qs for its scales). The data region of each file starts on a 4096-byte boundary; at the release's geometry a record is 675 pages, so every record is page aligned. index.json carries the format name and version, the geometry, the group size, the record size, the layers done and complete; the engine refuses a sidecar whose index or files disagree with config.json and skips (with a line on stderr) one still being written.

Q38_EXPERT_INT4=0 keeps the snapshot's FP8 experts; =1 refuses to start without a complete sidecar. Every expert path reads the sidecar: the single LRU load, the parallel batch reads (Q38_EXPERT_PARALLEL_READS), the prefetch, the prefill batch and COLI_MAP_EXPERTS=1. The startup line names the expert representation, the bytes of one expert, what the cache costs full at the given cap, and how many experts per layer the RAM available at that moment would hold. coli plan and --auto-tier size the cache cap with the sidecar's records and plan no VRAM for them: the CUDA expert tier streams FP8 experts only, and with int4 experts it stays off, trunk included. A build with Vulkan too (VK=1 CUDA=1, what coli setup builds on an NVIDIA card) hands the sidecar's experts (int4-g64 and the low-bit ones) to the Vulkan tier and its dense chain, on the same card, when COLI_CUDA=1 and COLI_VULKAN is unset; COLI_VULKAN=0 keeps them on the CPU. On an RTX PRO 4000 (24 GB) with 62 GB of RAM, the release with the int4-g64 sidecar (64 GiB of experts), a 215-token prompt and 128 tokens, runs 2 to 4: 100-102 / 23.7-24.9 tok/s (prefill / decode) with the handoff, as with COLI_VULKAN=1; 9-18 / 6.3-6.8 with the experts on the CPU; and 7.2-7.4 / 3.1-3.2 with the snapshot's FP8 experts on the CUDA tier (Q38_EXPERT_INT4=0), where 3098 of them fit the card and the rest of the 176 GB comes from the disk. The Segment and Edge adapters keep the snapshot's experts.

make -C c qwen38-tiny-int4-check converts the FP8 tiny fixture (experts scaled by 3, so that they move the logits beyond the oracle's tolerance) and gates on the int4 matmul against a dequantized f32 reference, the converter's round trip against the FP8 source, the engine reproducing the float32 reference of the dequantized sidecar through every expert path with the last logits identical to the bit across them, and the CUDA tier declining the int4 experts on the fake backend.

Speculative decoding with the MTP head

On by default from 1.13.0 when the checkpoint has it (Q38_MTP=0 turns it off, and Q38_MTP=1 refuses a checkpoint without it). The release carries one more decoder layer under mtp.* that reads the model's four hyper-connection streams at a position together with the next token's embedding and predicts the token after it. The engine loads it and decodes speculatively: after each token the head drafts the next one, and one forward over both (S=2) returns the logits of the first as a plain decode step would while it checks the draft. When the token picked from those logits equals the draft, the second row's logits answer the next step with no forward; when it does not, the second row is undone: the Gated DeltaNet and PLE state go back to a copy taken after the first row (a pointer swap, nothing is recomputed) and the attention, indexer and prefix records go back one position.

Every row of the verify forward is computed as a decode step computes it, so the output is the output of plain decoding, token for token and logit for logit, greedy or sampled (a sampled token that happens to equal the draft is an accepted draft). The CUDA expert tier is the exception it already is: its float order follows which experts are resident when.

The head is built only from blocks the engine already runs for its own layers: the decoder layer is a QSA attention layer with its indexer, gated residuals, the MoE with its shared expert, the head's own hyper-connection mixer and the shared lm_head. The tensors do not say how pre_fc_norm_hidden (4 x 2560 wide) groups the model's four streams before fc_hidden maps each of them, so Q38_MTP_WIRING selects it (c/qwen38_core.h, "the MTP head"): b, the default, normalizes each stream on its own, the way the hyper-connection norms do; a normalizes the four streams as one vector and stays as a diagnostic. The output does not depend on the choice. The acceptance rate does, and on the release it picks b.

Every run prints its line, and a served request prints one per turn, on stderr:

[qwen38 MTP] run: 1.94 tokens/forward (51 forwards per 99 tokens) | acceptance 94.1% (48/51 drafts) | wiring b

Tokens are the decode tokens after the prompt and forwards the forwards that produced them. The head's routed experts stay the snapshot's FP8 with or without the int4-g64 sidecar (one layer read for at most two rows per draft is a small share of the expert traffic); their cache holds Q38_MTP_CAP experts, by default the cap every layer has, or every expert of the head on an integrated GPU with 32 or more compute units, the expert tier and the dense chain on (vulkan.md). The startup line prints what it costs.

Measured

On the release with the int4-g64 sidecar (Ryzen 7 PRO 8700GE, 16 threads, 61 GiB, NVMe; model files dropped from the page cache before every run, load average under 2, OMP_NUM_THREADS=8, 100 new tokens). Greedy output was byte-identical to MTP off in every run.

prompt, cap wiring acceptance tokens/forward tok/s
raw text prompt, cap 96 MTP off 3.57
a 92.2% (47/51) 1.90 3.97
b 96.0% (48/50) 1.94 4.01 (+12%)
c 78.2% (43/55) 1.77 3.76
chat-template coding prompt, cap 170 MTP off 4.15
a 84.9% (45/53) 1.83 4.58
b 94.1% (48/51) 1.94 4.74 (+14%)
c 63.3% (38/60) 1.62 4.21

c was a third reading (the normalized streams averaged into one vector before a single fc_hidden). It came last on both prompts and was removed. The means of the head's norm weights are -0.76 (embedding), -0.33 (hidden) and 3.79 (mixer), consistent with norms that scale by 1 + w like every other norm of the model.

Nearly two tokens per forward gives only 12-14% more tokens per second here because the forward is not where the time goes. A verify forward feeds two tokens, and each token routes its own 10 experts per layer. Consecutive tokens share on average 3.2 of their 10 experts per layer (measured on a ROUTE_TRACE of 199 tokens), so a pair needs almost twice the experts of a single token. At cap 170 the cache missed 74 experts per token whether the tokens went one at a time or in accepted pairs. The disk reads for experts are therefore the same with MTP and without it. What MTP saves is the dense part of the forward, read once per forward instead of once per token: the trunk matrices, DeltaNet, attention and lm_head. Where the routed experts fit in RAM, those disk reads go away and the dense part is a larger share of each token, so the gain should be larger. That case has not been measured.

Deeper verifies and prompt lookup

Q38_MTP_DRAFTS lets the head draft up to 3 tokens per verify: each draft past the first reads the head's own streams from the row before, with the draft just proposed. The verify copies the DeltaNet and PLE state after each of its rows but the last, so a rejection after row k restores the state after row k. Q38_MTP_DRAFTS=0 lets a gate pick the depth per verify from the measured acceptance by position and the measured cost of a verify by its rows. Prompt lookup (on by default) adds drafts (up to 5, from the context's n-grams), and a verify carries whichever proposal is worth more. The output stays that of plain decoding. Measured on the release with the int4-g64 sidecar: 4.94 tok/s at two drafts against 4.71 at one and 4.12 without MTP on the CPU (2.75 tokens per forward), 4.55 against 4.39 and 3.91 with the Vulkan tier and chain on a Radeon 780M, so two drafts is the default (one on an integrated GPU with 32 or more compute units, the tier and the chain on). The details, the settings and the tests are in speculative.md.

make -C c qwen38-tiny-mtp-check adds a head with random weights under the release's names to the three tiny fixtures and gates on the head's draft logits against a float32 reference of both wirings (c/tools/qwen38_mtp_ref.py), and on byte-identical output with drafts and without them across caps, prefill batching, both trunk formats, BF16, FP8 and int4 experts, text and serve, with drafts all rejected, all accepted and alternating.

GPU: Vulkan expert tier

In a VK=1 build, COLI_VULKAN=1 puts the trunk on the Vulkan device (as before) and the routed experts of the model's layers on the shared Vulkan expert tier (c/vk_tier.c, vulkan.md): a cache of experts in device memory, filled at startup from .coli_usage (coli setup ships a starting one) and adapted while you chat, whose experts the device computes while the CPU computes the rest of the step. It takes every expert form this engine reads: the int4-g64 sidecar's planar records (as int4 groups of 64 on the device), the release's FP8 with the 128x128 block-scale bank (fmt 12, the block scale repeated over its rows), BF16. Decode and prefill both use it; an MTP verify's two rows take the device's per-row route, so they get a decode step's bits; the MTP head's own layer goes on the tier as an extra layer on a discrete GPU (COLI_VK_TIER_MTP, vulkan.md). Every expert's output joins its row in rank order, device or not.

make -C c qwen38 VK=1
COLI_VULKAN=1 SNAP=<checkpoint> ./c/qwen38 96 8 prompt.txt    # experts; the trunk too on a discrete GPU
COLI_VULKAN=1 COLI_VK_DENSE=0 ...                              # experts only, trunk on the CPU
COLI_VULKAN=1 COLI_VK_DENSE=1 ...                              # trunk on the device on any GPU

On a GPU that shares the CPU's RAM (an integrated GPU, Lavapipe) the trunk stays on the CPU by default while the tier is on; the startup line says where it went and why.

On an integrated Radeon 780M the tier with the trunk on the CPU decoded at 3.80 tok/s against the CPU's 3.51 and reached the first token of a 512-token prompt in 38.3 s against 43.9 s; the trunk on the device cost more there than the tier gained. The budget, the knobs (COLI_VK_TIER*), those measurements and what they leave out are in vulkan.md. With COLI_CUDA=1 as well, the CUDA tier below wins.

GPU: CUDA VRAM expert tier

With CUDA=1 the engine links the same expert tier as Qwen3.6 (c/qwen36_tier.c, qwen36-cuda-tier.md) in its fp8 streaming mode. Nothing about the model or the RAM cache changes: experts still stream from disk into the per-layer LRU (cap), and the tier keeps its own copies of the hot ones in VRAM as a third stage above it, disk -> RAM LRU -> VRAM. Routed experts that are resident are computed on the GPU (grouped_hidden_f8w_dual / grouped_down_f8w from #817: e4m3 bytes plus the checkpoint's [ceil(O/128), ceil(I/128)] block scales, no conversion); the rest run on the CPU as before. Every expert the CPU computes is reported to the tier, which copies the 4.7 MiB slab and the three scale tables during that call -- the RAM slot may be recycled by the next token -- and promotes it when it has room or when it is hotter than the coldest resident expert on its device (budget-neutral swap). Prefill, the DeltaNet recurrence, QSA and the PLE table stay on the CPU; the dense trunk goes to the GPU as well, see below.

make -C c qwen38 CUDA=1 CUDA_ARCH=native      # NVCC=/usr/bin/nvcc on distro CUDA
COLI_CUDA=1 COLI_GPUS=0,1 CUDA_EXPERT_GB=auto COLI_TIMERS=1 \
OMP_NUM_THREADS=<physical cores - 1> OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
SNAP=<checkpoint> N_NEW=200 ./c/qwen38 128 8 prompt.txt

coli chat --gpu <n> does the same through the planner (supports_accelerator in the qwen38 descriptor). The tier reports next to the cache hit rate: resident experts, uploads, VRAM hits and misses, LFRU swaps. Requirements: native FP8 routed experts (Q38_NATIVE_FP8=1, the default) with every layer's block-scale bank resident; a checkpoint that falls back to the per-matrix scale loader stays on the CPU and says so.

The card's allowance. CUDA_EXPERT_GB is what each card gives the tier, the dense trunk and, with the CUDA chain, the chain's own buffers, in GB. Unset, 0 or auto, it is the card's free memory less 1 GB; the start prints both, [qtier] dev 0: <free> GB free, allowance <GB> GB. coli chat --vram <GB> passes its number as CUDA_EXPERT_GB; --gpu alone leaves it to the engine. COLI_VK_TIER_GB is the Vulkan tier's and does not size this one.

What a VRAM budget buys. The routing of Qwen3.8-Flash-Next has no hot set that carries over between prompts: the 1,400 experts (5.8 % of the model) that were hottest in one 100-token run covered a different prompt's routes at chance level (4.7 %; two cards, 2,800 experts: 9.3 % against 11.5 % chance; rank correlation of expert frequency per layer between the two runs -0.2). A heat file from an earlier run therefore does not help, and the engine does no warmstart of the fp8 tier. (A GGUF is another case: the tier holds its experts as the file has them and fills the card at the start, the history's most used first, .coli_usage; COLI_CUDA_TIER_WARM=0 turns that off. See gguf.md.) Inside a run the locality is strong -- on the route trace of a 315-token prompt plus 100 generated tokens, an LRU of 32 slots per layer serves 55 % of decode routes, 64 serve 79 %, 128 serve 90 % -- so the tier earns its VRAM by promotion at touch time: with the RAM LRU at cap 128, one 8 GB card served 45 % of decode routes from VRAM on the real checkpoint, two cards 59 %.

VRAM per expert. The tier charges what cudaMalloc takes, not the payload: an allocation above 1 MiB rounds up to a multiple of 2 MiB, so each 1.56 MiB expert matrix occupies 2 MiB and an expert costs 6.03 MiB of VRAM for 4.69 MiB of bytes. A 6.5 GB budget holds about 1,080 experts. (Qwen3.6's int4 matrices are exactly 512 KiB and are served exactly, so its accounting was already right.) Pooling experts into one arena per device would recover the 22 %; it is not done yet.

What it is worth on this machine -- measured, not projected. Threadripper PRO 3945WX (12 cores, OMP_PLACES=cores), 94 GB, checkpoint in the page cache, prompt of 315 tokens plus 100 generated, cap 128, COLI_TIMERS=1 decode bank (the Speed line the engine prints divides by the whole generation time including the prompt; the numbers below are decode only):

CPU only tier, one 8 GB card tier, two cards
VRAM share of routed experts -- 45 % 59 %
routed-expert GEMV on the CPU, ms/token 192 126 93
decode, ms/token 806 808 808
decode tok/s (three runs each) 1.14 - 1.24 1.20 - 1.24 1.21 - 1.24
greedy tokens identical identical identical

A decode token costs about 0.8 s here: 260 ms dense kernels (BF16 trunk and the DeltaNet projections), 244 ms expert reads (page-cache copies of the cap misses), 192 ms routed-expert GEMVs on the CPU (0.5 ms for the three FP8 matrices of one expert), 42 ms LM head, 24 ms QSA, 19 ms shared expert. The tier removes 66 - 99 ms of GEMV time and spends about as much on its own per-layer issue/take round trips and on staging the experts it promotes, so the wall time does not move. The parity holds, the plumbing is exercised on a real checkpoint, and the honest summary is: on a machine of this class the expert tier is not where Qwen3.8's decode time is. The dense trunk (a third of the token) and the expert reads (another third, a RAM-cap question) are.

The dense trunk in VRAM (stage 1)

The trunk is the largest fixed cost of a decode token and it is bandwidth-bound: 8 GiB of BF16 matmul weights read on every token at the memory bus's pace. With the tier on, the engine offers every dense matmul matrix of at least 1 MiB to the tier's placer by name and layer (the DeltaNet projections dnqkv/dnz/dnout, attention attnq/attnk/ attnv/attno/qsaidx, the hyper-connection mixers hcad/hcau/hcmd/ hcmu, the shared expert shg/shu/shd, the router, and lmhead); whatever the placer accepts is quantized to int8 per row (scale = max|w| / 127, the qwen36 dnproj/lmhead format) when the tier starts -- about 2 s for the 553 matrices, 3.96 GiB on one card -- and answers decode GEMVs from there, one round trip per matmul (x up, y down; activations stay on the CPU). Prefill rows (S > 1) and any backend failure take the CPU path, which holds the same int8 rows (see the trunk on the CPU). The placer takes the trunk before the experts (it is read on every token), so on an 8 GB card about 2 GB remain for hot experts; coli plan prices it the same way (4.0 GB int8 trunk + ... hot tier).

variable effect
Q38_TRUNK_GPU=0 keep the trunk on the CPU (experts-only tier, the behaviour before this)
Q38_TRUNK_MIN_KB=<n> offer matrices of at least n KiB (default 1024; a round trip costs more than a tiny GEMV saves)
Q38_TRUNK_SKIP=name,name leave the named components on the CPU (bisecting, or a component that does not pay)
QT_UPLOAD_SYNC=1 the tier's qt_issue waits for every in-flight upload first (tests and diagnostics: deterministic residency, no upload/compute overlap)
Q38_TRUNK_CPU_INT8=0 keep the trunk BF16 on the CPU with the f32 kernel (the numeric reference); the default is int8 rows with integer dot products, GPU or not
Q38_TRUNK_SELFTEST=1 at start, every placed matrix is checked once: GPU GEMV against the same int8 rows on the CPU (relative error printed per matrix)

Numerics. GPU int8 against CPU int8 on the same rows: relative error ~1e-7 on all 553 matrices (float summation order), greedy tokens identical over 30 tokens on the 315-token prompt and 20 on a short one. int8 against BF16: the per-row quantization error is 1 - 3 % per matrix (largest on the router and the hyper-connection down-mixers), and on the same prompts the greedy text is identical to the BF16 run for the 30 tokens compared. Perplexity on wikitext-2 (8 chunks of 512, 2048 scored tokens, llama-perplexity protocol): 1.880 with the trunk and the expert tier on the GPU against 1.845 for the BF16 CPU run, +1.9 %. The expert tier on its own is neutral (+0.0008 nats on a 256-token probe), the lm_head contributes nothing, and keeping the 48 routers in BF16 on the CPU (Q38_TRUNK_SKIP=router) only recovers 0.3 points (1.875): the per-row int8 error is spread over the trunk, not concentrated in one class. Group scales (gs 64/128) for the trunk are the lever left; that needs a kernel format, not a placement change.

Measured (same machine and prompt as above, cap 224 so the RAM LRU serves 84 % of expert reads, COLI_TIMERS=1 decode bank, 100 tokens):

CPU only (BF16 trunk) experts only, one card trunk on one card, no experts trunk + experts, one card trunk + experts, two cards
VRAM: trunk / experts -- 0 / 6.1 GB 4.0 / 0.1 GB 4.0 / 2.1 GB 1.8 + 2.2 / 4.3 + 4.3 GB
VRAM share of routed experts -- 43 % 2 % 22 % 52 %
resident-mm (dense trunk incl. DeltaNet proj.), ms/token 284 285 52 63 89
shared expert / lm-head, ms/token 21 / 46 21 / 46 6 / 5 7 / 5 12 / 14
routed-expert GEMV on the CPU, ms/token 179 118 182 151 93
expert-read (page-cache copies), ms/token 168 164 162 160 145
decode, ms/token 744 754 557 523 466
decode tok/s 1.34 1.33 1.80 1.91 2.14
greedy tokens (100) reference identical identical identical identical

Peak RSS 60.1 - 60.5 GB in every run (cap 224); TTFT 183 - 185 s in every run (the prompt's forward stays on the CPU). On two cards the placer spreads the trunk by free room (1.8 GB on the 3070, 2.2 GB with the lm_head on the Quadro) and the experts on both; the Quadro is the slower card (lm-head 14 ms there against 5 ms on the 3070), which is the +26 ms of resident-mm against one card -- and the 52 % expert share still wins by 57 ms. The trunk is worth +34 % on its own (1.34 -> 1.80), the expert tier nothing on its own (1.33) and +6 % on top of the trunk (1.91), because the GEMVs it saves were the small part all along; the two-card run gets its +60 % from both. The first calibration (06.09.) gave 1.45 -> 1.76 on the same prompt with the old binary; today's CPU baseline is 8 % lower, run-to-run scatter of the page-cache expert reads (168 vs 149 ms), so the ratios are the comparable figures.

One thing found on the way: the backend's resident dense matvec shared its x/y staging buffers with the asynchronous expert group. The shared expert's three GEMVs run between qt_issue and qt_take, so with the trunk in VRAM they overwrote the in-flight group's input and output -- no CUDA error, only wrong tokens, worse the more experts were resident. The dense path has its own buffers now (qwen36-cuda-tier.md); qwen36 never called the dense path inside that window.

The DeltaNet layer on the card (Q38_DN_GPU=1)

With the trunk in VRAM a DeltaNet layer still crosses the bus four times per decode token: the dnqkv and dnz results come down, the convolution, the recurrence and the gated norm run on the CPU, and the normed rows go back up for dnout. On qwen36 those round trips were 8 of 39 ms per token (qwen36-cuda-tier.md); Qwen3.8 has 36 such layers of 48 value heads. Q38_DN_GPU=1 keeps the layer on the card: for every DeltaNet layer whose dnqkv, dnz and dnout the placer put on one device, the conv ring, the recurrent state (48 x 128 x 128 f32, 3 MB per layer) and the conv/norm weights go there too, and a decode token runs the layer end to end in one device chain -- x up, the two in_proj GEMVs, causal conv + SiLU, per value head the L2 norms, the decay, the delta rule, the gated RMSNorm with sigmoid(z) (Qwen3.8's gate; Qwen3.6 uses silu(z)), the out_proj GEMV, out down. The two gates (a, b) stay on the CPU and travel as kernel parameters.

The host arrays remain the state's owner: a prompt (S > 1), an MTP verify (snap_after: the CPU path takes the snapshot after its first row), a --pin snapshot and the prompt cache pull the state down first; a reset, a pin restore and a rejected draft's rollback invalidate the card's copy. A failing GPU step turns the layer off and the CPU continues from what the card holds. Opt-in; tests/test_qwen38_dn_gpu.c (make qwen38-dn-gpu-check) runs the tiny fixture's oracle with the layers on the fake card.

The chain on the card (COLI_CUDA_CHAIN=1)

The CUDA twin of the Vulkan chain (qwen38_chain.h), on the same ops table qwen36's chain uses (cuda_chain.h, docs/qwen36-cuda-tier.md "The dense chain on the card"), in qwen38_cuda_chain.h. Per layer, one frame on the card: the previous block's MoE output joining the four hyper-connection streams (routed + sigmoid(gate) x shared, then hyper += inject x block), the gated residual's read (per-stream norm, the low-rank down/up pair, the stream mix, the inject weights), the gated attention with QSA (the index keys into the device cache, each completed block's pooled key computed once on the device, the indexer's top-k selection per query row, the attention over the listed positions with the output gate) or the Gated DeltaNet (the convolution with its ring, the recurrence with its state, the sigmoid-gated norm), the write-back, the second gated residual, the router logits; waited for. The host routes and runs the routed experts through the fp8 tier (q38_moe_ex routed only) and hands the sum up; the shared expert runs on the card meanwhile (frame A2). At the PLE layer the n-gram rows come from the host (disk reads, the history replayed as q38_ple does), the key and value projections, the gate and the dilated convolution with its ring run on the device. The last frame runs the final mixer and lm_head; the streams come back when the MTP head needs them.

The matrices: the tier's int8 copies where the placer put them (Q38_TRUNK_GPU, COLI_PLACE); every other matrix of a layer, the final mixer and lm_head the chain uploads itself, the trunk's int8 rows when the CPU holds them (the same bits), else the BF16/F32 rows as f32. So the chain runs whatever the placement did; what it adds in VRAM are the small matrices the placer never sees (the gated residuals' low-rank pairs, the DeltaNet b/a rows, the PLE projections) and the state (the DeltaNet state and conv rings, the PLE ring, the K/V, index-key and pooled-key mirrors).

State ownership as in qwen36's chain: the host's K/V and index caches canonical behind a watermark, the pooled block keys behind one of their own; the DeltaNet state, conv rings and PLE ring on the device while the chain runs, brought back before the host reads them (a pinned snapshot, the prefix cache) and pushed up after the host writes them (a reset as zeros, a restored snapshot as an upload). An MTP or prompt-lookup verify runs row-wise (the warp GEMV computes each row on its own: a decode step's bits) and copies the DeltaNet state, the conv rings and the PLE ring after each of its rows into a device slot; a rejected draft swaps slot r in (q38cc_rollback) as the CPU swaps its own. A lost device zeroes the state, rebuilds it on the CPU from the prefix record (the PLE history replayed) and runs there from then on. Q38_DN_GPU's per-layer step is not set up beside the chain; KV_SLOTS > 1 keeps the chain off.

Measured (RTX 3070 alone, the FP8 checkpoint, the trunk's 553 matrices as int8 in VRAM, the tier at 16 % hits behind it, the usual 49-token prompt, 100 new tokens, two runs each):

TTFT tok/s, the run ms per token resident-mm per forward DeltaNet QSA PLE lm_head shared
Q38_DN_GPU=1 (the per-layer step, #1870) 19.0-19.1 s 1.22 / 1.26 612-632 172 ms 161 23 12 26 23
COLI_CUDA_CHAIN=1 17.7-17.9 s 1.41 / 1.43 526-535 74 ms (the chain waits 68) in the chain in the chain in the chain 10 0.4

The chain takes the whole dense part at once (-15 % per token, 1.2 s off the first token); the token stays the experts': about 1 s of expert reads per forward (24,576 FP8 experts behind a 4 GiB trunk on 8 GB) and 290 ms of routed experts, which no chain touches. The chain uploads 174 MiB itself (the matrices the placer never sees) and holds 381 MiB of state (the K/V, index-key and pooled-key mirrors, the DeltaNet state, the PLE ring).

A long prompt layer by layer (q38cc_forward_lm, COLI_CUDA_CHAIN_LAYERWISE). A prompt chunk copies every layer's host pool experts into VRAM before that layer's experts run (q38cc_stage_from), so a 4K-token prompt in 2048-row chunks copied each layer's pool experts twice. Where every layer runs its experts in the frame and a chunk stages them, a prompt of more chunks than one now runs layer by layer: each layer's chunks in order, its copy waited for at its first chunk and the next staged layer's started after its last (on the copy stream, beside the next frames), every row's streams kept on the device between layers. Each layer's chunk is a frame waited for, so the K/V rows and the routes' books come down a chunk at a time as before, and a chunk runs the same ops on the same rows. RTX PRO 4000 (PCIe gen 4 x8), Qwen3.8-Flash-Next IQ2_XS with the host pool, prompts of 1K / 4K tokens: 321.3 / 309.0 -> 321.2 / 343.0 tok/s, the decode the same (43.6 tok/s). The Vulkan chain does the same (q38c_forward_lm, COLI_VK_CHAIN_LAYERWISE). A prompt longer than a segment (8192 rows) runs a segment at a time, so the rows' streams on the device stay bounded. The chain's VRAM reserve (q38cc_reserve_bytes) counts a segment's streams, a verify's copies of the DeltaNet state and the routed experts' scratch of a prompt chunk: with 2048-row chunks it kept 3.21 GiB where the chain used 3.34-3.51, a verify's copies were refused and every verify ran on the CPU (decode 43.6 -> 20.4 tok/s in such a run).

tests/test_qwen38_cuda_chain.c (make qwen38-cuda-chain-check, in CI beside qwen38-dn-gpu-check) runs the tiny FP8 fixture's oracle on the fake tier with the chain's ops from the fake's host-side table: the chain on (8/8 tokens, cosine 0.99998), a frame that fails mid-run (the state rebuilt on the CPU, the run finishing there within the limits), and the per-matrix path. tests/test_cuda_chain.cu checks the QSA and PLE kernels on a real card against their references.

The trunk on the CPU: int8 rows

Without a GPU the same trunk is the decode's floor: 8 GiB of BF16 read on every token, multiplied by a scalar loop. Since 1.12.1 the engine keeps the trunk on the CPU as int8 rows with one scale per row (the format the tier uploads, so GPU or not the rows are the same bytes), releases the BF16 copy once the rows exist, and multiplies with the integer kernels of idot.h: the activation is quantized to int8 once per row and meets the weights with maddubs (SSSE3 / AVX2), vpdpbusd (AVX-512 VNNI) or NEON dot products. Decode and prefill take this path. Every matrix of at least Q38_TRUNK_MIN_KB (default 1 MiB) is in; the small ones stay BF16 because they cost nothing either way. Q38_TRUNK_CPU_INT8=0 keeps the BF16 rows and the f32 kernel, the numeric reference.

The routed experts stay e4m3 with their 128 x 128 block scales, but the kernel decodes eight bytes at a time in registers and multiplies with FMA (q38_matmul_fp8_vec); the block scale still applies once per block and the blocks still add in double, so it differs from the table kernel only by the float summation order inside a block. For a batch of prefill rows the block is decoded once and every row runs through it. Q38_FP8_KERNEL=scalar restores the table kernel.

Measured on the released Qwen3.8-Flash-Next-FP8, a 16-core server shared with a training job, OMP_NUM_THREADS=8, RAM LRU cap 96 per layer, 100 generated tokens, COLI_TIMERS=1 decode bank:

BF16 trunk, table FP8 (1.12.0) BF16 trunk, vector FP8 int8 trunk, vector FP8 (default)
decode, tok/s 0.61 0.77 1.42
resident-mm (dense trunk), ms/token 434 437 85
lm-head, ms/token 76 79 13
routed-expert GEMV, ms/token 388 154 140
shared-expert, ms/token 43 44 15
expert-read (page cache), ms/token 156 155 140
peak RSS, GB 32.2 32.2 28.5

The middle column reproduces the first's text byte for byte over the 100 tokens: the vector kernel is a faster way to compute the same thing. The int8 trunk is a quantization, and its cost is measured as perplexity, not assumed. Teacher-forced NLL on four 512-token chunks of the repository docs (8 prompt tokens, 504 scored), same machine:

chunk BF16 trunk int8 trunk BF16 trunk, vector FP8
0 1.4527 1.4416 1.4527
1 2.4594 2.4544
2 2.3845 2.3929
3 2.7463 2.7753
mean nats/token 2.2607 2.2661

+0.24% nats, about +0.5% perplexity, two chunks lower and two higher: the per-row int8 error is noise at this size. The vector FP8 kernel alone reproduces the BF16 run to four decimals.

Prefill takes the same paths, and it is where the scalar BF16 loop hurt most. The 512-token chunk 0 as a prompt, one generated token:

BF16 trunk, table FP8 int8 trunk, vector FP8
wall, 512 prompt tokens 495 s 149 s
resident-mm 205 s 12 s
routed-expert GEMV 172 s 34 s
shared-expert 15.5 s 1.8 s
expert-read 28 s 26 s

The trunk quantization takes 3.3 s at load for the 553 matrices and the RSS after load grows by 0.6 GB while the int8 rows and the BF16 copy coexist; once the BF16 is released the peak RSS of a run is 3.7 GB lower.

Performance telemetry

Every served request reports its own routed-expert cache hit rate; persistent engine counters are differenced at request boundaries rather than exposed as a cumulative percentage. DONE decode throughput is based on completed inter-token intervals: the first token comes from prefill, so a one-token reply correctly reports zero decode tok/s instead of an artificial near-infinite rate. The request wall time, prompt/completion counts, expert read/wait/matmul time, sequence-mixer time, LM-head time and actual forward count are emitted through the shared PROF frame consumed by tools/datapoint.py.

Set COLI_TIMERS=1 for the finer Qwen-specific breakdown on stderr: expert reads, FP8 expansion, routed and shared experts, resident matmuls, DeltaNet, QSA indexing, QSA attention, PLE and the LM head. Architecture-phase times overlap the resident-matmul counter by design; expert disk service also overlaps the synchronous miss-wait value in PROF.

Correctness gate

The tiny oracle is generated from the Qwen4ExpForCausalLM class in transformers==5.16.1, the first release that exports the Qwen4-Exp text class:

python -m pip install -r c/tools/requirements-qwen38-tiny.txt
make -C c qwen38-tiny-check

It exercises Gated DeltaNet, sparse QSA above its token budget, PLE, four gated residual streams, routed and shared experts, prefill, cached decode, and LRU eviction. The gate checks both greedy token IDs and the final upstream logit vector, and runs both native-BF16 and expanded-FP32 resident modes at cache capacities one and four. CI repeats the capacity-one path under ASan and UBSan and verifies that a config/tensor shape disagreement is refused. make -C c qwen38-tiny-int4-check gates the optional int4-g64 experts (see Routed experts as int4-g64), and make -C c qwen38-tiny-mtp-check the MTP head (see Speculative decoding with the MTP head).

Supported checkpoint layouts

The released multimodal checkpoint stores text tensors below model.language_model; the standalone upstream text class stores them below model. Both prefixes are accepted. Routed experts may be the official per-expert block-FP8 matrices or the fused BF16 tensors emitted by the upstream text class. Q38_NATIVE_FP8=0 restores the former expanded-FP32 expert cache for numerical/performance A/Bs; Q38_NATIVE_BF16=0 does the same for resident and routed BF16 matrices. Other model types, unsupported scale encodings, and incompatible tensor shapes fail during load.

Q38_PREFILL_BATCH=0 restores row-at-a-time prompt execution for controlled A/B diagnosis. It does not change the single-token decode path.

The weights remain covered by the Qwen Community License 1.0 in the downloaded checkpoint. They are not redistributed by Colibri.

Vision

The released checkpoint is multimodal and the engine already reads its model.language_model prefix, so the vision tensors are present and reachable. The tower and gateway image path are implemented; their checks are described below. Image preprocessing is checked with tools/qwen38_image.py, pinned against the official Qwen2VLImageProcessor in tests/test_qwen38_image.py.

python3 tests/test_qwen38_image.py --config <model>/preprocessor_config.json

Neither needs the weights. The reference processor is built from preprocessor_config.json alone, which is 390 bytes, so the preprocessing can be developed and verified without the 185 GB.

Measured against it on eight shapes: geometry and patch order identical, and pixels bit-identical wherever no resampling happens (0.0000 on 256x256 and 640x480), 0.0157 worst case where it does, which is Pillow's bicubic against torchvision's. A wrong patch order would show as a discrepancy near 1, not 0.01.

Two things differ from GLM-5.3's tower and are the reason this is its own file:

The resolution is dynamic. GLM-5.3 fits everything onto a 448 canvas and pads. Qwen keeps the aspect ratio and picks a canvas whose area falls inside [shortest_edge, longest_edge], so there is no padding but the token count depends on the image. A 1080p photo becomes 2040 tokens, which on a disk-streaming engine is a prefill nobody will sit through -- the same reason GLM53_MAX_IMAGE_TOKENS exists, and preprocess(max_tokens=...) is the same lever here. It shrinks rather than crops: what is lost is detail, not pieces.

Normalisation is 0.5/0.5, not the CLIP constants.

The tower

qwen38_vision.h implements it: 27 blocks, hidden 1152, 16 heads, patch 16, spatial merge 2, projecting to 2560. Verified against the upstream Qwen4ExpVisionModel, again without the checkpoint -- the fixture is a 240 kB tower with random weights:

make -C c qwen38-vision-check

Two findings from building it are worth carrying, because both were invisible until an oracle was there to see them.

The merger uses a different GELU from the blocks. The blocks take ACT2FN[hidden_act], which is gelu_pytorch_tanh here; the merger instantiates nn.GELU(), the exact erf one. Using one for both matches to 4.6e-3 -- close enough to look right and far enough to move the image tokens.

A fixture can be too weak to test what it claims to. The first version scaled the random weights to 0.05, which makes the q.k products so small that softmax comes out essentially uniform: attention degenerates into the mean of the values and stops depending on the scores. A tower with no RoPE at all passed that fixture. At 0.6 the scores have a real range, and the four negative controls (rope off, wrong GELU, raster patch order, flat position interpolation) all fail as they should.

The tolerance is measured, not chosen: the reference in float32 differs from itself in float64 by 1.83e-4 on these activations, so a gap of that order is the arithmetic rather than a defect, and the threshold sits between it and the 4.6e-3 the real bug produced.

Wired up

Images work end to end. Send an OpenAI image_url part with a base64 data URI (a local path is accepted only under COLI_IMAGE_ROOT, see docs/ENVIRONMENT.md; coli chat reads a pasted path itself and sends the data URI); the gateway preprocesses it, replaces the part with <|vision_start|> + N x <|image_pad|> + <|vision_end|>, and hands the patches to the engine in an IMAGE frame ahead of the SUBMIT they belong to. The engine runs the tower once and substitutes its output for the embedding of each placeholder.

N is not a constant. The resolution is dynamic, so the placeholder count comes from the grid the preprocessor chose, and the engine refuses a request where the prompt and the grid disagree rather than guessing which vectors go where. Q38_MAX_IMAGE_TOKENS caps it; a 1080p photo is 2040 tokens without one.

make -C c qwen38-vision-serve-check

That gate does not check the model answers -- with random weights it would answer regardless, and would answer identically while ignoring the picture entirely. It checks that two different images produce two different answers, which is the only question a random fixture can answer honestly, and the one that catches the likeliest defect: patches loaded, tower run, result dropped somewhere between the merger and the embeddings. It also checks the refusals, since accepting a wrong image is worse than refusing it.

Remote URLs are refused rather than fetched, as elsewhere: a request should not make the server open a connection of the sender's choosing.

One image per request for now -- the engine holds a single pending image, and a second arriving before its SUBMIT drops the first and says so.