Skip to content

Batch 2026-11-15-e: qwen36 on Metal, the graph series on current dev - #2854

Merged
JustVugg merged 29 commits into
devfrom
batch/2026-11-15-e
Nov 15, 2026
Merged

JustVugg merged 29 commits into
devfrom
batch/2026-11-15-e

Conversation

@JustVugg

Copy link
Copy Markdown
Owner

qwen36 on Metal: @CoTi1938's series (#2015, #2016, #2298 to #2308) merged onto current dev, as rebase/qwen36-metal-graph. Their 24 commits are unchanged; on top are the merge with dev (both sides kept in qwen36.c and the docs) and three commits of ours, described in #2308 (comment):

  • the state copies dev's CPU engine gained after the series was cut (the prefix store, a parked conversation, the prompt-end snapshot, the rewind to the shared attention rows) stay off under the graph, like the pins the series already skipped; the graph steps aside, with its line, for LFM3 and for a GGUF that carries its MTP head or vision tower in the model's file;
  • coli_env.h lists the graph's variables;
  • braces around the pin and snapshot save under if (!m->graph).

What it does

The Metal graph runs qwen36's forward pass on the GPU (make qwen36 METAL=1): the expert cache and loaders, experts.qc slots on disk, read-ahead of the likeliest guessed experts, fast experts, speculative decoding with the MTP head, Metal kernels for ggml blocks, a GGUF read by the graph and its dense matrices kept in memory through a prompt. On Linux and Windows nothing changes.

Tests

  • CoTi on an Apple M1 (7 GPU cores, 8 GiB, macOS 26.6.2), this branch at 1b17e9c1 against the series' head 9e7bd192 (qwen36: keep a GGUF's dense matrices in memory through a prompt #2308 (comment)): make test-qwen36-kernels passed, make test-qwen36-graph METAL=1 32 passed with no skip, make check-env matched; on the checkpoint and on a GGUF the full logits of both are bitwise equal in exact and fast modes, and prefill and steady decode are at the series' speed. Two open observations from that report: on the checkpoint's code sequence the first decode after 512 tokens is 8 to 9 % slower than the series' (16 and 8 % after the last 64), with the same outputs and miss counts; and a GGUF's cold start is 20 to 50 ms later (+13 to 34 %). The cause of either is not assigned.

  • On ds (Linux x86-64), where the graph is not built: make check Ran 3613 tests, OK (skipped=499), so the Linux build and its tests are unchanged. The tree with this batch, batches C and D and windows: qwen38.exe in the CUDA package; setup says why an NVIDIA card stays unused (#2846) #2847 merged together: Ran 3629 tests, OK (skipped=502).

  • We have no Mac: the Metal results are CoTi's above, and CI's macOS jobs.

CoTi1938 and others added 29 commits October 9, 2026 20:31
The first piece of a Metal engine for qwen36 on Apple Silicon Macs: the
kernels (metal/qwen36.metal), the Objective-C layer that compiles them at
start and dispatches them (qwen36_metal.m, with qwen36_gpu.h for the C
side), and the part of a fast prompt batch that the CPU computes beside
the GPU through Accelerate (qwen36_cpu_mm.h). Nothing in the engine uses
them yet, so nothing a user runs changes.

Each operation has an exact kernel whose sums run in one fixed order, so
a row gets the same bits in a batch of any size. They are not the CPU
engine's bits: that engine quantizes the activations to int8 for its
products and sums some norms in double. The heavy operations (the
projections, attention, the delta rule of a prompt, the routed experts)
also have a fast kernel on the matrix units, and the norms, the router's
softmax and the delta rule of generation a fast SIMD one, with other
orders of the sums; QWEN36_METAL_EXACT=1 keeps the exact ones.

The layer can also keep a small kernel going on a second queue, from a
thread of its own, so that the GPU's clocks do not drop in the gaps of
decode, as ds4 does (THIRD_PARTY_NOTICES.md names the parts adapted from
ds4). Nothing starts it yet; QWEN36_METAL_KEEPALIVE=0 turns it off.

tests/test_qwen36_kernels.c runs every one of the 64 kernels, at the 35B
model's shapes and at small ones, against a double-precision reference
on random data: make test-qwen36-kernels. It needs a Metal device, so it
is not part of make check; the macOS CI job runs it in a step of its
own. GitHub's runners expose the Apple Paravirtual device, which never
completes a command buffer: there the test only compiles the kernels and
says so, as test_backend_metal.mm does. make clean removes
qwen36_metal.o.
The second piece of the qwen36 Metal engine, plain C without a model:

- qwen36_global_cache.h: one cache of expert payloads for the whole
  model. A FIFO lease holds a complete group of experts through
  compute; cancellation is cooperative. The victim is chosen by demand
  frequency (qwen36_cache_policy.h) or, in decode, by the expected time
  to an expert's next use (qwen36_expected_policy.h). A best-effort
  prefetch, which never goes ahead of queued demand, has no caller in
  the engine yet.
- qwen36_selected_loader.h: a thread that loads one group of experts
  (in decode, one layer's top-k) and hands the hits over before the
  misses are read; if the group then fails, the hits stay valid until
  the caller has finished with them.
- qwen36_layer_loader.h: four reader threads that load a prompt layer's
  experts while the GPU runs the layer's trunk.
- qwen36_advice_schedule.h: the read-ahead advice goes out only after
  the GPU commands it must not delay are submitted.

Nothing in the engine uses them yet: the graph that does comes next.

Five tests cover the six headers (the expected-reuse policy inside the
cache's test), part of make check on every platform. A hang ends in a
watchdog thread instead of alarm(), which MinGW lacks, and the loaders
set a stack guard page only where POSIX can: Windows guards thread
stacks itself, MinGW has no sysconf, and its pthread_attr_setguardsize
is a stub that returns ENOTSUP.
With make qwen36 METAL=1, an Apple Silicon Mac runs the forward pass of
Qwen3.6 on the GPU (qwen36_graph.h), on the Metal kernels and the expert
cache of the two previous changes, while the routed experts are still
read from the disk, so they do not have to fit in memory. In a fast
prompt batch the CPU's matrix unit takes part of the products.

Per layer the trunk and the router run as one GPU batch; the CPU reads
the router's choices, a worker thread loads the missing experts into GPU
slots of one cache for the whole model (a sixteenth of the RAM by
default, QWEN_EXPERT_CACHE_BYTES), and each expert computes every token
that chose it. In generation the GPU also runs the next layer's router
on this layer's residual, and the experts it guesses are announced to
the disk early. A long prompt runs layer by layer
(qwen36_prefill_stream.h), so each layer's experts are loaded once.

canonical.qc is read in place by the GPU (qwen_dense_mapped, QW.mapped:
never freed); the CPU engine keeps its copy. Without the file the graph
copies the matrices and frees the CPU's copies. In serve mode a copy of
the DeltaNet state where the prompt's last turn opens lets the next chat
turn read only what it adds.

The graph runs only where it can. COLI_METAL=0 keeps the CPU engine; a
qpack container, a GGUF model, a dense format other than INT8,
COLI_KEEP_F32, COLI_VULKAN=1 in a VK=1 build, CACHE_ROUTE=1, a serve of
several conversations at once (KV_SLOTS above 1), routed experts that
are not planar INT4 with groups of 64 (or QWEN_EXPERT_KERNEL=0), a model
without routed or shared experts, a model with sigmoid routing,
attention sinks, a sliding window or dense layers (Flint, Muse Spark), a
geometry outside the kernels, a Clef decision head (DECIDE reads the
hidden state of every position), no Metal device or one that cannot
build the kernels, and the Apple Paravirtual device keep it with one
line that says why. The graph takes no images (a line at start says the
vision tower is not loaded), no prompt-lookup or MTP drafts (with
Q36_MTP=1 the head stays off) and no read-out of the prompt's logits (a
logprobs request is refused). A STOP or CANCEL during a prompt takes
effect once the prompt is read. A router row with no valid choice
degrades as on the CPU engine. In a serve the dashboard's expert map
shows the graph's cache.

Where the graph runs, the CPU's rows interleaved by 16 (Q36_ILV16, AVX2
builds) stay off: the GPU reads the trunk and the experts as stored. The
graph's expert readers are threads of the system's pool, which the
system ends while idle, or a long prompt's four readers, which end with
it, so after each read they give back the block the expert load keeps
per thread.

docs/qwen36-metal.md describes it. make test-qwen36-graph METAL=1 checks
the graph against the CPU engine on a model the test writes; the macOS
CI job runs it, and on the runner's Paravirtual device it checks that
the CPU engine takes over. On the tiny fixture the CPU engine gives the
same bytes as before, with and without METAL=1.
The measurement of the previous commit's graph on Qwen3.6-35B-A3B: the
same harness on the CPU engine (COLI_METAL=0) and on the Metal graph,
prompts of 64 and 512 tokens, each followed by 32 generated tokens. Two
manifests (the generation after the 512-token prompt, and that prompt),
the summary, the raw logs of every run with the start time, the peak
memory and the swap counters, the harness, and quality.txt: the
teacher-forced likelihood and speed of 1024 tokens of the text on the
CPU engine and on the graph in both modes. The logits are the same in
every run of each engine.
The Metal graph reads each routed expert from the checkpoint as two
tensors (the int4 weights and their scales) and repacks the weights into
the planar layout of its GPU slots. qwen36-canonical --experts writes
experts.qc beside the model: every expert already in that layout,
weights and scales together, so a slot is filled by one pread.

The int4 values are repacked, not requantized. When every scale is a
bfloat16 max|w| over 7, as Colibri's converter writes them from a BF16
checkpoint, the file stores that bfloat16 in 2 bytes and the reader
divides it by 7 again into the same f32 bits (checked over every scale
before writing); otherwise it keeps the f32 scales. The file is written
as a temporary, synced (F_FULLFSYNC on macOS) and renamed; a signal
that stops the conversion removes the temporary, and the file gets the
mode of a plain file under the umask. At start its header and size are
checked against the model and against a stamp of the checkpoint (tensor
descriptors, shard inodes, sizes, timestamps and samples, but not the
volume's device number, which macOS can change at a restart), and once
against the layout of the cache slots: a file beside the model that
does not match is reported and ignored, and the experts come from the
checkpoint as without it. QWEN36_EXPERTS_QC=0 reads the checkpoint,
QWEN36_EXPERTS_QC=<path> names a file kept elsewhere, which then has to
match. The records carry no checksum. The CPU engine and the adapters
keep reading the checkpoint.

expert_ffn.h's int4 repack can now run in place, which the writer uses:
the scalar loop, now a function of its own, keeps a copy of the one
block it rewrites; the SIMD paths load each block before storing it.
qwen36-canonical takes --experts in any position and refuses any other
argument that starts with '-'.

tests/test_qwen36_expert_file.c and tests/test_qwen36_expert_convert.c
cover the format, the checked reads (both layouts, also at the 35B
model's expert shape, where their sizes differ), the writer, its
cleanup after a signal and the stamp, through qes_fingerprint with the
volume renumbered; a check pins the scale convention to known values,
which the writer and the reader could otherwise both get wrong
together. test_expert_ffn.c checks the in-place repack, and the scalar
loop on every host, against an independent decoder.
The measurement of the previous commit's code on Qwen3.6-35B-A3B: the
same Metal binary with experts.qc beside the model and with
QWEN36_EXPERTS_QC=0, which reads the experts from the checkpoint,
prompts of 64 and 512 tokens, each followed by 32 decode steps. Two
manifests (the 512-token prompt, and the decode steps after it), the
summary, the raw logs of every run with the start time, the peak memory
and the swap counters, the harness, and the converter's output when it
wrote the file's records on 3 October (the file was rewritten in place
with the same bytes on 5 October, and its 8-byte stamp on 7 October).
The logits are the same in every run of both versions.
In decode the Metal graph runs the next layer's router on the current
residual and announces the guesses that are not in the cache, so the SSD
reads them while the GPU computes the next layer's trunk. It announced all
eight of the top-k. On Qwen3.6-35B-A3B the seventh and eighth guesses were
among the experts the next layer chose only 56% and 41% of the time on
code (65% and 49% on prose), and each wrong one not already in memory is a
1.7 MB read the step does not use. Only the six likeliest are announced
now. The choice moves into qwen36_advice_schedule.h, where its test checks
the count and the order. Announcements are advice: the results do not
change.
The measurement of the previous commit against its parent on
Qwen3.6-35B-A3B with the Metal graph: the first 64 and 512 tokens of
prose, code and chat, each followed by 32 decode steps, 4 runs per
version in ABBA order. One manifest per text (decode steps 2 to 32 after
the 512-token prompt), the summaries, the raw logs of every run with the
start time, the peak memory and the swap counters, the harness, code.txt
and chat.txt (the prose text is cited by its source and its SHA-256);
guesses.txt holds the exploratory runs behind the count of six, with
their logs. The logits and the generated tokens are the same in every
run of both versions.
Much of each decode step of the Metal graph goes to reading routed
experts from the SSD: Qwen3.6-35B-A3B chooses 8 experts of 1.7 MB per
layer and token, and on an 8 GB Mac the expert cache holds about one
token's worth. QWEN36_FAST_EXPERTS=J, in fast mode only, uses a token's
J likeliest routed experts and the rest of its top-8 only when the
expert cache already holds them, so a step reads fewer experts, at a
cost in quality. A left-out choice points at the heaviest expert with
weight 0: no read and no cache slot, and that expert's FFN runs once
more for it. The kept weights stay the router's, divided by the mass of
the whole top-8: renormalizing over the kept experts would raise their
gain above what the model was trained with (arXiv 2609.04575 measures on
this model that renormalizing over fewer experts costs more than
dividing by the top-8 mass). Unset or 0, the default, and in exact mode
every chosen expert is used; other values are refused with a line, and a
line says when the option is on.

The decode batch of the cached experts, computed while the misses read,
now takes all the pairs of an expert, so it still runs when a choice is
left out (the same bits as before). The expected-reuse cache policy also
learns from a decode group with fewer than top-k experts (such a group
fell back to the frequency policy), qgc_present answers a layer's
presence under one lock, and the read-ahead announces at most the J
likeliest guesses. The graph test covers the option; the next commit
measures it.
The measurement of the previous commit on Qwen3.6-35B-A3B, the same
Metal binary with QWEN36_FAST_EXPERTS=5 and unset: the first 64 and 512
tokens of prose, code and chat, each followed by 32 teacher-forced
decode steps, 4 runs per version in ABBA order; the quality over 1024
teacher-forced tokens of each text, token by token, with the script that
compares them; and, without the variable, the exact-mode fingerprints of
that commit and of its parent. One manifest per text (decode steps 2 to
32 after the 512-token prompt), the summaries, the raw logs of every run
with the start time, the peak memory and the swap counters, the
per-token files, the harness, code.txt and chat.txt (the prose text is
cited by its source and its SHA-256).
Qwen3.6's checkpoint carries a multi-token-prediction head, which
tools/convert_qwen36.py carries into the container as
model-mtp.safetensors (f16 dense weights, the routed experts in the
model's own int4 layout). The Metal graph reads those tensors and runs
its own head on the GPU; the other engines' head (qwen36_mtp.h,
Q36_MTP=1) is not loaded beside it, and Q36_MTP=1 says so in one line.

With the head in the container, the Metal graph drafts the token after
each one and, when the head's confidence is at least 0.6
(QWEN36_MTP_MIN_P, from 0 to 1), one forward verifies the picked token
and the draft together; QWEN36_MTP=0 turns it off. In exact mode the
tokens and logits are plain decoding's: a verify computes each row with
a decode step's kernels, and a rejected draft swaps back the DeltaNet
state copied after the first row. The drafts go through the verify and
the rollback of q36_spec_step and are counted as the head's, with the
gate off for them (their line says "gate off:"); with the head the
report gives the head's lines only, with the drafts below the confidence
and the steps without the head.

The head runs on the GPU as one more layer, with its own KV cache and
its routed experts in the expert cache as the layer after the model's
last (its two heaviest only). It drafts among the first 32768 token ids,
which hold most tokens of English, other Latin-script text and code but
no Chinese character: where half or more of the last 16 tokens lie past
them, it does not run (it could draft only the other tokens there, and
it costs about 5% of a step), and its attention starts again when it
runs again; a prompt with 8 or more of its last 15 tokens past them
skips the head, which will not run at the next step. It reads only a
prompt's last 32 positions, which run the prompt's last layer whole
(every row does when the prompt's last batch has fewer than 33 rows),
and only their final residuals are kept. Its attention starts again
where a position does not follow its last one, as in a serve turn
resumed from the copy of an earlier position. Its dense weights are
quantized 256 rows at a time, so no float copy of a whole matrix stays
in the process's footprint. A verify leases both rows' experts at once
and the expected-reuse policy counts them as one visit of the layer; so
the cache keeps top-k fewer experts between steps, and a budget under
two top-k groups stops the start with a line. Without memory for a
verify's copies it drafts nothing, with one line. With
QWEN36_FAST_EXPERTS above 3 a draft row keeps two sure experts fewer,
except in the last quarter of the layers. Q36_MTP_DUMP writes the head's
logits as it does for the other engines' head.

tests/test_qwen36_graph.py writes a head too, as model-mtp.safetensors,
and checks the drafts against plain decoding, in a serve turn resumed
from the copy and in the turn after one that ended inside a verify,
which drafts again; the head's logits with its window in one batch or in
two; the steps it does not run where most tokens lie past the ids it
drafts from (Q36_MTP_VOCAB, for tests, narrows them) and the prompts
that skip it; the values of the variables, a cache too small for a
verify, and Q36_MTP=1 under the graph. Without the head, prompt lookup's
forced drafts check verifies of up to six rows on the graph.
The checkpoint's MTP head against QWEN36_MTP=0 with the same binary:
exact mode (greedy, the same tokens and fingerprints in every run of
both), fast mode and fast mode with QWEN36_FAST_EXPERTS=5 (each step fed
the text's own next token), on prose, code, chat and a Chinese text,
where the head stays idle: 64- and 512-token prompts, 32 decode steps, a
warm-up and 4 runs per version in ABBA order, each after the nominal
thermal state and a 60 s rest. Decode speed counts all 32 steps, the
first included: a verify in the first step can make the second free. One
manifest per mode and text, every run's raw log and the summaries; the
three cases more than 5% slower re-run from clean, with their own
manifests; the head's own cost in exact mode, with a draft verified only
at confidence 1 (QWEN36_MTP_MIN_P=1), on prose and the Chinese text.

Also the quality over 1024 teacher-forced tokens of each text, token by
token (plain decoding, the drafts, fast experts without and with the
drafts); 256 tokens of free generation per variant, with the drafts
counted and the generated texts; the exact-mode fingerprints against the
parent; the head's checksum, how it was made and how it compares with
the file tools/convert_qwen36.py writes; the code, chat and Chinese
texts, the harness and the comparison script.
QWEN36_FAST_GUESSED=R narrows fast experts (QWEN36_FAST_EXPERTS=J) in
decode: a token uses its R - 1 likeliest routed experts, its next ones
up to the J likeliest only when the expert cache holds them or the layer
before guessed them for its layer, and the rest of its top-k as fast
experts do. A guessed expert's read was announced while the layer before
computed, so keeping it costs little; a choice neither guessed nor in
the cache would wait for a whole SSD read. The first layer of a step,
which nothing guessed, and a verify's draft rows keep the rule of fast
experts (qfe_keep_guessed in qwen36_fast_experts.h; the guesses are the
ones qg_advise_ahead announces, every row's, once each, and each forward
keeps its own).

Unset or 0, the default, changes nothing. Exact mode, the variable
without QWEN36_FAST_EXPERTS of 2 or more and a value that is not a whole
number from 2 to J are refused with a line; a line says when it is on,
and one at exit counts the rows the rule ran for and the choices that
stayed because they were guessed. As with fast experts, the output
depends on what the cache holds.

docs/qwen36-metal.md and docs/ENVIRONMENT.md describe it.
test_qwen36_fast_experts checks the rule and its count without a model;
make test-qwen36-graph METAL=1 checks the values and their lines, the
same output in every run, the prompt and exact mode unchanged, from the
line at exit that the rule ran at every layer of a decode step but the
first and kept guessed choices, and with the MTP head's drafts that it
ran once per forward, with the same output in every run.
The measurement of the previous commit on Qwen3.6-35B-A3B, the same
Metal binary with QWEN36_FAST_EXPERTS=5 and the MTP head's drafts,
without and with QWEN36_FAST_GUESSED=4: the first 64 and 512 tokens of
prose, code and chat, each followed by 32 teacher-forced decode steps
(all counted, the first included), 4 runs per version in ABBA order,
each after the nominal thermal state and a 60 s rest; the quality over
1024 teacher-forced tokens of each text and of a Chinese text, token by
token, against fast experts alone and against plain fast decoding; 256
tokens of free generation, two runs per version, with the generated
texts; and, without the variable, the exact-mode fingerprints of that
commit and of the speculative-decoding commit below it. One manifest per
text (decode after the 512-token prompt), the summaries, the raw logs of
every run with the peak memory and the swap counters, the per-token
files, the harness, the comparison script, code.txt, chat.txt and
guxiang.txt (the prose text is cited by its source and its SHA-256).
In decode each layer guesses the next layer's experts and announces the
reads of those not in the expert cache (qg_advise_ahead); the next layer
then reads its misses into cache slots while the step waits.
QWEN36_PREFETCH=1, with experts.qc, copies the guesses while this layer
computes, on two serial queues of their own, into eight slots beside the
expert cache (qwen36_prefetch_ring.h). When the next layer misses one of
them, its load takes the copy instead of reading the record, after
waiting for a copy still reading: the cache's slot and the copy's trade
their storage (qwen_slot_swap). A copy done when the layer starts is
also lent to it (qwen_prefetch_lend): the GPU computes it early, with
the experts the cache holds, from the copy's own tensor, while the
layer's other misses read, and no other copy reuses its slot until the
load takes it or the next sync ends the loan. The cache makes the same
choices as without the copies and its slots end up with the same bytes,
and a lent copy is computed with the same pairs in another batch, so the
output does not change, in any mode.

A guess is not copied when a later layer's guesses were issued (its
layer has read its experts), when the cache holds it, or when the
system's file cache does not hold every page of its record (mincore, on
the copy's queue): a cold one would hold a queue for a disk read, and is
only announced, as before. Only decode steps of one token copy; a verify
of the MTP head's drafts copies nothing. A copy that fails is dropped,
and its layer reads the expert itself.

The eight slots hold one expert each, 14.2 MB on the 35B model, on top
of the expert cache's budget (QWEN_EXPERT_CACHE_BYTES); the line that
says it is on gives their bytes. Unset or 0, the default, changes
nothing; another value and the variable without experts.qc are refused
with a line, and a line at exit counts the copies, those their layer
took and those it computed early, and why the others were not made.

docs/qwen36-metal.md and docs/ENVIRONMENT.md describe it.
test_qwen36_prefetch_ring checks the slots without a model: which entry
a copy takes, a load that takes a copy and the storage it gets, one that
waits for a copy still reading, a copy that fails, and a copy lent
before its load, which no other copy reuses until the load takes it or
the loan ends; make test-qwen36-graph METAL=1 checks the values and
their lines, a prompt alone copying nothing, and the output without the
copies, byte for byte, with copies made and taken and none failing: in
exact mode, in fast mode, with fast experts and QWEN36_FAST_GUESSED, and
with the cache's frequency policy (over these five some copies are
computed early), with the MTP head's drafts, and in serve mode (its
layers and text), where a turn's prompt comes after the decode steps of
the one before, also with a cache large enough for the prompt's readers,
which read a longer turn layer by layer.
The measurement of the previous commit on Qwen3.6-35B-A3B, the same
Metal binary with QWEN36_FAST_EXPERTS=5, QWEN36_FAST_GUESSED=4 and the
MTP head's drafts, without and with QWEN36_PREFETCH=1: the first 64 and
512 tokens of prose, code and chat, each followed by 32 teacher-forced
decode steps (all counted, the first included), 4 runs per version in
ABBA order, each after the nominal thermal state and a 60 s rest; a
clean re-run of code, whose first step after the 512-token prompt had
been 10.1% slower with the copies; the copies, from the line at exit of
every run with them; and the exact-mode fingerprints with and without
the copies, with the drafts and without them, and of the guessed-experts
commit below. One manifest per text (decode after the 512-token prompt)
and one for the re-run, the summaries, the raw logs of every run with
the peak memory and the swap counters, the copies of each run, the
harness, code.txt and chat.txt (the prose text is cited by its source
and its SHA-256).
The Metal layer gains kernels that compute GGUF matrices as the file
holds them: rows of Q8_0, Q4_K, Q5_K, Q6_K, IQ4_NL or IQ4_XS blocks,
decoded where they are read, with no second quantization. Nothing in
the engine calls them yet: the graph takes a GGUF in a later change.

- q36_gq_step decodes 32 weights of a row with the operations of
  ggml_quants.h's float decoders, uncontracted, so the weights are
  gq_dequant_row's bits; q36_gq_dequant writes them out for the test.
  kvalues_iq4nl comes from a table in shaders/blockdec_table.h's
  layout, which q36_gpu_init fills from ggml_grids.h as the Vulkan
  backend fills its own.
- q36_dot_gq: in exact mode a thread per output, with the sums of the
  CPU's exact mode (gq_matmul_f32: each product rounded, added in
  input order) and the ".scale" factor as matmul_d applies it, so the
  CPU's bits; in fast mode a SIMD group per output for up to 8 rows
  (each row keeps the bits it has alone) and the matrix units for
  prompt batches, a step's weights decoded once into threadgroup
  memory for 32 rows.
- q36_dot_f32: a float matrix (a GGUF keeps the router in F32) with
  matmul_f32.h's dot_f32_lanes, the CPU's arithmetic and bits.
- q36_expert_gate_up_gq and q36_expert_down_gq: a GGUF layer's routed
  experts, from slots holding the file's rows of gate, up and down,
  each of its tensor's type, with the slots' ".scale" factors: the
  sums of moe_gq_run and the SwiGLU of q36_expert_gate_up.
- The DeltaNet kernels take llama.cpp's layout as the CPU engine reads
  it (g_dn_tiled, g_ssm_a_direct): with q36_shape.tiled value head h
  reads key head h % k_heads, and with a_direct the A_log weights hold
  A = -exp(A_log). The graph passes 0 for both, which gives the bits
  of before.

Any other type, a partial block, a tensor too small or weights at an
offset their loads cannot take is refused with nothing encoded.
test_qwen36_kernels checks the decoders, the exact products and the
float product against the CPU's bits, the fast products against a double
reference, a row in a batch against the row alone, the experts (down
against the CPU's bits, gate and up against a double reference, since
the SwiGLU's exp is the GPU's), the refusals, and the DeltaNet kernels
with tiled heads in both modes.
A METAL=1 build runs a qwen35moe GGUF on the Metal graph, where it ran
the CPU engine: the dense matrices and the routed experts are computed
from the file's ggml blocks with the kernels of the change before, with
no second quantization. The model's own format (canonical.qc,
experts.qc) and its kernels do not change.

- qwen_graph_eligible takes a GGUF whose dense matrices and experts are
  Q8_0, Q4_K, Q5_K, Q6_K, IQ4_NL or IQ4_XS, each of its own type, and
  dense matrices of floats (the router); a file with another type runs
  on the CPU engine, with a line that names the matrix and its type
  (qwen_gguf_graph_off).
- The file is mapped (qwen_gguf_map). The graph sees each dense matrix
  in place, in a Metal buffer of its own pages (q36_map_view_own: the
  file keeps the experts between the dense tensors, and the GPU holds a
  buffer resident whole), and the embedding rows are read from the
  mapping: load_tq no longer copies the matrices. A matrix of floats is
  uploaded from the CPU engine's F32 copy, which is then freed.
- An expert cache slot holds an expert as the file does, gate | up |
  down in its layer's types (gqe_max bytes), filled by
  load_expert_merged; qg_experts runs the _gq kernels with the ".scale"
  factors of each slot's expert, in decode, in a prompt (layer by layer
  too) and in a verify.
- The DeltaNet kernels take llama.cpp's layout as the CPU engine reads
  it (g_dn_tiled, g_ssm_a_direct).
- With a GGUF the CPU's share of a fast prompt is off (its products
  take INT8 weights); the MTP head is off and experts.qc and
  QWEN36_PREFETCH are ignored, with a line when they are asked for. At
  exit a line counts the products with the file's matrices and experts
  that ran on the GPU.
- GGUF_CHECK=1 builds no graph: the CPU engine's loader checks the
  header as before, and a line says whether the graph would run it.

test_qwen36_graph writes a GGUF fixture in llama.cpp's layout (two key
heads for four value heads, the six types, ".scale" companions) and
checks: the exact graph against the CPU engine (GQ_EXACT=1), within
1e-4 of the largest value (about 7e-6 measured); the fast graph against
the exact one; fast experts; a serve turn resumed from the copy; verifies
of drafts forced in ref.json mode (nothing drafts on a GGUF), with plain
decoding's bits, on a copy whose router bias makes a verify read one
layer's experts in one lease (qg_moe_verify); the line and the CPU
engine for a Q4_0 matrix and IQ3_S experts; the options of the model's
own format; GGUF_CHECK=1, which builds no graph; and the layout of coli
import --mode copy, bit for bit as the file it was copied from.
test_qwen36_kernels checks q36_map_view_own.
In fast mode the GGUF kernels now follow the layouts of llama.cpp's
Metal matrix-vector kernels in decode and of the graph's own int8 and
int4 kernels in prompts. Before, fast mode took a whole 32-weight step
per lane and applied the block scales to each weight, and the routed
experts ran the exact-mode kernels (a thread per output and choice) in
both modes. Exact mode does not change: its kernels and their bits are
those of before.

- Decode and verifies (up to 8 rows): a row of weights takes up to 32
  lanes of a SIMD group, spread over its blocks (Q8_0: 4 lanes a block,
  8 weights each; Q4_K and Q5_K: 8 lanes, 4 runs of 8; Q6_K: 16 lanes,
  4 runs of 4), and the block scales multiply the blocks' sums of q x
  (d * sc * sum(q x) - dmin * m * sum(x)) instead of each weight
  (q36_gqf_part). IQ4_NL and IQ4_XS keep whole steps. Where a row takes
  32 lanes, or 16 (a 512-wide Q5_K down), the count is a compile-time
  constant: q36_dot_gq_row32 for one row of x,
  q36_expert_gate_up_gq_vec32 and q36_expert_down_gq_vec16 and _vec32.
  A row's bits still do not depend on the batch.
- Prompt batches: every lane decodes its own weights straight into the
  8x8 matrix operands, as q36_dot_i8_mm reads its int8 ones
  (q36_gq_four); only the x tile goes through threadgroup memory.
- The routed experts of a prompt run in the tiles of the int4 kernels:
  up to 64 choices of a slot a threadgroup, the row blocks past a
  slot's last choice skipped, and its last tile of 1 to 4 row blocks
  in q36_expert_*_gq_mm4 with the inputs read straight from memory.
  Q4_K and Q5_K read a block's scales and fifth bits once for its 8
  steps (q36_gq_rowdec).

test_qwen36_kernels gains a decode row under 32 lanes for every type,
the experts' 32-lane down of a decode, experts with tiles of 3, 5, 6
and 7 row blocks and with a full tile and a short one, and the bits of
the one-row 32-lane kernel against the rows' kernel for Q4_K and Q5_K.
The experts' outputs start as NaN in each mode, so an output a kernel
leaves unwritten fails.
The measurements of the previous commit on Qwen3.6-35B-A3B, on an 8 GB
M1 MacBook Air, with one harness built against the code as measured:
unsloth's UD-Q4_K_M GGUF with the fast-mode kernels of the previous
commit and of the change before it; the expert cache's default budget
against one slot more, which a GGUF's larger slots need for the
layer-by-layer prompt; the model's own format before and after the
three GGUF changes; and the GGUF against the model's own format in fast
mode, with fast and guessed experts and in exact mode. The first 64 and
512 tokens of prose, code and chat, each followed by 32 teacher-forced
decode steps; a warm-up (of each file in the comparisons of the two
files), then 4 runs per version in ABBA order, each after the nominal
thermal state and a 60 s rest. Two manifests per comparison and text
(decode after the 512-token prompt, and that prompt), the summaries and
every run's raw log with the peak memory and the swap counters; the
quality over 1024 teacher-forced tokens of each text, token by token;
the exact graph against the CPU engine on the GGUF; timed runs per file
and text, with 32 decode steps and with one (where the time goes); the
harness, the comparison script, code.txt and chat.txt.

docs/qwen36-metal.md gives the speed against the model's own format,
what one slot more in the expert cache does, where a GGUF's decode time
goes, and what the timers show during the wait for the disk.
A GGUF keeps its routed experts in the same file as its dense matrices,
and a prompt's expert reads pushed the dense pages out of memory, for
the prompt itself and the first decode step to read back from disk: on
an 8 GB M1, 0.3 to 1.2 GB of the 35B model's 1.9 GB of dense matrices
were out after a 512-token prompt (mincore), and the first decode step
took 0.7 to 1.5 s. With the dense tensors in a file of their own (a copy
at the same offsets) they stayed, as canonical.qc's did beside
experts.qc in the same runs.

The graph now locks a GGUF's dense matrices in memory (mlock) from the
start of a prompt to the end of the decode step after it, which then
took 0.3 to 0.5 s; during decode the GPU holds them in memory anyway,
and between requests the system may take them back (a request that ends
without a decode step unlocks them at its end). A line says how many
bytes are locked, or why they could not be (then nothing stays locked
and no later prompt tries again), and the report at exit how many times
they were locked and unlocked, and whether any stayed locked. What is
computed does not change, nor anything for the model's own format. The
decode after a locked prompt finds fewer of its experts in the file
cache and is slower by a few percent: the record of the next commit
gives the numbers.

The GGUF graph test checks, in its graph runs, that the lock's line says
the bytes of the file's dense matrices, each rounded out to whole pages,
and that they were unlocked after the first decode step, or at the end
of a request without one.
The measurements of the previous commit on unsloth's UD-Q4_K_M GGUF of
Qwen3.6-35B-A3B, on an 8 GB M1 MacBook Air, with one harness built
against the code as measured: the change before it against the previous
commit, in fast and in exact mode, on the first 64 and 512 tokens of
prose, code and chat, each followed by 32 teacher-forced decode steps,
and on a 32-token turn resumed from the state saved after 64 tokens
(prose and chat, fast mode); a warm-up, then 4 runs per version in ABBA
order, each after the nominal thermal state and a 60 s rest. Three
manifests per mode and text (the decode after the 512-token prompt or
the turn, the first decode step after it, and that prompt), the
summaries and every run's raw log with the peak memory and the swap
counters; the system's wired memory during runs of both
versions; and the diagnosis runs, with the harness that counts the dense
pages in memory, the page-ins and the experts read, and its two patches:
why a GGUF's first decode step was slow, and what the lock does to the
decode after it.

docs/qwen36-metal.md gives the first decode step, the second new word,
the decode and the resident size with the lock.
The qwen36 Metal graph (#2015, #2016, #2298 to #2308), merged on dev 1147 commits after it was cut. qwen36.c: QW keeps dev's wf16 and q4r16 with the graph's mapped flag; a DeltaNet layer's state is not allocated under the graph (dev's LFM3 check and q36_rec_cells kept); reset_recurrent resets dev's KV ring before the graph's early return; a verify's result keeps dev's per-step depth after the graph's qg_mtp_keep; the serve path keeps dev's park and rewind, then arms the graph's snapshot; the prefill control keeps dev's condition and the graph's note. docs: dev's rows with the graph's notes added (ENVIRONMENT.md, gguf.md, serve_protocol.md, speculative.md).
…ne's

The graph keeps a conversation's state on the GPU, so like the pins it already skipped, what dev's qwen36 gained after the stack was cut copies the CPU engine's state and stays off under the graph: the prefix store (q36_ps_on), the parked conversation (q36_park), the prompt-end snapshot (q36_pes_take, q36_pes_restore, q36_pes_after_abort) and the rewind to the shared attention rows. The graph's own snapshot (qg_snapshot_arm) still serves a chat's next turn.

The graph also steps aside, with its line, for LFM3 and for a GGUF that carries its MTP head or vision tower in the model's file (Ollama's): it was measured on unsloth's files, which do not. All of it is under QWEN36_METAL_GRAPH: other builds are unchanged.
The registry came to dev after the Metal stack was cut: QWEN36_CACHE_EXPECTED, QWEN36_CPU_MATMUL, QWEN36_EXPERTS_QC, QWEN36_FAST_EXPERTS, QWEN36_FAST_GUESSED, QWEN36_MTP, QWEN36_MTP_MIN_P, QWEN36_PREFETCH, QWEN36_PREFILL_STREAM, QWEN36_PROMPT_ROWS and QWEN_EXPERT_CACHE_BYTES, as qwen36 reads them (docs/qwen36-metal.md).
…e graph's check

The graph's check (if (!m->graph)) was written for the pin save alone; dev has since given that if an else (q36_pes_take). The block says what was already so: under the graph neither copy is made, without it both are as before.
qwen36 on Metal: CoTi1938's series (#2015, #2016, #2298 to #2308) on current dev
@JustVugg
JustVugg merged commit 78be384 into dev Nov 15, 2026
119 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants