Repository navigation
Batch 2026-11-15-e: qwen36 on Metal, the graph series on current dev - #2854
Merged
Merged
Conversation
The first piece of a Metal engine for qwen36 on Apple Silicon Macs: the kernels (metal/qwen36.metal), the Objective-C layer that compiles them at start and dispatches them (qwen36_metal.m, with qwen36_gpu.h for the C side), and the part of a fast prompt batch that the CPU computes beside the GPU through Accelerate (qwen36_cpu_mm.h). Nothing in the engine uses them yet, so nothing a user runs changes. Each operation has an exact kernel whose sums run in one fixed order, so a row gets the same bits in a batch of any size. They are not the CPU engine's bits: that engine quantizes the activations to int8 for its products and sums some norms in double. The heavy operations (the projections, attention, the delta rule of a prompt, the routed experts) also have a fast kernel on the matrix units, and the norms, the router's softmax and the delta rule of generation a fast SIMD one, with other orders of the sums; QWEN36_METAL_EXACT=1 keeps the exact ones. The layer can also keep a small kernel going on a second queue, from a thread of its own, so that the GPU's clocks do not drop in the gaps of decode, as ds4 does (THIRD_PARTY_NOTICES.md names the parts adapted from ds4). Nothing starts it yet; QWEN36_METAL_KEEPALIVE=0 turns it off. tests/test_qwen36_kernels.c runs every one of the 64 kernels, at the 35B model's shapes and at small ones, against a double-precision reference on random data: make test-qwen36-kernels. It needs a Metal device, so it is not part of make check; the macOS CI job runs it in a step of its own. GitHub's runners expose the Apple Paravirtual device, which never completes a command buffer: there the test only compiles the kernels and says so, as test_backend_metal.mm does. make clean removes qwen36_metal.o.
The second piece of the qwen36 Metal engine, plain C without a model: - qwen36_global_cache.h: one cache of expert payloads for the whole model. A FIFO lease holds a complete group of experts through compute; cancellation is cooperative. The victim is chosen by demand frequency (qwen36_cache_policy.h) or, in decode, by the expected time to an expert's next use (qwen36_expected_policy.h). A best-effort prefetch, which never goes ahead of queued demand, has no caller in the engine yet. - qwen36_selected_loader.h: a thread that loads one group of experts (in decode, one layer's top-k) and hands the hits over before the misses are read; if the group then fails, the hits stay valid until the caller has finished with them. - qwen36_layer_loader.h: four reader threads that load a prompt layer's experts while the GPU runs the layer's trunk. - qwen36_advice_schedule.h: the read-ahead advice goes out only after the GPU commands it must not delay are submitted. Nothing in the engine uses them yet: the graph that does comes next. Five tests cover the six headers (the expected-reuse policy inside the cache's test), part of make check on every platform. A hang ends in a watchdog thread instead of alarm(), which MinGW lacks, and the loaders set a stack guard page only where POSIX can: Windows guards thread stacks itself, MinGW has no sysconf, and its pthread_attr_setguardsize is a stub that returns ENOTSUP.
With make qwen36 METAL=1, an Apple Silicon Mac runs the forward pass of Qwen3.6 on the GPU (qwen36_graph.h), on the Metal kernels and the expert cache of the two previous changes, while the routed experts are still read from the disk, so they do not have to fit in memory. In a fast prompt batch the CPU's matrix unit takes part of the products. Per layer the trunk and the router run as one GPU batch; the CPU reads the router's choices, a worker thread loads the missing experts into GPU slots of one cache for the whole model (a sixteenth of the RAM by default, QWEN_EXPERT_CACHE_BYTES), and each expert computes every token that chose it. In generation the GPU also runs the next layer's router on this layer's residual, and the experts it guesses are announced to the disk early. A long prompt runs layer by layer (qwen36_prefill_stream.h), so each layer's experts are loaded once. canonical.qc is read in place by the GPU (qwen_dense_mapped, QW.mapped: never freed); the CPU engine keeps its copy. Without the file the graph copies the matrices and frees the CPU's copies. In serve mode a copy of the DeltaNet state where the prompt's last turn opens lets the next chat turn read only what it adds. The graph runs only where it can. COLI_METAL=0 keeps the CPU engine; a qpack container, a GGUF model, a dense format other than INT8, COLI_KEEP_F32, COLI_VULKAN=1 in a VK=1 build, CACHE_ROUTE=1, a serve of several conversations at once (KV_SLOTS above 1), routed experts that are not planar INT4 with groups of 64 (or QWEN_EXPERT_KERNEL=0), a model without routed or shared experts, a model with sigmoid routing, attention sinks, a sliding window or dense layers (Flint, Muse Spark), a geometry outside the kernels, a Clef decision head (DECIDE reads the hidden state of every position), no Metal device or one that cannot build the kernels, and the Apple Paravirtual device keep it with one line that says why. The graph takes no images (a line at start says the vision tower is not loaded), no prompt-lookup or MTP drafts (with Q36_MTP=1 the head stays off) and no read-out of the prompt's logits (a logprobs request is refused). A STOP or CANCEL during a prompt takes effect once the prompt is read. A router row with no valid choice degrades as on the CPU engine. In a serve the dashboard's expert map shows the graph's cache. Where the graph runs, the CPU's rows interleaved by 16 (Q36_ILV16, AVX2 builds) stay off: the GPU reads the trunk and the experts as stored. The graph's expert readers are threads of the system's pool, which the system ends while idle, or a long prompt's four readers, which end with it, so after each read they give back the block the expert load keeps per thread. docs/qwen36-metal.md describes it. make test-qwen36-graph METAL=1 checks the graph against the CPU engine on a model the test writes; the macOS CI job runs it, and on the runner's Paravirtual device it checks that the CPU engine takes over. On the tiny fixture the CPU engine gives the same bytes as before, with and without METAL=1.
The measurement of the previous commit's graph on Qwen3.6-35B-A3B: the same harness on the CPU engine (COLI_METAL=0) and on the Metal graph, prompts of 64 and 512 tokens, each followed by 32 generated tokens. Two manifests (the generation after the 512-token prompt, and that prompt), the summary, the raw logs of every run with the start time, the peak memory and the swap counters, the harness, and quality.txt: the teacher-forced likelihood and speed of 1024 tokens of the text on the CPU engine and on the graph in both modes. The logits are the same in every run of each engine.
The Metal graph reads each routed expert from the checkpoint as two tensors (the int4 weights and their scales) and repacks the weights into the planar layout of its GPU slots. qwen36-canonical --experts writes experts.qc beside the model: every expert already in that layout, weights and scales together, so a slot is filled by one pread. The int4 values are repacked, not requantized. When every scale is a bfloat16 max|w| over 7, as Colibri's converter writes them from a BF16 checkpoint, the file stores that bfloat16 in 2 bytes and the reader divides it by 7 again into the same f32 bits (checked over every scale before writing); otherwise it keeps the f32 scales. The file is written as a temporary, synced (F_FULLFSYNC on macOS) and renamed; a signal that stops the conversion removes the temporary, and the file gets the mode of a plain file under the umask. At start its header and size are checked against the model and against a stamp of the checkpoint (tensor descriptors, shard inodes, sizes, timestamps and samples, but not the volume's device number, which macOS can change at a restart), and once against the layout of the cache slots: a file beside the model that does not match is reported and ignored, and the experts come from the checkpoint as without it. QWEN36_EXPERTS_QC=0 reads the checkpoint, QWEN36_EXPERTS_QC=<path> names a file kept elsewhere, which then has to match. The records carry no checksum. The CPU engine and the adapters keep reading the checkpoint. expert_ffn.h's int4 repack can now run in place, which the writer uses: the scalar loop, now a function of its own, keeps a copy of the one block it rewrites; the SIMD paths load each block before storing it. qwen36-canonical takes --experts in any position and refuses any other argument that starts with '-'. tests/test_qwen36_expert_file.c and tests/test_qwen36_expert_convert.c cover the format, the checked reads (both layouts, also at the 35B model's expert shape, where their sizes differ), the writer, its cleanup after a signal and the stamp, through qes_fingerprint with the volume renumbered; a check pins the scale convention to known values, which the writer and the reader could otherwise both get wrong together. test_expert_ffn.c checks the in-place repack, and the scalar loop on every host, against an independent decoder.
The measurement of the previous commit's code on Qwen3.6-35B-A3B: the same Metal binary with experts.qc beside the model and with QWEN36_EXPERTS_QC=0, which reads the experts from the checkpoint, prompts of 64 and 512 tokens, each followed by 32 decode steps. Two manifests (the 512-token prompt, and the decode steps after it), the summary, the raw logs of every run with the start time, the peak memory and the swap counters, the harness, and the converter's output when it wrote the file's records on 3 October (the file was rewritten in place with the same bytes on 5 October, and its 8-byte stamp on 7 October). The logits are the same in every run of both versions.
In decode the Metal graph runs the next layer's router on the current residual and announces the guesses that are not in the cache, so the SSD reads them while the GPU computes the next layer's trunk. It announced all eight of the top-k. On Qwen3.6-35B-A3B the seventh and eighth guesses were among the experts the next layer chose only 56% and 41% of the time on code (65% and 49% on prose), and each wrong one not already in memory is a 1.7 MB read the step does not use. Only the six likeliest are announced now. The choice moves into qwen36_advice_schedule.h, where its test checks the count and the order. Announcements are advice: the results do not change.
The measurement of the previous commit against its parent on Qwen3.6-35B-A3B with the Metal graph: the first 64 and 512 tokens of prose, code and chat, each followed by 32 decode steps, 4 runs per version in ABBA order. One manifest per text (decode steps 2 to 32 after the 512-token prompt), the summaries, the raw logs of every run with the start time, the peak memory and the swap counters, the harness, code.txt and chat.txt (the prose text is cited by its source and its SHA-256); guesses.txt holds the exploratory runs behind the count of six, with their logs. The logits and the generated tokens are the same in every run of both versions.
Much of each decode step of the Metal graph goes to reading routed experts from the SSD: Qwen3.6-35B-A3B chooses 8 experts of 1.7 MB per layer and token, and on an 8 GB Mac the expert cache holds about one token's worth. QWEN36_FAST_EXPERTS=J, in fast mode only, uses a token's J likeliest routed experts and the rest of its top-8 only when the expert cache already holds them, so a step reads fewer experts, at a cost in quality. A left-out choice points at the heaviest expert with weight 0: no read and no cache slot, and that expert's FFN runs once more for it. The kept weights stay the router's, divided by the mass of the whole top-8: renormalizing over the kept experts would raise their gain above what the model was trained with (arXiv 2609.04575 measures on this model that renormalizing over fewer experts costs more than dividing by the top-8 mass). Unset or 0, the default, and in exact mode every chosen expert is used; other values are refused with a line, and a line says when the option is on. The decode batch of the cached experts, computed while the misses read, now takes all the pairs of an expert, so it still runs when a choice is left out (the same bits as before). The expected-reuse cache policy also learns from a decode group with fewer than top-k experts (such a group fell back to the frequency policy), qgc_present answers a layer's presence under one lock, and the read-ahead announces at most the J likeliest guesses. The graph test covers the option; the next commit measures it.
The measurement of the previous commit on Qwen3.6-35B-A3B, the same Metal binary with QWEN36_FAST_EXPERTS=5 and unset: the first 64 and 512 tokens of prose, code and chat, each followed by 32 teacher-forced decode steps, 4 runs per version in ABBA order; the quality over 1024 teacher-forced tokens of each text, token by token, with the script that compares them; and, without the variable, the exact-mode fingerprints of that commit and of its parent. One manifest per text (decode steps 2 to 32 after the 512-token prompt), the summaries, the raw logs of every run with the start time, the peak memory and the swap counters, the per-token files, the harness, code.txt and chat.txt (the prose text is cited by its source and its SHA-256).
Qwen3.6's checkpoint carries a multi-token-prediction head, which tools/convert_qwen36.py carries into the container as model-mtp.safetensors (f16 dense weights, the routed experts in the model's own int4 layout). The Metal graph reads those tensors and runs its own head on the GPU; the other engines' head (qwen36_mtp.h, Q36_MTP=1) is not loaded beside it, and Q36_MTP=1 says so in one line. With the head in the container, the Metal graph drafts the token after each one and, when the head's confidence is at least 0.6 (QWEN36_MTP_MIN_P, from 0 to 1), one forward verifies the picked token and the draft together; QWEN36_MTP=0 turns it off. In exact mode the tokens and logits are plain decoding's: a verify computes each row with a decode step's kernels, and a rejected draft swaps back the DeltaNet state copied after the first row. The drafts go through the verify and the rollback of q36_spec_step and are counted as the head's, with the gate off for them (their line says "gate off:"); with the head the report gives the head's lines only, with the drafts below the confidence and the steps without the head. The head runs on the GPU as one more layer, with its own KV cache and its routed experts in the expert cache as the layer after the model's last (its two heaviest only). It drafts among the first 32768 token ids, which hold most tokens of English, other Latin-script text and code but no Chinese character: where half or more of the last 16 tokens lie past them, it does not run (it could draft only the other tokens there, and it costs about 5% of a step), and its attention starts again when it runs again; a prompt with 8 or more of its last 15 tokens past them skips the head, which will not run at the next step. It reads only a prompt's last 32 positions, which run the prompt's last layer whole (every row does when the prompt's last batch has fewer than 33 rows), and only their final residuals are kept. Its attention starts again where a position does not follow its last one, as in a serve turn resumed from the copy of an earlier position. Its dense weights are quantized 256 rows at a time, so no float copy of a whole matrix stays in the process's footprint. A verify leases both rows' experts at once and the expected-reuse policy counts them as one visit of the layer; so the cache keeps top-k fewer experts between steps, and a budget under two top-k groups stops the start with a line. Without memory for a verify's copies it drafts nothing, with one line. With QWEN36_FAST_EXPERTS above 3 a draft row keeps two sure experts fewer, except in the last quarter of the layers. Q36_MTP_DUMP writes the head's logits as it does for the other engines' head. tests/test_qwen36_graph.py writes a head too, as model-mtp.safetensors, and checks the drafts against plain decoding, in a serve turn resumed from the copy and in the turn after one that ended inside a verify, which drafts again; the head's logits with its window in one batch or in two; the steps it does not run where most tokens lie past the ids it drafts from (Q36_MTP_VOCAB, for tests, narrows them) and the prompts that skip it; the values of the variables, a cache too small for a verify, and Q36_MTP=1 under the graph. Without the head, prompt lookup's forced drafts check verifies of up to six rows on the graph.
The checkpoint's MTP head against QWEN36_MTP=0 with the same binary: exact mode (greedy, the same tokens and fingerprints in every run of both), fast mode and fast mode with QWEN36_FAST_EXPERTS=5 (each step fed the text's own next token), on prose, code, chat and a Chinese text, where the head stays idle: 64- and 512-token prompts, 32 decode steps, a warm-up and 4 runs per version in ABBA order, each after the nominal thermal state and a 60 s rest. Decode speed counts all 32 steps, the first included: a verify in the first step can make the second free. One manifest per mode and text, every run's raw log and the summaries; the three cases more than 5% slower re-run from clean, with their own manifests; the head's own cost in exact mode, with a draft verified only at confidence 1 (QWEN36_MTP_MIN_P=1), on prose and the Chinese text. Also the quality over 1024 teacher-forced tokens of each text, token by token (plain decoding, the drafts, fast experts without and with the drafts); 256 tokens of free generation per variant, with the drafts counted and the generated texts; the exact-mode fingerprints against the parent; the head's checksum, how it was made and how it compares with the file tools/convert_qwen36.py writes; the code, chat and Chinese texts, the harness and the comparison script.
QWEN36_FAST_GUESSED=R narrows fast experts (QWEN36_FAST_EXPERTS=J) in decode: a token uses its R - 1 likeliest routed experts, its next ones up to the J likeliest only when the expert cache holds them or the layer before guessed them for its layer, and the rest of its top-k as fast experts do. A guessed expert's read was announced while the layer before computed, so keeping it costs little; a choice neither guessed nor in the cache would wait for a whole SSD read. The first layer of a step, which nothing guessed, and a verify's draft rows keep the rule of fast experts (qfe_keep_guessed in qwen36_fast_experts.h; the guesses are the ones qg_advise_ahead announces, every row's, once each, and each forward keeps its own). Unset or 0, the default, changes nothing. Exact mode, the variable without QWEN36_FAST_EXPERTS of 2 or more and a value that is not a whole number from 2 to J are refused with a line; a line says when it is on, and one at exit counts the rows the rule ran for and the choices that stayed because they were guessed. As with fast experts, the output depends on what the cache holds. docs/qwen36-metal.md and docs/ENVIRONMENT.md describe it. test_qwen36_fast_experts checks the rule and its count without a model; make test-qwen36-graph METAL=1 checks the values and their lines, the same output in every run, the prompt and exact mode unchanged, from the line at exit that the rule ran at every layer of a decode step but the first and kept guessed choices, and with the MTP head's drafts that it ran once per forward, with the same output in every run.
The measurement of the previous commit on Qwen3.6-35B-A3B, the same Metal binary with QWEN36_FAST_EXPERTS=5 and the MTP head's drafts, without and with QWEN36_FAST_GUESSED=4: the first 64 and 512 tokens of prose, code and chat, each followed by 32 teacher-forced decode steps (all counted, the first included), 4 runs per version in ABBA order, each after the nominal thermal state and a 60 s rest; the quality over 1024 teacher-forced tokens of each text and of a Chinese text, token by token, against fast experts alone and against plain fast decoding; 256 tokens of free generation, two runs per version, with the generated texts; and, without the variable, the exact-mode fingerprints of that commit and of the speculative-decoding commit below it. One manifest per text (decode after the 512-token prompt), the summaries, the raw logs of every run with the peak memory and the swap counters, the per-token files, the harness, the comparison script, code.txt, chat.txt and guxiang.txt (the prose text is cited by its source and its SHA-256).
In decode each layer guesses the next layer's experts and announces the reads of those not in the expert cache (qg_advise_ahead); the next layer then reads its misses into cache slots while the step waits. QWEN36_PREFETCH=1, with experts.qc, copies the guesses while this layer computes, on two serial queues of their own, into eight slots beside the expert cache (qwen36_prefetch_ring.h). When the next layer misses one of them, its load takes the copy instead of reading the record, after waiting for a copy still reading: the cache's slot and the copy's trade their storage (qwen_slot_swap). A copy done when the layer starts is also lent to it (qwen_prefetch_lend): the GPU computes it early, with the experts the cache holds, from the copy's own tensor, while the layer's other misses read, and no other copy reuses its slot until the load takes it or the next sync ends the loan. The cache makes the same choices as without the copies and its slots end up with the same bytes, and a lent copy is computed with the same pairs in another batch, so the output does not change, in any mode. A guess is not copied when a later layer's guesses were issued (its layer has read its experts), when the cache holds it, or when the system's file cache does not hold every page of its record (mincore, on the copy's queue): a cold one would hold a queue for a disk read, and is only announced, as before. Only decode steps of one token copy; a verify of the MTP head's drafts copies nothing. A copy that fails is dropped, and its layer reads the expert itself. The eight slots hold one expert each, 14.2 MB on the 35B model, on top of the expert cache's budget (QWEN_EXPERT_CACHE_BYTES); the line that says it is on gives their bytes. Unset or 0, the default, changes nothing; another value and the variable without experts.qc are refused with a line, and a line at exit counts the copies, those their layer took and those it computed early, and why the others were not made. docs/qwen36-metal.md and docs/ENVIRONMENT.md describe it. test_qwen36_prefetch_ring checks the slots without a model: which entry a copy takes, a load that takes a copy and the storage it gets, one that waits for a copy still reading, a copy that fails, and a copy lent before its load, which no other copy reuses until the load takes it or the loan ends; make test-qwen36-graph METAL=1 checks the values and their lines, a prompt alone copying nothing, and the output without the copies, byte for byte, with copies made and taken and none failing: in exact mode, in fast mode, with fast experts and QWEN36_FAST_GUESSED, and with the cache's frequency policy (over these five some copies are computed early), with the MTP head's drafts, and in serve mode (its layers and text), where a turn's prompt comes after the decode steps of the one before, also with a cache large enough for the prompt's readers, which read a longer turn layer by layer.
The measurement of the previous commit on Qwen3.6-35B-A3B, the same Metal binary with QWEN36_FAST_EXPERTS=5, QWEN36_FAST_GUESSED=4 and the MTP head's drafts, without and with QWEN36_PREFETCH=1: the first 64 and 512 tokens of prose, code and chat, each followed by 32 teacher-forced decode steps (all counted, the first included), 4 runs per version in ABBA order, each after the nominal thermal state and a 60 s rest; a clean re-run of code, whose first step after the 512-token prompt had been 10.1% slower with the copies; the copies, from the line at exit of every run with them; and the exact-mode fingerprints with and without the copies, with the drafts and without them, and of the guessed-experts commit below. One manifest per text (decode after the 512-token prompt) and one for the re-run, the summaries, the raw logs of every run with the peak memory and the swap counters, the copies of each run, the harness, code.txt and chat.txt (the prose text is cited by its source and its SHA-256).
The Metal layer gains kernels that compute GGUF matrices as the file holds them: rows of Q8_0, Q4_K, Q5_K, Q6_K, IQ4_NL or IQ4_XS blocks, decoded where they are read, with no second quantization. Nothing in the engine calls them yet: the graph takes a GGUF in a later change. - q36_gq_step decodes 32 weights of a row with the operations of ggml_quants.h's float decoders, uncontracted, so the weights are gq_dequant_row's bits; q36_gq_dequant writes them out for the test. kvalues_iq4nl comes from a table in shaders/blockdec_table.h's layout, which q36_gpu_init fills from ggml_grids.h as the Vulkan backend fills its own. - q36_dot_gq: in exact mode a thread per output, with the sums of the CPU's exact mode (gq_matmul_f32: each product rounded, added in input order) and the ".scale" factor as matmul_d applies it, so the CPU's bits; in fast mode a SIMD group per output for up to 8 rows (each row keeps the bits it has alone) and the matrix units for prompt batches, a step's weights decoded once into threadgroup memory for 32 rows. - q36_dot_f32: a float matrix (a GGUF keeps the router in F32) with matmul_f32.h's dot_f32_lanes, the CPU's arithmetic and bits. - q36_expert_gate_up_gq and q36_expert_down_gq: a GGUF layer's routed experts, from slots holding the file's rows of gate, up and down, each of its tensor's type, with the slots' ".scale" factors: the sums of moe_gq_run and the SwiGLU of q36_expert_gate_up. - The DeltaNet kernels take llama.cpp's layout as the CPU engine reads it (g_dn_tiled, g_ssm_a_direct): with q36_shape.tiled value head h reads key head h % k_heads, and with a_direct the A_log weights hold A = -exp(A_log). The graph passes 0 for both, which gives the bits of before. Any other type, a partial block, a tensor too small or weights at an offset their loads cannot take is refused with nothing encoded. test_qwen36_kernels checks the decoders, the exact products and the float product against the CPU's bits, the fast products against a double reference, a row in a batch against the row alone, the experts (down against the CPU's bits, gate and up against a double reference, since the SwiGLU's exp is the GPU's), the refusals, and the DeltaNet kernels with tiled heads in both modes.
A METAL=1 build runs a qwen35moe GGUF on the Metal graph, where it ran the CPU engine: the dense matrices and the routed experts are computed from the file's ggml blocks with the kernels of the change before, with no second quantization. The model's own format (canonical.qc, experts.qc) and its kernels do not change. - qwen_graph_eligible takes a GGUF whose dense matrices and experts are Q8_0, Q4_K, Q5_K, Q6_K, IQ4_NL or IQ4_XS, each of its own type, and dense matrices of floats (the router); a file with another type runs on the CPU engine, with a line that names the matrix and its type (qwen_gguf_graph_off). - The file is mapped (qwen_gguf_map). The graph sees each dense matrix in place, in a Metal buffer of its own pages (q36_map_view_own: the file keeps the experts between the dense tensors, and the GPU holds a buffer resident whole), and the embedding rows are read from the mapping: load_tq no longer copies the matrices. A matrix of floats is uploaded from the CPU engine's F32 copy, which is then freed. - An expert cache slot holds an expert as the file does, gate | up | down in its layer's types (gqe_max bytes), filled by load_expert_merged; qg_experts runs the _gq kernels with the ".scale" factors of each slot's expert, in decode, in a prompt (layer by layer too) and in a verify. - The DeltaNet kernels take llama.cpp's layout as the CPU engine reads it (g_dn_tiled, g_ssm_a_direct). - With a GGUF the CPU's share of a fast prompt is off (its products take INT8 weights); the MTP head is off and experts.qc and QWEN36_PREFETCH are ignored, with a line when they are asked for. At exit a line counts the products with the file's matrices and experts that ran on the GPU. - GGUF_CHECK=1 builds no graph: the CPU engine's loader checks the header as before, and a line says whether the graph would run it. test_qwen36_graph writes a GGUF fixture in llama.cpp's layout (two key heads for four value heads, the six types, ".scale" companions) and checks: the exact graph against the CPU engine (GQ_EXACT=1), within 1e-4 of the largest value (about 7e-6 measured); the fast graph against the exact one; fast experts; a serve turn resumed from the copy; verifies of drafts forced in ref.json mode (nothing drafts on a GGUF), with plain decoding's bits, on a copy whose router bias makes a verify read one layer's experts in one lease (qg_moe_verify); the line and the CPU engine for a Q4_0 matrix and IQ3_S experts; the options of the model's own format; GGUF_CHECK=1, which builds no graph; and the layout of coli import --mode copy, bit for bit as the file it was copied from. test_qwen36_kernels checks q36_map_view_own.
In fast mode the GGUF kernels now follow the layouts of llama.cpp's Metal matrix-vector kernels in decode and of the graph's own int8 and int4 kernels in prompts. Before, fast mode took a whole 32-weight step per lane and applied the block scales to each weight, and the routed experts ran the exact-mode kernels (a thread per output and choice) in both modes. Exact mode does not change: its kernels and their bits are those of before. - Decode and verifies (up to 8 rows): a row of weights takes up to 32 lanes of a SIMD group, spread over its blocks (Q8_0: 4 lanes a block, 8 weights each; Q4_K and Q5_K: 8 lanes, 4 runs of 8; Q6_K: 16 lanes, 4 runs of 4), and the block scales multiply the blocks' sums of q x (d * sc * sum(q x) - dmin * m * sum(x)) instead of each weight (q36_gqf_part). IQ4_NL and IQ4_XS keep whole steps. Where a row takes 32 lanes, or 16 (a 512-wide Q5_K down), the count is a compile-time constant: q36_dot_gq_row32 for one row of x, q36_expert_gate_up_gq_vec32 and q36_expert_down_gq_vec16 and _vec32. A row's bits still do not depend on the batch. - Prompt batches: every lane decodes its own weights straight into the 8x8 matrix operands, as q36_dot_i8_mm reads its int8 ones (q36_gq_four); only the x tile goes through threadgroup memory. - The routed experts of a prompt run in the tiles of the int4 kernels: up to 64 choices of a slot a threadgroup, the row blocks past a slot's last choice skipped, and its last tile of 1 to 4 row blocks in q36_expert_*_gq_mm4 with the inputs read straight from memory. Q4_K and Q5_K read a block's scales and fifth bits once for its 8 steps (q36_gq_rowdec). test_qwen36_kernels gains a decode row under 32 lanes for every type, the experts' 32-lane down of a decode, experts with tiles of 3, 5, 6 and 7 row blocks and with a full tile and a short one, and the bits of the one-row 32-lane kernel against the rows' kernel for Q4_K and Q5_K. The experts' outputs start as NaN in each mode, so an output a kernel leaves unwritten fails.
The measurements of the previous commit on Qwen3.6-35B-A3B, on an 8 GB M1 MacBook Air, with one harness built against the code as measured: unsloth's UD-Q4_K_M GGUF with the fast-mode kernels of the previous commit and of the change before it; the expert cache's default budget against one slot more, which a GGUF's larger slots need for the layer-by-layer prompt; the model's own format before and after the three GGUF changes; and the GGUF against the model's own format in fast mode, with fast and guessed experts and in exact mode. The first 64 and 512 tokens of prose, code and chat, each followed by 32 teacher-forced decode steps; a warm-up (of each file in the comparisons of the two files), then 4 runs per version in ABBA order, each after the nominal thermal state and a 60 s rest. Two manifests per comparison and text (decode after the 512-token prompt, and that prompt), the summaries and every run's raw log with the peak memory and the swap counters; the quality over 1024 teacher-forced tokens of each text, token by token; the exact graph against the CPU engine on the GGUF; timed runs per file and text, with 32 decode steps and with one (where the time goes); the harness, the comparison script, code.txt and chat.txt. docs/qwen36-metal.md gives the speed against the model's own format, what one slot more in the expert cache does, where a GGUF's decode time goes, and what the timers show during the wait for the disk.
A GGUF keeps its routed experts in the same file as its dense matrices, and a prompt's expert reads pushed the dense pages out of memory, for the prompt itself and the first decode step to read back from disk: on an 8 GB M1, 0.3 to 1.2 GB of the 35B model's 1.9 GB of dense matrices were out after a 512-token prompt (mincore), and the first decode step took 0.7 to 1.5 s. With the dense tensors in a file of their own (a copy at the same offsets) they stayed, as canonical.qc's did beside experts.qc in the same runs. The graph now locks a GGUF's dense matrices in memory (mlock) from the start of a prompt to the end of the decode step after it, which then took 0.3 to 0.5 s; during decode the GPU holds them in memory anyway, and between requests the system may take them back (a request that ends without a decode step unlocks them at its end). A line says how many bytes are locked, or why they could not be (then nothing stays locked and no later prompt tries again), and the report at exit how many times they were locked and unlocked, and whether any stayed locked. What is computed does not change, nor anything for the model's own format. The decode after a locked prompt finds fewer of its experts in the file cache and is slower by a few percent: the record of the next commit gives the numbers. The GGUF graph test checks, in its graph runs, that the lock's line says the bytes of the file's dense matrices, each rounded out to whole pages, and that they were unlocked after the first decode step, or at the end of a request without one.
The measurements of the previous commit on unsloth's UD-Q4_K_M GGUF of Qwen3.6-35B-A3B, on an 8 GB M1 MacBook Air, with one harness built against the code as measured: the change before it against the previous commit, in fast and in exact mode, on the first 64 and 512 tokens of prose, code and chat, each followed by 32 teacher-forced decode steps, and on a 32-token turn resumed from the state saved after 64 tokens (prose and chat, fast mode); a warm-up, then 4 runs per version in ABBA order, each after the nominal thermal state and a 60 s rest. Three manifests per mode and text (the decode after the 512-token prompt or the turn, the first decode step after it, and that prompt), the summaries and every run's raw log with the peak memory and the swap counters; the system's wired memory during runs of both versions; and the diagnosis runs, with the harness that counts the dense pages in memory, the page-ins and the experts read, and its two patches: why a GGUF's first decode step was slow, and what the lock does to the decode after it. docs/qwen36-metal.md gives the first decode step, the second new word, the decode and the resident size with the lock.
The qwen36 Metal graph (#2015, #2016, #2298 to #2308), merged on dev 1147 commits after it was cut. qwen36.c: QW keeps dev's wf16 and q4r16 with the graph's mapped flag; a DeltaNet layer's state is not allocated under the graph (dev's LFM3 check and q36_rec_cells kept); reset_recurrent resets dev's KV ring before the graph's early return; a verify's result keeps dev's per-step depth after the graph's qg_mtp_keep; the serve path keeps dev's park and rewind, then arms the graph's snapshot; the prefill control keeps dev's condition and the graph's note. docs: dev's rows with the graph's notes added (ENVIRONMENT.md, gguf.md, serve_protocol.md, speculative.md).
…ne's The graph keeps a conversation's state on the GPU, so like the pins it already skipped, what dev's qwen36 gained after the stack was cut copies the CPU engine's state and stays off under the graph: the prefix store (q36_ps_on), the parked conversation (q36_park), the prompt-end snapshot (q36_pes_take, q36_pes_restore, q36_pes_after_abort) and the rewind to the shared attention rows. The graph's own snapshot (qg_snapshot_arm) still serves a chat's next turn. The graph also steps aside, with its line, for LFM3 and for a GGUF that carries its MTP head or vision tower in the model's file (Ollama's): it was measured on unsloth's files, which do not. All of it is under QWEN36_METAL_GRAPH: other builds are unchanged.
The registry came to dev after the Metal stack was cut: QWEN36_CACHE_EXPECTED, QWEN36_CPU_MATMUL, QWEN36_EXPERTS_QC, QWEN36_FAST_EXPERTS, QWEN36_FAST_GUESSED, QWEN36_MTP, QWEN36_MTP_MIN_P, QWEN36_PREFETCH, QWEN36_PREFILL_STREAM, QWEN36_PROMPT_ROWS and QWEN_EXPERT_CACHE_BYTES, as qwen36 reads them (docs/qwen36-metal.md).
…e graph's check The graph's check (if (!m->graph)) was written for the pin save alone; dev has since given that if an else (q36_pes_take). The block says what was already so: under the graph neither copy is made, without it both are as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
qwen36 on Metal: @CoTi1938's series (#2015, #2016, #2298 to #2308) merged onto current dev, as
rebase/qwen36-metal-graph. Their 24 commits are unchanged; on top are the merge with dev (both sides kept inqwen36.cand the docs) and three commits of ours, described in #2308 (comment):coli_env.hlists the graph's variables;if (!m->graph).What it does
The Metal graph runs qwen36's forward pass on the GPU (
make qwen36 METAL=1): the expert cache and loaders,experts.qcslots on disk, read-ahead of the likeliest guessed experts, fast experts, speculative decoding with the MTP head, Metal kernels for ggml blocks, a GGUF read by the graph and its dense matrices kept in memory through a prompt. On Linux and Windows nothing changes.Tests
CoTi on an Apple M1 (7 GPU cores, 8 GiB, macOS 26.6.2), this branch at
1b17e9c1against the series' head9e7bd192(qwen36: keep a GGUF's dense matrices in memory through a prompt #2308 (comment)):make test-qwen36-kernelspassed,make test-qwen36-graph METAL=132 passed with no skip,make check-envmatched; on the checkpoint and on a GGUF the full logits of both are bitwise equal in exact and fast modes, and prefill and steady decode are at the series' speed. Two open observations from that report: on the checkpoint's code sequence the first decode after 512 tokens is 8 to 9 % slower than the series' (16 and 8 % after the last 64), with the same outputs and miss counts; and a GGUF's cold start is 20 to 50 ms later (+13 to 34 %). The cause of either is not assigned.On ds (Linux x86-64), where the graph is not built:
make checkRan 3613 tests, OK (skipped=499), so the Linux build and its tests are unchanged. The tree with this batch, batches C and D and windows: qwen38.exe in the CUDA package; setup says why an NVIDIA card stays unused (#2846) #2847 merged together: Ran 3629 tests, OK (skipped=502).We have no Mac: the Metal results are CoTi's above, and CI's macOS jobs.