Repository navigation
Batch 2026-10-30: #2501, #2502, #2503, #2504, #2505, #2515 - #2516
Merged
Merged
Conversation
cc_norm_k read its weights after the row's sum, and cc_addnorm_k read t, then the post-norm's weights, then x, then the norm's weights, each after the step before it: four trips to memory one after the other on a single block, for a 2048-wide row most of the kernel's time (6.0 us a call, 46 a LFM3 token). Both load everything first now (cc_norm_k for rows up to 2048, the old loops past that; no weight read where the norm has none), with the same values in the same order, so the same bits: test_cuda_chain passes unchanged. RTX PRO 4000, engine only (no HTTP), Q4_K_M, alternated with dev, decode: LFM3-VL-2B 233.9, 232.4 -> 237.9, 236.9 tok/s; OLMoE 350.7, 349.1 -> 354.0, 353.3; Qwen3.6 103.9, 103.5 -> 104.6, 104.1. Prompt unchanged. test_cuda_bd's bench reads copies of the matrix past the card's L2 in turn (BENCH_MB, a decode step's reads), and BENCH_ONLY="t:I:O,..." times chosen shapes alone. Measured with it and not kept: unrolling the step loop (none), x quantized once by a kernel of its own (its launch costs what it saves), 256 blocks moved to 70-512 (256 is the best), more lanes a row for few rows (under 1 us).
…nd the chains ensure_kv grew the KV cache to exactly np + max_tokens, and the CUDA and Vulkan chains their scratch to exactly a chunk's rows: every request longer than any before paid the cache's reallocation and copy, the chain's KV mirrors freed, allocated and zeroed again, and some thirty device and page-locked buffers regrown. A chat resends a longer transcript every turn, so that was nearly every turn. LFM3-VL-2B on CUDA, prompts of about 480 tokens: 34-35 ms each, but 44-48 ms each time one was longer than all before (481, then 484); on Vulkan 65 and 76. - ensure_kv grows by half again at least, from 1024 rows, in steps of 256, up to the context the engine holds (the RAM plan reserves that context's cache already); the stride stays the capacity, as before. - Both chains' scratch is sized for a power of two of rows from 64, at most a chunk's (and the KV split's window on Vulkan; ch->rows is the stride every buffer uses). - A prompt's embedding rows are decoded over the OpenMP team (each row its own; the ids still checked in order first): LFM3's Q6_K table on one thread held the prompt before the device. RTX PRO 4000, engine only, 14 fresh prompts of 475-484 tokens: the 484-token one 47.4 -> 34.8 ms on CUDA, 76.1 -> 64.9 ms on Vulkan; every prompt after the first now at its usual time (CUDA median 13.7k -> 13.9k tok/s with the embedding rows); the first still grows once. test_qwen36_ctx's growth case states the new contract (room for a longer next turn, never shrinks, nothing reallocated up to the capacity, growth by half again); test_qwen36_prefix_serve, the tiny oracle (16/16) and vulkan_engines.sh qwen-chain on Lavapipe (every check up to the Qwen3.8 fixture, which wants transformers 5.16.1 here) pass.
olc_scratch sized the chain's buffers to a chunk's rows exactly, so every prompt longer than any before regrew them. They are sized for a power of two of rows now (from 64, at most a chunk's or the KV split's window), as qwen36's chains. OLMoE Q4_K_M on Vulkan, RTX PRO 4000, engine only, 14 fresh prompts: the first of 471 tokens 77.6 -> 72.2 ms, the 470-token ones 70-73 ms either way. vulkan_engines.sh inkling-olmoe-chain passes (61 checks).
…ed tier, cu_dense)
Under COLI_CUDA=1 a GGUF ran on the CPU: colibri.c's own CUDA path takes no
ggml block types (qt_cuda_upload, its expert residency), and a build with
both backends handed the GGUF to Vulkan unless COLI_VULKAN=0, which then left
everything on the CPU ("resident set: 0 tensors").
colibri.c now runs the shared expert tier through xtier.h, as GLM-5.3, the
DeepSeek engines, Kimi K3 and MiMo do: XT(f) is vkt_f or cut_f, the same tier
compiled over the Vulkan or the CUDA backend. With COLI_CUDA=1, no Vulkan
device and a GGUF (colibri_cu_tier_start), the routed experts go to the tier
on the CUDA device as the file has them (fmt 100 + t), the resident dense
matrices to it through cu_dense.h's cud_mm as far as its budget goes (the
tier leaves that room: VktConfig.dense_bytes), and the attention stays the
CPU's. Containers keep the old CUDA path. A build with both backends still
runs a GGUF on Vulkan by default (its dense chain takes the attention, so the
prompt is faster there); COLI_VULKAN=0 runs it on CUDA. The tier's start lines
say [CUDA] or [VK] as the device is. CPU-only builds call the stubs as before
(XT is vkt_ there).
Mistral-Large-4 UD-IQ1_S, RTX PRO 4000 (24 GB), 62 GB of RAM, experts from the
SSD, dev without the OMP places fix:
- the tier on CUDA holds 888 experts (22.2 GiB): the dense part (16.96 GiB) is
more than half of the card's free memory, so cu_dense leaves it on the CPU;
- 24 tokens after an 11-token prompt: 0.62 tok/s (Vulkan with its chain 0.60);
- a 240-token prompt then 24 tokens: 66.4 s and 1.34 tok/s; with the dense
part on the card (COLI_CUDA_DENSE_GB=17, 209 experts) 89.2 s and 0.65 tok/s;
- the same text as Vulkan and the CPU.
Tiny Mistral GGUFs (UD-IQ1_S and UD-Q4_K_XL mixes): the CUDA tier's logits
within 4.1e-2 and 4.7e-1 of the CPU's (max 6.4), the dense matrices on the
device by cud_mm; the Vulkan path unchanged (156 experts, 0.60 tok/s, [VK]).
The shared tier's code (the MoE layer's dispatch, the expert sources, its start) was compiled only with COLI_VULKAN, so a `make CUDA=1` build ran a GGUF on the CPU however the previous commit routed it. COLI_XT (Vulkan or CUDA) now guards that code and xtier.h's include (vk_tier.h's stubs stand in for the backend a build lacks); what is Vulkan's alone keeps COLI_VULKAN: the dense preload, the second device's registry and its issue in moe_vk, the absorb core's KV mirror and the partial chain in the tier's budget, the device's shutdown, the iGPU question (xt_shares_ram), the block types (xt_block_elems asks the device the tier runs on) and the globals only they read. colibri_cu_tier_start runs in any CUDA build. Built CPU-only, VK=1, CUDA=1 and CUDA=1 VK=1, no new warnings. The CUDA-only build on Mistral-Large-4 UD-IQ1_S: the tier on CUDA (888 experts), 0.88 tok/s over 24 tokens, the same text.
…inst the Vulkan chain Alternated on one build with the core places of #2495, Mistral-Large-4 UD-IQ1_S on an RTX PRO 4000: CUDA 66-82 s for a 240-token prompt and 1.34-1.36 tok/s of decode, Vulkan 36.5-36.9 s and 1.16-1.17.
colibri.c re-runs itself once with OMP_WAIT_POLICY=active, GOMP_SPINCOUNT=200000, KMP_BLOCKTIME=200, OMP_PROC_BIND=close, OMP_PLACES=cores and OMP_DYNAMIC=FALSE (Linux, FreeBSD; not macOS, #707; not with CUDA or Metal on). olmoe, qwen36 and qwen38 only sized the team (omp_tune.h), so on Linux libgomp's defaults ran their decode, a token of many small matmul regions. coli_omp_hot_team in omp_tune.h does colibri.c's block for them: the same settings without overwriting the user's, the affinity reset before the exec (#471), the re-exec by the path /proc/self/exe names so the engine keeps its process name, and the same exclusions and kill switch (COLI_NO_OMP_TUNE). i5-13500 (6 P + 8 E), served (coli serve, a 452-token prompt and 128 tokens), alternated with dev: CPU OLMoE decode 43.3, 42.7 -> 47.9, 47.6 tok/s; prompt 267, 268 -> 279, 279 LFM3 decode 33.7, 33.8 -> 34.5, 34.5; prompt 282, 280 -> 287, 287 Qwen3.6 decode 13.5-13.6 -> 13.9; prompt 115.7 -> 115.4 (4 rounds) Qwen3.8 IQ2_XS, 68 GB on 62 GB of RAM (from the disk): decode 7.9-8.1 -> 9.0-9.1; prompt 38.2 -> 36.7 (3 rounds) Vulkan decode unchanged; prompt LFM3 6636, 6593 -> 6961, 6879, OLMoE the same, Qwen3.6 2003 -> 1965 (4 rounds) Pinning alone (close/cores) and more threads (14, 20) did not move decode; the waiting policy is what does. Tests: test_omp_places reads omp_tune.h too, and runs olmoe without SNAP: the re-exec once, none with COLI_CUDA=1 or COLI_NO_OMP_TUNE.
utf8_drain appended a token's bytes into its 16-byte buffer before it took the
complete codepoints out, so every byte past the 16th was dropped. A Cyrillic
word is often a single token longer than that (" экскурсии" 19 bytes,
" удовольствием" 27), and Qwen3.6's Russian came out with replacement
characters in chat, streamed and not. The codepoints are taken after every
byte now, so the buffer only ever holds a partial one (at most three bytes);
everything else about the function is as it was. qwen38.c's decoder was
already incremental.
The diagnosis, the reproduction and this fix are the reporter's (#2492).
Test: test_qwen36_ctx's utf-8 case, a 27-byte token and a 40-byte one whole and
a codepoint split across two tokens; the old function fails the first two.
…nce a step, a stopped reasoning turn continued inside its block /health (authed) carries `live`, one entry per turn in flight (reading the prompt or writing, tokens, tok/s over the whole turn and over the last 32 tokens) and `system` (the box's RAM, the engine's RSS and CPU, the load). /experts?since=N returns the decode steps routed after N, each as its experts, so a page can light them step by step. A turn stopped while it was still reasoning (no answer text, reasoning_content set) goes on inside the open block on Qwen3.6 and Qwen3.8 (CONTINUE_IN_THINK, falsy for every other caller); any other family still refuses an empty continued turn.
…b search, expert map, live monitor, models, coding agents One page (ui/index.html, no build step) and a standard-library server (ui/server.py) in front of a running `coli serve`: the API proxied on one origin, streams passed through chunk by chunk, a stopped request closing the engine's turn. - chat with the reasoning in its own block, Markdown, pictures when the engine loaded its vision tower, conversations kept in the browser, seven languages; - web search (ui/web.py): SearXNG with --searx, else DuckDuckGo's plain HTML results; pages read at public addresses only; - the expert map and the live monitor from /health and /experts; - the Models page (ui/models.py): what fits the machine it runs on and how fast, measured there; our catalog and the GGUFs colibri reads; downloads; switching the model the gateway serves, with the previous one started again when the new one does not start; - coding agents (ui/agents.py) opened in a terminal already pointed at colibri, and the Responses API over the chat API for Codex (ui/responses_shim.py). `coli ui` starts it and opens the browser; the release archive carries ui/ next to coli.
blockdec_host.h: F16 (fmt 101) and BF16 (fmt 130) are blocks of 32 values, 64 bytes, no scale, the file's rows as they are. Every block-format path takes them: - Vulkan: bdstep.glsl reads a 16-bit step 16 bytes at a time and widens it in registers (the per-type GEMV, the 64 tile, the matrix units' 64 and 128 tiles); blockdec.glsl's fast_dot for the general GEMV; vk_chain.c's per-type pipelines list both types. - CUDA: the float GEMV reads the values; bd_dec16h/bd_dec16 and hcm_dense_k take F16 as it is and BF16 converted to f16 (held to +-65504); the typed MoE kernels leave 16-bit experts to the generic tiles. - qwen36: a GGUF's F16 or BF16 matrix keeps its f32 copy for the CPU and gives the devices its 16 bits back from it (q36_wf16_rows, exact) as block format 100 + type: on Vulkan instead of f32 rows outside the tiled GEMMs, on CUDA instead of the CPU. Qwen3-0.6B F16 on an RTX PRO 4000, 470-token prompt and 128 tokens: CUDA 137 -> 15921 tok/s of prompt and 17.6 -> 221 of decode, Vulkan 925 -> 11601 and 37.2 -> 204; BF16 alike. The greedy text is the CPU's on CUDA, the Vulkan chain and Vulkan per matrix. test_cuda_bd and test_vk_chain cover both types against the CPU's decoders.
Qwen3.6-35B-A3B, LFM3-VL-2B and OLMoE in Q4_K_M and LFM3-VL-2B in F16, served on an i5-13500 with an RTX PRO 4000 Blackwell SFF: the tree of this batch, the better of two runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One CI for six PRs that merge cleanly with each other: each is merged here as it is (no squash), so when this batch is merged into
deveach PR shows as merged with its own commits.The details, measurements and tests are in each PR. Together here they also run against each other: #2503, #2504 and #2515 all touch the CUDA backend, #2501, #2502, #2504 and #2515 the qwen36 engine.
Checked before this CI:
make -j14 check(clean, the portable build, every C and Python test) on Linux with gcc 15.2, the merged tree of this branch.