Repository navigation
Batch 2026-11-13-d: nineteen PRs (DeepSeek V4 decode pieces, the Vulkan tier's rooms per size, CUDA prompt attention, FreeBSD, translations) - #2820
Merged
Conversation
A row in the release matrix on ubuntu-24.04-arm: every engine, the TUI and the desktop window built for plain armv8-a, so the archive starts on a Cortex-A72 too, CPU only, with the same verify steps and glibc floor as linux-x86_64. release_asset_suffix() names it, so start-here.sh downloads it on aarch64 Linux instead of stopping with "no prebuilt release". Closes #2684
cc_attn_fk, a prompt chunk's attention on the tensor cores, stages each block of 64 keys of K and V into shared memory as f16 with a loop that reads one element a load: two bytes a load from the chain's float16 cache. Since cc_attn_f2 takes prompt chunks from sm_75, cc_attn_fk runs them on sm_70 (and with COLI_CUDA_CHAIN_FLASH2=0). At head dim 256 a thread now starts 8 loads of 16 bytes before it stores any (cc_fa_stage16): eight halves of a float16 cache copied as they are, or four floats of a float32 cache rounded as the one-element loop rounds them, so the bits are the loop's. Groups of 16 loads spilled on sm_70, where the kernel already holds 230 registers (255 registers and 392 bytes of stack with a float16 cache, and the prompt 4.5 % slower than the loop at 8192 tokens); groups of 8 take 242 registers and no stack. The staging needs K and V 16-byte aligned, as the chain's mirrors are: cc_attn_t checks and keeps the loop otherwise, as head dim 64 and 128 and cc_attn_w2 (windows, sinks, the ring) do. A new test, prompt_rows_16byte_staging, runs 2043 rows after 6144 positions with Qwen3.6's heads (the last block of keys is partial) from K and V aligned and one element further on (the loop), from a float32 cache of values that are not float16 numbers and from a float16 cache, and wants the same bits from both; four rows within 1e-4 of the reference. It runs where cc_attn_fk takes a prompt chunk and says so elsewhere. Qwen3.6-35B-A3B int4-g64 on a Tesla V100-SXM2-32GB (every expert on the card), a prompt from an empty state, 4 + 4 runs in ABBA order: 2048 tokens 2442.37 -> 2595.14 tok/s (+6.3 %), 4096 2341.59 -> 2643.43 (+12.9 %), 8192 2218.47 -> 3014.55 (+35.9 %), every run faster, with the same fingerprint (a hash of the logits after the prompt and of the first word) and nll at every length. Under nsys, cc_attn_fk takes 31.3 ms of a 2048-token prompt instead of 83.9.
Measurements of the previous commit's change against dev bf0b0ea on one Tesla V100-SXM2-32GB (sm_70, CUDA 12.8, a Vast.ai container with about 181 GiB and 15 CPUs), Qwen3.6-35B-A3B int4-g64, every expert on the card. prompt.md and manifest.json: prompts of 2048, 4096 and 8192 tokens in one process, each from an empty state, a warm-up and 4 + 4 runs in ABBA order (the manifest's "ram" is the host's; its "every logit" is the bench's fingerprint: the logits after the prompt and of the first word). correctness.txt: the CUDA chain test of both versions (three failures on sm_70 that dev has too, outside this change), the two counter-tests that make the new test fail, registers and stack of cc_attn_fk on every arch, the first port with groups of 16 loads, the kernel each version runs (nsys), and tests/cuda_tier_engines.sh. The code was measured as a local commit, a37779ed, whose c/ tree is the previous commit's, 61e4fa3 (git rev-parse HEAD~1:c on this commit); the previous commit adds only the docs paragraph.
page() checked a name's addresses, then urllib resolved the name again to connect. A name that answers a public address first and 127.0.0.1 next (DNS rebinding) passed the check and the page read a service on this machine; a redirect to such a name did the same. The connection now goes to an address from the lookup that was checked. The name stays in the Host header and the TLS handshake, and a proxy, which resolves the name itself, is reached as before.
Section 3 assumes the CUDA Toolkit installer. A conda-layout tree works too (no admin rights needed): its Library/bin on PATH for nvcc and cudart, and CUDA_HOME pointing at Library. Co-Authored-By: Claude <noreply@anthropic.com>
… banks
A served request reported one timer bank ("request"), prompt and decode
together. The prefill/decode split existed only for the process total, which
prints at exit, and a serve process stopped by its gateway never gets there.
Under COLI_TIMERS=1 both serve paths (serve_one and the KV_SLOTS mux) now
also print "request prefill" and "request decode" banks, cut where the
request's prompt has been read (q38_serve_start), and one line with where
the request's routed experts came from: RAM cache hits and disk reads, and
with the CUDA tier on its VRAM hit rate, CPU-computed experts, uploads, swaps
and q_skips (qt_counters, new, under the tier's mutex). Without COLI_TIMERS
the output is unchanged.
Tests: test_qwen38_metrics (the line with and without the tier, overflow),
test_qwen36_tier_replan (qt_counters on the fake backend).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p (V4_LAYER_FUSED, opt-in) A decode token's attention layer is several device round trips with host arithmetic between them. Under WDDM each trip costs more than the kernels it launches: decprof's attention is 84 ms per token on an RTX 5080 at a 3.3k context. V4_LAYER_FUSED=1 runs the layer as one round trip (dsv4_cuda_attn_decode_fused): the token's six projections from one upload of the hidden state; the query's norm and RoPE; the key row's norm, RoPE and FP8 quantize-dequantize, entered into the device KV ring; on a token that emits a compressed row, the row pooled from a device mirror of the compressor state; the indexer's query, scores over resident keys and a bitonic top-k with the host's tie order; the cached sparse attention over the selected rows, inverse RoPE and the grouped output projection. The sequence is a CUDA graph per layer. A token the fused layer does not take falls back to the chained forms it is built from (V4_WO_CHAIN, V4_DECODE_PROJ6), and from those to the existing paths. Every step keeps the CPU reference's order of operations, so the results are the CPU's bit for bit. V4_LAYER_FUSED_VERIFY=1 runs the fused layer beside the usual path and counts differing words per call; DSV4_WO_VERIFY=1 does the same for the chained tail. With the switch unset nothing changes. The new entry points are not called, and the kernels the existing entry points would otherwise pick up (the warp matvecs, the cached sparse attention's second form, the coalesced indexer reads) default to off; each sub-switch follows V4_LAYER_FUSED unless set. coli_v4_indexer_select_batch, the expert pipeline and the prompt's batch paths are not touched. With it on, the packed compressor copies and the resident indexer mirrors (769 MiB on V4 Flash) default to off: the fused layer reads the plain bf16 mirrors and its own resident indexer weights. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d trip (V4_LAYER_FUSED) The manifest and the raw logs: three interleaved runs per arm (the switch unset against 1), and the two verifier runs (V4_LAYER_FUSED_VERIFY, DSV4_WO_VERIFY), 1,505 calls each without a differing word. The measured figures in docs/deepseek-v4.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…copies (V4_MIRROR_BLOCK, opt-in) A mirror in the per-device cache is three FP4 tensors, each with its own weight and scale allocation, so an upload or an in-place recycle is six cudaMemcpyAsync. The host record already keeps the three scale tensors together and the three weight tensors together. V4_MIRROR_BLOCK=1 gives a mirror one weight run and one scale run on the device, laid out as the record has them (dsv4_cuda_mirror_alloc), and fills it with two copies (dsv4_cuda_mirror_refill). The entry is still three tensors to everything that reads it: the expert group, the peer pool, the resident route. gate owns the two allocations, up and down point into them. The cache, its growth guard, its LRU and its recycle in place are unchanged. An expert whose host tensors are not two contiguous runs, or whose weights would not start on a 16-byte offset, takes the three-tensor mirror as before, in the same cache; a cache can hold both kinds. A DLL without the new exports keeps the three-tensor mirrors. V4_MIRROR_BLOCK_VERIFY=1 reads every fill back and compares it with the host bytes. tests/test_v4_gpu_placement.c covers: the switch unset, a block made and filled from the record's runs in the file's tensor order, a hit, a three-tensor victim replaced by a block, a block victim refilled in place, a scattered record in a block victim's place, a refused refill, the verifier's counts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…runs The prefix store lives in RAM only, so the first request after every engine start reads its whole system prompt again, even when the model and the prompt are the same as before the restart. With Q38_PREFIX_DISK=1 (default 0) the checkpoints a prompt keeps where conversations part from it (where its system prompt ends, where its messages start; not a turn's end) also go to Q38_PREFIX_DIR (default <SNAP>/.coli_ckpt, as deepseek_v4's) as q38_<identity>_<ids hash>.bin: the ids, the DeltaNet/PLE/MTP state and every position's K/V/index rows, with a SHA-256 trailer. They are copied from the store after DONE and written on a thread of their own (temp + rename), so decode never waits on the disk. A request the RAM store does not take as far takes the deepest matching file: size and hash checked before a row is written, a bad file deleted. The identity covers the format, row layout and block, state geometry, the weights (SNAP's safetensors and config, a GGUF's files, the int4 or low-bit expert sidecar's files, by name, size and time), the binary, the loaded backend DLL, the Vulkan shaders, the knobs that pick the prefill's kernels, the K/V row precision (Q38_KV) and the build. Q38_PREFIX_DISK_MB (default 4096) caps the directory, least recently used first; every file a request's prompt starts with gets that request's age and the deeper go first at one age, so a conversation's system-prompt file goes last. Tests: tests/test_qwen38_prefix_disk.py (restart reuse with a fresh read's bytes and logits, plain and MTP; another model or block size never loads; corrupt or truncated files dropped; the cap keeps the system prompt's file; off by default), run in CI next to test_qwen38_prefix_store. The variables are in coli_env.h and docs/ENVIRONMENT.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restored state's PLE history and pending MTP tokens index the embeddings. The restore now refuses (and deletes) a file whose ids fall outside the vocabulary, as the segment adapter does for its PLE history, before any row is written into the slot. The other-identity test now checks the first identity's own file by name, with its bytes and mtime: the other runs write a file at the same position into the same directory, so looking a position up could not fail. It also checks that each of them wrote its own file under another name. ci.yml: the line continuation before tests.test_qwen38_prefix_disk is back, as in the steps around it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e environment reference, and the experiment record docs/deepseek-v4.md: what the switch changes, which experts keep three tensors, the exit line, the verifier, and the measurement on a 16 GB card. docs/experiments/v4-mirror-block-2026-11-12: the manifest and the raw logs. Three runs per arm at DSV4_CUDA_VRAM_RESERVE_MB=600 (440 mirrors): 21.42 s against 21.05 s for 35 tokens, +1.7 % tok/s. Three runs per arm at the default reserve (32 mirrors on this card): 22.84 s against 22.91 s, no gain. One verifier run: 5,618 fills read back, no difference. The text is the same in all thirteen runs. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… bank releases the oldest when it is created (V4_BANK_MAKE_ROOM, opt-in) With COLI_CUDA_MOE_BATCH=1 the decode mirror cache keeps the prompt's expert bank free at all times (two banks with COLI_CUDA_MOE_DOUBLE=1), so the next long prompt finds its room. On a 16 GB card that reserve is 8,329 MiB and leaves decode 32 mirrors: every routed expert of every token is then an upload. V4_BANK_MAKE_ROOM=1 turns the order around. The cache keeps only the 600 MiB headroom free and grows into the rest. Where a prompt creates a bank (coli_v4_gpu_moe_batch_union, both banks), v4_gpu_mirrors_make_room releases the least recently used mirrors of that device's caches until the bank and the headroom are free, then lets the growth guard probe the card again. A prompt too short for the bank creates none and releases nothing. Not touched: a resident table (experts_per_layer), a cache on another device (a layer placed elsewhere, the peer pool), unified memory. An explicit DSV4_CUDA_VRAM_RESERVE_MB still decides the reserve; the release at bank creation then runs only if the switch is on. tests/test_v4_gpu_placement.c: the mock's free VRAM follows its live tensors; the reserve with and without the switch, a full cache released down to the bank's room in LRU order, nothing released when the bank already fits, a resident table and another device's cache left alone. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nment reference, and the experiment record docs/experiments/v4-bank-make-room-2026-11-12: the manifest and the raw logs. Three runs per arm, one 3,278-token prompt and 36 tokens: 22.83 s against 21.39 s for the 35 tokens after the first, +6.7 % tok/s, the same text. Three such prompts in a row on one serve engine, one run per arm: with the switch on, the second and third prompt each release 528 mirrors, both banks are created, nothing fails to allocate, and every turn prints the text it prints with the switch unset. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rs spin (#2286) MinGW's libgomp is configured "mingw32 posix": its barriers wait on a semaphore with no spin loop, so OMP_WAIT_POLICY=active and GOMP_SPINCOUNT, which the launcher sets for deepseek_v4, do nothing on Windows, and each of a token's many small parallel regions pays a thread wake-up. On an i7-7700HQ (4 threads) an OpenMP barrier costs 22 us with libgomp whatever those variables say, 0.3 us with LLVM's libomp. - c/Makefile: clang's MinGW triple (x86_64-w64-windows-gnu) is Windows too, so `make deepseek-v4 CC=clang` reaches the V4 build; the static OpenMP archive check looks for libomp.a when the compiler is clang. - c/Makefile.deepseek-v4: with CC=clang on Windows, libomp.a is linked in (-static as before, no new DLL), no -pthread (clang's driver links winpthreads; the flag only warns), LTO off unless asked for (GNU ld has no LLVM plugin in MSYS2's clang package). The gcc line is unchanged. - c/deepseek_v4.c: drop v4_moe_bank_hash_layer, written and never read (clang's -Wunused-but-set-global), so the clang build has 0 warnings. - c/coli, c/openai_server.py: KMP_BLOCKTIME=200 beside GOMP_SPINCOUNT for deepseek_v4. With OMP_WAIT_POLICY=active alone libomp's team spins forever: a served clang engine burned 3 cores while idle (#341). - release.yml, ci.yml: the Windows deepseek_v4.exe is built with CC=clang; the OpenMP package is installed under either of its MSYS2 names; the release checks the packaged engine is a libomp build importing no MinGW DLL. - docs and CHANGELOG. Tiny V4 fixture on the i7-7700HQ, 7 interleaved rounds, same tokens in every run: decode 500 -> 1547 tok/s, a 72-token prompt 67 -> 20 ms. test_deepseek_v4_tiny/_prefix/_brio pass with both builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e expert group ordered after them by an event (V4_UPLOAD_THREAD, opt-in) A decode step attaches each expert it read from the store before it starts that lane's next read. A recycled mirror is the three tensors' copies and a wait for them on the compute stream, on the thread that feeds the lanes, so every upload delays a read. V4_UPLOAD_THREAD=1 hands the view to one uploader thread and starts the next read at once. The thread runs the same tiered attach as the step (v4_gpu_expert_attach_tiered: the same caches, growth guard, LRU and recycle in place), one view at a time. Its refills copy on the device's second stream and return without waiting (dsv4_cuda_tensor_refill_fp4 sync 3). Once the step's reads are in it joins the thread, records an event behind the copies on the second stream and makes the compute stream wait for it (dsv4_cuda_upload_fence), so the expert group runs after the copies on the device and the host waits for no transfer. Before the step releases its leases it waits for the second stream. V4_UPLOAD_FENCE=0 keeps the thread and has it drain the second stream after each mirror (sync 2), with no event. A backend without dsv4_cuda_upload_fence (an older DLL) is asked once; the uploader then copies on the compute stream and drains it, as the inline attach does. A new mirror, while the cache still grows, is a synchronous upload in every mode. The hybrid split (DSV4_HYBRID) keeps its own uploads. The step makes no backend call between its first hand-over and the join: a view the uploader refuses is attached inline only after a join. The backend is entered by one thread at a time, as before. Unset, nothing changes: the refill's two new sync values are only passed by the uploader, and the fence is only called after a hand-over. tests/test_v4_gpu_placement.c covers: the switch, a view of a layer no cache owns, three views handed over and their copies enqueued by the uploader with sync 3, one event per card at the join, the wait at the settle, a hit, V4_UPLOAD_FENCE=0, an event that fails, a backend without the entry, and the inline attach unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…the environment reference, and the experiment record docs/deepseek-v4.md: what the switch changes, where the copies go in each mode, the exit line, and the measurement on a 16 GB card. docs/experiments/v4-upload-thread-2026-11-12: the manifest and the raw logs. Three interleaved runs per arm, one 3,278-token prompt and 36 tokens. At DSV4_CUDA_VRAM_RESERVE_MB=600: 21.49 s against 20.34 s for the 35 tokens after the first, +5.6 % tok/s; with V4_UPLOAD_FENCE=0, 20.56 s. At the default reserve: 22.98 s against 21.62 s, +6.3 % tok/s. The text is the same in all fifteen runs. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
With formats by layer the main layers fell in two classes of rooms: the form most layers have at its own size, every other form in the room of the biggest of them. A file may give most of its layers other forms than the most common one: Flint 40B-A2.5B UD-Q4_K_XL has Q5_K experts on 16 of its 49 layers and six other forms on the 33 others, all of those in one room, 2.3 MiB an expert on average for 1.8 of tensors. - One class of rooms for each size the forms' tensors take (lcls, ncls; forms of the same bytes share a class), class 0 still the most common form's, so no tensor sits in a bigger form's room. Two sizes are held as before. - A newcomer displaces a resident of its own class first, then one of a class whose rooms hold it, the tightest first (fall, pick_victim): a smaller form's expert in a bigger form's room, never the other way. A candidate list per class (vcls_of: room class c at c + 1). - The budget counts the rooms on average over the layers and streaming slots take the biggest room of all, as before. - test_vk_tier: rooms, three layers in three forms of three sizes. A budget of the experts' own bytes holds them all (29 of 30 before), the smallest form's experts take the rooms the bigger forms' give up, and a bigger form's never a smaller one's; no upload refused. Radeon RX 9060 XT (16 GB), 32 GB of RAM, COLI_VK_STAGED=1 COLI_VK_TIER_HOST=0, each server on a copy of one history, three interleaved runs, medians: 5263 -> 6717 experts in 12.11 GiB, a 5899-token prompt's disk loads 16050 -> 8562, prompt 109.8 -> 167.5 tok/s, decode 8.2 -> 11.5 tok/s (docs/experiments/vk-tier-rooms-per-size-2026-11-12). Co-Authored-By: Claude <noreply@anthropic.com>
#2286) The launcher's KMP_BLOCKTIME=200 default for deepseek_v4 is now set on Windows only, and only when the resulting OMP_WAIT_POLICY is active. libomp gives passive its blocktime of 0 only while KMP_BLOCKTIME is unset, so the default turned a user's passive into 200 ms of spinning after every region. macOS's V4 also links libomp (Homebrew), where the change was not measured (#707), and Linux builds V4 with libgomp. coli's GLM path, which sets the same active policy on Windows, gets the same line: a clang colibri.exe now builds with the Windows flags. ci.yml's Windows job builds the default gcc deepseek-v4 again, and the release's XDNA=1 VK=1 CC=clang one after it, with the Vulkan headers and shaderc installed, so the clang compile of the Vulkan units and the LTO-less link are built on every PR. The Makefile's missing libomp.a warning points to docs/deepseek-v4.md. Docs and CHANGELOG: same tokens on the tiny fixture's oracle, the issue's 1.5x came from an LTO build, the Windows CUDA tier now spins (not measured). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…: both sets of rows in the environment table Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…SLAB_SHIFT, on; 0 = the move as before) A direct read of an expert lands a 4 KiB-aligned window of the file at the slab's first byte, and the record starts some bytes into that window. The reader then moved the record to the slab's first byte (a contiguous record), or the weights to the end of the scale run (scales and weights apart in the file): a memmove of the whole record, 12.6 MB for a V4 Flash expert, after every read, on the thread that read it. The slot now remembers where the record sits (data_shift for the scales, weight_shift for the weights) and fill_tensor_view adds it. Nothing is moved. - A contiguous record: one window, both shifts are its leading pad. - Scales and weights apart: the weights' window lands on the first page boundary past the scale run, the scales are read buffered to the slab's first byte as before. - A per-matrix or GGUF record, a buffered read, a direct read that fell back: both shifts are 0, as before. The reader resets them on every read. A slab holds 16 KiB past its record instead of 8 (COLI_V4_EXPERT_SLAB_SLACK): the scale run rounded up to a page, the window's pad and its round-up. The private read's caller (the Vulkan and CUDA expert tier) sizes its buffer with the same constant. V4_SLAB_SHIFT=0 restores the move. The bytes the views show are the same either way; only their address in the slab differs. tests/test_deepseek_v4.c: test_expert_store_slab_shift reads the fixture's expert with scales and weights apart and two contiguous ones through the direct path, then two of them again from RAM, with the switch unset and with 0. Every byte of every tensor is compared, and the weights' address says which path ran (their file offset modulo a page when kept, right after the scales when moved). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e experiment record docs/experiments/v4-slab-shift-2026-11-13: the manifest and the raw logs. Three interleaved runs per arm, one 3,278-token prompt and 36 tokens, the switch at 0 against 1. At the default mirror reserve: 22.15 s against 20.34 s for the 35 tokens after the first, +8.9 % tok/s, and 65.1 s against 58.5 s to the first token. At DSV4_CUDA_VRAM_RESERVE_MB=600: 20.75 s against 18.81 s, +10.3 % tok/s. The text is the same in all twelve runs. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The hot-team re-exec on FreeBSD ran `/proc/curproc/file`, which exists only when procfs is mounted; a default install does not mount it. The exec failed and the engine ran without the hot team, printing "re-exec failed, running untuned". coli_self_exe_path (`omp_tune.h`) gets the path from the kernel with the `kern.proc.pathname` sysctl, which works without procfs, and both re-exec sites use it, keeping `/proc/curproc/file` as a fallback. The CPU mask is deliberately not reset before the exec as it is on Linux, so a cpuset the user set is kept. Other platforms are unchanged.
coli_own_fopen opens with O_NOFOLLOW and documents ELOOP for a name that is a symlink. FreeBSD's open(2) returns EMLINK in that case, not POSIX's ELOOP (see its STANDARDS section). So on FreeBSD test_own_file failed, and coli_own_fopen_tmp (and its copy in route_trace.h), which checks for ELOOP before replacing a link left under the temp name, never replaced it: saving the state file failed instead. FreeBSD now maps EMLINK from that open to ELOOP. Other platforms are unchanged.
Four fixes, each a test that assumed Linux or a GNU toolchain: - Eleven tests ran `make`, which on the BSDs is BSD make and cannot read this GNU Makefile. They now use gmake there, and skip if it is missing; Linux and macOS still run make. - The e8x4g64 and fp8 repack harnesses mirror the Makefile's flags but lacked -pthread, which the Makefile has passed on the BSDs since #219. Without it pthread_create does not link on FreeBSD. - test_tok_load_validation now links with -lm, as the Makefile links every engine: json.h calls isfinite(), which on FreeBSD is a call to __isfinite in libm. - test_dsv41_cluster_protocol calls signal() without <signal.h>, which Linux and macOS pull in through <sys/wait.h> and FreeBSD does not. Also, check_prebuilt ran setup as if on Linux or Windows but used the real detect_cpu() of the test machine; where that reports no AVX2 (an older x86, or FreeBSD) the download tests failed. They now pin detect_cpu to a processor with AVX2, as the no-AVX2 test pins it to one without.
backend_cuda_dsv4.cu builds with hipcc through backend_gpu_compat.h (name mappings) and links into the engine like CUDA=1: make deepseek-v4 HIP=1 HIP_ARCH=gfx1102. The cooperative fused-MoE kernel is CUDA-only and compiled out. It needs ROCm 6.3+ for the OCP fp8 type; the fp8 names live in backend_cuda_dsv4_quant.cuh so the ROCm 6.2 HIP syntax check of backend_cuda.cu does not see them. The native engine read none of the GPU variables coli exports for V4. On the HIP build it now reads the names the other engines take on HIP too (GPU_BACKENDS.md): COLI_CUDA=0 (--gpu none), COLI_GPUS (--gpu N) and CUDA_EXPERT_GB (--vram N). --vram is the VRAM the engine may use: the expert mirrors grow until the card has that much less free than at open. The hold is a floor for the mirror guard's reserve, so the engine's own headroom stays, and an explicit DSV4_CUDA_VRAM_RESERVE_MB is a floor too. Below 10 GiB the dense/attention mirrors stay in RAM and the VRAM goes to the mirrors; a long prompt is prefilled on the CPU. With the mirror live, rows16 repacking turns itself off. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
libomp.a defines omp_get_num_devices() only as a COFF weak symbol, and GNU ld leaves it unresolved with MSYS2's llvm-openmp 22.1.7 or binutils 2.46, which is what GitHub's windows runner installs: `make deepseek-v4 CC=clang` failed at the link in #2805's CI, on libomp's own kmp_alloc and kmp_ftn_extra. Only llvm-openmp 23.x with binutils 2.47 links it. Makefile.deepseek-v4 now links one call to it the same way; only when that fails on this symbol does backend_loader_dsv4.c define it, returning 0, what libomp returns when no offload runtime is linked. Where it links, nothing is added, so there is no second definition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ced with dev New README.es.md, README.fr.md, README.de.md and README.ru.md, translated from the current README.md: code blocks, commands, flags, env vars, links, anchors and table structure unchanged, prose and headings translated, with the English-slug <a id> anchors the in-page links need (as README.it.md does). The six existing translations (it, ja, pt-BR, zh-CN, zh-TW, id) gain what README.md added since they were last synced: Kimi K3.1 Pro in the intro and the model table, the Qwen-Image-2.1 / Turbo GGUF row, "these models" in the GGUF intro, the joules sentence on the Profiling page and the "documents in" bullet (/file PATH). Also a few small fixes where a translation had drifted from the English (an extra "speculation" in the docs table's Tuning row, an added "new" in the DeltaNet bullet, a missing troubleshooting row in id). Every language switcher lists the four new languages. Korean is left to the open PR #2803. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`compat_meminfo_gb` checks for macOS and Windows; everything else read `/proc/meminfo`, which FreeBSD does not have unless `linprocfs` is mounted. On FreeBSD both numbers came back 0, so every engine fell back to its smallest expert cache, and `tests/test_compat_mem` failed. Compiling on FreeBSD now reads `hw.physmem` for the total and free + inactive pages (`vm.stats.vm.*`) for available, the counterpart of Linux's MemAvailable and of the macOS branch. The laundry queue and ZFS's ARC are left out, as MemAvailable leaves out memory that cannot be reclaimed quickly. `hw_probe` in `telemetry.h` had the same gap and now uses the same function. Linux, macOS and Windows are otherwise unchanged.
`resource_plan.memory_available` and `setup_hw.detect_memory` read `/proc/meminfo`, which FreeBSD does not have unless `linprocfs` is mounted, so on FreeBSD the planner had no memory figure (None) and setup knew neither the total nor what is available. Both now read the kernel's counters with sysctl: `hw.physmem` for the total, and free + inactive pages for available, the definition `compat_meminfo_gb` uses in C. `setup_hw` reuses resource_plan's parser. Other platforms are unchanged.
…code-telemetry
test_v4_gpu_placement.c: #2782's test_mirror_block and this test_bank_make_room were added at the same place; both are kept.
backend_cuda_dsv4.cu's Dev takes #2788's upload fence event and this one's buffers; coli_env.h keeps both sets of rows in order.
#2789 gives every form of expert a room of its own size, and the budget's count from the rooms' average. The staging of a prompt chunk's host pool experts took its share of a layer's experts at that average too, while a layer's experts keep their own form's size: on Qwen3.8-Flash-Next UD-IQ2_XS (five sizes) with COLI_VK_TIER_GB=6 on an RTX PRO 4000 it shrank from 1019 to 840 MiB, more of a prompt's experts were read from host memory, and the 3036-token prompt went from 262.5 to 255.4 tok/s (three runs a side, decode 23.38 -> 23.87). The staging is now sized per expert as it was before (a layer of a form other than most layers' at those forms' biggest tensors), and the budget gives its experts up for those bytes. With one or two sizes that is the rooms' average, so nothing changes there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nineteen PRs on dev after #2809, each merged with its own commits. Four merges needed a hand (below), and one commit of ours sizes the Vulkan tier's prompt staging so that #2789 costs no prompt speed.
What is in it
DeepSeek V4
V4_SLAB_SHIFT, on;0= the move as before).V4_LAYER_FUSED, opt-in), with a verifier beside the usual path.V4_MIRROR_BLOCK, opt-in).V4_BANK_MAKE_ROOM, opt-in).V4_UPLOAD_THREAD, opt-in).make deepseek-v4 HIP=1), and--gpu/--vramreaching it.deepseek_v4.exebuilt with clang and LLVM's libomp, whose barriers spin (#2286), with a link probe for MSYS2's llvm-openmp 22.1.7.Qwen3.8, Qwen3.6 and the tiers
cc_attn_fk) stages K and V with 16-byte loads, the scalar loop's bits.Q38_PREFIX_DISK=1, opt-in).COLI_TIMERS=1, each served request's prefill and decode banks and where its experts came from.Platforms, release and web
kern.proc.pathname, without procfs.O_NOFOLLOWon a link reportsELOOP, as on the other platforms.-pthread,-lm,<signal.h>, a pinned CPU in the prebuilt tests).linux-aarch64release archive (plain armv8-a, CPU only), and setup downloads it on ARM64 Linux.README.md.By @dcutugno (#2778, #2782, #2784, #2788, #2796), @KairaCode (#2762, #2781, #2800, #2811), @risner (#2790, #2798, #2799, #2802), @CoTi1938 (#2434), @kflirik (#2603), @wenmiI (#2789), @ZacharyZcR (#2714), @divya0795 (#2742) and @tylnexttime (#2740).
Merges resolved by hand
q38_ps_savekeeps qwen38 prefix store: with COLI_PREFIX_LOG, each checkpoint save says whether it was kept #2780'sCOLI_PREFIX_LOGlines and returns whether the checkpoint was kept, as qwen38: keep the prefix store's checkpoints on disk across engine runs (opt-in, Q38_PREFIX_DISK=1) #2781 needs. Its RAM take keeps qwen38 prefix store: with the MTP head on, a take leaves in place only the blocks below the row the slot's next token made #2770's rule for the MTP head's row; a take from disk puts every row back from the file.test_v4_gpu_placement.c,backend_loader_dsv4.c,coli_env.h(in order),docs/deepseek-v4.mdandDevinbackend_cuda_dsv4.cu. The fake refill in the placement test counts deepseek-v4: a decode mirror as two runs, uploaded with two copies (V4_MIRROR_BLOCK, off by default) #2782's refills and deepseek-v4: a decode step's mirror uploads on an uploader thread, the expert group ordered after them by an event (V4_UPLOAD_THREAD, off by default) #2788's syncs and threads.With
V4_MIRROR_BLOCK=1andV4_UPLOAD_THREAD=1together, the two-run refill still drains the compute stream on the uploader thread, as #2782 wrote it: the copies are done before the leases go, and that combination does not get #2788's enqueue-only copies.Ours: the prompt staging under #2789
#2789 sizes the budget's count from the rooms' average. The staging of a prompt chunk's host pool experts took its share of a layer's experts at that average too, while each layer's experts keep their own form's size. On Qwen3.8-Flash-Next UD-IQ2_XS (five sizes) with
COLI_VK_TIER_GB=6on an RTX PRO 4000 the staging shrank from 1019 to 840 MiB, more of a prompt's experts were read from host memory, and the prompt lost 2.7 %. The staging is now sized per expert as before #2789; with one or two sizes nothing changes.(dev's two numbers are from the two sessions, each against its batch arm.)
Tests
ds (Ryzen, 16 threads)
make check: every C test and 3424 Python tests pass.qwen38-tiny-check,-fp8-,-tier-engine-,-dn-gpu-,-cuda-chain-,-mtp-): 8 of 8 tokens in all 36 runs.test_v4_gpu_placementwith the hand-merged fakes.deepseek-v4-tiny-check(token-exact against the reference, prefix reuse, read-only), and the oracle again withV4_SLAB_SHIFT=0and withV4_LAYER_FUSED=1.RTX PRO 4000 Blackwell (sm_120, CUDA 13.1), i5-13500
make dsv4-cuda-test,make cuda-chain-check(334 ok), andcuda-chain-checkagain withCOLI_CUDA_CHAIN_FLASH2=0, where CUDA chain: the prompt attention stages K and V with 16-byte loads #2434's staging runs: the scalar loop's bits from a float32 and a float16 cache.make vk-tier-check VK=1, vk_tier: every form's experts in rooms of its own size #2789'sroomscase among them.colibri,qwen38anddeepseek_v4build withCUDA=1, with no warning dev's build does not have.COLI_CUDA_CHAIN_FLASH2=0A first session of two runs a side gave the same, except one branch run of Qwen3.8 on CUDA at 9.75 tok/s (10.58 to 10.77 in the other nine); the three-a-side session above was run because of it.
Not run here: DeepSeek V4 on a model (none on our machines), so #2796's and the opt-in V4 paths' speed is the authors' measurement; HIP (#2603); FreeBSD (#2790, #2798, #2799, #2802); the Windows builds (#2800, CI's Windows jobs); the
linux-aarch64archive (the release workflow runs on this PR).