Skip to content

Batch 2026-11-13-b: twenty-eight fixes (DeepSeek V4 CUDA, qwen38 chain and prefix store, server, Windows builds and tests) - #2808

Merged
JustVugg merged 63 commits into
devfrom
batch/2026-11-13-b
Nov 13, 2026
Merged

JustVugg merged 63 commits into
devfrom
batch/2026-11-13-b

Conversation

@JustVugg

Copy link
Copy Markdown
Owner

Twenty-eight PRs, fixes and the tests and builds around them, each merged with its own commits. None conflicts with another; one commit of ours on top (below).

What is in it

DeepSeek V4

PR What it fixes
#2728 The batched MoE union's loader lanes are capped at the layer's lease capacity: with lanes + 1 above the layer's slots the last lookups found no slot and the union failed.
#2729 The exact CUDA head is packed through a staging buffer of at most 1024 rows instead of a second full copy.
#2745 DeepGEMM scales are packed only for shapes the DeepGEMM kernels take (no empty grid, no division by zero).
#2752 An absolute COLI_DSV4_DLL path is loaded as given, and the current directory is out of the CUDA runtime's search order.
#2755 cuda-dsv4-dll on Windows accepts CUDA_ARCH=portable-pre-ampere and NO_TC=1.
#2760 portable-pre-ampere needs CUDA <= 12.9, said wherever the preset is recommended.
#2763 The exact CUDA head is uploaded even when the RAM plan streams it (read for the upload, then freed).
#2767 The generic 128-block fp8 matvecs on CUDA round their input to E4M3 per 128, as the CPU reference and the fused kernels do (DSV4_CUDA_ACT_QUANT=0 keeps the raw input).

Qwen3.8 and Qwen3.6

PR What it fixes
#2746 The CUDA chain's device memory is kept out of the FP8 expert tier's budget, as the GGUF arena already does.
#2747 A device-memory refusal in the CUDA chain gives the K/V mirror back instead of holding it for the process.
#2748 A placed trunk matrix's BF16 rows are freed as soon as its int8 copy exists.
#2761 Text : on its own line after the reference's C engine : ids.
#2768 A picture's rows go with a request refused with CONTEXT_EXCEEDED.
#2770 With the MTP head on, a prefix-store take keeps only the blocks below the row the slot's next token made.
#2780 With COLI_PREFIX_LOG, each checkpoint save says whether it was kept.

Prefix store, shared by eight engines

PR What it fixes
#2766 A save its budget cannot hold is refused before it evicts anything, a failed save takes back its blocks, the RAM floor holds across a whole save, and a chain's system-prompt checkpoint is kept while the chain is in use.

Server and launcher

PR What it fixes
#2730 --ram reaches qwen38 as RAM_GB too.
#2732 DeepSeek V4.1: a message's content blocks join on a blank line, as encoding.py does.
#2771 Mistral-Large-4 vision: a picture announced for a refused request does not go to the next one.
#2779 keep_engine takes the context and max_tokens of the restarted engine.

Builds and tests

PR What it fixes
#2731 The CUDA chain tests build with nvcc's MSVC host (setenv shim).
#2737 The OpenMP tests build without libomp.
#2739 CUDA builds without nvcc warning #221 on Windows, and without an unused ANS-only global.
#2744 c/deepseek_v4.ldflags is ignored and cleaned.
#2751 The qwen38 tests keep compat.h's setenv on Windows (MinGW).
#2754 The int2/int4 converter tests delete their temp directories on Windows.
#2756 datapoint.py refuses non-finite memory budgets.
#2769 test_qwen38_prefix_serve reads the reuse source without Windows' CR.

All by @KairaCode except #2737 (@divya0795), #2739 (@tylnexttime) and #2756 (@Stamina9).

Ours: #2755's new row in docs/deepseek-v4.md recommends the pre-Ampere preset and #2760's test wants every such place to name CUDA <= 12.9. The row now does.

Tests

KairaCode and others added 30 commits November 9, 2026 18:54
…lease capacity

v4_moe_batch_union keeps dual_loader_lanes() reads in flight, and each of
them holds an expert-cache lease until the expert is computed; the expert
being computed holds one more. The hot store has min(experts, 6) slots per
layer at its smallest, and without the prefill pool a layer can lease only
its own slots: lookup_hot returns -1 once every slot in the partition is
leased. With 9 lanes (the default; up to 16 via V4_LOADER_LANES) and 9 or
fewer slots a layer, the union's later reads found no free slot and the
prefill failed with "reading expert N from the expert store failed".

The union now caps its lanes at coli_v4_expert_store_lease_capacity() - 1.
The new function returns the layer's slots, or every slot while the prefill
pool lends them to this layer (what hot_oldest_pool_victim's last-resort
fallback can reach), and 0 for an alternative backend, which leaves the
lanes as they were. Runs with more slots than lanes are unchanged.

test_deepseek_v4: test_expert_store_lease_capacity pins the store contract
the cap relies on: 6 concurrent leases on a 6-slot layer, the 7th refused,
12 while pooled (8 leased), 6 again after the pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ffer

dsv4_cuda_upload_head_exact built the packed head through
dsv4_cuda_upload_compressor: a full row-major copy, then the packed copy,
then a free of the first. Both copies were live at once, so the upload
peaked at twice the head (2020 MiB for V4-Flash's 129280x4096 bf16 head),
and coli_v4_gpu_head_upload asked for rows*cols*4 bytes above the resident
reserve before trying.

Upload the head in chunks of at most 1024 rows into a device staging
buffer and pack each chunk straight into the final allocation with the
existing pack_bf16_rows32 kernel, at offset r*I (r is a multiple of 32).
The peak is the head plus the staging buffer (1018 MiB for V4-Flash), and
the placement check now asks for that. Same packed layout, same scoring
kernels, same preconditions: the logits do not change.

test_dsv4_head_cuda gains a 2080x256 shape in its --small set, so the
small run crosses two chunk boundaries and a 32-row tail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#2356 added qwen36 to env_for_engine's RAM_GB tuple because the gateway
turns RAM_GB into Qwen's positional expert-cache cap. The gateway's cap
planner reads RAM_GB the same way for qwen38 (openai_server: arch in
deepseek_v41, mimo, qwen36, qwen38), but coli never forwarded the flag
for that arch, so `coli serve/chat --ram N` on Qwen3.8 was ignored and
the planner used an inherited RAM_GB or measured RAM instead.

Add qwen38 to the tuple and run the existing regression test for both
Qwen archs as subTests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/test_cuda_chain.cu and tests/test_cuda_dsv4c.cu call setenv/unsetenv,
which MSVC does not declare. On Windows, nvcc uses cl.exe as its host
compiler, so `make cuda-chain-check` (and `make gpu-compile`) stopped at
`identifier "setenv" is undefined`. Both files now carry the same _WIN32
shim over _putenv_s that tests/test_backend_cuda.cu and
tests/test_fp8_warp_cuda.cu already use. An empty _putenv_s value removes
the variable, so unsetenv keeps its meaning. Test-only. Builds that do not
define _WIN32 are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The V4.1 image expander joined a message's text parts and image
placeholder spans with no separator. The checkpoint's encoding.py
(process_image_messages) treats each text part and each image as its own
block and joins the blocks with "\n\n", so a multi-part or image request
reached the engine with different prompt bytes than the vendor encoder
produces. Join with "\n\n" as the reference does.

Tests: the reference's encoding case 5 (two images between three text
parts, thinking mode, effort max) is now copied into tests/fixtures and
run through expand_dsv41_images and the renderer; it is byte-identical
after folding each placeholder span to the single placeholder the
reference writes. The fixture reader opens the files as UTF-8, which the
CJK text of case 5 needs on Windows. The text-part test in
test_openai_server expects the blank line for deepseek_v41.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Makefile builds single-threaded on a Mac without libomp, but five
tests included <omp.h> unguarded and test_glm53_live_prefix called
omp_set_num_threads, so make check stopped there. Guard them with
#ifdef _OPENMP like test_omp_tune.c, and check every tests/test_*.c
for it (tests/test_omp_guard_source.py): CI only builds with OpenMP.
…ly global

The Windows SDK UCRT defines INFINITY as ((float)(1e+300)), so device code
that writes -INFINITY makes nvcc warn #221 at every instantiation (45 in
coli_cuda.dll). Use -__int_as_float(0x7f800000) at the four sites, the
spelling cuda_dsv4c.cuh already has; the value is the same -inf.

g_cuda_raw_experts is read and written only under COLI_ANS; define it
under the same guard so a COLI_CUDA build without COLI_ANS does not warn
that it is unused.

Co-Authored-By: Claude <noreply@anthropic.com>
#2652 added V4_LINK_STAMP = deepseek_v4.ldflags (Makefile.deepseek-v4),
written into c/ by every `make deepseek-v4`. Its two siblings,
deepseek_v4.cflags and deepseek_v4.cudaflags, are listed in the root
.gitignore and in tools/clean.py; the new one is in neither. So a
standalone V4 build leaves `?? c/deepseek_v4.ldflags` in `git status`,
and `make clean` (tools/clean.py) leaves the file behind.

Add the entry to both lists, and a test_clean_ownership case that reads
every V4_*STAMP name from Makefile.deepseek-v4 and requires each one in
clean.py's FILES and in .gitignore, so the next stamp cannot repeat it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
q38cc_mirror allocates every attention layer's K/V, index-key and
pooled-key mirrors at the host's capacity. When one cc_buf failed it set
cap = 0 and returned, leaving the layers before it (and the part of the
failing layer that did allocate) on the card. q38cc_forward then logged
"device memory for N rows refused; the per-matrix path" and set
ch->failed, which no code resets and every chain entry point checks: the
mirrors, the largest of the chain's buffers, stayed on the card unused
until the process exited, VRAM the per-matrix path and the expert tier
could no longer use (and, under WDDM, process commit).

q38cc_mirror_free releases every attention layer's mirrors (the MTP
head's slot included) and resets cap and the watermarks. q38cc_mirror
calls it on a refusal, so the mirror is all or nothing, and
q38cc_forward calls it before turning the chain off. The host's K/V rows
are canonical, so nothing is lost; a later q38cc_mirror (prewarm, MTP
head) uploads them again.

test_qwen38_cuda_chain: a chain written by hand on the fake card with
fake_chain_dev_cap set so the second attention layer's mirror is
refused, then so every mirror fits and a row's scratch does not; both
forwards return 0 with the card's bytes back where they were, and a later
forward takes nothing. The three checks fail on dev.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s take

upload() in DeepGEMM builds packed DeepGEMM scales for every fmt 8/9
tensor with O and I multiples of 128. The packing kernels work in packs
of four 128-column groups (I/512), and the fmt 9 path splits O into
1024-row groups:

- I < 512 (I = 128, 256, 384): groups/4 is 0, the packing grid is empty,
  the launch fails with "invalid argument" and the whole upload returns
  0, so the tensor silently stays on the CPU.
- fmt 9 with O < 1024: groups = O/1024 = 0 and O/groups divides by zero
  on the host, which kills the process. fmt 9 is attn.wo_a with the
  batched attention path on.

No V4-Flash shape is affected, but small models (the V4 tiny fixture,
all 128x128) fail the upload or crash. Every DeepGEMM consumer needs
I % 512 == 0 (and the grouped wo_a path N == 1024), and refuses a tensor
whose dg_scale is NULL, so pack only those shapes. Other tensors keep
dg_scale NULL and run on the generic kernels. The packing of the real
shapes is unchanged.

test_dsv4_dense_batch_cuda now uploads small fp8 and fp8-bf16 tensors
(128x128 up to 512x512) and checks a matvec on each against an exact
host reference; on a DeepGEMM build it failed the 128x128 upload before
this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r's budget

The GGUF path offers the chain's memory (q38cc_reserve_bytes) to the placer
as a component of its own ("chain") before the tier sizes its expert budget.
The native FP8 path did not: qt_init_fp8 took free minus 1 GB minus the
trunk as its budget, and the chain, set up after q38_tier_start, allocated
its K/V mirrors, DeltaNet state, scratch and own matrices into the room the
promotions were filling. Promotions run until a cudaMalloc fails, so the
chain's later growth (layer-major buffers, a verify's copies, larger
read-backs) is refused and q38cc_forward turns the chain off for the
process; with the driver's sysmem fallback the uploads spill to host memory
instead.

Move the GGUF branch's offer into q38_tier_offer_chain and call it from the
FP8 branch too. qt_reserve_stage stays GGUF-only. This is the qwen38
counterpart of what qwen36's main does with qt_reserve (#2741).

test_qwen38_tier_engine: a pass with COLI_CUDA_CHAIN=1 checks that the
chain's offer is placed on the card and that the expert budget is the
chain-off budget less its bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… card

The capacity growth case the fix covers had no check: place the mirror
at 32 positions, then grow it to 64 with room for the first attention
layer's new mirrors only. q38cc_mirror returns 0 with cap 0, no mirror
pointer and the card's bytes back where they were. On dev the last
layer's old mirrors and the second layer's half stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
q38cc_reserve_bytes's comment said the "chain" offer exists so a GGUF's
arena leaves it; q38_tier_offer_chain now makes the same offer before the
FP8 tier sizes its promotions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
q38_trunk_place_all quantizes every matrix it offers to the CUDA tier to
int8 and kept its BF16 rows; they were freed only later, in
q38_trunk_cpu_int8, after q38_tier_start returned. With COLI_CUDA=1,
FP8 experts and a BF16 trunk, the load therefore held the whole BF16
trunk plus the int8 copy of every placed matrix at once, a transient the
gateway's planner (which prices the trunk at its int8 steady state) does
not budget.

Every CPU read of a placed matrix already goes through q8
(q38_weight_matmul takes q8 before data), so free the BF16 rows right
after placement, with the same resident_weight_bytes bookkeeping as
q38_trunk_cpu_int8, which then skips those matrices. Gated on
q38_trunk_cpu_int8_wanted(): Q38_TRUNK_CPU_INT8=0 still keeps BF16.
Block-format (GGUF-quantized) trunk items skip this code; a GGUF's
BF16/F32 trunk matrices take the new path and end in the same state.
The placement log reports the GiB released.

test_qwen38_tier_engine: after the trunk-on-the-fake-tier pass, run
model_init + q38_tier_start directly and check that every placed matrix
has q8, no BF16 rows and the int8 resident_weight_bytes, and that
Q38_TRUNK_CPU_INT8=0 keeps the rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_qwen38_tier_engine, test_qwen38_dn_gpu and test_qwen38_cuda_chain
#undef setenv inside their _WIN32 block. compat.h already maps setenv to
compat_setenv there (and c6d6b10 dropped the per-test helpers for that
reason), so after the #undef the tests call an undeclared setenv and do
not compile with MinGW gcc. Keep the unsetenv override, drop the #undef.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runtime search

dsv4_load_one glued COLI_DSV4_DLL onto the engine's folder, so an absolute
path (C:/x/coli_cuda_dsv4.dll or C:\x\...) never loaded as given; the by-name
fallback (application dir + System32) then found nothing or found the DLL
without its CUDA runtime on PATH: error 126, "continuing-CPU", a silent ~3x
slower run.

- A name with '\', '/' or ':' is an explicit file: GetFullPathNameA (which also
  turns '/' into '\') and LOAD_WITH_ALTERED_SEARCH_PATH, so its imports resolve
  beside it, then System32 and PATH. A path over MAX_PATH says "path too long
  or invalid" instead of a stale error. Bare names keep the engine-folder load.
- SetDllDirectoryA("") before loading: the altered search order looks in the
  CWD before PATH, so a cudart64_*.dll / cublasLt64_*.dll there was loaded
  into the engine (silent CPU fallback on a bogus one).
- The failure names the CUDA runtime generically (cudart64_*.dll, plus
  cublasLt64_*.dll for a DSV4_TC=1 build) and the path that was tried; the
  success line prints the loaded DLL's path.

tests/test_dsv4_dll_path.py (make dsv4-dll-path-check DSV4_DLL=...): a real
DLL outside the engine's folder with the runtime only on PATH loads by '/'
and '\' path; runtime off PATH and an over-long path are named; the default
still loads beside the engine with a fake runtime in the CWD; a bare name
ignores a DLL in the CWD. Skips without DSV4_TEST_DLL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A relative value (sub\coli_cuda_dsv4.dll) resolved against the engine's
folder on dev; the previous commit sent any name with '\', '/' or ':' to
GetFullPathNameA, which resolves it against the CWD instead. Only a drive
path (C:\x, C:/x) or a UNC / \?\ path is now an explicit file; anything
else is glued onto the engine's folder as before.

test_relative_path_stays_under_engine: sub\<dll> beside a copied engine
loads from there with the CWD elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The DeepSeek V4 loader and resource_plan tell Pascal/Turing users to build
with CUDA_ARCH=portable-pre-ampere NO_TC=1, but on Windows the DLL is built
by c/Makefile's cuda-dsv4-dll, which passed the preset to nvcc as -arch
(fatal: Unsupported gpu architecture) and read only DSV4_TC. The preset and
NO_TC lived only in Makefile.deepseek-v4's Linux/macOS block.

c/Makefile now expands portable-pre-ampere to the same gencode list as
Makefile.deepseek-v4 and maps NO_TC=1 to DSV4_TC=0; the Windows DLL table
in docs/deepseek-v4.md gains the row, and a make dry-run test pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndows

test_qwen38_int2_convert kept CovStats (and its np.memmap of cov.bin)
alive when its TemporaryDirectory closed. test_qwen38_int4_convert runs
convert(workers=1), which builds the worker's Source in the test process;
Source opens the shards with os.open and never closes them, and every
convert() call leaves a new one behind. Windows cannot delete an open
file, so the temp-dir cleanup raised WinError 32 in 5 tests. Drop the
memmap, and close the in-process Source's descriptors after each convert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it is recommended

The preset's gencode list starts at compute_61, which CUDA 13 dropped
(nvcc 13 lists compute_75 first), so nvcc stops at once with
"Unsupported gpu architecture 'compute_61'". The loader's no-usable-DLL
message, the backend's no-kernel-image reason, resource_plan's
CUDA_TIER_FLOOR hint, Makefile.deepseek-v4 and docs/deepseek-v4.md all
sent Pascal and Turing users to the preset without that limit. They now
say "CUDA <= 12.9"; the Makefile comment and the doc add that a Turing
card builds on CUDA 13 with CUDA_ARCH=sm_75 NO_TC=1.

A test reads the preset's oldest compute_XX from Makefile.deepseek-v4,
takes the last release that builds it from setup_flow.CUDA_RELEASES (the
measured nvcc table) and checks every hint names that limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a tokenizer loaded, a reference run printed the decoded text on the
"C engine :" line itself ("C engine : 1 2 3 Text      : ..."), followed
by a blank line before "Matching tokens". The text now starts its own
line and the blank line is gone; without a tokenizer the output is
byte-identical.

spec_drafts_harness.engine_tokens now accepts only ids on that line
instead of cutting at "Text", so the qwen38/qwen36 MTP and spec-draft
gates (whose fixtures carry a tokenizer) catch it if text comes back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s it

On a low-RAM box the RAM plan streams the BF16 head, and
coli_v4_gpu_head_upload returned early without a host copy: the head ran on
the CPU and read 1.01 GiB from disk every token, whatever the free VRAM.

- coli_v4_gpu_head_upload reads a streamed head for the upload and frees the
  host copy afterwards, uploaded or refused. It reads only for a BF16
  vocab x hidden head, a backend with the exact head, a device with room and
  head + 512 MiB of available host RAM; otherwise it logs why.
- head_scores_impl tries the device head before streaming.
- dsv4_cuda_head_exact_available: optional export (dsv4.def, loader), so a
  DLL built before this keeps the old behaviour.
- test_v4_gpu_placement: borrowed head refused, uploaded, unreadable, and not
  read when the upload cannot happen.

V4-Flash, RAM_GB=11 (head streamed): head time 289 -> 5.9 ms/token warm,
1280 -> 21 ms/token on the first turn, same text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…out the CR that Windows' stderr adds

The engine's stderr lines end in \r\n under MinGW, and prefix_serve_harness
decodes them without newline translation, so REUSED's "from (.+)" captured
"the live state\r" and six tests failed on Windows. Stop the capture at \r or \n.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ave, hold the RAM floor across a save, keep a chain's root checkpoint

- coli_ps_save refuses a save whose chain and state (coli_ps_cost: every
  position's rows and token, and the state) exceed the budget before it
  evicts anything. Before, it added the blocks one by one, evicting other
  entries for each, then failed at the checkpoint and left the blocks
  behind; every later save on that chain did the same.
- A save that fails (the RAM floor, an allocation) takes back the blocks it
  added (coli_ps_unwind): without their checkpoint they are only RAM.
- The free-RAM floor holds across a whole save: coli_ps_room lowers
  avail_mb by every grant and raises it by every eviction. It was checked
  per entry against the same value, so one save could commit the whole
  budget below COLI_PS_RAM_FLOOR_MB.
- A chain's shallowest checkpoint (where its system prompt ends) is marked
  used whenever the chain is, by a save and by a lookup
  (coli_ps_touch_root), and between checkpoints used at the same time the
  deeper is evicted first. A turn's end no longer evicts the system
  prompt's checkpoint as the oldest entry.

tests/test_prefix_store: one check for each; 5 of them fail on the old
header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y the blocks below the row the slot's next token made

q38_ps_take kept in place every whole block the slot's record shares with
the prompt. The MTP head's row at p pairs the streams at p with the token at
p + 1, so its row at common - 1 was made with the record's token at common,
not the prompt's. When the two part exactly at a block boundary, that row
stayed. With the head on, keep is now at most common - 1: the block that
ends there is loaded again from the store. Drafts are verified, so only
their acceptance was affected, never the output.

Test: test_mtp_head_row_where_the_slot_parts_from_the_prompt (a system
prompt that differs from the stored one at token 32): 16 rows in place with
the head on (32 on dev), 32 with it off; the answers a fresh read's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_EXCEEDED

q38_serve_start attaches the request's image (q38_vision_attach) before the
context check. The CONTEXT_EXCEEDED refusal returned without
q38_vision_detach, and both callers (serve_one_turn and the KV_SLOTS mux)
return on a failed start before their own detach. The picture's map stayed
for the next request: its prompt read the picture's rows in place of its
own tokens at those absolute positions (q38_embed_row), its prefix reuse
was tainted, and the prefix store stayed off for it.

The refusal now detaches the picture. test_qwen38_vision_serve.py sends the
same text-only prompt before and after an image request refused for context
and expects the same answer; turn() now keeps each DATA frame's bytes so
two requests with the same id compare text, not only lengths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…quest its IMAGE frame names

An IMAGE frame was kept for whatever SUBMIT came next. A SUBMIT refused
before the picture was attached (CONTEXT_EXCEEDED, EMPTY_PROMPT,
BAD_REQUEST, SLOT_BUSY, DUPLICATE_ID) left it pending, and the next
request, text only, went into mlv_attach and was answered ERROR BAD_IMAGE.

The pending picture now keeps its IMAGE frame's id, as glm53 does; at
SUBMIT a picture with another id is dropped with a stderr line before the
attach. tests/test_ml4_pending_picture.py drives it on the tiny ML4 model
and tower, wired into the Mistral-Large-4 CI step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s to E4M3 per 128 on the device

coli_fp8_matvec_ref and coli_fp8_matmul_batch_ref round every fp8 input to
E4M3 per 128 (coli_fp8_activation_qdq_ref) before they multiply, and the
fused CUDA kernels run fp8_sim before run_mv<8>. A mirrored weight is sent
to the GPU with the raw input, and dsv4_cuda_matvec / _matvec_grouped,
dsv4_cuda_wo_decode (wo_a input and the bf16 intermediate), _proj_batch and
the generic _matmul_batch multiplied it as it came. So CUDA decode used other
operands than the CPU build and the fused path.

act_quant_input runs fp8_sim per 128 on the device copy before those
kernels: per group in _matvec_grouped (as the CPU group views), on the wo_a
input and after bf16_round in _wo_decode, once in _proj_batch (all fp8),
and in _matmul_batch on a device copy, so the caller's activation stays raw
(dsv4_cuda_moe_activation does the same). The DeepGEMM branch of
_matmul_batch is left alone: its kernels round. fp4 (run_mv<4>) is not
touched. DSV4_CUDA_ACT_QUANT=0 keeps the raw input.

tests/test_dsv4_act_quant_cuda.c checks each call bit for bit against
coli_fp8_matvec_ref on operands whose sums are exact in any order, and that
each call differs under DSV4_CUDA_ACT_QUANT=0; it joins dsv4-cuda-test.
tests/test_dsv4_gpu_grouped.c rounded nothing in its CPU reference; it now
rounds x per 128 per group unless DSV4_CUDA_ACT_QUANT=0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…9 too

#2755 added the row and #2760 the check that every place recommending the
portable-pre-ampere preset names the last toolkit that builds it.
@JustVugg
JustVugg merged commit e47b2ba into dev Nov 13, 2026
117 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants