Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
008a01b
deepseek_v41: plan the RAM workspace at one prompt chunk, not the ful…
jameswitney Nov 6, 2026
c615cca
Vulkan: a turn-end VRAM ledger, 48 streaming slots reserved before th…
jameswitney Nov 6, 2026
90959d6
Vulkan: a tier pool's budget is page-granular, so a three-expert tier…
jameswitney Nov 6, 2026
955a0f6
tests/vulkan_engines: the DSpark stages' default check is for a devic…
jameswitney Nov 6, 2026
12b2908
vulkan: the candidate mask is one bit a column, so the chunk's quadra…
jameswitney Nov 6, 2026
a6486c7
docs: the packed mask's saving on the released checkpoint's row cost
jameswitney Nov 6, 2026
9508327
Merge fix/ram-plan-128k: deepseek_v41's RAM workspace is one prompt c…
jameswitney Nov 6, 2026
05d9b97
tests/context_compaction: the v41 long-context table a chunk-priced w…
jameswitney Nov 6, 2026
e2cfaa3
dsv41: the prefill VRAM wave's experiment record, and the two documen…
jameswitney Nov 6, 2026
ab1fe3f
dsv41: the record's gate table, from the run on this tree
jameswitney Nov 6, 2026
6d4e6c2
review: the DSpark guard matches the tier line's real wording; docs a…
jameswitney Nov 6, 2026
e8f0bc5
dsv41: the ledger prints the card's gtt as well as its vram
jameswitney Nov 8, 2026
3f3eb73
dsv41: COLI_VK_KV_QUANT's device KV mirrors in the checkpoint's cache…
jameswitney Nov 8, 2026
6cae18b
dsv41: the kv-quant tests -- the codec bit for bit, the approximate m…
jameswitney Nov 8, 2026
b49b29d
docs: COLI_VK_KV_QUANT -- the quantized KV mirrors, what they cost an…
jameswitney Nov 8, 2026
62450e2
dsv41: kv-quant under the engine -- vkc_write's offsets are floats, t…
jameswitney Nov 8, 2026
f572efd
kv-quant: the review's corrections -- one basis for the mirror number…
jameswitney Nov 8, 2026
de8a3fc
kv-quant host: the block rules and their tests -- staging through 68 …
jameswitney Nov 8, 2026
685c4ed
kv-quant host: the storage swap, the writers and the readers -- the h…
jameswitney Nov 8, 2026
60d143c
kv-quant host: the planner prices the packed host -- 890 B a position…
jameswitney Nov 8, 2026
cbae3e8
kv-quant host: the ledger's host-kv term -- the flag's host savings o…
jameswitney Nov 8, 2026
4e2d182
kv-quant host: the =2 gate arms -- the family, the split, DSpark's un…
jameswitney Nov 8, 2026
b0b4517
kv-quant host: the split's cold staging strides in bytes now, and v4'…
jameswitney Nov 8, 2026
685d380
kv-quant host: the kv-read timer -- the decode-on-read cost its own l…
jameswitney Nov 8, 2026
230af87
kv-quant host: the docs -- =2's row, the honest table, the map's refe…
jameswitney Nov 8, 2026
8a8fb6c
kv-quant host: the review's nits -- the state probe stays fp32-only, …
jameswitney Nov 8, 2026
a58492f
merge origin/dev: the env registry's rows, and the host codes inside …
jameswitney Nov 9, 2026
38a6c3d
docs: the 890 B row is the packed rows alone -- the window rings are …
jameswitney Nov 9, 2026
92201a6
dsv41: the DeepInfer-style packed cache is the default -- COLI_VK_KV_…
jameswitney Nov 9, 2026
5c4eb90
dsv41: kv-quant's ckv scale demotes a step in the near-dead band -- t…
jameswitney Nov 9, 2026
fae79e4
the planner's mask row and the tier gates meet the packed mask -- the…
jameswitney Nov 9, 2026
ed323e1
dsv41: one flag for the packed cache -- COLI_VK_KV_FP32=1 is the exac…
jameswitney Nov 10, 2026
2df7eb7
dsv41: the planner prices the host's rings as the codes it holds -- a…
jameswitney Nov 10, 2026
6de97a3
kv-quant tests: the host half is not a skip, and its fixpoint runs wi…
jameswitney Nov 10, 2026
d10c0a5
Merge origin/dev into pr2712 (conflict: c/cu_tier.c alias block -- ke…
jameswitney Nov 10, 2026
382dfb4
dsv41: the planner prices the DSpark stages and rollback rows; the pa…
jameswitney Nov 11, 2026
bad6358
docs/experiments: the packed KV cache's 12-prompt A/B and a measured …
jameswitney Nov 11, 2026
96a6d3b
dsv41: a stage count without a draft block or target prices no stage …
jameswitney Nov 11, 2026
80d9cf8
Merge dev (15f810f0) into r2b/kv-quant-host
Nov 12, 2026
e141623
cuda: the V4.1 CUDA chain declines a packed host and byte-strides its…
jameswitney Nov 12, 2026
e798600
v41: the packed ring's snapshot takes the slot's stored code row, not…
jameswitney Nov 12, 2026
1029085
tests: the packed spec arm accepts its own drafts and emits the strea…
jameswitney Nov 12, 2026
19b4249
planner: price V4.1's prompt workspace at what a forward of n rows al…
jameswitney Nov 12, 2026
b9b145a
tests: the packed read paths the suite never ran
jameswitney Nov 12, 2026
d7745e4
tests: the index-score gates from the measured band
jameswitney Nov 12, 2026
616a366
ledger: the host's KV bytes, not one decimal
jameswitney Nov 12, 2026
162b91c
docs: the host holds code rows; the DI band and its costs
jameswitney Nov 12, 2026
2072865
tests: an ASan/UBSan leg for the packed host
jameswitney Nov 12, 2026
8f05d12
planner: the review's three nits on the workspace price
jameswitney Nov 12, 2026
43501e8
v41: a prefill reads the code row it stores, not the floats it packed…
jameswitney Nov 12, 2026
8f1d2a7
v41: the prefix store keeps code rows, whole ones -- its widths, and …
jameswitney Nov 12, 2026
f20511c
tests: the CPU host's row-1 gate holds the tolerance, not the pick
jameswitney Nov 12, 2026
0f70be3
tree: the kv-quant lane's own dumps are ignored, so a lane run leaves…
jameswitney Nov 12, 2026
0b464e8
v41: the tier reserves no chain bytes for a host the chain declines
jameswitney Nov 12, 2026
21e45a9
planner: V4.1's launch floor is one expert slot, not a refusal
jameswitney Nov 12, 2026
309e1bc
kv-quant lane: a sanitizer report fails the arm it lands in, not the …
jameswitney Nov 14, 2026
a1c4198
planner: the packed host priced exactly as the engine prices it
jameswitney Nov 14, 2026
cb18598
prefix store: a rows_io part's span is the one run it stages, and its…
jameswitney Nov 14, 2026
57dad3b
docs: the vendor paragraph admits the packed cache, the 1M row its 5.…
jameswitney Nov 14, 2026
2390095
Merge origin/dev into r2b/kv-quant-host
jameswitney Nov 15, 2026
9e43e6c
Merge origin/dev into r2b/kv-quant-host
jameswitney Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@ jobs:
# make check built serially and took 23 minutes on 2026-10-17, then past 25 with two
# more engines' GGUF code: the builds now use every core (make test runs the C and
# Python suites one after the other, so only the compiling is parallel).
# 2026-11-09: 35 was reached in turn (#2712, cut at 35m23s mid-build); 60.
# 2026-11-11: 30.0 minutes on #2750, then cancelled at 35 twice on #2759, which adds
# a GGUF MTP suite. 50.
timeout-minutes: 50
# a GGUF MTP suite. Take the larger of the two: 60.
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- name: make check
Expand Down
11 changes: 11 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,17 @@ jobs:
- family: kv-split-deepseek-sanitize
req: c/tools/requirements-deepseek-v4-tiny.txt
extra: sympy pillow
# the packed KV cache (COLI_VK_KV_FP32 unset, the default): the codec
# bit-for-bit, the six chain arms, the KV split's give-way, the CPU host
- family: kv-quant-deepseek
req: c/tools/requirements-deepseek-v4-tiny.txt
extra: sympy pillow
# ... and the same family on an ASan/UBSan build: the only leg whose arms hold
# the HOST's KV rows as 528/288/68 B codes (the other sanitized families walk
# the 64/32 fixture, where the geometry guard declines the packed host)
- family: kv-quant-deepseek-sanitize
req: c/tools/requirements-deepseek-v4-tiny.txt
extra: sympy pillow
# the dense chain's layers on two devices (Lavapipe opened twice, COLI_VK_DEV2=0):
# every chain engine against its CPU run, split at every kind of layer, the state
# crossing between the devices, the second device lost; the same under ASan and
Expand Down
22 changes: 22 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -204,3 +204,25 @@ c/mimo.exe

# Checkpoint MTP fixtures generated by qwen36-tiny-mtp-check
c/qwen36_tiny_mtp*/

# VRAM audit scratch (docs/experiments/dsv41-prefill-vram-2026-11-06)
/vram-*
c/*.err

# kv-quant run artifacts and fixtures
c/dsv41_kvq_*/
kvq-*.csv
# ...and the logits/token dumps its arms write into c/ (each arm `rm -f`s its own at the
# start and leaves them behind): `kv-quant-deepseek` is the lane this branch owns, and a
# lane run leaves seven untracked names that a `git add -A` before a push would ship.
# The usage ledgers are the arms' own COLI_USAGE output (cleaned at the family's start so
# a FAILED lane leaves none behind either).
c/off.f32
c/on.f32
c/off.txt
c/on.txt
c/plain.txt
c/forced.txt
c/self.json
c/chain.usage
c/chain-serve.usage
23 changes: 20 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,22 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).

### New

- **The packed KV cache is the default on DeepSeek V4.1 Flash** (`COLI_VK_KV_FP32` unset,
`1` is the exact fp32 opt-out): the chain's device KV mirrors *and* the host's own KV
arrays stay in the checkpoint's cache formats — MXFP8 window rings (528 B a row), FP4
compressed rows (288 B), FP4 index keys (68 B) — so the host state costs 890 B a position
against fp32's 6,400 (86.1% less at the released geometry), read paths dequantizing on
read and the boundaries (pull, park, reset, rollback) moving code rows. An approximate
mode: the logits drift a little and the `kv-quant-deepseek` family gates that documented
tolerance instead of bit-identity. The planner prices it — the host's window rings and
the `(dspark_block_size + 1)` rollback rows a layer included, 3,032,832 B packed against
11,763,712 B fp32, and 17,614,592 B at 16K, the engine's own `host kv` figure — reading
the stage count the way the released `config.json` spells it
(`num_nextn_predict_layers`, which it carries where `n_mtp_layers` is absent). New CI leg:
`kv-quant-deepseek` in the `vulkan-engines` job runs the codec bit-for-bit, the six chain
arms, the KV split's give-way and the CPU host
([deepseek-v41.md](docs/deepseek-v41.md#quantized-kv-mirrors-coli_vk_kv_fp32)).

- **Qwen3.8 on AMD Quark's MXFP4 experts** (`Q38_EXPERT_LOWBIT=mxfp4`): the routed experts
from the `experts-mxfp4/` sidecar, one read of 2.6 MB per miss instead of 4.9 MB, computed by
`quant.h`'s `matmul_mxfp4`; the trunk and the shared expert stay FP8. CPU only for now (the
Expand Down Expand Up @@ -604,9 +620,10 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).
which no writer produces: the first topk in block order, every slot written). A conversation
reset and a prefix-store take now drop the blocks too, as they already dropped the shared
top-k list: the rows a reset keeps apart were saved by park first, and a new conversation
no longer holds the last one's allocation (2 GiB at 262144) live. The device masks the
chain still paints stay dense (a separate change), and so does allocating the prompt's
rows rather than the tail's (2 GiB to 19.8 MB at 262144), which is named there.
no longer holds the last one's allocation (2 GiB at 262144) live. On the host this PR's
packed cache packs those device masks too (one bit a column, `v41c_mask_words`), so the
readback unpacks the bit row into the ids' keep row; allocating the prompt's rows rather
than the tail's stays as it was (2 GiB to 19.8 MB at 262144), which is named there.
- **CUDA on Windows: the answer stopped a few tokens in with the card at 100%** (#2696). Windows'
WDDM driver model, which WSL2 drives the card through too, holds CUDA launches back and sends
them in batches. A hybrid step (a layer whose experts are partly in RAM) has the card signal the
Expand Down
15 changes: 14 additions & 1 deletion c/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -574,7 +574,7 @@ TEST_EXCLUDE = test_uring test_deepseek_v4 test_v4_ownership test_v4_serve_frami
test_e8x4g64_loader test_qwenimage_vae test_vk_tier \
test_i4_grouped_sse41 test_i4_grouped_sse41_o1 test_i4_grouped_sse41_no_contract test_idot_ssse3 test_int_kernel_exact_ssse3 \
test_ggml_quants test_gguf_c test_gq_dot
TEST_EXCLUDE += test_vk_chain test_vk_memtype test_glm53_vk_f32 test_vk_dev2_window test_vk_pcache
TEST_EXCLUDE += test_vk_chain test_vk_memtype test_glm53_vk_f32 test_vk_dev2_window test_vk_kv_quant test_vk_pcache
# needs a CUDA device and the CUDA objects: make tests/test_cu_tier CUDA=1
TEST_EXCLUDE += test_cu_tier
# takes a fixture and the CPU build's logits on its command line (ci.yml, the int4 gs64 A/B step)
Expand Down Expand Up @@ -2916,6 +2916,19 @@ tests/test_vk_chain$(EXE): tests/test_vk_chain.c $(VK_OBJ) $(VK_SPV)
vk-chain-check: tests/test_vk_chain$(EXE)
./tests/test_vk_chain$(EXE) shaders/qmatmul.spv

# The packed cache's codec (needs VK=1): the chain's quantized KV mirrors
# (chain_dsv4.comp modes 13/14 and the quant readers) against c/dsv41_kvq.h's reference,
# DeepInfer's cache formats -- the device's pack bit for bit, its unpack, a re-upload's
# lanes, the pack's idempotence and the index scores' top-k overlap. Like the chain test
# it is kept out of `check`/`test` (which run without a device), reachable from its own
# target and the kv-quant-deepseek family of tests/vulkan_engines.sh.
tests/test_vk_kv_quant$(EXE): tests/test_vk_kv_quant.c $(VK_OBJ) $(VK_SPV)
@[ "$(VK)" = 1 ] || { echo "tests/test_vk_kv_quant needs VK=1" >&2; exit 1; }
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
.PHONY: vk-kv-quant-check
vk-kv-quant-check: tests/test_vk_kv_quant$(EXE)
./tests/test_vk_kv_quant$(EXE)

# Checks the choice of memory type and the size, budget and room of the device memory
# (vk_memtype.h) on memory-property tables written by the test. It needs the Vulkan headers
# (VK=1) and no device, so `check` does not run it.
Expand Down
7 changes: 7 additions & 0 deletions c/coli_env.h
Original file line number Diff line number Diff line change
Expand Up @@ -421,9 +421,14 @@ static const ColiEnvVar coli_env_table[] = {
{"COLI_VK_KV_BLOCK", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO, 0, NULL},
{"COLI_VK_KV_COLD", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO, 0, NULL},
{"COLI_VK_KV_DEVICE_ROWS", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO, 0, NULL},
{"COLI_VK_KV_FP32", CE_BOOL, CE_DSV41, 0, NULL},
{"COLI_VK_KV_PIN", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO, 0, NULL},
/* retired: read once only to say so (v41_host_kvq's one line, the inverted meaning
* included), so it takes no moved default and no deprecated-name rewrite here */
{"COLI_VK_KV_QUANT", CE_STR, CE_DSV41, 0, NULL},
{"COLI_VK_KV_SLICE", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO, 0, NULL},
{"COLI_VK_KV_SPLIT", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO, 0, NULL},
{"COLI_VK_LEDGER", CE_INT, CE_DSV41, 0, NULL},
{"COLI_VK_LOADER", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_MLA_DEC", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_MOE_GV", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
Expand Down Expand Up @@ -475,6 +480,7 @@ static const ColiEnvVar coli_env_table[] = {
{"COLI_VK_TIER_RESERVE_GB", CE_FLOAT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_TIER_STREAM", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_TIER_STREAM_HALF", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_TIER_STREAM_MIN", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_TIER_STREAM_PAR", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_TIER_STREAM_ROWS", CE_INT, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
{"COLI_VK_TIER_STREAM_SLOTS", CE_STR, CE_COLIBRI | CE_KIMI | CE_INKLING | CE_OLMOE | CE_DSV4 | CE_QWEN | CE_GLM53 | CE_QWEN38 | CE_DSV41 | CE_QWENIMAGE | CE_MIMO | CE_LAYA | CE_GLINER, 0, NULL},
Expand Down Expand Up @@ -907,6 +913,7 @@ static const ColiEnvVar coli_env_table[] = {
{"V41_PREFILL_TAIL", CE_INT, CE_DSV41, 0, NULL},
{"V41_READ_DEPTH", CE_INT, CE_DSV41, 0, NULL},
{"V41_SPEC_FORCE", CE_BOOL, CE_DSV41, 0, NULL},
{"V41_STATE_PROBE", CE_BOOL, CE_DSV41, 0, NULL},
{"V41_STATS", CE_BOOL, CE_DSV41, 0, NULL},
{"V41_TRACE", CE_BOOL, CE_DSV41, 0, NULL},
{"V4_ATTN_BATCH_PROJ", CE_BOOL, CE_DSV4, 0, NULL},
Expand Down
2 changes: 2 additions & 0 deletions c/cu_tier.c
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,12 @@ static const CuTierOps *cut_ops(void) {
#define vkt_score cut_score
#define vkt_settle cut_settle
#define vkt_shutdown cut_shutdown
#define vkt_slot_reserve cut_slot_reserve
#define vkt_stage_room cut_stage_room
#define vkt_step_rows cut_step_rows
#define vkt_stream_prefetch cut_stream_prefetch
#define vkt_take_back cut_take_back
#define vkt_vram cut_vram
#define vkt_wanted cut_wanted
#define vkt_wants cut_wants
#define vkt_warm_report cut_warm_report
Expand Down
Loading