Repository navigation
Batch 2026-11-13-a: seven PRs for open issues (CUDA chain flash2 mask, no-FMA FP8 experts, Windows mappings, macOS memory) - #2805
Merged
Merged
Conversation
With NoDefaultCurrentDirectoryInExePath set, Windows does not search the current directory for a relative executable name, and MSYS2's Python keeps the forward slash through normpath, so tools/run_tests.py failed to start tests/*.exe with FileNotFoundError before the first test ran. Co-Authored-By: Claude <noreply@anthropic.com>
The capture file is still open as stderr when the test calls remove() on it. Linux unlinks an open file; Windows refuses, so every run there left tmp_test_ablate_stderr.<pid>.txt behind in c/. Point stderr at the null device first, then remove the file. Co-Authored-By: Claude <noreply@anthropic.com>
…s too (#1764) map_of_fd, the shard mapping both knobs read their views from, mapped only under __APPLE__, __linux__ and __FreeBSD__ and returned NULL elsewhere. On Windows qt_load_mmap and the COLI_MMAP expert path then took their resident copies, without a word: TRUNK_RESIDENT_LAYERS=0 COLI_MMAP=1 RAM_GB=2, the low-RAM setting the docs give, freed nothing there. The Windows branch maps the whole shard read-only with compat_map_readonly, the CreateFileMapping/MapViewOfFile view st.h and qwen36 already use, and keeps it for the life of the process like the POSIX mmap. The views point at the same bytes the copies held, so the tokens do not change. tests/test_trunk_resident_layers.py failed its resident-dense check on Windows before this (0.21 -> 0.21 MB on the int4 glm_tiny) and passes now; a new case runs TRUNK_RESIDENT_LAYERS=0 with COLI_MMAP=1 against the resident baseline. docs/ENVIRONMENT.md drops the Windows exception and says that Windows counts the mapped pages the engine has read in the working set RSS_GUARD_GB checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts are views (#1764) map_of_fd's fallbacks stayed silent: a failed mapping, a 513th shard or an OS without a mapping branch sent TRUNK_RESIDENT_LAYERS and COLI_MMAP back to copies in RAM while the [MMAP] banner still spoke of views. One [MMAP] line per process now says so. test_trunk_and_experts_mapped_keep_tokens checked only what the trunk test already checks, so a COLI_MMAP expert path that fell back to pread passed it. It now runs with PROF=1: the `[PROF] DISK-CLASS` line, printed only when the expert loader read an expert with pread, must be in the baseline and absent with COLI_MMAP=1. On a Windows build without the map_of_fd branch it fails there 3 times out of 3. docs/ENVIRONMENT.md: on Windows, with COLI_MMAP=1, rss_guard has no copies to free but still lowers the expert-cache cap without a [RAM-GUARD] line; RSS_GUARD_GB=0 turns it off. CHANGELOG entry under Fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_plan set aside colibri.c's slack (1.2 + 2.5 GB and 64 experts in flight) for every family before sizing the expert cache. olmoe.c sizes its own cache from budget - resident - 0.5 GB of activations. Under --auto-tier the plan's cap reaches the engine as an explicit cap (COLI_PLAN_CAP), so on a 16 GB Mac OLMoE ran with far fewer slots than it picks by itself, and coli tune, which only tries smaller caches than the plan's, could not get them back. OLMoE-1B-7B with 8 GB available: 16 slots per layer planned, 62 sized by the engine at the same budget. The olmoe plan now reserves the engine's 0.5 GB, plus the KV of the planned context as before (with a cap given, the engine does not take slots back as the KV grows). Other families are unchanged. Refs #2785 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a build without FMA (Sandy and Ivy Bridge, the x86-64 baseline and v2) a prompt's native FP8 routed experts take q38_moe_prefill: 32-row chunks, and for each distinct expert three matmul_fp8 calls, each with its own parallel regions (two for a batch: the activations' interleave and the product) and a malloc. On Qwen3.8-Flash-Next a real prompt's 32-row chunk touches about 91 distinct experts, about 3.3 rows each (#1686's measurement, in the comment above q38_env_positive_int), so #2724's batched FP8 kernel (rows in 8 AVX lanes) runs about 40% full, and every expert pays its fork/join barriers in every chunk: several hundred a layer and chunk. Q38_PREFILL_BATCH_ROWS (with Q38_PREFILL_WORKSPACE_MIB) already widens the chunk and fills the lanes; this change removes the barriers. The int4-g64 and GGUF experts already take q38_moe_decode_grouped: up to 256 rows grouped by expert, one parallel region a layer over (expert, row block) items. On these builds the native FP8 experts take it too, decode rows and MTP verifies included: - quant.h: the no-FMA FP8 kernels' bodies become helpers (four outputs of one row; four outputs of rows interleaved in lanes; the interleave), and matmul_fp8_nofma_part computes output rows [r0,r1) for rows given by pointers into a strided output, with no parallel region of its own. matmul_fp8_nofma_s1 and matmul_fp8_nofma_rows call the same helpers. - qwen38_core.h: q38_rows_kind takes FP8 matrices as f32-input rows and q38_rows runs them through matmul_fp8_nofma_part. A layer qualifies when its experts are native FP8 in the release's layout (the resident scale bank, expert 0's gate/up adjacent) and the parallel batch reads are on; otherwise the grouped path would run them a row at a time. With a cache smaller than the layer, a prompt keeps q38_moe_prefill. Every output row is meant to be matmul_fp8_scalar's, as before: one row the four-row pass, two or more the rows in lanes, each lane the scalar kernel's multiply and add in element order, and the grouped path adds each row's experts in rank order and then the shared expert, as the per-expert paths do. tests/test_nofma_kernels.c checks matmul_fp8_nofma_part against matmul_fp8_scalar (a range of output rows off the four-row pass and across a 128-row block, rows by pointer in reverse order, a wider output stride, every other output untouched) in the x86-64 baseline and Sandy Bridge builds. Builds with FMA (AVX2, NEON, the release's) compile none of this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With an expert cache smaller than the layer, the grouped path works in groups of cap/top-k rows. Each group reads an expert once, as each prefill chunk does, so the grouped path reads more only when its groups are narrower than the prefill's chunk (q38_moe_prefill_rows: 32 rows by default, Q38_PREFILL_BATCH_ROWS and the workspace budget). The gate compared the cache with the layer instead and sent caches of 32 * top-k rows or more to the prefill as well. Comments: quant.h names matmul_fp8_nofma_part next to the two kernels; the test header's columns line up again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rule Every macOS probe counted free + inactive pages: resource_plan.py, setup_hw.py, compat_meminfo_gb, deepseek_v4.c's own copy and telemetry.h. The inactive queue holds file pages, which macOS drops, and anonymous pages, which it compresses. So a model's file cache still in the active queue was missed (#2785: a 16 GB Mac planned 0 expert slots, and OLMoE's automatic cache moved from 42 to 64 slots as the pages aged), and other apps' inactive anonymous memory was counted (#1614: 32 GiB accepted on a 48 GB Mac with browsers open, then compressed and swapped). One rule now: Pages free + File-backed pages + Pages purgeable, i.e. free_count - speculative_count + external_page_count + purgeable_count, physical memory less Activity Monitor's "Memory Used". - resource_plan.darwin_available_bytes() parses vm_stat; memory_available() and setup_hw.memory_from_darwin() use it (setup_hw's copy is gone). - compat_meminfo_gb() computes it from host_statistics64; deepseek_v4.c and telemetry.h call compat instead of keeping their own copies. - Tests: DarwinAvailableMemoryTest with vm_stat fixtures for both issues; test_setup_hw's darwin fixture asserts the new sum. Not yet tested on a Mac. Refs #2785 Refs #1614 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once the plan had a slot, cap_for_arch() passed COLI_PLAN_CAP to olmoe as its argv cap. An explicit cap stops olmoe.c from sizing its cache from RAM_GB with its measured RSS, and from giving slots back as the KV grows (kv_room_fit). With no --cap and no tune profile, OLMoE now gets its implicit cap 0 and sizes the cache from the plan's RAM_GB, as it already did whenever the plan had 0 slots. The plan keeps olmoe.c's 0.5 GB reserve and the planned KV, so coli plan, doctor and tune show the cache olmoe keeps at a full context. The planner test also covers a Q8_0 GGUF linked by coli import, through a header-level GGUF fixture, at --ram 8 and at 8 GB available. Refs #2785 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…potheses compat_meminfo_gb() multiplies host_statistics64's page counts by vm_kernel_page_size, the size vm_stat prints, instead of the process's sysconf(_SC_PAGESIZE). They differ under Rosetta 2 (4 KiB process pages on a 16 KiB kernel), where the figure was 4x too low; DeepSeek V4, which used host_page_size() before, would have regressed there. Comments, the test fixtures and the CHANGELOG entry no longer state the mechanisms as established: the fixtures are synthetic (neither report posted vm_stat), the active-queue drift is a hypothesis, and #2785's 0-slot plan comes mostly from the planner's runtime reserve, not from the probe, which the planner and olmoe.c already shared on dev. V4's comment keeps #1623's pointer: the compressor is not modeled, a fix needs vm.compressor_page_count. compat.h no longer lists telemetry.h among the hand-rolled probes; olmoe.c's 2% floor names the old probe as history. Not yet tested on a Mac. Refs #2785 Refs #1614 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OS version A Mac mini M4 on macOS 15.6 shows File-backed pages == external_page_count, without the speculative pages, which the comments assumed it held. The rule reads vm_stat's Pages free (free_count - speculative_count) and File-backed pages (external_page_count), so the speculative pages count once or not at all depending on the version, never twice; the comments now say so. Tests: the vm_stat fixture builder no longer derives Anonymous pages from the active + inactive + speculative identity; a fixture with the reviewer's 15.6 figures asserts the rule's bytes and that Pages speculative is never read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
98c1bcc gave cc_attn_f2 a 32-bit mask a block of keys for a selection list (sel, head dim 256, blocks of 32 keys or fewer). Without a list the mask is 0xFFFFFFFF, and an edge block still tests (mk >> kl) & 1 for every key of the block. The head dim 64 and 128 kernels run blocks of 64 keys, so kl reaches 63: a shift of a 32-bit value by 32 or more, which is undefined. On sm_89 and sm_86 with CUDA 12.x it reads as 0, and keys 32..63 of the block a row's position falls in are dropped: six prompt-row cases of test_cuda_chain fail (#2753), max |diff| 8e-2 to 3e-1. The mask is tested only when there is a list. sel is non-null only for BC <= 32, so kl stays below 32 there. test_cuda_chain on an RTX 4090 Laptop (sm_89, CUDA 12.8.93, MSVC host, with the setenv shim of #2731 applied locally): on 15f810f, 233 ok and 6 FAIL before, 239 ok after; on c976527 with this change, 244 ok. The six cases read 1.52e-04, 3.16e-04, 7.69e-05, 6.89e-05, 2.64e-04 and 3.20e-04, and the selection-list case stays at 7.7e-05. Fixes #2753 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…reserve # Conflicts: # CHANGELOG.md
JustVugg
force-pushed
the
batch/2026-11-13-a
branch
from
November 13, 2026 07:08
50bc527 to
bd8d43f
Compare
3 of 5 tasks
KairaCode
added a commit
to KairaCode/colibri
that referenced
this pull request
Nov 13, 2026
…um_devices (JustVugg#2286) With MSYS2's llvm-openmp 22.1.7 (what GitHub's windows runner installs) a static link cannot resolve omp_get_num_devices(), which libomp.a's kmp_alloc and kmp_ftn_extra call, so `make deepseek-v4 CC=clang` failed at the link (JustVugg#2805's CI). Makefile.deepseek-v4 now links one call to it the same way; only when that fails on this symbol does backend_loader_dsv4.c define it, returning 0 (no offload runtime is linked: libomp's own host-only answer). Where the link resolves it (23.x) nothing is added, so there is no second definition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KairaCode
added a commit
to KairaCode/colibri
that referenced
this pull request
Nov 13, 2026
…2286) libomp.a defines omp_get_num_devices() only as a COFF weak symbol, and GNU ld leaves it unresolved with MSYS2's llvm-openmp 22.1.7 or binutils 2.46, which is what GitHub's windows runner installs: `make deepseek-v4 CC=clang` failed at the link in JustVugg#2805's CI, on libomp's own kmp_alloc and kmp_ftn_extra. Only llvm-openmp 23.x with binutils 2.47 links it. Makefile.deepseek-v4 now links one call to it the same way; only when that fails on this symbol does backend_loader_dsv4.c define it, returning 0, what libomp returns when no offload runtime is linked. Where it links, nothing is added, so there is no second definition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seven PRs that fix open issues, each merged with its own commits.
What is in it
cc_attn_f2tested its key mask with no selection list too, shifting a 32-bit word by up to 63.TRUNK_RESIDENT_LAYERSandCOLI_MMAPmap the shards on Windows too, and say so when a shard cannot be mapped.coli setupand every engine.--auto-tierlets OLMoE size its own expert cache fromRAM_GB, and the plan reserves olmoe.c's 0.5 GB.run_tests.pyruns the C tests by absolute path (Windows withNoDefaultCurrentDirectoryInExePath).test_ablate_modeno longer leaves its stderr capture behind on Windows.The only conflict was two entries at the same place in
CHANGELOG.md(#2791, #2792); both are kept.#2800 (#2286) was in the first push and is out: CI's MSYS2 has clang and LLVM OpenMP 22.1.7, whose static
libomp.aneedsomp_get_num_devicesat link time, somake deepseek-v4 CC=clangfails there (it was built with 23.1.5).Tests
make checkon the whole branch (with windows: DeepSeek V4 built with clang and LLVM's libomp, whose barriers spin (#2286) #2800 in it at the time).check-env, every C test and 3392 Python tests pass (458 skipped).test_trunk_resident_layerswith the int4 GLM fixture built as CI'sefficiencyjob builds it, so its three tests run instead of skipping, the newTRUNK_RESIDENT_LAYERS=0 COLI_MMAP=1one included. All pass.GLIBC_TUNABLES=glibc.cpu.hwcaps=-AVX2,-FMA,-FMA4).test_nofma_sse2andtest_nofma_avxpass.--geometry gguf-256(expert width 256, so the grouped path runs, as its log line says), the last logits are byte for byte those ofQ38_MOE_GROUPED=0and those of dev built the same way, with a full cache (8) and with 4 of 8.make cuda-chain-checkon dev and on this branch. Both give 331okand no failures, the V4.1, GLM-5.3 and K3 chain checks included. All 25 prompt-row lines carry the same numbers on both, the same as the reporter's after the fix on sm_89.cuda_chain.oalso builds for sm_86 and sm_89 with no warnings.Fixes #2753
Refs #2715, #1764, #1614, #2785