Skip to content

Batch 2026-11-13-a: seven PRs for open issues (CUDA chain flash2 mask, no-FMA FP8 experts, Windows mappings, macOS memory) - #2805

Merged
JustVugg merged 20 commits into
devfrom
batch/2026-11-13-a
Nov 13, 2026
Merged

JustVugg merged 20 commits into
devfrom
batch/2026-11-13-a

Conversation

@JustVugg

@JustVugg JustVugg commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Seven PRs that fix open issues, each merged with its own commits.

What is in it

PR Issue What it fixes
#2804 by @tylnexttime #2753 CUDA chain: a long prompt's attention at head dim 64 and 128 masked out keys 32..63 of a block on sm_86 and sm_89. cc_attn_f2 tested its key mask with no selection list too, shifting a 32-bit word by up to 63.
#2801 by @KairaCode #2715 qwen38 on a CPU without FMA: the native FP8 experts take the grouped MoE path, one parallel region a layer instead of one per expert.
#2786 by @KairaCode #1764 TRUNK_RESIDENT_LAYERS and COLI_MMAP map the shards on Windows too, and say so when a shard cannot be mapped.
#2791 by @KairaCode #1614, #2785 macOS: available memory is free + file-backed + purgeable pages, one rule for the planner, coli setup and every engine.
#2792 by @KairaCode #2785 --auto-tier lets OLMoE size its own expert cache from RAM_GB, and the plan reserves olmoe.c's 0.5 GB.
#2735 by @tylnexttime run_tests.py runs the C tests by absolute path (Windows with NoDefaultCurrentDirectoryInExePath).
#2738 by @tylnexttime test_ablate_mode no longer leaves its stderr capture behind on Windows.

The only conflict was two entries at the same place in CHANGELOG.md (#2791, #2792); both are kept.

#2800 (#2286) was in the first push and is out: CI's MSYS2 has clang and LLVM OpenMP 22.1.7, whose static libomp.a needs omp_get_num_devices at link time, so make deepseek-v4 CC=clang fails there (it was built with 23.1.5).

Tests

Fixes #2753
Refs #2715, #1764, #1614, #2785

tylnexttime and others added 20 commits November 10, 2026 15:33
With NoDefaultCurrentDirectoryInExePath set, Windows does not search the
current directory for a relative executable name, and MSYS2's Python keeps
the forward slash through normpath, so tools/run_tests.py failed to start
tests/*.exe with FileNotFoundError before the first test ran.

Co-Authored-By: Claude <noreply@anthropic.com>
The capture file is still open as stderr when the test calls remove() on
it. Linux unlinks an open file; Windows refuses, so every run there left
tmp_test_ablate_stderr.<pid>.txt behind in c/. Point stderr at the null
device first, then remove the file.

Co-Authored-By: Claude <noreply@anthropic.com>
…s too (#1764)

map_of_fd, the shard mapping both knobs read their views from, mapped only
under __APPLE__, __linux__ and __FreeBSD__ and returned NULL elsewhere. On
Windows qt_load_mmap and the COLI_MMAP expert path then took their resident
copies, without a word: TRUNK_RESIDENT_LAYERS=0 COLI_MMAP=1 RAM_GB=2, the
low-RAM setting the docs give, freed nothing there.

The Windows branch maps the whole shard read-only with compat_map_readonly,
the CreateFileMapping/MapViewOfFile view st.h and qwen36 already use, and
keeps it for the life of the process like the POSIX mmap. The views point at
the same bytes the copies held, so the tokens do not change.

tests/test_trunk_resident_layers.py failed its resident-dense check on
Windows before this (0.21 -> 0.21 MB on the int4 glm_tiny) and passes now;
a new case runs TRUNK_RESIDENT_LAYERS=0 with COLI_MMAP=1 against the
resident baseline. docs/ENVIRONMENT.md drops the Windows exception and says
that Windows counts the mapped pages the engine has read in the working set
RSS_GUARD_GB checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts are views (#1764)

map_of_fd's fallbacks stayed silent: a failed mapping, a 513th shard or an
OS without a mapping branch sent TRUNK_RESIDENT_LAYERS and COLI_MMAP back to
copies in RAM while the [MMAP] banner still spoke of views. One [MMAP] line
per process now says so.

test_trunk_and_experts_mapped_keep_tokens checked only what the trunk test
already checks, so a COLI_MMAP expert path that fell back to pread passed it.
It now runs with PROF=1: the `[PROF] DISK-CLASS` line, printed only when the
expert loader read an expert with pread, must be in the baseline and absent
with COLI_MMAP=1. On a Windows build without the map_of_fd branch it fails
there 3 times out of 3.

docs/ENVIRONMENT.md: on Windows, with COLI_MMAP=1, rss_guard has no copies
to free but still lowers the expert-cache cap without a [RAM-GUARD] line;
RSS_GUARD_GB=0 turns it off. CHANGELOG entry under Fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_plan set aside colibri.c's slack (1.2 + 2.5 GB and 64 experts in
flight) for every family before sizing the expert cache. olmoe.c sizes its
own cache from budget - resident - 0.5 GB of activations. Under --auto-tier
the plan's cap reaches the engine as an explicit cap (COLI_PLAN_CAP), so on a
16 GB Mac OLMoE ran with far fewer slots than it picks by itself, and coli
tune, which only tries smaller caches than the plan's, could not get them
back. OLMoE-1B-7B with 8 GB available: 16 slots per layer planned, 62 sized
by the engine at the same budget.

The olmoe plan now reserves the engine's 0.5 GB, plus the KV of the planned
context as before (with a cap given, the engine does not take slots back as
the KV grows). Other families are unchanged.

Refs #2785

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a build without FMA (Sandy and Ivy Bridge, the x86-64 baseline and v2)
a prompt's native FP8 routed experts take q38_moe_prefill: 32-row chunks,
and for each distinct expert three matmul_fp8 calls, each with its own
parallel regions (two for a batch: the activations' interleave and the
product) and a malloc. On Qwen3.8-Flash-Next a real prompt's 32-row chunk
touches about 91 distinct experts, about 3.3 rows each (#1686's
measurement, in the comment above q38_env_positive_int), so #2724's
batched FP8 kernel (rows in 8 AVX lanes) runs about 40% full, and every
expert pays its fork/join barriers in every chunk: several hundred a layer
and chunk. Q38_PREFILL_BATCH_ROWS (with Q38_PREFILL_WORKSPACE_MIB) already
widens the chunk and fills the lanes; this change removes the barriers.

The int4-g64 and GGUF experts already take q38_moe_decode_grouped: up to
256 rows grouped by expert, one parallel region a layer over (expert, row
block) items. On these builds the native FP8 experts take it too, decode
rows and MTP verifies included:

- quant.h: the no-FMA FP8 kernels' bodies become helpers (four outputs of
  one row; four outputs of rows interleaved in lanes; the interleave), and
  matmul_fp8_nofma_part computes output rows [r0,r1) for rows given by
  pointers into a strided output, with no parallel region of its own.
  matmul_fp8_nofma_s1 and matmul_fp8_nofma_rows call the same helpers.
- qwen38_core.h: q38_rows_kind takes FP8 matrices as f32-input rows and
  q38_rows runs them through matmul_fp8_nofma_part. A layer qualifies when
  its experts are native FP8 in the release's layout (the resident scale
  bank, expert 0's gate/up adjacent) and the parallel batch reads are on;
  otherwise the grouped path would run them a row at a time. With a cache
  smaller than the layer, a prompt keeps q38_moe_prefill.

Every output row is meant to be matmul_fp8_scalar's, as before: one row
the four-row pass, two or more the rows in lanes, each lane the scalar
kernel's multiply and add in element order, and the grouped path adds each
row's experts in rank order and then the shared expert, as the per-expert
paths do. tests/test_nofma_kernels.c checks matmul_fp8_nofma_part against
matmul_fp8_scalar (a range of output rows off the four-row pass and across
a 128-row block, rows by pointer in reverse order, a wider output stride,
every other output untouched) in the x86-64 baseline and Sandy Bridge
builds. Builds with FMA (AVX2, NEON, the release's) compile none of this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With an expert cache smaller than the layer, the grouped path works in
groups of cap/top-k rows. Each group reads an expert once, as each prefill
chunk does, so the grouped path reads more only when its groups are
narrower than the prefill's chunk (q38_moe_prefill_rows: 32 rows by
default, Q38_PREFILL_BATCH_ROWS and the workspace budget). The gate
compared the cache with the layer instead and sent caches of 32 * top-k
rows or more to the prefill as well. Comments: quant.h names
matmul_fp8_nofma_part next to the two kernels; the test header's columns
line up again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rule

Every macOS probe counted free + inactive pages: resource_plan.py,
setup_hw.py, compat_meminfo_gb, deepseek_v4.c's own copy and telemetry.h.
The inactive queue holds file pages, which macOS drops, and anonymous
pages, which it compresses. So a model's file cache still in the active
queue was missed (#2785: a 16 GB Mac planned 0 expert slots, and OLMoE's
automatic cache moved from 42 to 64 slots as the pages aged), and other
apps' inactive anonymous memory was counted (#1614: 32 GiB accepted on a
48 GB Mac with browsers open, then compressed and swapped).

One rule now: Pages free + File-backed pages + Pages purgeable, i.e.
free_count - speculative_count + external_page_count + purgeable_count,
physical memory less Activity Monitor's "Memory Used".

- resource_plan.darwin_available_bytes() parses vm_stat; memory_available()
  and setup_hw.memory_from_darwin() use it (setup_hw's copy is gone).
- compat_meminfo_gb() computes it from host_statistics64; deepseek_v4.c
  and telemetry.h call compat instead of keeping their own copies.
- Tests: DarwinAvailableMemoryTest with vm_stat fixtures for both issues;
  test_setup_hw's darwin fixture asserts the new sum.

Not yet tested on a Mac.

Refs #2785
Refs #1614

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once the plan had a slot, cap_for_arch() passed COLI_PLAN_CAP to olmoe as
its argv cap. An explicit cap stops olmoe.c from sizing its cache from
RAM_GB with its measured RSS, and from giving slots back as the KV grows
(kv_room_fit). With no --cap and no tune profile, OLMoE now gets its
implicit cap 0 and sizes the cache from the plan's RAM_GB, as it already
did whenever the plan had 0 slots.

The plan keeps olmoe.c's 0.5 GB reserve and the planned KV, so coli plan,
doctor and tune show the cache olmoe keeps at a full context. The planner
test also covers a Q8_0 GGUF linked by coli import, through a header-level
GGUF fixture, at --ram 8 and at 8 GB available.

Refs #2785

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…potheses

compat_meminfo_gb() multiplies host_statistics64's page counts by
vm_kernel_page_size, the size vm_stat prints, instead of the process's
sysconf(_SC_PAGESIZE). They differ under Rosetta 2 (4 KiB process pages on
a 16 KiB kernel), where the figure was 4x too low; DeepSeek V4, which used
host_page_size() before, would have regressed there.

Comments, the test fixtures and the CHANGELOG entry no longer state the
mechanisms as established: the fixtures are synthetic (neither report
posted vm_stat), the active-queue drift is a hypothesis, and #2785's
0-slot plan comes mostly from the planner's runtime reserve, not from the
probe, which the planner and olmoe.c already shared on dev. V4's comment
keeps #1623's pointer: the compressor is not modeled, a fix needs
vm.compressor_page_count. compat.h no longer lists telemetry.h among the
hand-rolled probes; olmoe.c's 2% floor names the old probe as history.

Not yet tested on a Mac.

Refs #2785
Refs #1614

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OS version

A Mac mini M4 on macOS 15.6 shows File-backed pages == external_page_count,
without the speculative pages, which the comments assumed it held. The rule
reads vm_stat's Pages free (free_count - speculative_count) and File-backed
pages (external_page_count), so the speculative pages count once or not at
all depending on the version, never twice; the comments now say so.

Tests: the vm_stat fixture builder no longer derives Anonymous pages from the
active + inactive + speculative identity; a fixture with the reviewer's 15.6
figures asserts the rule's bytes and that Pages speculative is never read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
98c1bcc gave cc_attn_f2 a 32-bit mask a block of keys for a selection list
(sel, head dim 256, blocks of 32 keys or fewer). Without a list the mask is
0xFFFFFFFF, and an edge block still tests (mk >> kl) & 1 for every key of the
block. The head dim 64 and 128 kernels run blocks of 64 keys, so kl reaches 63:
a shift of a 32-bit value by 32 or more, which is undefined. On sm_89 and sm_86
with CUDA 12.x it reads as 0, and keys 32..63 of the block a row's position
falls in are dropped: six prompt-row cases of test_cuda_chain fail (#2753),
max |diff| 8e-2 to 3e-1.

The mask is tested only when there is a list. sel is non-null only for
BC <= 32, so kl stays below 32 there.

test_cuda_chain on an RTX 4090 Laptop (sm_89, CUDA 12.8.93, MSVC host, with the
setenv shim of #2731 applied locally): on 15f810f, 233 ok and 6 FAIL before,
239 ok after; on c976527 with this change, 244 ok. The six cases read
1.52e-04, 3.16e-04, 7.69e-05, 6.89e-05, 2.64e-04 and 3.20e-04, and the
selection-list case stays at 7.7e-05.

Fixes #2753

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@JustVugg JustVugg changed the title Batch 2026-11-13-a: eight PRs for open issues (CUDA chain flash2 mask, no-FMA FP8 experts, Windows libomp and mappings, macOS memory) Batch 2026-11-13-a: seven PRs for open issues (CUDA chain flash2 mask, no-FMA FP8 experts, Windows mappings, macOS memory) Nov 13, 2026
@JustVugg
JustVugg merged commit 6150372 into dev Nov 13, 2026
117 checks passed
KairaCode added a commit to KairaCode/colibri that referenced this pull request Nov 13, 2026
…um_devices (JustVugg#2286)

With MSYS2's llvm-openmp 22.1.7 (what GitHub's windows runner installs)
a static link cannot resolve omp_get_num_devices(), which libomp.a's
kmp_alloc and kmp_ftn_extra call, so `make deepseek-v4 CC=clang` failed
at the link (JustVugg#2805's CI). Makefile.deepseek-v4 now links one call to it
the same way; only when that fails on this symbol does
backend_loader_dsv4.c define it, returning 0 (no offload runtime is
linked: libomp's own host-only answer). Where the link resolves it
(23.x) nothing is added, so there is no second definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KairaCode added a commit to KairaCode/colibri that referenced this pull request Nov 13, 2026
…2286)

libomp.a defines omp_get_num_devices() only as a COFF weak symbol, and
GNU ld leaves it unresolved with MSYS2's llvm-openmp 22.1.7 or binutils
2.46, which is what GitHub's windows runner installs: `make deepseek-v4
CC=clang` failed at the link in JustVugg#2805's CI, on libomp's own kmp_alloc
and kmp_ftn_extra. Only llvm-openmp 23.x with binutils 2.47 links it.

Makefile.deepseek-v4 now links one call to it the same way; only when
that fails on this symbol does backend_loader_dsv4.c define it,
returning 0, what libomp returns when no offload runtime is linked.
Where it links, nothing is added, so there is no second definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants