Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ea86d48
tests: run C test binaries by absolute path
tylnexttime Nov 10, 2026
dcf092b
tests: remove test_ablate_mode's stderr capture on Windows too
tylnexttime Nov 10, 2026
49395d9
colibri: TRUNK_RESIDENT_LAYERS and COLI_MMAP map the shards on Window…
KairaCode Nov 12, 2026
58e9b96
colibri: say when a shard cannot be mapped; test that COLI_MMAP exper…
KairaCode Nov 12, 2026
c899016
resource_plan: OLMoE's runtime reserve is the engine's own
KairaCode Nov 12, 2026
ea6e6e8
qwen38: FP8 experts grouped by expert on a CPU without FMA (#2715)
KairaCode Nov 12, 2026
98bc9b4
qwen38: FP8 prompt keeps the prefill MoE only when its chunk is wider
KairaCode Nov 12, 2026
a272f83
macOS: available memory is free + file-backed + purgeable pages, one …
KairaCode Nov 12, 2026
02a5a15
openai_server: OLMoE sizes its own cache under --auto-tier
KairaCode Nov 12, 2026
de4a26a
macOS memory rule: the kernel's page size, and #2785/#1614 told as hy…
KairaCode Nov 12, 2026
8ca728b
macOS memory rule: speculative pages counted at most once, on any mac…
KairaCode Nov 13, 2026
02dbb3c
CHANGELOG: the macOS memory rule was checked on one Mac
KairaCode Nov 13, 2026
0aa9458
cuda chain: cc_attn_f2 reads the key mask only with a selection list
tylnexttime Nov 13, 2026
4bc9710
Merge pull request #2804 from tylnexttime/fix-f2-mask-no-list
JustVugg Nov 13, 2026
70af101
Merge pull request #2801 from KairaCode/kaira/issue-2715-qwen38-pre-avx2
JustVugg Nov 13, 2026
dbccd54
Merge pull request #2786 from KairaCode/kaira/issue-1764-windows-map-…
JustVugg Nov 13, 2026
22499c1
Merge pull request #2791 from KairaCode/kaira/mac-mac-available-memor…
JustVugg Nov 13, 2026
5e77386
Merge pull request #2792 from KairaCode/kaira/mac-plan-olmoe-runtime-…
JustVugg Nov 13, 2026
cf677c6
Merge pull request #2735 from tylnexttime/run-tests-abspath
JustVugg Nov 13, 2026
bd8d43f
Merge pull request #2738 from tylnexttime/ablate-test-stderr-windows
JustVugg Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,25 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).

### Fixed

- **macOS: one rule for available memory, free + file-backed + purgeable pages** (#2785, #1614).
The planner, `coli setup` and every engine counted free + inactive pages (DeepSeek V4 with its
own copy). That queue holds file pages, which macOS can drop, and anonymous pages, which it can
only compress or swap, and it leaves out file pages still active. The new figure is physical
memory less Activity Monitor's "Memory Used", the same in Python and C. Possibly why OLMoE's
automatic cache moved from 42 to 64 slots within minutes on a 16 GB Mac, and why a 48 GB Mac
with browsers open accepted a 32 GiB cache that was compressed, then swapped: neither report
shows how its pages split, so this is unconfirmed. #2785's 0-slot plan came mostly from the
planner's OLMoE runtime reserve, a separate fix. The compressor is still not modeled. Checked on
one Mac (macOS 15.6), not yet on the reporters' machines.
- **`--auto-tier` gave OLMoE a smaller expert cache than OLMoE picks by itself** (#2785). Once
the plan had a slot, `coli serve`, `chat` and `web` passed the plan's cap to olmoe, which then
no longer sized its cache from `RAM_GB` or gave slots back as the KV grew. The plan also set
aside colibri.c's runtime reserve (1.2 + 2.5 GB and 64 experts) where olmoe.c sets aside 0.5 GB,
so that cap came out 34-36 slots per layer lower than with olmoe's reserve on OLMoE-1B-7B, and
the KV's worth lower again. The gateway now hands OLMoE the plan's `RAM_GB` only, and olmoe sizes
the cache itself. The plan reserves olmoe.c's 0.5 GB and still the KV of the planned context:
`coli plan` and `coli doctor` show, and `coli tune` starts from, the cache olmoe keeps at a full
context, which is the KV's worth of slots below the size it starts with.
- **CUDA on Windows: the answer stopped a few tokens in with the card at 100%** (#2696). Windows'
WDDM driver model, which WSL2 drives the card through too, holds CUDA launches back and sends
them in batches. A hybrid step (a layer whose experts are partly in RAM) has the card signal the
Expand Down Expand Up @@ -690,6 +709,10 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).
- **The chat page's live expert map lights up on Qwen3.8.** The gateway serves the experts each decode step routed (`/experts?since=N`) from the engine's `HITL` lines, and no engine wrote them, so the map stayed dark while an answer was written and filled in only after the turn. qwen38 now writes one after the tokens of each step, the experts routed since the previous one (the CUDA chain's too). `COLI_HITS_LIVE=0` turns them off.
- The chat page's first-run dialog waits for the models service to finish measuring the
machine; on a fresh state it showed "- GB di RAM" and no model that fits.
- **`TRUNK_RESIDENT_LAYERS` and `COLI_MMAP` map the shards on Windows** (#1764). GLM-5.2's two
weights-from-disk knobs mapped only on Linux, macOS and FreeBSD; on Windows they read the
weights into RAM as without them, and said nothing. They now map there too, and one `[MMAP]` line
says so when a shard cannot be mapped.

### Docs

Expand Down
12 changes: 10 additions & 2 deletions c/colibri.c
Original file line number Diff line number Diff line change
Expand Up @@ -3261,7 +3261,15 @@ static void *map_of_fd(int fd){
#endif
}
}
#elif defined(_WIN32)
/* the whole shard as one read-only view, kept for the life of the process like the mmap above */
struct stat st; compat_ro_map wm; const void *wp;
if(g_nmaps<512 && fstat(fd,&st)==0 && compat_map_readonly(fd,0,(size_t)st.st_size,&wm,&wp)==0){
base=(void*)wp; g_maps[g_nmaps].fd=fd; g_maps[g_nmaps].base=base; g_maps[g_nmaps].len=wm.len; g_nmaps++;
}
#endif
if(!base){ static int warned; /* under g_map_mtx: one line per process */
if(!warned){ warned=1; fprintf(stderr,"[MMAP] a model shard could not be mapped: TRUNK_RESIDENT_LAYERS / COLI_MMAP read its weights into RAM instead\n"); } }
pthread_mutex_unlock(&g_map_mtx);
return base;
}
Expand Down Expand Up @@ -13343,8 +13351,8 @@ static void pin_load(Model *m, const char *statspath, double gb, int trusted){

static double g_mem_avail_boot=0; /* MemAvailable all'avvio, prima di caricare il modello */
/* RAM disponibile ADESSO (GB): e' il tetto vero, non il totale. Linux: MemAvailable
* da /proc/meminfo. macOS: pagine free+inactive+purgeable da host_statistics64
* (stessa semantica: recuperabili senza swap). Senza questo ramo il fallback
* da /proc/meminfo. macOS: pagine free+file-backed+purgeable da host_statistics64
* (recuperabili senza comprimere ne' swappare, compat_meminfo_gb). Senza questo ramo il fallback
* "assumo 8 GB" castrava la cache expert proprio sulle macchine con piu' RAM. */
static double mem_available_gb(void){
/* Era la sola copia giusta di questa misura; glm53.c ne aveva una che
Expand Down
26 changes: 20 additions & 6 deletions c/compat.h
Original file line number Diff line number Diff line change
Expand Up @@ -647,7 +647,7 @@ static inline void coli_print_launcher_help(const char *engine, const char *by_h

/* --- RAM disponibile ADESSO, in GB, per tutte le piattaforme ---------------
* "Disponibile" = recuperabile senza swap: MemAvailable su Linux; free +
* inactive + purgeable su macOS; su Windows ullAvailPhys MA limitata da
* file-backed + purgeable su macOS; su Windows ullAvailPhys MA limitata da
* ullAvailPageFile, il commit ancora concedibile: e' quello che decide se
* il prossimo malloc riesce, e su una macchina con pagefile piccolo puo'
* essere molto meno della RAM fisica libera.
Expand Down Expand Up @@ -800,9 +800,9 @@ static inline void compat_cg_bound(double *total, double *avail, uint64_t lim, u
*
* "Available" alone was consolidated here by #1375 because there were two
* copies of it and one was wrong. "Total" is now in the same position: every
* caller that wants it hand-rolls its own #ifdef ladder (telemetry.h uses
* sysctl hw.memsize on macOS, olmoe.c uses sysconf(_SC_PHYS_PAGES), glm53.c
* read /proc/meminfo unconditionally), and those definitions do not agree.
* caller that wants it hand-rolls its own #ifdef ladder (olmoe.c uses
* sysconf(_SC_PHYS_PAGES), glm53.c read /proc/meminfo unconditionally),
* and those definitions do not agree.
* A budget computed from a total and an available that came from two
* different definitions is not a budget, it is a coincidence.
*
Expand All @@ -821,9 +821,23 @@ static inline void compat_meminfo_gb(double *total_gb, double *avail_gb){
if(sysctlbyname("hw.memsize", &memsize, &len, NULL, 0) == 0) total = (double)memsize / 1e9;
mach_msg_type_number_t cnt = HOST_VM_INFO64_COUNT;
vm_statistics64_data_t vm;
/* What macOS can take back without compressing or swapping: free pages,
* file-backed pages (clean ones can be dropped from either queue) and
* purgeable pages; physical memory less Activity Monitor's "Memory Used".
* Not inactive_count: that queue also holds anonymous pages, which can only
* be compressed or swapped, and it leaves out file pages still active
* (#1614, #2785). It can, not it will: the compressor is not modeled.
* The terms are vm_stat's: "Pages free" is free_count - speculative_count,
* "File-backed pages" external_page_count. Speculative (read-ahead) pages
* come out of free_count; whether external_page_count holds them depends on
* the macOS version (on 15.6 it does not), so they count once or not at all,
* never twice. The counts are in kernel pages, so vm_kernel_page_size (vm_stat's),
* not the process's (4 KiB under Rosetta on a 16 KiB kernel).
* resource_plan.darwin_available_bytes reads the same three vm_stat lines. */
if(host_statistics64(mach_host_self(), HOST_VM_INFO64, (host_info64_t)&vm, &cnt) == KERN_SUCCESS)
avail = ((double)vm.free_count + (double)vm.inactive_count + (double)vm.purgeable_count)
* (double)sysconf(_SC_PAGESIZE) / 1e9;
avail = ((double)vm.free_count - (double)vm.speculative_count
+ (double)vm.external_page_count + (double)vm.purgeable_count)
* (double)vm_kernel_page_size / 1e9;
#elif defined(_WIN32)
MEMORYSTATUSEX msx = {0};
msx.dwLength = sizeof(msx);
Expand Down
4 changes: 2 additions & 2 deletions c/cuda_chain.cu
Original file line number Diff line number Diff line change
Expand Up @@ -840,8 +840,8 @@ __global__ static void __launch_bounds__(32 * NW) cc_attn_f2(const float *__rest
for (int u = 0; u < 2; u++) {
if (edge || sel) {
int key = k0 + j * 8 + tig * 2 + u, kl = j * 8 + tig * 2 + u;
if (key > qp0 || (win > 0 && key + win <= qp0) || !((mk0 >> kl) & 1u)) sc[j][u] = -1e30f;
if (key > qp1 || (win > 0 && key + win <= qp1) || !((mk1 >> kl) & 1u)) sc[j][2 + u] = -1e30f;
if (key > qp0 || (win > 0 && key + win <= qp0) || (sel && !((mk0 >> kl) & 1u))) sc[j][u] = -1e30f;
if (key > qp1 || (win > 0 && key + win <= qp1) || (sel && !((mk1 >> kl) & 1u))) sc[j][2 + u] = -1e30f;
}
x0 = fmaxf(x0, sc[j][u]); x1 = fmaxf(x1, sc[j][2 + u]);
}
Expand Down
58 changes: 13 additions & 45 deletions c/deepseek_v4.c
Original file line number Diff line number Diff line change
Expand Up @@ -1298,10 +1298,6 @@ static int multiply_u64(uint64_t a, uint64_t b, uint64_t *output) {
return 0;
}

#ifdef __APPLE__
#include <mach/mach.h>
#endif

uint64_t coli_v4_os_available_memory(void) {
#ifdef _WIN32
MEMORYSTATUSEX status;
Expand All @@ -1313,37 +1309,15 @@ uint64_t coli_v4_os_available_memory(void) {
return status.ullAvailPageFile && status.ullAvailPageFile < status.ullAvailPhys
? (uint64_t)status.ullAvailPageFile : (uint64_t)status.ullAvailPhys;
#elif defined(__APPLE__)
/* No /proc and no _SC_AVPHYS_PAGES on macOS. "Available" is what the
* kernel could hand out without swapping: free + inactive pages -- the
* same approximation Activity Monitor reports, and the closest analogue
* of Linux's MemAvailable (which also counts reclaimable cache).
*
* Two things a caller must know before trusting this number.
*
* It does NOT add purgeable_count, while inkling.c, kimi_k3.c, compat.h and
* telemetry.h all do. The same machine therefore reports a smaller figure
* here than through any other engine. Keep the two in mind together: they
* are not interchangeable.
*
* And "could hand out without swapping" is not "will keep resident".
* macOS answers memory pressure by COMPRESSING anonymous pages rather than
* swapping them, so a budget this function accepts can still end up half
* compressed, and every cache hit then pays a decompression. Swap stays at
* zero throughout, so a swap-based check sees nothing wrong. Reported and
* measured in issue #1614: on a 48 GB machine a 32 GiB budget decoded
* SLOWER than a 16 GiB one (0.96 vs 1.31 tok/s) while the hit rate rose
* monotonically. Sizing a cache from this number alone is therefore
* unsafe on Darwin; a fix needs vm.compressor_page_count, which this
* function does not read. */
mach_msg_type_number_t count = HOST_VM_INFO64_COUNT;
vm_statistics64_data_t vm;
vm_size_t page = 0;
if (host_statistics64(mach_host_self(), HOST_VM_INFO64,
(host_info64_t)&vm, &count) == KERN_SUCCESS &&
host_page_size(mach_host_self(), &page) == KERN_SUCCESS && page)
return ((uint64_t)vm.free_count + (uint64_t)vm.inactive_count) *
(uint64_t)page;
return 0;
/* compat_meminfo_gb's rule, as every other engine and the planner: free +
* file-backed + purgeable pages, what macOS can take back without
* compressing or swapping. This used to be free + inactive, which also
* counts inactive anonymous pages: on a 48 GB Mac with browsers open it
* offered 32.5 GiB, and a 32 GiB cache was compressed, then swapped (#1614).
* Still not modeled: the compressor. #1614 measured 32 GiB decoding slower
* than 16 GiB (0.96 vs 1.31 tok/s) with swap at zero; a fix needs
* vm.compressor_page_count (compressor_page_count in host_statistics64). */
return (uint64_t)(compat_mem_available_gb() * 1e9);
#else
/* MemAvailable bounded by a container's cgroup (compat_mem_available_gb) */
double gb = compat_mem_available_gb();
Expand Down Expand Up @@ -18859,11 +18833,8 @@ static void v4_hwinfo_emit(void) {
* it already carries a Darwin branch. Declared extern because the amalgamation compiles
* this file once per -DCOLI_V4_UNIT_*, so the definition need not be in this unit.
*
* That branch returns free + inactive, NOT free + inactive + purgeable: this engine is
* deliberately one term more conservative than inkling.c, kimi_k3.c, compat.h and
* telemetry.h, which all add purgeable_count. Whoever changes one of the two formulas
* should know the other exists, or the same machine will report two different "free RAM"
* figures depending on which engine is asked. */
* That branch calls compat_mem_available_gb(): free + file-backed + purgeable pages, the
* figure every other engine and the planner report for the same machine. */
{
extern uint64_t coli_v4_os_available_memory(void);
if (!cpu[0]) {
Expand Down Expand Up @@ -18896,11 +18867,8 @@ static void v4_hwinfo_emit(void) {
* it already carries a Darwin branch. Declared extern because the amalgamation compiles
* this file once per -DCOLI_V4_UNIT_*, so the definition need not be in this unit.
*
* That branch returns free + inactive, NOT free + inactive + purgeable: this engine is
* deliberately one term more conservative than inkling.c, kimi_k3.c, compat.h and
* telemetry.h, which all add purgeable_count. Whoever changes one of the two formulas
* should know the other exists, or the same machine will report two different "free RAM"
* figures depending on which engine is asked. */
* That branch calls compat_mem_available_gb(): free + file-backed + purgeable pages, the
* figure every other engine and the planner report for the same machine. */
{
extern uint64_t coli_v4_os_available_memory(void);
if (!cpu[0]) {
Expand Down
2 changes: 1 addition & 1 deletion c/inkling.c
Original file line number Diff line number Diff line change
Expand Up @@ -1642,7 +1642,7 @@ static void model_init(Model *m, const char *snap, int cap, int bits) {
}

static double mem_avail_bytes(void) {
/* Shared probe: Linux MemAvailable, macOS free+inactive+purgeable,
/* Shared probe: Linux MemAvailable, macOS free+file-backed+purgeable,
* Windows ullAvailPhys/commit. The #else here used to return 0, so
* Windows auto-cap was always 16 experts/layer and never warned. */
double gb = compat_mem_available_gb();
Expand Down
2 changes: 1 addition & 1 deletion c/kimi_k3.c
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ static int k3_cap_for_ram(double budget_gb, double resident_gb, double reserve_g
* compat_mem_available_gb, the quantity colibri.c's cap_for_ram() budgets
* against: Linux MemAvailable bounded by a container's cgroup, Windows
* ullAvailPhys bounded by the commit still grantable, macOS
* free+inactive+purgeable. #855. */
* free+file-backed+purgeable. #855. */
static double k3_mem_avail(void){ return compat_mem_available_gb(); }
static double rss_gb(void){ struct rusage r; getrusage(RUSAGE_SELF,&r);
#if defined(__APPLE__)
Expand Down
5 changes: 3 additions & 2 deletions c/olmoe.c
Original file line number Diff line number Diff line change
Expand Up @@ -338,8 +338,9 @@ static double mem_available_gb(void) {
* box it returned 0.0 and the auto budget collapsed to the dense resident
* footprint -- a one-slot-per-layer cache with no warning (#1601).
* Dev already had a fallback to half physical (7ed6084), but it missed the
* warm-macOS case where free+inactive+purgeable is 0.x GB on a 128 GB box:
* not zero, but implausible. The 2% floor treats that as unmeasured too. */
* warm-macOS case where the old free+inactive+purgeable probe read 0.x GB
* on a 128 GB box: not zero, but implausible. The 2% floor treats that as
* unmeasured too. */
double total = 0, avail = 0;
compat_meminfo_gb(&total, &avail);
static int warned = 0;
Expand Down
5 changes: 4 additions & 1 deletion c/openai_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -6660,7 +6660,10 @@ def cap_for_arch(arch, cap, env=None, model=None):
planned = int(env.get("COLI_PLAN_CAP", ""))
except (TypeError, ValueError):
planned = 0
if planned >= 1:
# OLMoE sizes its cache from RAM_GB itself, after its dense weights are
# resident, and gives slots back as the KV grows (kv_room_fit): an
# argv cap turns both off (#2785).
if planned >= 1 and arch != "olmoe":
return planned
if arch in ("deepseek_v41", "mimo", "qwen36", "qwen38") and model is not None:
# V4.1, MiMo, Qwen3.6 and Qwen3.8 only read their argv cap, not RAM_GB.
Expand Down
Loading
Loading