Skip to content

fix: read missed experts concurrently, prefill lookahead, PILOT worker pool - #2850

Open
veerareddyvishal144 wants to merge 1 commit into
JustVugg:devfrom
veerareddyvishal144:par-expert-loads
Open

veerareddyvishal144 wants to merge 1 commit into
JustVugg:devfrom
veerareddyvishal144:par-expert-loads

Conversation

@veerareddyvishal144

Copy link
Copy Markdown
Contributor

Summary

On a small-RAM box the qwen36 expert cache misses more than it hits, and every miss was served as one pread at a time (three per expert in link mode), so the drive sat at queue depth 1 while the engine waited. This PR keeps the maths and the token ids exactly as they were and only changes how misses are served:

  • moe_gq_decode: a token's missed experts are read concurrently (expert_fetch already loads outside its lock, as the PILOT worker does).
  • moe_gq_run: the layer's needed experts are known up front; misses are read PF_CHUNK (default 8, capped at cap/2) at a time, and the next chunk is handed to the prefetch pool while the current one computes (PF_AHEAD=1, default).
  • The PILOT worker becomes a pool (PILOT_THREADS, default 1) sharing the ring with a CAS claim per entry.
  • Loader threads no longer open an OpenMP team per expert for the row interleave: four workers each spawning sixteen threads on top of the compute team halved warm decode in testing.

PAR_LOAD=0 and PF_AHEAD=0 restore the previous behaviour; defaults are on because the output is unchanged and the cost on a fully resident cache is nil.

Measurements

Host: 16-core Xeon 8171M VM (AVX-512), 1 NUMA node. Disk: 2 MiB direct reads 1.08 GB/s at QD1, 2.30 GB/s at QD8. Model: Ornith-1.5-35B-A3B (qwen35moe, same geometry as Qwen3.6-35B-A3B) Q4_K_M, imported with coli import (link mode unless noted). Limit: systemd-run -p MemoryMax=16G, page cache dropped (echo 3 > drop_caches) before every run, .coli_usage present from one prior run, 16 OMP threads, cap 128 experts/layer (~11.2 GB RSS). Numbers are the engine's own footer.

2509-token prompt, then 128 tokens (main @ 0a5504a):

arm TTFT decode after hit rate
link, before 83.7 s 1.06 tok/s 68.4%
link, PAR_LOAD=1 57.0 s 1.53 tok/s 74.7%
link, PAR_LOAD=1 PF_AHEAD=1 PILOT=1 PILOT_THREADS=4 53.5 s 1.44 tok/s 92.2%
copy container, before 64.2 s 1.25 tok/s 68.4%
copy container, all on 54.2 s 1.42 tok/s 92.9%

32-token prompt, then 256 tokens, copy container: TTFT 12.8 s -> 5.8 s, decode 3.78 -> 3.80 tok/s (at this cap the residual ~25 misses/token are mostly single misses per layer, so decode is latency-bound either way; the gain there comes from a higher hit rate).

With Q3_K experts (imatrix requant, 13.8 GB) and cap 256 under the same 16G limit: 768 tokens at 8.00 tok/s before, 10.17 tok/s after; steady state from (768 - 256 tokens) 12.4 tok/s.

Rebased on dev (b046149) and re-run (copy container, cap 128, same protocol):

arm TTFT decode / hit rate
32-token prompt + 256 tokens, before 14.3 s 3.34 tok/s, 87.2%
32-token prompt + 256 tokens, all on 5.7 s 4.70 tok/s, 88.9%
2509-token prompt + 32 tokens, before 72.3 s hit rate 39.8%
2509-token prompt + 32 tokens, all on 54.2 s hit rate 86.9%

Validation

  • Clean build on dev: make -C c qwen36, 0 warnings.
  • Output is byte-identical with and without the change: cap 32 (heavy streaming), 24 greedy tokens, both a 32-token and a 2509-token prompt, on the main-based build and again on the dev-based build.
  • make check on dev: check-env passes (849 variables, registry matches the sources; the three new knobs are registered). The full make check was still running when this PR was opened; its result will be added here.
  • The GLM oracle was not run: this change does not touch colibri.c, and the glm_tiny fixture needs torch, which this host does not have.

🤖 Generated with Claude Code

On a small-RAM box the expert cache misses more than it hits, and every miss
was one pread at a time (three in link mode), so the drive sat at queue depth
1 while the whole engine waited. This keeps the maths and the token ids
exactly as they were and only changes how misses are served:

- moe_gq_decode: a token's missed experts are read concurrently
  (expert_fetch already loads outside its lock, as the PILOT worker does).
- moe_gq_run: the layer's needed experts are known up front; misses are read
  PF_CHUNK (default 8, capped at cap/2) at a time, and the next chunk is handed
  to the prefetch pool while the current one computes (PF_AHEAD=1 default).
- The PILOT worker is a pool (PILOT_THREADS, default 1) sharing the ring with
  a CAS claim per entry. Loader threads no longer open an OpenMP team per
  expert for the row interleave: four workers each spawning sixteen threads
  on top of the compute team halved warm decode.

PAR_LOAD=0 and PF_AHEAD=0 restore the previous behaviour.

Measured on a 16-core Skylake VM (disk 1.1 GB/s at QD1, 2.3 GB/s at QD8),
Ornith-1.5-35B-A3B (qwen35moe) Q4_K_M imported from the GGUF, cgroup
MemoryMax=16G, page cache dropped before every run, cap 128 experts/layer:

  2509-token prompt then 128 tokens          TTFT   decode   hit rate
    link, before                              83.7 s  1.06 t/s  68.4%
    link, parallel misses                     57.0 s  1.53 t/s  74.7%
    link, + lookahead + PILOT=1 x4 workers    53.5 s  1.44 t/s  92.2%
    copy container, before                    64.2 s  1.25 t/s  68.4%
    copy container, all on                    54.2 s  1.42 t/s  92.9%
  32-token prompt then 256 tokens (copy, all on)     TTFT 12.8 -> 5.8 s

With Q3_K experts (imatrix requant, 13.8 GB) and cap 256 under the same
16G limit: 768 tokens at 8.00 t/s before, 10.17 t/s after; steady state
(768 vs 256 tokens) 12.4 t/s.

Output text is byte-identical with and without the change at cap 32 on
both prompts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@veerareddyvishal144 veerareddyvishal144 changed the title qwen36: read missed experts concurrently, prefill lookahead, PILOT worker pool fix: read missed experts concurrently, prefill lookahead, PILOT worker pool Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant