Repository navigation
fix: read missed experts concurrently, prefill lookahead, PILOT worker pool - #2850
Open
veerareddyvishal144 wants to merge 1 commit into
Open
veerareddyvishal144 wants to merge 1 commit into
veerareddyvishal144 wants to merge 1 commit into
Conversation
On a small-RAM box the expert cache misses more than it hits, and every miss
was one pread at a time (three in link mode), so the drive sat at queue depth
1 while the whole engine waited. This keeps the maths and the token ids
exactly as they were and only changes how misses are served:
- moe_gq_decode: a token's missed experts are read concurrently
(expert_fetch already loads outside its lock, as the PILOT worker does).
- moe_gq_run: the layer's needed experts are known up front; misses are read
PF_CHUNK (default 8, capped at cap/2) at a time, and the next chunk is handed
to the prefetch pool while the current one computes (PF_AHEAD=1 default).
- The PILOT worker is a pool (PILOT_THREADS, default 1) sharing the ring with
a CAS claim per entry. Loader threads no longer open an OpenMP team per
expert for the row interleave: four workers each spawning sixteen threads
on top of the compute team halved warm decode.
PAR_LOAD=0 and PF_AHEAD=0 restore the previous behaviour.
Measured on a 16-core Skylake VM (disk 1.1 GB/s at QD1, 2.3 GB/s at QD8),
Ornith-1.5-35B-A3B (qwen35moe) Q4_K_M imported from the GGUF, cgroup
MemoryMax=16G, page cache dropped before every run, cap 128 experts/layer:
2509-token prompt then 128 tokens TTFT decode hit rate
link, before 83.7 s 1.06 t/s 68.4%
link, parallel misses 57.0 s 1.53 t/s 74.7%
link, + lookahead + PILOT=1 x4 workers 53.5 s 1.44 t/s 92.2%
copy container, before 64.2 s 1.25 t/s 68.4%
copy container, all on 54.2 s 1.42 t/s 92.9%
32-token prompt then 256 tokens (copy, all on) TTFT 12.8 -> 5.8 s
With Q3_K experts (imatrix requant, 13.8 GB) and cap 256 under the same
16G limit: 768 tokens at 8.00 t/s before, 10.17 t/s after; steady state
(768 vs 256 tokens) 12.4 t/s.
Output text is byte-identical with and without the change at cap 32 on
both prompts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On a small-RAM box the qwen36 expert cache misses more than it hits, and every miss was served as one
preadat a time (three per expert in link mode), so the drive sat at queue depth 1 while the engine waited. This PR keeps the maths and the token ids exactly as they were and only changes how misses are served:moe_gq_decode: a token's missed experts are read concurrently (expert_fetchalready loads outside its lock, as the PILOT worker does).moe_gq_run: the layer's needed experts are known up front; misses are readPF_CHUNK(default 8, capped at cap/2) at a time, and the next chunk is handed to the prefetch pool while the current one computes (PF_AHEAD=1, default).PILOT_THREADS, default 1) sharing the ring with a CAS claim per entry.PAR_LOAD=0andPF_AHEAD=0restore the previous behaviour; defaults are on because the output is unchanged and the cost on a fully resident cache is nil.Measurements
Host: 16-core Xeon 8171M VM (AVX-512), 1 NUMA node. Disk: 2 MiB direct reads 1.08 GB/s at QD1, 2.30 GB/s at QD8. Model: Ornith-1.5-35B-A3B (
qwen35moe, same geometry as Qwen3.6-35B-A3B) Q4_K_M, imported withcoli import(link mode unless noted). Limit:systemd-run -p MemoryMax=16G, page cache dropped (echo 3 > drop_caches) before every run,.coli_usagepresent from one prior run, 16 OMP threads,cap128 experts/layer (~11.2 GB RSS). Numbers are the engine's own footer.2509-token prompt, then 128 tokens (main @ 0a5504a):
PAR_LOAD=1PAR_LOAD=1 PF_AHEAD=1 PILOT=1 PILOT_THREADS=432-token prompt, then 256 tokens, copy container: TTFT 12.8 s -> 5.8 s, decode 3.78 -> 3.80 tok/s (at this cap the residual ~25 misses/token are mostly single misses per layer, so decode is latency-bound either way; the gain there comes from a higher hit rate).
With Q3_K experts (imatrix requant, 13.8 GB) and
cap256 under the same 16G limit: 768 tokens at 8.00 tok/s before, 10.17 tok/s after; steady state from (768 - 256 tokens) 12.4 tok/s.Rebased on
dev(b046149) and re-run (copy container, cap 128, same protocol):Validation
dev:make -C c qwen36, 0 warnings.dev-based build.make checkondev:check-envpasses (849 variables, registry matches the sources; the three new knobs are registered). The fullmake checkwas still running when this PR was opened; its result will be added here.colibri.c, and theglm_tinyfixture needs torch, which this host does not have.🤖 Generated with Claude Code