Repository navigation
Batch 2026-11-15-a: CI, setup and FreeBSD fixes, the gateway, the dashboard, Q2_K and qwen36's serve codec (14 PRs) - #2848
Merged
Merged
Conversation
…hropic error types
/v1/messages refused `stop_sequences` with a 400, so an Anthropic client that
sends them (the SDKs pass them through, agent frameworks set them) could not use
the endpoint at all. They are now added to the family's implicit role markers:
a client sequence ends the turn with stop_reason "stop_sequence" and is named in
`stop_sequence` (response body and message_delta); a role marker still ends it
as end_turn.
POST /v1/messages/count_tokens answers {"input_tokens": N}: the prompt
/v1/messages would render, times the tokens per character the server measured
on the prompts it served (COMPACT_DEFAULT_RATIO before the first). No engine
call.
The Anthropic error envelope now carries the `type` the HTTP status has in
Anthropic's API (authentication_error, not_found_error, rate_limit_error,
overloaded_error, api_error, ...) instead of the OpenAI one (server_error for
every 5xx, invalid_request_error for a 404). An error after the stream has
committed is sent as an `event: error` before the close, as Anthropic's streams
do, instead of a bare truncation. message_delta usage carries input_tokens too
(message_start goes out before the prompt is counted).
The OpenAI endpoints are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #2812. start-here.sh only probed python3 and python on PATH, so a Homebrew Python that the interactive shell reaches through an alias (or a Homebrew bin directory that is not on the script's PATH) was invisible and first run failed with 'Python 3.10 or newer was not found' even though python3 works in the same terminal. Probe versioned names on PATH as well, then the well-known install locations: Homebrew bin directories, pyenv shims and versioned installs, and keg-only python@3.x cellars. COLIBRI_PYTHON_DIRS adds colon-separated directories ahead of the built-in locations, for unusual installs and for tests. The 3.10 floor is unchanged, and a PATH python is still preferred when it is usable. Add tests/test_start_here_python_detection.sh, a dependency-free regression harness (run: sh tests/test_start_here_python_detection.sh). It runs the script against a stub launcher with Python only present outside PATH: 3 cases fail before this fix and pass after, and the no-usable-Python cases still report not-found with exit 1. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
The Sanitizers job compiles the whole C suite with ASan + UBSan on one core; nearly all of its time is gcc, the tests themselves run in about a minute. test-asan runs clean and then test-c as two sub-makes, so -j only parallelises the compiling. The Python tests job's engine build gets the same -j. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new push to a pull request now cancels that PR's older CI run, the same way check.yml already does. Push runs on dev and main each get a group of their own, so this change never cancels or queues them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tok/s) generation
- Track and display pp/s badge in topbar from uncached prompt tokens and TTFT (and clear on chat reset) - Exclude cached KV prefix tokens when computing pp/s across multi-turn chats - Clarify token generation rate as tg/s across live badges, charts, and profiling tables - Add prompt processing rate pp/s to message footer stats and monitor in ui/index.html - Add translated pp/s, tg/s, and metric tooltip strings across ui/index.html I18N and web/src/i18n/*
Q2_K was the only K-quant without one: decode unpacked each row to int8 and then ran gq_dot_i8 over it. gq_sbk_q2_k decodes a sub-block the way gq_ublk's Q2_K case does, and GQ_DOT_SBK keeps gq_dot_i8's float order (the per-16 scales and the min term's fmaf chain), so every row's result is unchanged. On an EPYC 7532 (2048 x 4096, S = 1) a matvec takes 1.04 ms instead of 1.28 ms on one thread and 0.15 ms instead of 0.24 ms on eight. tests/test_gq_q2_k.c checks the kernel and gq_matmul against the old path to the bit for one to four activation rows; CI runs it on the runner's ISA and on x86-64-v3.
serve_read_req takes its input and output streams, and the turn's STOP/CANCEL poll reads through serve_control_read with a readiness check passed in, so a test can drive both on synthetic frames. The engine still passes stdin and stdout; nothing it does changes. tests/test_qwen36_serve_framing.c records what the reader makes of the frames the gateway sends: SUBMIT with its fields, payload and the logprobs/pin keys, max_tokens=0 only with logprobs, an empty prompt, STOP and CANCEL, IMAGE held for the following SUBMIT, DECIDE with a Clef head, frames in sequence, and the poll's rule that only the turn's own controls count and CANCEL wins over STOP.
serve_read_req reads its frames through serve_codec.h, as the sibling engines do. Valid frames give the same requests (the frozen tests pass unchanged), with the limits this reader had: an image up to 1 GiB, a prompt up to 16 MiB and a decision record up to 64 MiB, the last two checked once the payload is consumed. Malformed frames no longer read the rest of the stream as commands: - a payload that does not end on its newline, or a short read, ends the session (every later read returns end of input); - a header longer than the line is one refused line, not a second command made of its tail; - ids=1 (GLM's token ids) is refused with its body consumed, instead of tokenizing the digits as text; - a DECIDE to an engine without a Clef head is consumed. The turn's STOP/CANCEL poll reads whole frames too: a SUBMIT or DECIDE that arrives while the one slot is busy is consumed and refused with SLOT_BUSY, where its header was dropped and its payload left in the stream, and an IMAGE is held for its SUBMIT. Admission, queueing, the cancellation policy and every frame the engine writes stay with qwen36. A malformed extension key is now answered "bad submit header", like the other codec engines; the comments that named "bad submit extension" say so.
ci: cancel superseded PR runs
ci: Sanitizers builds on every core
setup: no "no AVX2" note when the features could not be read
setup: build with gmake on the BSDs
FreeBSD: count physical cores with the kern.smp.cores sysctl
mistral_vision: pass the 8-float accumulator by pointer
coli stop: find processes with ps where /proc lists none
tests: run the test suite on FreeBSD
fix(setup): find a usable Python outside PATH in start-here.sh (#2812)
Fix reset detection with queued pipelined requests
ggml_quants: a fused row kernel for Q2_K, the same bits
gateway: /v1/messages takes stop_sequences, adds count_tokens and Anthropic error types
dashboard: add uncached pp/s metrics, clarify tg/s, and fix model card arch & fast-turn stats
refactor(serve): share framing with qwen36
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourteen PRs on dev after #2844, each merged with its own commits and none by hand: CI, setup and FreeBSD fixes, the gateway, the dashboard and two engine changes that keep the output's bits.
What is in it
CI
ci.ymlrun; pushes todevandmainkeep a group of their own.-j"$(nproc)").Setup, FreeBSD and the launcher
features_known), instead of taking "none read" for "none present".coli setupandcoli buildrungmakeon the BSDs; without it the toolchain check says it cannot build.kern.smp.coressysctl, in the plan and inomp_tune.h.mistral_vision.h: the 8-float sum takes its vector by pointer (clang's-Wpsabiwithout AVX); the same floats in the same order.coli stopaskspswhen the/procscan lists no process, one-oper column.detect_cpupinned for the Windows release test, a tolerance near zero in the int2 converter test.start-here.shfinds a Python 3.10+ outsidePATH(versioned names,COLIBRI_PYTHON_DIRS, Homebrew and pyenv locations) (#2812), with a shell test.Gateway and dashboard
/v1/messagestakesstop_sequences(stop_reason: "stop_sequence"),POST /v1/messages/count_tokensgives an estimate, errors carry Anthropic'stypefor their status, and an error after a stream started is sent as anerrorevent.pp/s, without the tokens reused from the KV prefix) next totg/s; each model card shows its own architecture.Engines, the same bits
GQ_DOT_SBK), withgq_dot_i8's float order: every row's result is unchanged.serve_codec.h, as the sibling engines do. Valid frames give the same requests; a frame that breaks off ends the session instead of reading the rest of the stream as commands, and a SUBMIT or DECIDE that arrives during a lone serve's turn is consumed and refused.By @KairaCode (#2819, #2827, #2828), @risner (#2802, #2813, #2814, #2816, #2817, #2829), @ZacharyZcR (#2843, #2845), @Varshith-Kali (#2815), @agammann (#2838) and @Neppord (#2842).
Tests
ds (Ryzen, 16 threads)
make check: every C test and 3581 Python tests pass, the newtest_qwen36_serve_framingandtest_gq_q2_kamong them.test_gq_q2_kalso built at-march=x86-64-v3, as CI builds it.tests/test_start_here_python_detection.sh(fix(setup): find a usable Python outside PATH in start-here.sh (#2812) #2815): 6 of 6 cases.qwen36-tiny-check(all 17 steps: the oracle at several caps, MTP and multi-turn serve, the 2.4T, Qwen3-Coder, Qwen3-8B and 27B geometries, ASan + UBSan, KV prefix reuse, the int4 gs64 A/B, the dashboard),clef-tiny-check(DECIDE through the new reader) andcheck.yml's generated qwen36 oracles (Flint, Muse Spark, LFM3).make test-asan: the suite clean under ASan + UBSan./v1/completions,/v1/messagesplain and streamed (thinking on and off), two requests at once, a prompt of sixty paragraphs, and the request after a client that left during a prompt;stop_sequences: ["\n"]ends at the first line withstop_reason: "stop_sequence", plain and streamed;count_tokensanswers 200 (dev: 400 and 404, as before).The web changes (#2842) are covered by CI's Web UI job; ds has no Node.