Skip to content

Batch 2026-11-15-a: CI, setup and FreeBSD fixes, the gateway, the dashboard, Q2_K and qwen36's serve codec (14 PRs) - #2848

Merged
JustVugg merged 32 commits into
devfrom
batch/2026-11-15-a
Nov 15, 2026
Merged

JustVugg merged 32 commits into
devfrom
batch/2026-11-15-a

Conversation

@JustVugg

Copy link
Copy Markdown
Owner

Fourteen PRs on dev after #2844, each merged with its own commits and none by hand: CI, setup and FreeBSD fixes, the gateway, the dashboard and two engine changes that keep the output's bits.

What is in it

CI

PR What it does
#2828 A new push to a pull request cancels that PR's older ci.yml run; pushes to dev and main keep a group of their own.
#2827 The Sanitizers job and the Python job's engine build compile on every core (-j"$(nproc)").

Setup, FreeBSD and the launcher

PR What it does
#2829 Setup leaves out the "no AVX2" note when the CPU features could not be read (features_known), instead of taking "none read" for "none present".
#2817 coli setup and coli build run gmake on the BSDs; without it the toolchain check says it cannot build.
#2816 FreeBSD: the physical cores from the kern.smp.cores sysctl, in the plan and in omp_tune.h.
#2814 mistral_vision.h: the 8-float sum takes its vector by pointer (clang's -Wpsabi without AVX); the same floats in the same order.
#2813 coli stop asks ps when the /proc scan lists no process, one -o per column.
#2802 Two test fixes after the FreeBSD suite: detect_cpu pinned for the Windows release test, a tolerance near zero in the int2 converter test.
#2815 start-here.sh finds a Python 3.10+ outside PATH (versioned names, COLIBRI_PYTHON_DIRS, Homebrew and pyenv locations) (#2812), with a shell test.

Gateway and dashboard

PR What it does
#2819 /v1/messages takes stop_sequences (stop_reason: "stop_sequence"), POST /v1/messages/count_tokens gives an estimate, errors carry Anthropic's type for their status, and an error after a stream started is sent as an error event.
#2838 The disconnect watch reads the socket's pending error before peeking, so a reset behind pipelined bytes cancels the upstream prefill.
#2842 The dashboard shows prompt speed (pp/s, without the tokens reused from the KV prefix) next to tg/s; each model card shows its own architecture.

Engines, the same bits

PR What it does
#2843 Q2_K gets the fused row kernel the other K-quants have (GQ_DOT_SBK), with gq_dot_i8's float order: every row's result is unchanged.
#2845 qwen36 reads its serve frames through serve_codec.h, as the sibling engines do. Valid frames give the same requests; a frame that breaks off ends the session instead of reading the rest of the stream as commands, and a SUBMIT or DECIDE that arrives during a lone serve's turn is consumed and refused.

By @KairaCode (#2819, #2827, #2828), @risner (#2802, #2813, #2814, #2816, #2817, #2829), @ZacharyZcR (#2843, #2845), @Varshith-Kali (#2815), @agammann (#2838) and @Neppord (#2842).

Tests

ds (Ryzen, 16 threads)

  • make check: every C test and 3581 Python tests pass, the new test_qwen36_serve_framing and test_gq_q2_k among them. test_gq_q2_k also built at -march=x86-64-v3, as CI builds it.
  • tests/test_start_here_python_detection.sh (fix(setup): find a usable Python outside PATH in start-here.sh (#2812) #2815): 6 of 6 cases.
  • The steps of the CI jobs this batch touches, run one by one: qwen36-tiny-check (all 17 steps: the oracle at several caps, MTP and multi-turn serve, the 2.4T, Qwen3-Coder, Qwen3-8B and 27B geometries, ASan + UBSan, KV prefix reuse, the int4 gs64 A/B, the dashboard), clef-tiny-check (DECIDE through the new reader) and check.yml's generated qwen36 oracles (Flint, Muse Spark, LFM3).
  • make test-asan: the suite clean under ASan + UBSan.
  • Served, dev against this branch, Qwen3.8-27B UD-IQ2_XXS on the CPU (qwen36 engine; three of its attention matrices are Q2_K, so ggml_quants: a fused row kernel for Q2_K, the same bits #2843's kernel runs), with one KV slot (the lone serve, whose mid-turn reads refactor(serve): share framing with qwen36 #2845 changed) and with two:
    • the same text from both for chat, streamed chat, /v1/completions, /v1/messages plain and streamed (thinking on and off), two requests at once, a prompt of sixty paragraphs, and the request after a client that left during a prompt;
    • a client that leaves after four streamed tokens of a 200-token answer: the engine stops 0.3 s after the cancel on both, and the next request answers in 5.0 to 5.5 s, against 76 to 81 s for the whole answer;
    • stop_sequences: ["\n"] ends at the first line with stop_reason: "stop_sequence", plain and streamed; count_tokens answers 200 (dev: 400 and 404, as before).
  • Decode speed, dev and this branch alternated three times, 64 tokens (seconds, second and third run of each): 24.09 / 24.16 against 23.59 / 23.60, then 23.67 / 23.76 against 25.07 / 25.40 as the machine's load rose, then 25.25 / 25.52 against 25.23 / 25.46 at the same load. The same speed.

The web changes (#2842) are covered by CI's Web UI job; ds has no Node.

risner and others added 30 commits November 13, 2026 12:28
…hropic error types

/v1/messages refused `stop_sequences` with a 400, so an Anthropic client that
sends them (the SDKs pass them through, agent frameworks set them) could not use
the endpoint at all. They are now added to the family's implicit role markers:
a client sequence ends the turn with stop_reason "stop_sequence" and is named in
`stop_sequence` (response body and message_delta); a role marker still ends it
as end_turn.

POST /v1/messages/count_tokens answers {"input_tokens": N}: the prompt
/v1/messages would render, times the tokens per character the server measured
on the prompts it served (COMPACT_DEFAULT_RATIO before the first). No engine
call.

The Anthropic error envelope now carries the `type` the HTTP status has in
Anthropic's API (authentication_error, not_found_error, rate_limit_error,
overloaded_error, api_error, ...) instead of the OpenAI one (server_error for
every 5xx, invalid_request_error for a 404). An error after the stream has
committed is sent as an `event: error` before the close, as Anthropic's streams
do, instead of a bare truncation. message_delta usage carries input_tokens too
(message_start goes out before the prompt is counted).

The OpenAI endpoints are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #2812. start-here.sh only probed python3 and python on PATH, so a
Homebrew Python that the interactive shell reaches through an alias (or
a Homebrew bin directory that is not on the script's PATH) was invisible
and first run failed with 'Python 3.10 or newer was not found' even
though python3 works in the same terminal.

Probe versioned names on PATH as well, then the well-known install
locations: Homebrew bin directories, pyenv shims and versioned installs,
and keg-only python@3.x cellars. COLIBRI_PYTHON_DIRS adds
colon-separated directories ahead of the built-in locations, for unusual
installs and for tests. The 3.10 floor is unchanged, and a PATH python
is still preferred when it is usable.

Add tests/test_start_here_python_detection.sh, a dependency-free
regression harness (run: sh tests/test_start_here_python_detection.sh).
It runs the script against a stub launcher with Python only present
outside PATH: 3 cases fail before this fix and pass after, and the
no-usable-Python cases still report not-found with exit 1.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
The Sanitizers job compiles the whole C suite with ASan + UBSan on one
core; nearly all of its time is gcc, the tests themselves run in about a
minute. test-asan runs clean and then test-c as two sub-makes, so -j
only parallelises the compiling. The Python tests job's engine build gets
the same -j.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new push to a pull request now cancels that PR's older CI run, the same
way check.yml already does. Push runs on dev and main each get a group of
their own, so this change never cancels or queues them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Track and display pp/s badge in topbar from uncached prompt tokens and TTFT (and clear on chat reset)
- Exclude cached KV prefix tokens when computing pp/s across multi-turn chats
- Clarify token generation rate as tg/s across live badges, charts, and profiling tables
- Add prompt processing rate pp/s to message footer stats and monitor in ui/index.html
- Add translated pp/s, tg/s, and metric tooltip strings across ui/index.html I18N and web/src/i18n/*
Q2_K was the only K-quant without one: decode unpacked each row to int8
and then ran gq_dot_i8 over it. gq_sbk_q2_k decodes a sub-block the way
gq_ublk's Q2_K case does, and GQ_DOT_SBK keeps gq_dot_i8's float order
(the per-16 scales and the min term's fmaf chain), so every row's result
is unchanged. On an EPYC 7532 (2048 x 4096, S = 1) a matvec takes
1.04 ms instead of 1.28 ms on one thread and 0.15 ms instead of 0.24 ms
on eight.

tests/test_gq_q2_k.c checks the kernel and gq_matmul against the old
path to the bit for one to four activation rows; CI runs it on the
runner's ISA and on x86-64-v3.
serve_read_req takes its input and output streams, and the turn's
STOP/CANCEL poll reads through serve_control_read with a readiness
check passed in, so a test can drive both on synthetic frames. The
engine still passes stdin and stdout; nothing it does changes.

tests/test_qwen36_serve_framing.c records what the reader makes of the
frames the gateway sends: SUBMIT with its fields, payload and the
logprobs/pin keys, max_tokens=0 only with logprobs, an empty prompt,
STOP and CANCEL, IMAGE held for the following SUBMIT, DECIDE with a
Clef head, frames in sequence, and the poll's rule that only the turn's
own controls count and CANCEL wins over STOP.
serve_read_req reads its frames through serve_codec.h, as the sibling
engines do. Valid frames give the same requests (the frozen tests pass
unchanged), with the limits this reader had: an image up to 1 GiB, a
prompt up to 16 MiB and a decision record up to 64 MiB, the last two
checked once the payload is consumed.

Malformed frames no longer read the rest of the stream as commands:
- a payload that does not end on its newline, or a short read, ends the
  session (every later read returns end of input);
- a header longer than the line is one refused line, not a second
  command made of its tail;
- ids=1 (GLM's token ids) is refused with its body consumed, instead of
  tokenizing the digits as text;
- a DECIDE to an engine without a Clef head is consumed.

The turn's STOP/CANCEL poll reads whole frames too: a SUBMIT or DECIDE
that arrives while the one slot is busy is consumed and refused with
SLOT_BUSY, where its header was dropped and its payload left in the
stream, and an IMAGE is held for its SUBMIT. Admission, queueing, the
cancellation policy and every frame the engine writes stay with qwen36.

A malformed extension key is now answered "bad submit header", like the
other codec engines; the comments that named "bad submit extension"
say so.
setup: no "no AVX2" note when the features could not be read
setup: build with gmake on the BSDs
FreeBSD: count physical cores with the kern.smp.cores sysctl
mistral_vision: pass the 8-float accumulator by pointer
coli stop: find processes with ps where /proc lists none
tests: run the test suite on FreeBSD
fix(setup): find a usable Python outside PATH in start-here.sh (#2812)
Fix reset detection with queued pipelined requests
ggml_quants: a fused row kernel for Q2_K, the same bits
gateway: /v1/messages takes stop_sequences, adds count_tokens and Anthropic error types
dashboard: add uncached pp/s metrics, clarify tg/s, and fix model card arch & fast-turn stats
refactor(serve): share framing with qwen36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants