Skip to content

colibri (Mistral-Large-4 vision) serve: a picture announced for a refused request does not go to the next one - #2771

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
KairaCode:kaira/colibri-ml4-pending-picture
Nov 13, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
KairaCode:kaira/colibri-ml4-pending-picture

Conversation

@KairaCode

Copy link
Copy Markdown

colibri (Mistral-Large-4 vision) serve: a picture announced for a refused request does not go to the next one

Base: dev @ 15f810f · Branch: kaira/colibri-ml4-pending-picture · Depends on: none

Summary

In SERVE_BATCH mode, colibri.c keeps an IMAGE frame's patches for the SUBMIT that follows, but not the
frame's id. If that SUBMIT is refused before the picture is attached (CONTEXT_EXCEEDED, EMPTY_PROMPT,
BAD_REQUEST, SLOT_BUSY, DUPLICATE_ID), the picture stays pending. The next request, even a text-only one, then
gets it and is answered ERROR BAD_IMAGE.

This PR stores the IMAGE frame's id with the picture and attaches the picture only to the SUBMIT with that id,
which is what glm53.c already does. A pending picture with another id is dropped, with one stderr line. A new serve
test on the tiny Mistral-Large-4 model and tiny tower fails on dev and passes on this branch.

Problem on dev (15f810f)

c/colibri.c:

  • g_mlv_pending holds patches, bytes, gh, gw, present, with no request id (:8850).
  • mux_submit, IMAGE <id> <bytes> <grid_h> <grid_w>: the id is parsed into iid (:11566), but only the patches
    and the grid are stored (:11573-11576). iid is used only in the frame's own error replies.
  • mux_submit, SUBMIT: the pending picture is used and cleared in a single place, after every refusal (:11702-11706):
    if(g_mlv_pending.present){ mlv_attach(...); mlv_pending_clear(); ... }.
    Each of these returns earlier and leaves the picture pending:
    • BAD_REQUEST: slot out of range or NUL in the payload (:11640), or a bad ids=1 payload (:11687);
    • SLOT_BUSY (:11643), DUPLICATE_ID (:11646);
    • EMPTY_PROMPT (:11692);
    • CONTEXT_EXCEEDED (:11698).
  • The next SUBMIT that gets past those checks finds g_mlv_pending.present set and calls mlv_attach on its own
    prompt. With no tower loaded, mlv_attach fails at :8868. With a tower, a text-only prompt has 0 [IMG] tokens
    against the picture's rows (:8877-8879). Either way the request is answered ERROR <id> BAD_IMAGE (:11705), and
    the picture is cleared only then.

docs/serve_protocol.md defines IMAGE as announcing the picture "for the request with the same id" (:92-93).
The gateway always sends IMAGE <request_id> and the matching SUBMIT under one lock (openai_server.py:7781), so
the id is available. In practice, an over-long prompt with a picture comes back as a 400 (correct). The next
request, sent by a different client and without a picture, then fails as a bad image.

glm53.c avoids this: it stores g_pending.id and uses the picture only when g_pending.id == q->id
(glm53.c:5273).

Fix

c/colibri.c only:

  • g_mlv_pending gains id. The IMAGE handler stores iid in it.
  • In mux_submit, just before the existing attach block: if a picture is pending and its id is not sub.id, print
    [ml4] the picture of request <old> was not used; request <new> goes without it to stderr and clear it.
    The request then goes on as text only. A picture whose id matches goes to mlv_attach exactly as before.

The check sits at the one place that uses the picture. That avoids adding a clear to each of the refusal branches
(and to any added later). A refused request's picture therefore stays in memory until the next SUBMIT that reaches
that point, or the next IMAGE frame, which already replaces it. No new env flags.

Tests

c/tests/test_ml4_pending_picture.py (new)

The test drives the real colibri binary in SERVE=1 SERVE_BATCH=1 KV_SLOTS=1 CTX=32, on the tiny Mistral-Large-4
model with the tiny tower. It sends ids=1 prompts, so the byte-level tiny vocabulary does not matter. First it
asserts CAPS vision=1, so the tower is loaded. The tiny vocabulary has no [IMG] token, so a picture that reaches
mlv_attach is refused there.

  1. test_a_refused_request_does_not_pass_its_picture_on: IMAGE 1, then SUBMIT 1 with 40 ids, which gives
    ERROR 1 CONTEXT_EXCEEDED. Then SUBMIT 2 with 6 ids must end in DONE 2.
  2. test_the_picture_still_goes_to_its_own_request (the control): IMAGE 3 + SUBMIT 3 must still be
    ERROR 3 BAD_IMAGE, because the picture reached the tower check. Then SUBMIT 4 ends in DONE 4.

Fixture (about 5 s), and the command used on both trees:

python tools/make_mistral_large4_tiny.py --out $T/ml4fx
python tools/gguf_import.py $T/ml4fx/model.gguf $T/ml4fx/coli
python tools/make_mistral_vision_tiny.py --out $T/ml4fv --out-hidden 64 --release $T/ml4f/fp8
make colibri
PYTHONUTF8=1 COLI_ML4_FIXTURE=$T/ml4fx/coli ML4_MMPROJ=$T/ml4fv/mmproj.gguf \
  python -m unittest -v tests.test_ml4_pending_picture
test 1 (refused request, then text) test 2 (control)
dev 15f810f FAIL: '^DONE 2 ' not found in 'ERROR 2 BAD_IMAGE' ok
this branch ok ok

The engine's stderr on this branch, test 1 then test 2:

[API] prompt does not fit: >=31 token, context is 30 (CTX=32). ...
[ml4] the picture of request 1 was not used; request 2 goes without it
[API] KV slot 0 prefix 0/6 token, prefill 6
[ml4] the prompt has 0 [IMG] tokens, the picture 4 rows

The test is skipped without COLI_ML4_FIXTURE, so make test-python discovery is unchanged.

CI

One line in the existing "Mistral-Large-4 tiny oracle" step of .github/workflows/ci.yml, after the tower-load
check. It reuses that step's /tmp/ml4/coli (GGUF import, which has a tokenizer.json) and /tmp/ml4fv/mmproj.gguf:

COLI_ML4_FIXTURE=/tmp/ml4/coli ML4_MMPROJ=/tmp/ml4fv/mmproj.gguf python3 -m unittest -v tests.test_ml4_pending_picture

The step's /tmp/ml4f/coli (the FP8 converter's container, which carries mmproj.gguf) was not used: it has no
tokenizer.json, and serve mode exits on it before READY. I checked that locally by generating that container with
the step's own commands.

Other checks

  • make colibri builds on both trees with no warnings.
  • The serve tests that share mux_submit, run on this branch on the tiny GLM oracle (tools/make_glm_oracle.py):
    COLI_ORACLE_FIXTURE=$PWD/glm_tiny python -m unittest tests.test_run_serve_context_exceeded tests.test_glm_prompt_cancel tests.test_glm_systemone tests.test_glm_park: 12 tests, OK.

Not verified

  • Linux. The test and the build ran on Windows 11 (MinGW gcc, Python 3.14). The CI line has not run in CI.
  • No real Mistral-Large-4 checkpoint run. The change is serve bookkeeping, and the tower and attach path are
    unchanged.
  • The other refusal branches (EMPTY_PROMPT, BAD_REQUEST, SLOT_BUSY, DUPLICATE_ID) are not driven one by one.
    They all return before the one check this PR adds, which the CONTEXT_EXCEEDED case exercises.

Compatibility and risks

  • No IMAGE frame: the new check is skipped (present is 0) and nothing changes.
  • IMAGE <id> followed by SUBMIT <id> (what the gateway sends): unchanged.
  • A client that sends IMAGE with an id other than its SUBMIT's: its picture used to be attached anyway; now it is
    dropped with a stderr line, and the request is served as text. docs/serve_protocol.md already defines IMAGE as
    "for the request with the same id" (:92-93), the gateway sends it that way, and glm53.c already behaves this
    way.

Related open PRs

None touches the IMAGE/SUBMIT handling in colibri.c or this part of ci.yml. #2602 edits c/colibri.c in
other functions (qalloc, qt_fill, qt_from_disk, the Vulkan/QT destroy paths). #2712 and #2618 edit other
ci.yml hunks.

Template checklist

  • make -C c check: not run in full. The new test, the four GLM serve tests and make colibri were run (above).
  • No CUDA, Metal or Vulkan code changed.
  • No performance claim.
  • The default CPU build remains dependency-free.
  • No model files, generated binaries, or benchmark artifacts are included.

🤖 Generated with Claude Code

…quest its IMAGE frame names

An IMAGE frame was kept for whatever SUBMIT came next. A SUBMIT refused
before the picture was attached (CONTEXT_EXCEEDED, EMPTY_PROMPT,
BAD_REQUEST, SLOT_BUSY, DUPLICATE_ID) left it pending, and the next
request, text only, went into mlv_attach and was answered ERROR BAD_IMAGE.

The pending picture now keeps its IMAGE frame's id, as glm53 does; at
SUBMIT a picture with another id is dropped with a stderr line before the
attach. tests/test_ml4_pending_picture.py drives it on the tiny ML4 model
and tower, wired into the Mistral-Large-4 CI step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JustVugg
JustVugg merged commit 4cddfc0 into JustVugg:dev Nov 13, 2026
117 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants