Repository navigation
colibri (Mistral-Large-4 vision) serve: a picture announced for a refused request does not go to the next one - #2771
Merged
JustVugg merged 1 commit intoNov 13, 2026
Conversation
…quest its IMAGE frame names An IMAGE frame was kept for whatever SUBMIT came next. A SUBMIT refused before the picture was attached (CONTEXT_EXCEEDED, EMPTY_PROMPT, BAD_REQUEST, SLOT_BUSY, DUPLICATE_ID) left it pending, and the next request, text only, went into mlv_attach and was answered ERROR BAD_IMAGE. The pending picture now keeps its IMAGE frame's id, as glm53 does; at SUBMIT a picture with another id is dropped with a stderr line before the attach. tests/test_ml4_pending_picture.py drives it on the tiny ML4 model and tower, wired into the Mistral-Large-4 CI step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
colibri (Mistral-Large-4 vision) serve: a picture announced for a refused request does not go to the next one
Base: dev @ 15f810f · Branch: kaira/colibri-ml4-pending-picture · Depends on: none
Summary
In
SERVE_BATCHmode,colibri.ckeeps anIMAGEframe's patches for theSUBMITthat follows, but not theframe's id. If that
SUBMITis refused before the picture is attached (CONTEXT_EXCEEDED,EMPTY_PROMPT,BAD_REQUEST,SLOT_BUSY,DUPLICATE_ID), the picture stays pending. The next request, even a text-only one, thengets it and is answered
ERROR BAD_IMAGE.This PR stores the
IMAGEframe's id with the picture and attaches the picture only to theSUBMITwith that id,which is what
glm53.calready does. A pending picture with another id is dropped, with one stderr line. A new servetest on the tiny Mistral-Large-4 model and tiny tower fails on dev and passes on this branch.
Problem on
dev(15f810f)c/colibri.c:g_mlv_pendingholdspatches, bytes, gh, gw, present, with no request id (:8850).mux_submit,IMAGE <id> <bytes> <grid_h> <grid_w>: the id is parsed intoiid(:11566), but only the patchesand the grid are stored (:11573-11576).
iidis used only in the frame's own error replies.mux_submit,SUBMIT: the pending picture is used and cleared in a single place, after every refusal (:11702-11706):if(g_mlv_pending.present){ mlv_attach(...); mlv_pending_clear(); ... }.Each of these returns earlier and leaves the picture pending:
BAD_REQUEST: slot out of range or NUL in the payload (:11640), or a badids=1payload (:11687);SLOT_BUSY(:11643),DUPLICATE_ID(:11646);EMPTY_PROMPT(:11692);CONTEXT_EXCEEDED(:11698).SUBMITthat gets past those checks findsg_mlv_pending.presentset and callsmlv_attachon its ownprompt. With no tower loaded,
mlv_attachfails at :8868. With a tower, a text-only prompt has 0[IMG]tokensagainst the picture's rows (:8877-8879). Either way the request is answered
ERROR <id> BAD_IMAGE(:11705), andthe picture is cleared only then.
docs/serve_protocol.mddefinesIMAGEas announcing the picture "for the request with the sameid" (:92-93).The gateway always sends
IMAGE <request_id>and the matchingSUBMITunder one lock (openai_server.py:7781), sothe id is available. In practice, an over-long prompt with a picture comes back as a 400 (correct). The next
request, sent by a different client and without a picture, then fails as a bad image.
glm53.cavoids this: it storesg_pending.idand uses the picture only wheng_pending.id == q->id(
glm53.c:5273).Fix
c/colibri.conly:g_mlv_pendinggainsid. TheIMAGEhandler storesiidin it.mux_submit, just before the existing attach block: if a picture is pending and its id is notsub.id, print[ml4] the picture of request <old> was not used; request <new> goes without itto stderr and clear it.The request then goes on as text only. A picture whose id matches goes to
mlv_attachexactly as before.The check sits at the one place that uses the picture. That avoids adding a clear to each of the refusal branches
(and to any added later). A refused request's picture therefore stays in memory until the next
SUBMITthat reachesthat point, or the next
IMAGEframe, which already replaces it. No new env flags.Tests
c/tests/test_ml4_pending_picture.py(new)The test drives the real
colibribinary inSERVE=1 SERVE_BATCH=1 KV_SLOTS=1 CTX=32, on the tiny Mistral-Large-4model with the tiny tower. It sends
ids=1prompts, so the byte-level tiny vocabulary does not matter. First itasserts
CAPS vision=1, so the tower is loaded. The tiny vocabulary has no[IMG]token, so a picture that reachesmlv_attachis refused there.test_a_refused_request_does_not_pass_its_picture_on:IMAGE 1, thenSUBMIT 1with 40 ids, which givesERROR 1 CONTEXT_EXCEEDED. ThenSUBMIT 2with 6 ids must end inDONE 2.test_the_picture_still_goes_to_its_own_request(the control):IMAGE 3+SUBMIT 3must still beERROR 3 BAD_IMAGE, because the picture reached the tower check. ThenSUBMIT 4ends inDONE 4.Fixture (about 5 s), and the command used on both trees:
'^DONE 2 ' not found in 'ERROR 2 BAD_IMAGE'The engine's stderr on this branch, test 1 then test 2:
The test is skipped without
COLI_ML4_FIXTURE, somake test-pythondiscovery is unchanged.CI
One line in the existing "Mistral-Large-4 tiny oracle" step of
.github/workflows/ci.yml, after the tower-loadcheck. It reuses that step's
/tmp/ml4/coli(GGUF import, which has atokenizer.json) and/tmp/ml4fv/mmproj.gguf:The step's
/tmp/ml4f/coli(the FP8 converter's container, which carriesmmproj.gguf) was not used: it has notokenizer.json, and serve mode exits on it beforeREADY. I checked that locally by generating that container withthe step's own commands.
Other checks
make colibribuilds on both trees with no warnings.mux_submit, run on this branch on the tiny GLM oracle (tools/make_glm_oracle.py):COLI_ORACLE_FIXTURE=$PWD/glm_tiny python -m unittest tests.test_run_serve_context_exceeded tests.test_glm_prompt_cancel tests.test_glm_systemone tests.test_glm_park: 12 tests, OK.Not verified
unchanged.
EMPTY_PROMPT,BAD_REQUEST,SLOT_BUSY,DUPLICATE_ID) are not driven one by one.They all return before the one check this PR adds, which the
CONTEXT_EXCEEDEDcase exercises.Compatibility and risks
IMAGEframe: the new check is skipped (presentis 0) and nothing changes.IMAGE <id>followed bySUBMIT <id>(what the gateway sends): unchanged.IMAGEwith an id other than itsSUBMIT's: its picture used to be attached anyway; now it isdropped with a stderr line, and the request is served as text.
docs/serve_protocol.mdalready definesIMAGEas"for the request with the same
id" (:92-93), the gateway sends it that way, andglm53.calready behaves thisway.
Related open PRs
None touches the
IMAGE/SUBMIThandling incolibri.cor this part ofci.yml. #2602 editsc/colibri.cinother functions (
qalloc,qt_fill,qt_from_disk, the Vulkan/QT destroy paths). #2712 and #2618 edit otherci.ymlhunks.Template checklist
make -C c check: not run in full. The new test, the four GLM serve tests andmake colibriwere run (above).🤖 Generated with Claude Code