diff --git a/tools/boss/docs/postmortems/incident-010-local-dispatch-quarantine-recurrence-and-unshipped-followup.md b/tools/boss/docs/postmortems/incident-010-local-dispatch-quarantine-recurrence-and-unshipped-followup.md new file mode 100644 index 0000000000..16b83ce829 --- /dev/null +++ b/tools/boss/docs/postmortems/incident-010-local-dispatch-quarantine-recurrence-and-unshipped-followup.md @@ -0,0 +1,119 @@ +# Incident 010 — Local dispatch quarantine recurred while its fix waited + +- **Date:** 2026-11-17; November times below are CST (UTC−06:00). The October creation time is CDT (UTC−05:00). +- **Impact:** New local launches were refused; existing workers continued. The remote host was disabled. Recovery again required the repo owner to delete a metadata row. +- **Provenance:** The supplied incident record, repository source and history, and public PR descriptions. No production database, installed app, or private engine logs were accessed for this investigation. +- **Baseline:** `c3c7282c` on main; the recorded update installed 1.0.1033 (`59abc9fe`). Current source is evidence of implementation, not proof of which build ran on November 11. +- **Related:** [Incident 008](incident-008-local-dispatch-quarantined-by-update-mid-spawn.md), merged in [PR #3403](https://github.com/spinyfin/mono/pull/3403). Incident 009 already exists, so this report uses 010. + +## Verdict + +The known restart-during-spawn defect remained capable of disabling all local dispatch. The earlier follow-up had never run: it waited roughly thirteen days for an execution, then that execution was abandoned and automatic start was disabled, leaving the fix in the backlog until the recurrence prompted intervention. + +The record establishes that delivery failure, but **does not establish which admission gate caused the initial delay or which caller performed the November 11 abandonment and flag change**. Focus mode and capacity are possible launch delays, not demonstrated explanations for thirteen days without an execution. An open attention record did not provide an effective escalation to the repo owner. + +## Timeline and evidence limits + +| Local time | Recorded event | Interpretation | +| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| Oct 29, 19:22 CDT | Follow-up “Stop one unverifiable worker spawn from quarantining all local dispatch” created at medium priority. | Filing established intent, not execution or shipment. | +| Oct 30 | Incident 008 merged as #3403. | Its writeup documented the defect; it did not change runtime behavior. | +| Nov 11, 02:59 CST | First execution created; it never started. | About 12 days 8 hours after filing, conventionally described as thirteen days. | +| Nov 11, 03:05 | `dispatch_stage_stalled` attention opened. | Detection existed about six minutes after execution creation. | +| Nov 11, 13:27 | Execution abandoned; automatic start disabled with actor `engine`. | About 10 hours 28 minutes after creation; the fix then remained `todo`. | +| Nov 17, 18:51 | Local spawn began for the Google Drive upload-dialog task, execution `exec_18e8fd6e24e04ca0_2e0`. | Supplied minute-level incident time. | +| Nov 17, 18:51:44 | Installed app replaced by update 1.0.1033; engine restarted. | Bundle ctime is the supplied replacement evidence. | +| After restart | Startup scan held the incomplete spawn; status showed an active spawn and running lifecycle, but no pane and no entry in the full agent list. | A database lifecycle label did not prove a worker existed. | +| ~18:58 onward | New local launches refused; one newly created project's first task failed four attempts. | Repeated identical quarantine refusals; retrying could not clear the gate. | +| ~19:05 | Repo owner deleted `local_worker_startup_quarantine` from metadata. | Dispatch resumed immediately; the repair launched successfully. The held execution remained for the repo owner to stop. | +| ~19:07 | Repo owner requested the earlier fix start. | Approximate supplied time, not a second independently timed launch. | + +**Timestamp inconsistency:** The supplied spawn epoch, `1794963106`, converts to **18:51:46 CST**, two seconds after the supplied bundle ctime. It does not independently prove a spawn-before-update ordering. The incomplete-spawn/restart sequence is accepted from the incident record; the precise interruption point remains an inference consistent with incident 008. Resolving the two-second discrepancy would require the original spawn and shutdown audit events. Do not derive a precise race duration from these timestamps. + +## Recurrence: the old failure mode remains + +Incident 008 §4 explains the mechanism: creation of a local run precedes persistence of its tmux identity; a restart inside that interval makes a new spawn look like a historical worker. Missing process identity means “unknown,” so startup retains it rather than proving it dead. + +At this report's baseline, [the startup scan](../../engine/core/src/local_worker_quarantine.rs) still merges previously held executions into its candidates regardless of terminal status. Release still requires a proven-dead process or a durable write more than an hour before kernel boot. [The extracted storage code](../../engine/work-db/src/local_worker_quarantine_store.rs) still turns any held execution into a global local-dispatch refusal and still prescribes rollback/drain/restart. For a recent null-pid spawn on the same boot, that advice cannot establish death. The first response repeated it despite incident 008 already documenting its failure. + +History comparison from #3403's merge (`5745ecbad957`) to `c3c7282c` shows that `local_worker_quarantine.rs` was refactored: storage moved into the database crate, the attention constant moved, and visibility changed. The candidate/hold/release behavior above did not become a scoped recovery mechanism. The file is therefore **not byte-for-byte unchanged**, but the relevant defect remains. The observed immediate recovery after metadata deletion is consistent with dispatch reading that hold afresh. + +[PR #3485, “Close recovery quarantine gaps”](https://github.com/spinyfin/mono/pull/3485), merged November 9 UTC, is a different quarantine. Its five changed files are under `engine/recovery/src/execution_bookmark*`; it protects preserved edits against pointer corruption and foreign changes. It does not clear startup worker holds, change spawn identity ordering, or serialize app updates with spawning. Its title is not evidence that this incident's repair shipped. + +## Why the follow-up did not ship + +### 1. Creation to first execution: cause not retained in the supplied evidence + +The scheduler distinguishes **creating an execution** from **admitting that execution to a worker slot**. [Focus-mode documentation](../dispatch-focus-mode.md) says out-of-scope automatic executions remain ready with a `focus_mode` wait reason; focus does not change automatic-start flags or task status. The scheduler likewise retains ready executions under `interactive_concurrency_cap`. Those gates can explain an execution waiting after November 11, but cannot alone explain the reported absence of any execution before then. + +[Task admission](../../engine/work-db/src/exec_status_helpers.rs), especially `task_accepts_execution`, refuses automatic creation for a `todo` task with automatic start disabled, parked or deleted tasks, unsuitable states, and certain explicit failure/answer holds. [Reconciliation](../../engine/work-db/src/parking_reconcile.rs) and [dispatch helpers](../../engine/work-db/src/dispatch_helpers.rs) also apply inherited parking, deferral, dependencies, and existing-execution rules. These are plausible places to investigate the earlier gap; none is identified by the supplied record. + +| Candidate explanation | What the evidence supports | +| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Focus mode | Can hold an already-created execution. No historical focus set or wait reason was supplied. Attribution is unproven. | +| Concurrency cap or unavailable capacity | Can delay slot admission. No occupancy history was supplied; medium priority does not establish starvation. Neither alone accounts for no execution row. | +| Automatic start disabled before Nov 11 | Can prevent creation while `todo`. The known false transition is at Nov 11, 13:27; it cannot be projected backward to Oct 29. | +| Deferred, parked, dependency-blocked, or unreconciled task | Source provides such gates. Historical flags, membership, requests, and reconciliation outcomes are missing. These remain hypotheses. | + +**Conclusion:** The thirteen-day wait is confirmed; its cause is unknown. A defensible reconstruction needs the creation payload and flag/status ledger, historical dispatch requests and wait reasons, and the deployed revision. Naming focus mode as the cause without those records would turn a possibility into a false finding. + +### 2. Abandonment and automatic-start disablement: separate transitions + +The source contains several paths that look similar in the UI but have different semantics: + +- **Startup ghost-active healing:** [dispatch.rs](../../engine/work-db/src/dispatch.rs), the boot-time heal around lines 163–266, abandons pending executions for active tasks with no run history and demotes them to `todo`, attributing the change to `engine`. It **does not clear automatic start** at this baseline or at inspected historical main `ca283c0224db` before November 11, 13:27 CST. It could explain abandonment, but not both recorded changes on its own. +- **Pre-start retry exhaustion:** `bounce_dispatch_failed_to_backlog` in the same file clears automatic start, sets `todo` and actor `engine`, and stamps `dispatch_failed_reason`, error, and time. [Pre-start failure storage](../../engine/work-db/src/pre_start_failure.rs) terminalizes those attempts as **failed**, not abandoned. Therefore this path also cannot be identified solely from the reported abandoned status. +- **Deferral or explicit updates:** [updates.rs](../../engine/work-db/src/updates.rs) can write an automatic-start patch and, when deferring, abandon pending executions through [chain helpers](../../engine/work-db/src/chain_helpers.rs). A combined update could produce the observed shape; no such request is in the supplied evidence. +- **Normal run start:** [executions_runs.rs](../../engine/work-db/src/executions_runs.rs) consumes automatic start as a single-use flag when recording a run. That is not an explanation for an execution with no start or run record. + +The matching timestamps and `engine` actor do not identify a unique caller or prove one atomic transition. The exact abandonment path remains unresolved; the needed discriminator is the status/flag audit basis, failure stamp, and execution run history at 13:27. This is a concrete evidence gap, not proof of a ten-hour abandonment timer. + +[PR #3586](https://github.com/spinyfin/mono/pull/3586), merged November 12, fixed reconciliation refusing to replace an abandoned execution that never started. It now remints when the task is otherwise runnable and no live sibling exists. This is relevant historical context, but it **does not override disabled automatic start**, so it is not proof this stranded follow-up would recover. + +### 3. Why the attention did not ensure recovery + +[Dispatch-stall escalation](../../engine/core/src/dispatch_stall_escalation.rs) files attention after five minutes on a sixty-second sweep, consistent with 02:59 → 03:05. It records and refreshes an attention; it does not itself start the task or request acknowledgement. Successful slot claim resolves that signal. Terminal executions are skipped by subsequent escalation passes, so an abandoned execution can leave an old open attention without a fresh escalation cycle. + +The [attention lifecycle documentation](../attention-lifecycle.md) distinguishes such durable signals from actual recovery. [Dispatch-failure recovery](../../engine/core/src/dispatch_failure_recovery_sweep.rs) can retry eligible tasks with failure stamps, but applies exclusions and churn limits; it is not a universal promise to restart every `todo` task with automatic start off. + +The record says the repo owner does not see open attention in practice. That supports an **ineffective delivery/acknowledgement channel**, not a claim that Boss emitted no signal. There is no evidence here of a delivered notification, acknowledgement, or an escalation deadline tied to this follow-up. The supported explanation for the final six-day gap is that automatic execution was disabled and no effective intervention reversed it until November 17. + +## Why updates could still interrupt spawning + +Comparing #3403's merge to this baseline yields **zero changes** in [UpdateLifecycle.swift](../../app-macos/Sources/UpdateLifecycle.swift) and [EngineProcessController.swift](../../app-macos/Sources/EngineProcessController.swift). The latter's `start(resetRestartBudget:)` compares the running and bundled engine fingerprints and calls `stopRunningEngine` on mismatch. That stop sends a shutdown RPC, waits for socket/process exit, and can escalate to signals. These are exit waits, not a handshake that closes spawn admission and waits for every accepted spawn to finish or roll back. + +The [shutdown handler](../../engine/app/src/app/sessions.rs) accepts an authenticated request and signals shutdown after a 50 ms response-flush delay. [Server shutdown](../../engine/app/src/app/server.rs) shuts down scoped tasks and preserves workers. Preserving established workers for later adoption does not make partially persisted spawn state complete. [The app quit check](../../app-macos/Sources/BossMacApp.swift) counts live agents; even a confirmation is not an engine-side spawn barrier and cannot close a concurrent admission race. + +The technical answer is that the required barrier was still absent. The delivery answer is that documenting a remedy did not implement it, and the repair never ran. Incident 008 §8 specifically separated its filed core repair from recommended update draining and a supported clearing CLI. The supplied current repair scope includes all three; that broader scope is recorded intent, not verified implementation or deployment. There is no evidence of an intentional decision to accept another outage. + +## Detection and recovery + +From the update at 18:51:44 to ~19:05 is roughly **13 minutes**; from the first recorded refusals at ~18:58 to recovery is roughly **7 minutes**. The exact restart/hold time and first human recognition are absent. Thus we cannot truthfully state exactly how long the outage went unnoticed. Refusals were observable by ~18:58, around six minutes after bundle replacement; recognition occurred by recovery at the latest. + +Earlier useful detection would be a persistent global dispatch-health banner when all eligible hosts are excluded, immediately naming the local hold and its safe recovery action. A failed launch should expose the same condition, deduplicate repeated attempts, and escalate an unacknowledged outage after a bounded interval. The old five-minute attention detected the stranded repair but did not ensure anyone acted. Recovery should be measurable by a successful new launch, as it was here, not merely by clearing an attention. + +## Action items and completion criteria + +These are proposals, not claims of implementation. The separate in-flight repair is described by the supplied record; its diff and release were not verified here. Additional proposals below still need filing. + +| Action | Coverage and ownership | Concrete acceptance evidence | +| ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Scope uncertain worker identity to the affected execution; make fresh spawns distinguishable from historical rows. | **Covered by in-flight repair.** Repair implementer. | Restart at each persistence boundary; preserve uncertain live workers while an unrelated task successfully launches. | +| Provide supported inspect/clear commands and accurate recovery text. | **Covered by in-flight repair.** Repair implementer. | Recover a null-identity hold without SQL; reject unsafe clearing of a demonstrably live worker; report the affected execution and resulting state. | +| Serialize engine replacement with spawn completion or rollback. | **Covered by in-flight repair.** Repair implementer. | Close admission before waiting; race an update with a new spawn; prove no partially recorded spawn is left behind. Include restart after interruption, not only a graceful happy path. | +| Make postmortem follow-ups accountable through shipment. | **Needs filing.** Repo owner assigns an owner and due date; engine/UI implementation tracks them. | Durable link from action to task, first actual start, PR, merged revision, and installed release. Alert if no execution starts by deadline; display the current gate even when no execution exists. Require explicit acknowledgement by the repo owner for indefinite deferral or disabling automatic start. A parked action stays overdue until acknowledged or shipped. | +| Escalate abandoned never-started repairs visibly. | **Needs filing; not covered merely by scoped quarantine.** Engine/UI implementation. | Record caller, previous/new flags and reason together; test automatic abandonment produces a persistent actionable banner and repeated bounded escalation until acknowledgement or recovery. Retain original failure context. | +| Reconstruct the missing Oct 29–Nov 11 admission and 13:27 transition history. | **Needs filing/access to retained evidence.** Repo owner supplies a sanitized audit export. | Identify historical admission reason, deployed revision, status/flag audit basis, and whether any run existed. If retention has lost the evidence, record that fact and add retention sufficient for future postmortems. | +| Detect loss of all dispatch capacity and avoid futile retries. | **Needs filing unless explicitly added to the repair.** Engine/UI implementation. | Immediate global banner, outage start/duration, acknowledgement, and a successful-launch recovery signal. Verify unrelated tasks retain their intent while the host gate is closed. | +| Verify shipment of this repair and all three promises. | **Needs explicit tracking; implementation scope alone is insufficient.** Repair owner, reviewed by repo owner. | Record the fixing PR and first installed version, exercise update-during-spawn and supported recovery, and close the postmortem action only with that evidence. | + +No scheduler can guarantee execution while respecting an explicit hold or unavailable capacity. The missing guarantee should instead be **execution by a deadline or an acknowledged, visible reason why it cannot run**, followed through to deployment. A filed task and an unread attention row satisfy neither condition. + +## Open questions + +- Which pre-execution admission condition held the repair before November 11, and what changed at 02:59? +- Which audited caller abandoned the execution and disabled automatic start at 13:27? Were these one transaction or separate actions? +- Which timestamp source explains the two-second spawn/update inconsistency, and when was the outage first recognized? +- Which PR and installed version will demonstrate that the supported clearing path, scoped failure, and spawn-safe update all shipped? + +These uncertainties limit attribution; they do not weaken the observed recurrence or the need to track corrective actions through a verified release. This document changes no runtime behavior.