Skip to content

Guard AI sizing estimates against run-to-run jumps; Monte Carlo varies AI ranges - #65

Merged
sa-anyan merged 1 commit into
mainfrom
e1/ai-estimate-guardrails
Nov 14, 2026
Merged

sa-anyan merged 1 commit into
mainfrom
e1/ai-estimate-guardrails

Conversation

@sa-anyan

Copy link
Copy Markdown
Owner

On the same M&E, the reactor came out at 15.6 m³, 125 m³ and 156 m³ on different runs. The AI re-estimated its sizing inputs on later runs (a pricing repair, a model change, a cleared row) and nothing compared the new size with the earlier one. Monte Carlo also never varied AI guesses, because they were saved without a range.

Guardrail (backend/blocks/equipment/estimate_guard.py)

Each item's first AI-derived size is anchored together with a digest of its M&E data (the item and its connected streams). Every later AI size is measured against that anchor:

Case Result
Same M&E, change ≤ 25% Kept (stable). Compared with the original anchor, so drift never creeps.
Same M&E, change > 25% Held. The anchored estimate is priced; the new one is saved with its size, its drift and the assumptions behind it.
M&E changed, or size unit changed Re-anchored. That is a legitimate change.
User entered any sizing input The user owns it: re-anchored, never held.
Manual price, or no size Left alone.

Monte Carlo varies AI predictions

  • AI ranges: each AI sizing assumption now returns low/high: the engineering range for that service, not a fixed percentage. Units are converted, and the range is dropped if the value falls outside it.
  • Prompt version: bumped to …-27-assumption-ranges, so older cached answers aren't reused.
  • No change to Monte Carlo itself: discovery already varies any input with a saved range.
  • Disagreement becomes uncertainty: when two AI runs disagree, the shared assumption's range widens to cover both, for example productivity 1–2 kg/m³/h.
  • One small fix: discovery no longer crashes when a price basis was saved with a null currency.

Live check (real AI, the user's workbook, a copy of the database)

Check Result
AI equipment run Every AI-sized item anchored. Ranges returned, for example residence time 24 h (4–168 h), working fraction 0.85 (0.7–0.95)
Second AI run Every item stable, 0% drift
Monte Carlo discovery 13 of 17 AI sizing inputs varied. The 4 fixed are densities from M&E measurements.

Tests

  • Backend pytest (Python 3.12.10): 4,675 passed, 3 skipped, 0 failures. The Monte Carlo and guard tests were rerun after the null-currency fix: 70 passed.
  • New test_estimate_guard.py (8 tests) covers:
    • anchoring, holding a jump (with the restored inputs and the widened range), drift within tolerance, re-anchoring when the M&E or user changes, items left alone;
    • range unit conversion and validity;
    • Monte Carlo varying an AI assumption that has a range;
    • wiring: the equipment run prices the anchored size and recalculates once.
  • Frontend: tsc OK, node tests 64/64 (held list, keep and use), next build OK.

Not covered

  • Existing studies: they have no anchor yet. Their next AI equipment run sets one. The 15.6, 125 and 156 m³ values already saved in the user's studies aren't corrected; the user should check the reactor basis once.
  • Cross-study anchors: each study anchors on its own. Re-uploading the same workbook as a new study starts fresh, apart from the AI cache.

🤖 Generated with Claude Code

…s AI ranges

- New equipment estimate guard: each item's first AI-derived size is anchored
  with a digest of its M&E. A later AI size more than 25% away for the same
  M&E is held for review and the anchored estimate is priced; a changed M&E,
  a changed size unit or any user input re-anchors instead. Drift is measured
  and recorded with the last proposals.
- AI sizing assumptions now carry the engineering low/high for the service
  (units converted, order checked); Monte Carlo already varies saved ranges.
  Disagreement between two AI estimates widens that range.
- Equipment page: "AI estimate check" shows both sizes and the drift; the user
  keeps the anchored estimate or uses the new one.
- Monte Carlo discovery tolerates a price basis saved with a null currency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
flowvalue-ai Ready Ready Preview Nov 14, 2026 11:52pm UTC

@sa-anyan
sa-anyan merged commit 9970e79 into main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant