feat(early-stop): opt-in early stop once armed criteria are decided

[mohsen-uipath]

Context

Some tests only care about a specific part of a run, not the whole trajectory. Activation tests are the clearest example: today they pin the agent to a single turn so the run ends right after the skill is meant to activate. Teams want to raise that turn limit so the agent can investigate the starting project before recalling a skill — but naively raising it means agents that already activated on turn one keep going, doing unrelated and unpredictable work the test was never scaffolded for. That inflates turns, time, and cost, and adds noise to the result.

The same need shows up beyond activation: we'd like to stop as soon as the outcome is known — either the criterion we care about is satisfied, or a required early step is definitively missed (e.g. the wrong skill loads). No point running the rest when the verdict is already in. This has been asked for a few times, both for activation turn limits and for a general "early-exit on pass or fail".

What we want

An opt-in mechanism to end a run early once a designated criterion is decided — on pass or on definitive fail. This lets us give the agent more turns without wasting them or polluting the result once the thing we're measuring has already happened.

Ideally selectable by flavor, so a single test can serve as both a smoke check (stop after the key step) and an e2e check (run the whole thing), instead of maintaining two near-duplicate tests.

Prior art

coder_eval already has a related mechanism in simulation/dialog mode (stop_on_criteria_pass) that ends a dialog once criteria pass. It's scoped to dialog mode and to the "all criteria pass" case, so it doesn't cover single-shot runs (which activation uses) or the stop-on-fail case — but it's a useful reference point for what the config surface and stop-reason reporting could look like.

Possible approaches

A. Watch the live run and stop as soon as the signal appears. Some criteria — such as whether a skill was triggered, or whether a specific command ran — can be observed while the agent is still working, so the run can be cut off the moment the criterion is decided. Lighter-weight, and it naturally covers the stop-on-fail case for these signals. Limited to criteria that are observable mid-run.

B. Check a designated criterion between turns and stop. Drive the agent turn-by-turn and evaluate the chosen criterion after each turn, ending the run when it's decided. More general — works for any criterion — but a larger change to how runs are driven.

Considerations

• Must be opt-in per test. Default behavior stays unchanged — many tests have intentionally weak criteria and should run to completion so cost/turn numbers stay accurate. Early-stop is for cases where the measured signal is well-defined (activation being the main one).
• Support both polarities: stop when the criterion is satisfied, and stop when it's definitively missed.
• Nice to have: gate by tag/flavor so the same test can be a smoke (early-stop) and an e2e (full) check.
• Record why and when the run stopped, so reports and telemetry stay clear about early-terminated runs.

Why this PR implements Approach A

There are two candidate methods for ending a run early once the measured criterion is decided.

Method A watches the live run: some criteria — whether a skill was triggered, whether a specific command ran — are observable in the event stream while the agent is still working, so a watcher evaluates them on each tool completion and cuts the run off at the next message boundary once the verdict is in.

Method B drives the agent turn-by-turn: the orchestrator regains control after every turn, evaluates the designated criterion against the sandbox, and ends the run when it is decided.

A has three decisive advantages over B. First, it fits the existing execution model: a single-shot run hands control to the SDK for the entire trajectory, so B would require rebuilding the drive loop into repeated single-turn calls with session continuity across every agent — while A rides the event stream the SDK already emits, stops at tool-call granularity (earlier than B's turn boundaries), and leaves the default path behaviorally unchanged. Second, it is sounder: A arms only criteria decided by positive events that cannot un-happen, so the live verdict and the authoritative re-check on the frozen trajectory agree by construction, whereas end-state criteria can flip after a mid-run "pass," making B's early verdicts guesses. Third, it is cheaper: live verdicts read the in-memory trajectory instead of B's per-turn sandbox inspections or judge calls. B remains the right tool only for genuinely end-state stop signals or agents that expose real turn boundaries — cases that are deferred behind hard resolution-time errors and can later slot in behind the same config surface.

In summary, A outperforms B on fit, correctness, and cost for everything this feature is meant to measure, which is why A is what ships here.

What's implemented

• Config surface: stop_when: pass|fail|decided on any criterion (the "armed set") plus the opt-in master switch run_limits.stop_early (default false — default behavior is unchanged).
• Live-verdict contract: live_stop_polarities + live_verdict on BaseCriterion, implemented for skill_triggered (first-engagement policy) and command_executed (monotone count bounds, incl. the must-NOT-run form). Lint rule CE025 keeps the pair consistent.
• Cooperative stop: a should_stop poll at the Claude agent's existing between-message guard — clean STOPPED_EARLY finalization (crashed=false, no SIGKILL), driven by EarlyStopWatcher (own EventCollector + stop rule) composed into the event stream.
• Verdict rule: an early-stopped run gates on the armed subset; non-armed criteria are still evaluated and recorded but advisory (clearly marked). A run that completes naturally gates on the full set, exactly as today.
• Flavor selection: one boolean per variant / -D run_limits.stop_early=true; ships with experiments/early-stop-ab.yaml (smoke vs. e2e from one task file) and docs recipes.
• Recording: EarlyStopInfo (reason, deciding criterion, SDK turn / tool-call index, elapsed seconds, turns-remaining upper bound) on EvaluationResult, surfaced as a run.md note, an HTML badge + advisory markers, stopped_early fields in run.json rows, and EarlyStopped/EarlyStopReason telemetry dimensions.
• Guardrails: every unsupported arming (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, dialog mode) is a hard error at resolution time, on both plan and run — never a silent no-op. A runtime live-verdict bug fails open to a full run.

Deferred, not foreclosed: Approach B for end-state criteria / other agents / dialog mode slots in later behind the same config surface; evalboard badge (task.json already carries early_stop).

Testing

• make verify green: 3382 passed, 3 expected skips, coverage 90.8% (gate 80%). New suites: tests/test_early_stop.py (models, live verdicts, watcher stop rule, agent seam incl. timeout-beats-stop precedence, orchestrator wiring, plan/run guardrail surfaces, report surfaces) and CE025 lint tests.
• Regression: the unarmed path is covered by dedicated default-off tests (full stream consumed, full gate) — with stop_early unset, behavior is unchanged.
• A/B acceptance with experiments/early-stop-ab.yaml on an activation-style task: identical pass/fail verdicts between the e2e and smoke variants, with the smoke variant substantially lower on turns, duration, and tokens.

[github-actions]

Claude finished @mohsen-uipath's task in 1m 13s —— View job

---

Code Review In Progress

Review Checklist:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Analyze the full diff (git diff origin/main...HEAD)
• Review each changed file in full context
• Perform cross-file consistency checks
• Analyze what's missing or could be improved
• Provide structured review feedback

---

[bai-uipath]

Mechanism is sound, but unproven on the suite it's for. Default-off, fail-open, and the guardrails are all right, but there's no end-to-end run showing it early-stops, and it isn't wired into activation. Please apply it there and demonstrate it before we call it the activation fix.

• Wire it into activation and show it working. Bump skills/tests/experiments/activation.yaml from max_turns: 1 to 3, arm the stacked skill_triggered criteria with stop_when: decided, and attach a run where positive rows stop on turn 1 with verdicts matching the max_turns: 1 baseline. The "identical verdicts, lower cost" claim is the deliverable, not an assertion.

[bai-uipath]

Mechanism is sound and default-off — approving it as a general smoke/e2e feature. The two latest fixes are correct, and I've seen it early-stop for real. Everything below is confirmation — things I checked that are fine, not change requests. The only ask is a wired-in activation run before we call this the activation fix, and that gates the label, not the merge.

Arming activation — checked, nothing to change

• The armed-subset gate is a no-op for activation, and that's harmless. armed_criteria_passed gates a per-run final_status, which activation ignores — it gates suite_thresholds: recall.yes across all rows (tasks/activation/activation.yaml). Early-stop preserves the confusion-matrix labels under one-skill-per-run, so the metric is unaffected. Noting it only so it's clear the "gate the armed subset, others advisory" machinery is inert here, not that anything's wrong.
• Savings are positive-only — expected, not a defect. Negatives have no stop event (absence isn't decidable mid-run), so early-stop only trims post-activation work on the rows that activate. max_turns is a ceiling, so cleanly-abstaining negatives terminate naturally rather than burning the raised limit. Whether raising the ceiling shifts recall/precision vs today's one-shot is a question about the config choice, not the mechanism — worth measuring when you wire it in, nothing to fix in the code.
• Wiring is trivial. Bump defaults.run_limits.max_turns, set stop_early per variant, add stop_when: decided to all 22 skill_triggered criteria. Just recording the recipe.

First-engagement vs any-engagement skew — verified harmless under one-skill-per-run

• Confirmed it can't fire as activation runs today. skill_triggered.live_verdict decides on the first engaged skill while _check_impl is any-engagement, and the gate re-checks the full trajectory rather than the frozen one (orchestrator.py:1388,1407). Those only disagree if two skills are engaged inside the deciding message — under one-skill-per-run it can't happen, so live verdict and gate agree by construction. No change needed.
• One thing to keep an eye on, not fix now: "engaged" here includes the skills/<name>/ file-read signal, not just the Skill tool call — so an agent reading a second skill's files while deciding would count as two, which gets likelier once turns are raised. If a wired-in run shows single-skill engagement holds, this stays a non-issue. Only if it ever doesn't: truncate the trajectory to the watcher's tool_call_index before check_all.