domino turns: explore observability (evaluator verdicts, contacts) + fingertip-rule disclosure#81
Merged
Merged
Conversation
yichao-liang
force-pushed
the
pr/session-capture-refine
branch
from
July 19, 2026 11:31
9f4b81f to
582443d
Compare
yichao-liang
force-pushed
the
pr/turns-observability
branch
from
July 19, 2026 11:31
3c0e3ce to
139d313
Compare
yichao-liang
force-pushed
the
pr/session-capture-refine
branch
from
July 19, 2026 12:22
582443d to
5463518
Compare
Hovering a run row now offers a kill button (live runs) and a delete button. POST /kill SIGTERMs only the PIDs attributed to that run via the existing ps+lsof pinning, descendants included; POST /delete removes the run's log dir and mirrored videos dir, pruning now-empty parents, refusing live runs unless kill=1 (TERM, 5s grace, KILL). Both endpoints are POST-only, same-origin, and run_*-scoped.
…ording Two observability gaps burned run_20260718_141716 (seed2 domino_turns): the agent's 20/20 explore trials reached the goal atoms while every evaluate_option_plan submission scored solved=False, and nothing showed that the arm's body - not the fingertip push - was driving the cascade. - sim.run(trials>=2, solved=True) scores each trial with the task evaluator (per-trial solved/reward, headline count, route-rejected hint), gated on a pristine init via _require_solved_evaluator, the same precondition gate refine(require_solved=True) now shares. - sim.run(contacts=True) reports per-step robot-link/object contact spans from a new opt-in PyBulletEnv recorder (raw ids in the hot path with a relevant-body filter, name resolution at stop, static scenery dropped). A failing step reports num_actions=0 while the env genuinely stepped, so the last bucket absorbs uncounted events and a notice flags the step-count mismatch instead of silently blanking exactly the step under diagnosis.
The certificate's counterfactual probe rejects arm-assisted layouts, but the rule was stated nowhere the solve agent could read it: rejections surfaced only as opaque solved=False verdicts, and run_20260718_141716 burned two 45-minute attempts theorizing about an "unknown evaluator criterion". Append the disclosure (goal_text.CASCADE_VERIFICATION_NL) to MIN_BLOCK/HEAVY_GOAL_NL and to the plain generator's evaluator-guarded goal_nl, so every surface that prints the goal carries the rule. This deliberately reverses the 2026-07-16 decision to keep the sentence out of goal_nl; per-submission probe verdicts stay harness-internal. Every shipped min-block/heavy task attaches a DominoEvaluator, so the constants never advertise an unenforced rule. Goal-text digest change regenerates min-block task caches (layouts and K* unaffected).
yichao-liang
force-pushed
the
pr/turns-observability
branch
from
July 19, 2026 12:34
139d313 to
dd75efa
Compare
yichao-liang
added a commit
that referenced
this pull request
Jul 19, 2026
…fingertip-rule disclosure (#81) * log_viewer: per-run kill and log-dir delete from the overview Hovering a run row now offers a kill button (live runs) and a delete button. POST /kill SIGTERMs only the PIDs attributed to that run via the existing ps+lsof pinning, descendants included; POST /delete removes the run's log dir and mirrored videos dir, pruning now-empty parents, refusing live runs unless kill=1 (TERM, 5s grace, KILL). Both endpoints are POST-only, same-origin, and run_*-scoped. * agent_sdk: evaluator verdicts on explore trials + rollout contact recording Two observability gaps burned run_20260718_141716 (seed2 domino_turns): the agent's 20/20 explore trials reached the goal atoms while every evaluate_option_plan submission scored solved=False, and nothing showed that the arm's body - not the fingertip push - was driving the cascade. - sim.run(trials>=2, solved=True) scores each trial with the task evaluator (per-trial solved/reward, headline count, route-rejected hint), gated on a pristine init via _require_solved_evaluator, the same precondition gate refine(require_solved=True) now shares. - sim.run(contacts=True) reports per-step robot-link/object contact spans from a new opt-in PyBulletEnv recorder (raw ids in the hot path with a relevant-body filter, name resolution at stop, static scenery dropped). A failing step reports num_actions=0 while the env genuinely stepped, so the last bucket absorbs uncounted events and a notice flags the step-count mismatch instead of silently blanking exactly the step under diagnosis. * pybullet_domino: state the fingertip-verification rule in the goal text The certificate's counterfactual probe rejects arm-assisted layouts, but the rule was stated nowhere the solve agent could read it: rejections surfaced only as opaque solved=False verdicts, and run_20260718_141716 burned two 45-minute attempts theorizing about an "unknown evaluator criterion". Append the disclosure (goal_text.CASCADE_VERIFICATION_NL) to MIN_BLOCK/HEAVY_GOAL_NL and to the plain generator's evaluator-guarded goal_nl, so every surface that prints the goal carries the rule. This deliberately reverses the 2026-07-16 decision to keep the sentence out of goal_nl; per-submission probe verdicts stay harness-internal. Every shipped min-block/heavy task attaches a DominoEvaluator, so the constants never advertise an unenforced rule. Goal-text digest change regenerates min-block task caches (layouts and K* unaffected). * update common.yaml: reset START_SEED to 0 and NUM_SEEDS to 3 * lint: f-strings for the log-viewer row buttons
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 6 of 6, landing the tip of
domino-next. Base: #80.Closes the two observability gaps that burned run_20260718_141716 (the agent's 20/20 explore trials reached the goal atoms while every submission scored
solved=False, and nothing showed the arm's body - not the fingertip push - was driving the cascade), plus run-management buttons in the log viewer.sim.run(trials>=2, solved=True)scores each trial with the task evaluator (per-trial solved/reward, headline count, and a route-rejected hint when goal atoms held but the evaluator said no), gated on a pristine init via the same precondition gaterefine(require_solved=True)shares.sim.run(contacts=True)reports per-step robot-link/object contact spans from a new opt-inPyBulletEnvrecorder (raw ids + relevant-body filter in the hot path, name resolution at stop, static scenery dropped). A failing step reportsnum_actions=0while the env genuinely stepped, so the last bucket absorbs uncounted events and a mismatch notice replaces silent blanking of exactly the step under diagnosis.CASCADE_VERIFICATION_NLis appended to every goal-text surface; per-submission probe verdicts stay harness-internal, so the agent knows the rule but still has to reason about each rejection.run_*-scoped.Results
With this stack, domino turn tasks are solved reliably: the last 9 runs (3 launches x seeds 0-2 of
domino_turns_new_goal-agent_oracle_hybrid_sim_long_python) all solved their test task.Log-viewer results table for the sweep (all runs
done, test results 1/1, $2.26-$9.36 per run):All nine solves side by side (rows = seeds 0/1/2, columns = the three launches; clips hold their last frame until the longest finishes):
Full-resolution videos (912x912, h264) are attached to the
results-domino-turns-9of9-20260719release:🤖 Generated with Claude Code