fix: detect eval orchestrator exiting without producing results#82091
fix: detect eval orchestrator exiting without producing results#82091theobarberbany wants to merge 1 commit into
Conversation
The eval harness orchestrator can launch execute.py in background and then exit cleanly, causing the background task to be killed. The commands.sh script only checks the claude CLI exit code, so this produces a false green JUnit. Add a post-execution check that run_result.json exists before declaring success. Co-Authored-By: Claude <noreply@anthropic.com>
WalkthroughThe agent evaluation command now verifies that a successful ChangesAgent evaluation validation
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: theobarberbany The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`:
- Line 170: Update the RESULT_FILES assignment in the step script to avoid
piping find into head under pipefail. Use find with -type f, the run_result.json
name filter, -print, and -quit so only the first matching file is returned
without risking SIGPIPE or preventing EVAL_EXIT from being updated.
- Around line 168-170: Scope the result lookup in the EVAL_EXIT success path to
the current evaluation run instead of searching the shared RUNS_DIR recursively.
Use the current run’s output directory or clear RUNS_DIR before /eval-run, then
require that a newly created run_result.json exists before treating the run as
successful.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 1a8fa628-6007-4977-847c-b44702f59495
📒 Files selected for processing (1)
ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh
| RUNS_DIR="${AGENT_EVAL_RUNS_DIR:-eval/runs}" | ||
| if [[ "${EVAL_EXIT}" -eq 0 ]]; then | ||
| RESULT_FILES=$(find "${RUNS_DIR}" -name "run_result.json" 2>/dev/null | head -1) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n -C 4 'AGENT_EVAL_RUNS_DIR|run_result\.json|eval-run|rm -rf' --glob '*.sh' .Repository: openshift/release
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file='ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh'
echo '=== outline ==='
ast-grep outline "$file" --view expanded
echo
echo '=== relevant ranges ==='
sed -n '1,220p' "$file" | cat -nRepository: openshift/release
Length of output: 9981
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 3 'run_result\.json|run-id|AGENT_EVAL_RUNS_DIR|eval/runs|/eval-run' \
ci-operator/step-registry/openshift/claude/agent-eval \
ci-operator/step-registry -g '*.sh' -g '*.md' -g '*.yaml' -g '*.yml'Repository: openshift/release
Length of output: 20719
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '=== AGENT_EVAL_RUNS_DIR references ==='
rg -n 'AGENT_EVAL_RUNS_DIR|run_result\.json|eval/runs|rm -rf .*eval/runs|mkdir -p .*eval/runs|rmdir .*eval/runs' ci-operator/step-registry/openshift/claude/agent-eval -g '*.sh' -g '*.yaml' -g '*.yml' -g '*.md'
echo
echo '=== other run_result.json uses in step-registry ==='
rg -n 'run_result\.json' ci-operator/step-registry -g '*.sh' -g '*.yaml' -g '*.yml' -g '*.md'Repository: openshift/release
Length of output: 1672
🌐 Web query:
agent-eval-harness run_result.json run-id output directory eval-runs
💡 Result:
The term agent-eval-harness refers to a category of tools used to evaluate AI agents rather than a single, universally standardized piece of software [1][2][3]. Multiple projects use this or similar names, and they differ significantly in their CLI arguments and output structures [4][5][3]. Based on common conventions and documentation for these frameworks, here is how the specified components are typically handled: 1. run_result.json: This is generally the filename used for the final aggregated results of an evaluation run [4][6][5]. It contains structured data such as overall metrics, pass rates, latency, and costs [4][6]. 2. run-id: This is a unique identifier used to label a specific evaluation execution [4][5]. It is often used to organize output directories and filenames, allowing users to track experiments over time [4][5]. 3. output directory: This is the destination path (often defaulting to a folder like./results or./eval-runs) where the framework saves the artifacts of a run, including logs and the final run_result.json [4][7][5]. 4. eval-runs: This is commonly used as a parent directory name for storing multiple evaluation results [5]. For example, in many frameworks, you might run an evaluation using commands similar to these: # Example of running an evaluation with a custom output directory and run ID harness run --output./eval-runs --run-id my-experiment-001 After execution, the tool would typically generate a result file located at:./eval-runs/my-experiment-001/run_result.json Because there are multiple projects (such as reaatech/agent-eval-harness, najeed/ai-agent-eval-harness, and others), you should verify the specific CLI documentation of the version you are using, as flags (e.g., --run-id vs --id) and default paths may vary [4][7][5][2].
Citations:
- 1: https://github.com/reaatech/agent-eval-harness
- 2: https://github.com/najeed/ai-agent-eval-harness/blob/main/docs/guides/help/00_COMPREHENSIVE_GUIDE.md
- 3: https://github.com/opendatahub-io/agent-eval-harness/blob/main/README.md
- 4: https://pypi.org/project/agent-eval-harness/
- 5: https://github.com/lukasmetzler/agenteval/blob/refs/tags/v0.7.3/docs/results.md
- 6: https://github.com/reaatech/agent-eval-harness/tree/main/packages/suite
- 7: https://github.com/reaatech/agent-eval-harness/blob/main/WALKTHROUGH.md
Scope the result check to the current run. At lines 168-170, find "${RUNS_DIR}" -name "run_result.json" can match a stale file from an earlier execution in the shared eval/runs tree and let an incomplete run pass. Use the current run’s output path, or clear RUNS_DIR before /eval-run and require a newly created result.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`
around lines 168 - 170, Scope the result lookup in the EVAL_EXIT success path to
the current evaluation run instead of searching the shared RUNS_DIR recursively.
Use the current run’s output directory or clear RUNS_DIR before /eval-run, then
require that a newly created run_result.json exists before treating the run as
successful.
| # ----------------------------------------------------------------------- | ||
| RUNS_DIR="${AGENT_EVAL_RUNS_DIR:-eval/runs}" | ||
| if [[ "${EVAL_EXIT}" -eq 0 ]]; then | ||
| RESULT_FILES=$(find "${RUNS_DIR}" -name "run_result.json" 2>/dev/null | head -1) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Avoid piping find into head under pipefail.
If this script uses the required set -euo pipefail, find can receive SIGPIPE after head consumes the first match, causing the step to exit before EVAL_EXIT is updated. Use find "${RUNS_DIR}" -type f -name "run_result.json" -print -quit instead.
As per coding guidelines, step-registry scripts should default to set -euo pipefail, so this pipeline must be safe under that contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift/claude/agent-eval/openshift-claude-agent-eval-commands.sh`
at line 170, Update the RESULT_FILES assignment in the step script to avoid
piping find into head under pipefail. Use find with -type f, the run_result.json
name filter, -print, and -quit so only the first matching file is returned
without risking SIGPIPE or preventing EVAL_EXIT from being updated.
Source: Coding guidelines
|
@theobarberbany: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Are we tracking this in agent-eval-harness? |
2 participants
Summary
openshift-claude-agent-eval-commands.shthat verifiesrun_result.jsonexists before writing a passing JUnitexecute.pyin background, exiting cleanly, and the background task being killed — producing a false greenContext
Observed on openshift-eng/ai-helpers#627. The
eval-payload-analysis-changedjob reported green but no eval cases actually completed —eval-runs.tarcontained only the empty directory structure (348 bytes, norun_result.json, nosummary.yaml).The orchestrator used
run_in_background: trueforexecute.py, said "I'll wait for results," then ended its session. The background task was killed andcommands.shsaw exit 0 fromclaudeand wrote a passing JUnit.This is a safety net — the root cause (orchestrator backgrounding execution) should also be fixed in the eval harness.
🤖 Generated with Claude Code
Summary by CodeRabbit
run_result.jsonis produced after evaluation.