Skip to content

LocalEvalService._evaluate_single_inference_result crashes when inferences is None #6071

@tottenjordan

Description

@tottenjordan

Description

LocalEvalService._evaluate_single_inference_result() accesses inference_result.inferences without a None check. When an inference fails (timeout, API error, etc.), inferences can be None, causing an unhandled crash.

Reproduction

When running GEPA optimization with MCP tool agents, if an MCP session drops during inference, the inference result may have inferences=None. The method then crashes at:

# local_eval_service.py, line ~41-42
if eval_case.conversation_scenario is None and len(
    inference_result.inferences  # <-- crashes if None

Impact

A single failed inference case crashes the entire GEPA optimization run instead of gracefully skipping the case with a NOT_EVALUATED status.

Suggested Fix

Add a None guard before accessing inference_result.inferences:

if inference_result.inferences is None:
    return inference_result, EvalCaseResult(
        eval_id=inference_result.eval_case_id,
        eval_set_id=inference_result.eval_set_id,
        final_eval_status=EvalStatus.NOT_EVALUATED,
        overall_eval_metric_results=[],
        eval_metric_result_per_invocation=[],
        session_id="skipped",
    )

Environment

  • google-adk 2.2.0
  • Python 3.11/3.12

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluation

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions