Add strategy lifecycle evidence gates#179
Conversation
Co-Authored-By: Codex <noreply@openai.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d936e4112
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import tomllib |
There was a problem hiding this comment.
This module is imported by quant_platform_kit.strategy_lifecycle.__init__, so on any supported Python 3.10 install the package now fails during import with ModuleNotFoundError: tomllib; pyproject.toml still declares requires-python = ">=3.10" and ruff targets py310. Use a 3.10-compatible fallback such as tomli or raise the package's minimum Python version.
Useful? React with 👍 / 👎.
|
|
||
| def _plugin_gate_is_usable(plugin_gate: Any) -> bool: | ||
| if isinstance(plugin_gate, Mapping): | ||
| status = str(plugin_gate.get("status") or plugin_gate.get("evidence_status") or "").strip().lower() |
There was a problem hiding this comment.
When an evidence package attaches the existing strategy-plugin artifact under plugins/plugin_gate, the approval status lives in execution_controls.consumption_evidence_status or payload.consumption_policy.evidence_status, but this lookup only checks top-level keys. In that common shape, even {"execution_controls": {"consumption_evidence_status": "research_only"}} has an empty status and returns true, so a live_candidate package can pass the plugin gate despite unapproved plugin evidence.
Useful? React with 👍 / 👎.
|
|
||
|
|
||
| def _run_evidence(args: argparse.Namespace) -> int: | ||
| _print(f"[evidence] Validating package file={args.file}") |
There was a problem hiding this comment.
When callers use quant-lifecycle evidence --json, this unconditional status line is printed before the JSON document, so the command's stdout is not parseable JSON for CI gates or downstream automation. Keep human-readable progress messages out of stdout in JSON mode, or send them to stderr.
Useful? React with 👍 / 👎.
| return True | ||
| if _is_non_empty(compatibility.get("runtime_enabled_profiles")): | ||
| return True | ||
| if _is_non_empty(compatibility.get("target_platforms")): |
There was a problem hiding this comment.
For live_candidate or runtime_enabled packages, this accepts platform_compatibility: {"target_platforms": [...]} as sufficient evidence even though top-level target_platforms is already separately required. A package can therefore pass the platform gate by restating the desired target without any verified/compatible flag or runtime catalog proof that the profile is actually supported.
Useful? React with 👍 / 👎.
| if summary.get("observation_count") not in (None, "", 0): | ||
| return True |
There was a problem hiding this comment.
Any non-empty, non-zero observation_count currently satisfies the backtest gate, including impossible or placeholder values such as -1 or "n/a". In those cases a promotion package can be marked valid without a usable sample size, so require a positive numeric count before treating the backtest summary as evidence.
Useful? React with 👍 / 👎.
| "automation_approved", | ||
| "deprecated_compatibility", | ||
| "notification_only", | ||
| "research_only", |
There was a problem hiding this comment.
For live_candidate or runtime_enabled packages, research_only is treated as an allowed plugin gate status and _plugin_gate_is_usable() then accepts it. That contradicts the new lifecycle policy that live use requires plugin evidence to be at least automation_approved or explicitly notification_only, so a plugin-dependent strategy can be promoted while its plugin evidence is still research-only.
Useful? React with 👍 / 👎.
1 participant
Summary
Validation
PYTHONPATH=src python3 -m pytest -q tests/test_lifecycle_contracts.py tests/test_lifecycle_reviewer.py tests/test_lifecycle_cli.py tests/test_lifecycle_evidence_gate.pygit diff --check origin/main...HEAD