feat(workflow): StateSync recipe completes at release — mark-ready is terminal#485
Merged
Merged
Conversation
… terminal The StateSync recipe ends at mark-ready: Complete means every mutation was performed and the node was released to re-bootstrap. Catch-up is verified node-side (sdk WaitCaughtUp, RPC, alerts) by whoever triggered the workflow. Why: catching_up is a proxy for health the workflow cannot own -- the old final await-condition(catchingUp) held the node's single adoption slot for the full catch-up (hours at real chain height) with no deadline anywhere (the sidecar task polls unbounded, the plan executor has no per-task clock), and the only exit from a wedged wait was the force-delete ceremony, which the docs framed as Failed-workflow recovery only. Completing at release frees the slot in minutes, deletes the unbounded-wait failure mode outright, and separates execution (the workflow's job) from verification (the node's health surfaces). Two properties this buys, both now test-pinned: - Failed <=> held: mark-ready is the terminal step and it is fire-and-forget (success == submitted), so a Failed workflow always means the node is still held. The old shape could fail AFTER release (await failing on a slow-but-healthy sync), which made "Failed holds the node" conditional. - The adoption slot frees at release, so drift work and queued workflows resume promptly. Accepted costs, on the record (peer-reviewed with kubernetes-specialist + systems-engineer, chain-side behavior verified by sei-network-specialist): - WorkflowInProgress clears at release, so height-lag alerts can fire during legitimate catch-up. Traced against the platform alert rules: at most 1-2 auto-resolving warning tickets on protocol-low, no pages. - Drift suppression lifts at release, so drift work (an image roll, a config reconcile) can interrupt catch-up at any point in the window, not only at the tail. Verified against sei-chain: the giga startup safety checks key on SS version markers written only at the END of a snapshot restore, so an interrupted restore re-enters state-sync and converges. One sub-second wedge window exists between the sequential cosmos/evm marker writes at the restore tail; recovery is re-running the recipe (runbook note, negligible probability). - The adoption slot frees at release, so a queued workflow can adopt and re-wipe a node mid-catch-up. Wasteful, never corrupting (the recipe quiesces before it wipes); operators sequence workflows per node. - Config drift cannot undo the migration flags: managed nodes have no config-drift detector, and the only running-node config surface (the update progression's p2p patch) never touches app.toml state-store keys. Test changes: planner progression expectations end at mark-ready with an explicit no-await pin; envtest lifecycle stall point moves to configure-state-sync (mark-ready is fire-and-forget, so GetTask-based stalls cannot hold it); AdoptAndComplete pins the terminal-step shape; workflow_test task counts 6 -> 5. sdk/sei's StateSyncWorkflow doc now states the release semantics (callers verify catch-up via WaitCaughtUp). No CRD schema change (make manifests generate produced no config/manifests diff). The seictl UX/docs companion (success handoff print, 15m default timeout, README) follows in sei-protocol/seictl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR SummaryMedium Risk Overview Docs in the CRD kind comment and Planner, controller, envtest, and integration tests were updated: plans are one step shorter, lifecycle tests assert the last task is Reviewed by Cursor Bugbot for commit 6e88d31. Bugbot is set up for automated code reviews on this repo. Configure here. |
bdchatham
added a commit
to sei-protocol/seictl
that referenced
this pull request
Jul 14, 2026
#234) ## Summary Companion to sei-protocol/sei-k8s-controller#485 (StateSync recipe completes at release; mark-ready is terminal). The state-sync command's Complete means the recipe's mutations landed and the node was released to re-bootstrap; the resync runs after Complete. A Failed workflow always means the node is still held. - On a successful watch the command prints the node-side verification handoff: `seictl node watch <node> --until=caught-up`, the SDK-backed readiness watch built on the same gates the nightly integration harness uses (committed height above 1 with `catching_up` false, plus EVM serving when the node publishes an EVM endpoint). No raw kubectl/curl/jq instructions in the output. - `--timeout` default drops to 15m: the watch ends at release, which takes minutes on a healthy node, so a timeout usually means a wedged recipe step. Archive-scale data directories legitimately run long; the help text directs raising `--timeout` for those. - README rewrites both flavors' step walkthroughs and the recipe mapping to the release-terminal shape, routes verification through the caught-up watch, and adds the migration's EVM-serving evidence paragraph and the mid-restore re-run recovery note. ## Review xreview ledger: `bdchatham-designs/designs/seinode-task/xreview/workflow-complete-at-release.md` — unanimous over three rounds (idiomatic-reviewer, prose-steward, sei-network-specialist as assigned dissenter). The dissenter caught a correctness-grade bug in the originally-printed check (a plain GET to `:26657/status` returns the bare result object, so a `.result.sync_info` jq path prints null); routing verification through `node watch` supersedes that surface entirely — the SDK's status reader models both body shapes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The StateSync recipe ends at mark-ready. Complete means every mutation was performed and the node was released to re-bootstrap; catch-up is verified node-side (sdk
WaitCaughtUp,seictl node watch --until=caught-up, alerts) by whoever triggered the workflow.Why:
catching_upis a proxy for health the workflow cannot own. The old finalawait-condition(catchingUp)held the node's single adoption slot for the full catch-up (hours at real chain height) with no deadline anywhere, and the only exit from a wedged wait was the force-delete ceremony. Completing at release frees the slot in minutes, deletes the unbounded-wait failure mode outright, and separates execution (the workflow's job) from verification (the node's health surfaces).Two properties this buys, both test-pinned:
Accepted costs, on the record
WorkflowInProgressclears at release, so height-lag alerts can fire during legitimate catch-up. Traced against the platform alert rules: at most 1-2 auto-resolving warning tickets on protocol-low, no pages.A lost mark-ready self-heals from two directions: the sidecar persists the Running row before returning and rehydrates on restart, and the controller repair loop (
sidecarNeedsReapproval, wired in every mode planner) re-issues mark-ready when the post-release health probe sees a 503.Changes
internal/planner/workflow.go: BuildPlan ends at mark-ready; no await-condition step.api/v1alpha1kind doc +sdk/seiStateSyncWorkflow doc state the release semantics.make manifests generateproduced no config/ diff).Review
xreview ledger:
bdchatham-designs/designs/seinode-task/xreview/workflow-complete-at-release.md— unanimous over two rounds (idiomatic-reviewer, systems-engineer as assigned dissenter, kubernetes-specialist), all findings resolved and verified.Companion seictl PR (handoff print, 15m default timeout, README) follows.
🤖 Generated with Claude Code