BlockDB: no orphaned blocks

[cody-littley]

Describe your changes and provide context

Don't return blocks that have had their QC messages pruned.

Testing performed to validate your change

unit tests

[cursor]

PR Summary

High Risk
Changes core ledger persistence semantics (prune, restart recovery, read visibility) on the consensus block store; incorrect watermark or clamp logic could hide valid blocks or serve blocks without QCs.

Overview
BlockDB now guarantees that any readable block also has a readable covering QC, including after async LittDB GC can leave blocks on disk without their QC.

littblock treats the prune watermark as a read gate (reads, hash lookups, iterators) in addition to GC eligibility. On reopen it re-derives the watermark from the lowest surviving QC so stranded below-watermark blocks are not served after restart; stores with blocks but no QC fail to open. PruneBefore is a no-op when no blocks exist, and aggressive prune requests are clamped to min(latestQCStartBlock, lastBlockNumber) so the newest cohort is never fully removed.

memblock gets the same prune clamp and empty-store behavior for parity.

The BlockDB interface docs now state the QC pairing invariant, that pruning never empties the store, and that pruned reads may return None without relying on timing of physical reclamation. Shared memblock/littblock contract tests and littblock stranding/GC tests (including raw-table checks) cover these paths; durable GC regression moved to littblock package tests.

^{Reviewed by Cursor Bugbot for commit 8e63c26. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Jul 10, 2026, 7:53 PM

[codecov]

Codecov Report

❌ Patch coverage is 87.14286% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.96%. Comparing base (e4ed4e6) to head (8e63c26).

Files with missing lines	Patch %	Lines
sei-db/ledger_db/block/littblock/litt_block_db.go	86.66%	3 Missing and 3 partials ⚠️
...b/ledger_db/block/littblock/litt_block_iterator.go	81.25%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3734      +/-   ##
==========================================
- Coverage   59.86%   58.96%   -0.91%     
==========================================
  Files        2286     2197      -89     
  Lines      189587   179535   -10052     
==========================================
- Hits       113502   105868    -7634     
+ Misses      65963    64397    -1566     
+ Partials    10122     9270     -852     

Flag	Coverage Δ
sei-chain-pr	87.72% <87.14%> (?)
sei-db	70.41% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/ledger_db/block/memblock/mem_block_db.go	98.13% <100.00%> (+0.09%)	⬆️
sei-tendermint/autobahn/types/testonly.go	95.34% <100.00%> (+0.08%)	⬆️
...b/ledger_db/block/littblock/litt_block_iterator.go	67.24% <81.25%> (+2.79%)	⬆️
sei-db/ledger_db/block/littblock/litt_block_db.go	71.64% <86.66%> (+2.55%)	⬆️

... and 135 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

The read-watermark gate that refuses below-watermark blocks/QCs and re-derives the watermark on restart is correct and well-tested, upholding the new "a readable block always has a readable covering QC" invariant. However, the newly added doc comment contains a misspelling that the enabled misspell linter will reject, failing CI.

Findings: 2 blocking | 4 non-blocking | 2 posted inline

Blockers

• gofmt/misspell: the new doc block in sei-tendermint/autobahn/types/block_db.go (lines 76-77) will fail CI — "gauranteed" trips the enabled misspell linter and the trailing whitespace on line 76 is removed by gofmt. Fix both before merge (raised by Codex).
• 1 blocking issue(s) flagged inline on specific lines.

Non-blocking

• Method-level docs are now stale relative to the new top-level guarantee: ReadBlockByNumber (lines 199-206), ReadBlockByHash (lines 214-217), and ReadQCByBlockNumber (lines 226-231) still say a below-watermark block/QC "may remain readable until it is actually reclaimed." Both implementations now never serve below-watermark blocks (littblock refuses via the read gate; memblock deletes synchronously in PruneBefore), so these docs should be tightened to match. (Codex point #2, adjusted — the docs are a permissive superset so not strictly wrong, but misleading.)
• Cursor's second-opinion review file (cursor-review.md) was empty — that pass produced no output.
• Minor: in qcIterator.Next, decodeNumberKey(key) is computed twice for a straddling QC (lines 96 and 103); could be hoisted to a local. Negligible.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

A well-scoped, correct fix that gates below-watermark reads/iterators in the durable littblock BlockDB so blocks whose covering QC was asynchronously GC'd are never served, with a re-derived on-open watermark and strong white-box regression tests. No blocking issues found; only minor/informational notes.

Findings: 0 blocking | 3 non-blocking | 0 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• The Cursor second-opinion review file (cursor-review.md) was empty — that pass produced no output. Codex reported no material findings and could not run the Go 1.25.6 tests in its sandbox.
• recoverReadWatermark's correctness depends on litt forward iteration being insertion order combined with WriteQC's ascending-contiguous enforcement (so the first primary QC encountered is the lowest surviving one). This is correct but implicit; a one-line note tying the scan to the insertion-order/contiguity guarantee would help future readers. Also, the 'no QC survives' path does a full-table scan on open — acceptable since it only happens when every QC has been reclaimed, but worth being aware of.
• Behavior change: below-watermark reads/iterators now reliably return None rather than 'maybe'. This is within the (updated) interface contract and no production callers rely on the old behavior (only blocksim/tests consume Blocks()/QCs()), so it is safe — flagging only for awareness.

Superseded: latest AI review found no blocking issues.

[wen-coding]

The contract at consensus level is that we should always keep at least one (block, qc) pair when pruning. So if you reach here, something is broken. Maybe the user accidentally removed the QC WAL file? If that's the case, is the right behavior:

1. just panic and exit
   or
2. just remove all blocks?
   I don't know how much I should trust the remaining blocks at this point.

[wen-coding]

I mean, we have code in consensus to guarantee that, so it's up to you whether you want to double-enforce it inside Block DB.

[cody-littley]

Good point, this is clear data corruption. Constructor now returns an error in this situation.

Superseded: latest AI review found no blocking issues.

Superseded: latest AI review found no blocking issues.

[wen-coding]

Just curious, when would this be true? I think the watermark is set to the start of the first QC on restart, so this should not happen?

[cody-littley]

The straddle branch is reached when a PruneBefore has set the watermark partway through a QC's range (First < watermark < Next). In that case we keep the QC, since GC won't reclaim it until Next <= watermark. But the block iterator still hides the blocks it covers that fall below the watermark, even though they're reachable on disk. Note this doesn't happen right after restart. The watermark isn't persisted, so it resets to 0, and this path only fires once a later prune advances it into a QC's range.

[seidroid]

Solid, well-tested change that adds a read-watermark gate so a served block always has a served covering QC, plus a never-empty prune clamp and startup watermark recovery. Two narrow concurrency-consistency notes (from Codex) and empty second-opinion/guideline files are the only findings; no blockers.

Findings: 0 blocking | 4 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• Iterators (Blocks/QCs) capture the watermark at creation (litt_block_iterator.go), which is the intended snapshot semantics — but a block yielded by a blockIterator created before a concurrent PruneBefore can sit below the now-current watermark, so a live ReadQCByBlockNumber for that block returns None (Codex #2). This is consistent within a single snapshot (the paired QC iterator captured the same watermark) but inconsistent when mixing an iterator with live point reads. By-design, but worth a one-line doc note on the iterators that the invariant holds within a snapshot, not against concurrent live reads.
• REVIEW_GUIDELINES.md and cursor-review.md are present but empty, and codex-review.md contains only the two findings addressed here — so the Cursor pass produced no output and there were no repo-specific guidelines to apply.
• recoverReadWatermark scans the shared table with a forward iterator until the first primary QC key. Since block keys can sort before that QC key, startup may scan many block entries before returning; fine as a one-time cost, but worth confirming acceptable for large stores.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] This checks the watermark before getBlock, whereas ReadBlockByHash (line 340) checks it after the read. A concurrent PruneBefore that advances the watermark between this check and getBlock returning lets ReadBlockByNumber serve a block that a subsequent ReadQCByBlockNumber(n) would refuse (n < new watermark), briefly violating the new "a readable block always has a readable covering QC" invariant. Consider keeping this pre-check as a fast path but re-checking n < s.watermark.Load() after the block is resolved, mirroring ReadBlockByHash, so the returned block is consistent with the watermark at read completion. (Race-free w.r.t. the detector since watermark is atomic; this is a logical-consistency tightening.)