fix(mem_wal): push LIMIT/OFFSET down to fresh-tier scan sources

[hamersaw]

Summary

The LSM scan planner (LsmScanPlanner) applied a query's LIMIT/OFFSET above the per-source union, so every flushed generation and the base table were fully scanned before the limit was imposed at the top. For a bounded read over a fresh tier with many flushed generations this materialized the entire tier on every query.

This threads an overfetch_factor through LsmScanner → LsmScanPlanner and pushes a per-source fetch of (offset + limit) * overfetch into each Base/Flushed source scan. The active memtable is exempt (its within-source dedup needs the full match set). Block-listed sources over-fetch by overfetch_factor so cross-generation PK dedup still leaves ~limit live rows; a LocalLimit re-imposes the exact skip/fetch cap above the merge. Unbounded reads (no limit) are unchanged.

Changes

• scanner/planner.rs: overfetch_factor field + with_overfetch_factor on LsmScanPlanner; per-source limit pushdown in plan_scan; fetch parameter on build_source_scan (pushed into Base/Flushed scans, ActiveMemTable exempt).
• scanner/builder.rs: overfetch_factor on LsmScanner + with_overfetch_factor, threaded into both the scan and FTS planners.

Validation

Validated end-to-end against a WAL benchmark on minikube with object storage behind a latency proxy: a bounded offset+limit read over an 18-generation fresh tier dropped from a full-tier scan (~360 generation scans) to a bounded per-source read.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 36.00000% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/mem_wal/scanner/builder.rs	16.66%	8 Missing and 2 partials ⚠️
rust/lance/src/dataset/mem_wal/scanner/planner.rs	53.84%	4 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[LuQQiu]

We call it refine factor?

[LuQQiu]

oh better to understand whether it's similar to refine factors in vector / fts
not saying we should use refine factor

[hamersaw]

Great point. I'm trying to keep these separate because they are logically different things. IIUC the refine_factor is used because IVF distances may not be exact, so we over retrieve so that we can become more accurate. The proposed overfetch_factor is strictly for probabilistically ensuring we have K values after we remove duplicates. This is a simple solution for the problem, but ultimately we will probably want to fix it in another way. So by separating them, we can remove the overfetch_factor more easily.