Add full-text search benchmark support

[jamesgao-jpg]

Context

VDBBench did not have a dedicated native full-text search benchmark path. This PR adds FTS as a first-class benchmark workload so BM25-based text search can be evaluated through the same task, runner, dataset, frontend, and result pipeline used by the rest of VDBBench.

Summary

• Add full-text search benchmark support centered on BM25 text retrieval.
• Introduce FTS performance cases that load text documents, run text queries, and report comparable performance results.
• Wire FTS through backend execution, dataset preparation, runner orchestration, Streamlit task generation, and result formatting.
• Use manifest-driven FTS ground truth so recall is measured against generated mathematical BM25 neighbors rather than semantic relevance labels.

Backends Covered

• Milvus: native BM25 full-text indexing/search configuration and execution path.
• Zilliz Cloud: FTS routing through the Milvus-compatible API with Cloud sparse auto-index handling, sharing the Milvus optimize/compaction path.
• ElasticCloud / Elasticsearch: BM25 text indexing/search path with configurable BM25 k1/b support.
• Vespa: BM25 schema/query path plus Vespa feed-client loading for large FTS document ingestion.
• Turbopuffer: namespace-based full-text benchmark path.

Testing Infra Touched

• Dataset layer: add MS MARCO and HotpotQA FTS dataset definitions, document/query loading, and S3-hosted mathematical BM25 ground-truth loading.
• Case layer: add FTS performance case definitions, payload profiles, and task assembly support.
• Runner layer: support FTS document loading plus serial recall and concurrent text-query search execution while preserving the existing backend insert contract.
• Backend layer: route Vespa FTS loading through its backend insert path, where the Vespa feed client is managed for high-throughput ingestion.
• Frontend layer: expose FTS cases and generate backend-specific FTS task configs from Streamlit.
• Result layer: format FTS benchmark outputs alongside existing VDBBench results.

Datasets Supported

• MS MARCO: small 100K, medium 1M, large 8.8M documents.
• HotpotQA: small 100K, medium 1M, large 5.2M documents.

Metrics

• Search metric type: BM25.
• Accuracy metric: recall@k against generated mathematical BM25 ground truth.
• Performance metrics: serial latency p95/p99, concurrent QPS, load duration, optimize duration, inserted count, payload profile, batch size, and load concurrency.

[XuanYang-cn]

Requesting changes on the FTS benchmark correctness contract. Inline comments preserve the existing architecture review draft structure for the blocking and medium findings.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jamesgao-jpg, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [XuanYang-cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment