feat(lancedb): enhance index types and CLI integration

[XuQianJin-Stars]

Summary

Rewrites the lancedb client for LanceDB, adding support for IVF_HNSW_SQ and IVF_HNSW_PQ index types, standard filter queries, PyArrow batch insert optimization, COS/GooseFS remote storage, and full CLI integration.

What's added

• Client implementation: vectordb_bench/backend/clients/lancedb/{lancedb.py,config.py,cli.py}
• CLI commands: vectordbbench LanceDB ... / LanceDBAutoIndex / LanceDBIVFPQ / LanceDBIVFHNSWSQ / LanceDBIVFHNSWPQ
• Index type registration: vectordb_bench/backend/clients/api.py — add IVF_HNSW_SQ and IVF_HNSW_PQ to IndexType enum
• Unit tests: tests/test_lancedb_config.py — 10 offline tests covering config defaults, to_dict connection options, index_param / search_param generation, metric parsing, case-config registry, and CLI structure
• Entry point: vectordb_bench/cli/vectordbbench.py — register all new LanceDB CLI commands

Algorithms supported

Algorithm	Build params	Search params
IVF_PQ (default)	num_partitions, num_sub_vectors, nbits, sample_rate, max_iterations	nprobes, refine_factor
AutoIndex	(auto — metric only)	nprobes, refine_factor
IVF_HNSW_SQ	num_partitions, m, ef_construction	ef, nprobes, refine_factor
IVF_HNSW_PQ	num_partitions, num_sub_vectors, m, ef_construction	ef, nprobes, refine_factor
NONE (brute-force)	—	refine_factor

Key improvements

• Filter support: Implement standard prepare_filter() pattern (NumGE / StrEqual)
• Batch insert: Use PyArrow FixedSizeListArray for batch writes instead of per-row dicts
• Multi-process: Custom __deepcopy__ to solve Rust handle serialization issue
• Remote storage: storage_options for COS/GooseFS backends
• Optimize: compact_files + cleanup_old_versions support

Example

vectordbbench LanceDBIVFHNSWSQ --case-type Performance768D1M --k 10 \
    --uri /tmp/lancedb \
    --num-partitions 256 --m 16 --ef-construction 128 \
    --ef 64 --nprobes 20 --refine-factor 25

vectordbbench LanceDBIVFPQ --case-type Performance768D1M --k 10 \
    --uri s3://my-bucket/lancedb \
    --storage-options '{"aws_access_key_id":"...","aws_secret_access_key":"..."}' \
    --num-partitions 256 --num-sub-vectors 48 --nprobes 20

[XuQianJin-Stars]

/assign @XuanYang-cn

[XuQianJin-Stars]

Hi @XuanYang-cn, all checks have passed and there are no conflicts. Could you help review this PR when you get a chance? Thanks!

[XuanYang-cn]

IndexType.NONE skips create_index(), but search_embedding() never applies case_config.metric_type to the LanceDB query. For cosine benchmark cases, no-index LanceDB searches therefore use LanceDB's default L2 distance while recall is measured against cosine ground truth. Please apply .metric(self.case_config.parse_metric()) for the no-index path, or otherwise make search use the dataset metric consistently.

[XuQianJin-Stars]

IndexType.NONE skips create_index(), but search_embedding() never applies case_config.metric_type to the LanceDB query. For cosine benchmark cases, no-index LanceDB searches therefore use LanceDB's default L2 distance while recall is measured against cosine ground truth. Please apply .metric(self.case_config.parse_metric()) for the no-index path, or otherwise make search use the dataset metric consistently.

Thanks for the detailed review, @XuanYang-cn! You're absolutely right about the no-index path.

I've updated the PR to always pin the query metric to the case metric in search_embedding() (commit 8b799ff). Now the search consistently applies .metric(self.case_config.parse_metric()) regardless of whether an index is created, so the IndexType.NONE / cosine case will no longer fall back to LanceDB's default L2 distance while recall is measured against cosine ground truth.

Could you take another look when you get a chance? Thanks!

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XuanYang-cn, XuQianJin-Stars

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [XuanYang-cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment