Skip to content

feat: add filtered full-text search benchmarking#814

Closed
jamesgao-jpg wants to merge 16 commits into
zilliztech:mainfrom
jamesgao-jpg:fts_ver2
Closed

feat: add filtered full-text search benchmarking#814
jamesgao-jpg wants to merge 16 commits into
zilliztech:mainfrom
jamesgao-jpg:fts_ver2

Conversation

@jamesgao-jpg

Copy link
Copy Markdown
Collaborator

Context

PR zilliztech/VectorDBBench#794 introduced first-class BM25 full-text search benchmarking. This PR extends that work with filtered FTS workloads, semantic relevance metrics, additional backend controls, and OSS OpenSearch support.

Summary

  • add filtered FTS dataset preparation and propagate filter data through cases, task runners, and search runners
  • implement FTS filtering for Milvus/Zilliz, Elastic Cloud, OSS OpenSearch, Turbopuffer, and Vespa paths
  • add semantic recall, MRR, and NDCG handling for FTS results while separating filter-quality checks from concurrent performance queries
  • expose FTS filter cases, BM25 controls, collection/index names, shard/replica settings, and backend search controls through CLI and frontend configuration
  • add OSS OpenSearch FTS configuration and execution support
  • publish curated filtered FTS result artifacts for the frontend result views

Compatibility

  • existing unfiltered FTS cases remain supported
  • backend-specific analyzer, BM25, filter-field, and search-level settings remain explicit in each backend case configuration
  • mathematical BM25 ground-truth plumbing is removed in favor of semantic qrel-based quality metrics

Verification

  • exact DCO sign-off verified for all 16 commits
  • 46 focused FTS case, CLI, dataset, runner, formatting, metric, OSS OpenSearch, and Turbopuffer tests passed
  • tests/test_fts_backend_filters.py was not collected locally because the optional elasticsearch SDK is not installed in the available environment

Signed-off-by: jamesgao-jpg james.gao@zilliz.com

Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jamesgao-jpg

Copy link
Copy Markdown
Collaborator Author

Superseded by #815, which removes filtered FTS frontend and result publication while retaining the unfiltered OSS OpenSearch and semantic FTS scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants