feat(search): vector-similarity ("Similar to…") search on /api/search#60
Draft
joewiz wants to merge 12 commits into
Draft
feat(search): vector-similarity ("Similar to…") search on /api/search#60joewiz wants to merge 12 commits into
joewiz wants to merge 12 commits into
Conversation
…policy Prototype of the field-discovery half of the broaden-/api/search design: a consumer (e.g. the Oxygen plugin's field picker) can ask "what can I search here?" and "what is this field's contract?" before querying. Two separated layers, per the ES model: - CATALOG: enumerate every configured field/facet under a collection scope, with its contract (kind, indexed element(s), analyzer, type, returnable), read with privilege. This XQuery collection.xconf parser is a STAND-IN for the native ft:fields($scope) the lucene session will build; configs live under /db/system/config (admin-only) and the schema is system-managed, so the catalog read is privileged and permission-agnostic. (The privilege need is exactly why ft:fields is worth building natively — it reads the resolved LuceneConfig via the broker and skips the system-config read entirely.) - FLS: a group->fields policy decides which catalog entries THIS caller sees, applied after the privileged read — keyed off $request?user. Field access lives in the policy, never as an ACL on the field (the Elasticsearch lesson). Default: public site-* fields for everyone incl. guest; everything else for authenticated callers; per-field group restrictions supported. Name-independent of ft:query-scope/ft:search-scope (those names are still in review on eXist-db/exist#6455), so this can proceed now; the field-param query cutover waits for that function to ship. Validated on the live 3-producer instance: guest sees only public site-* fields; an authenticated caller additionally sees the docs app's non-public index fields (category/definition/function-name/term) plus a seeded secret-notes; dba sees all. site-content's contract correctly dedups to one record listing the 7 elements it is indexed on across apps. NOT for merge as-is: the privileged read uses system:as-user (prototype); the production form swaps to ft:fields once it lands, and this gains an api.json route + XQSuite tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the collection.xconf-parsing + system:as-user stand-in with the native ft:fields($scope) (now in eXist-db/exist#6459). The catalog read is now permission-agnostic and credential-free, exactly as designed. Two ft:fields behaviours had to be handled in the API layer (worth a core follow-up — see the handoff note): 1. ft:fields does NOT aggregate across collections: it resolves the single config for a given collection/doc-set, so ft:fields("/db/apps") is empty when each app's config lives on its own data collection, and a sequence scope resolves to only the first collection's config. For site-wide discovery we union ft:fields over every descendant collection in scope (fields:descendant-collections). Collapses to a single ft:fields($scope) if it gains native cross-collection aggregation. 2. ft:fields also emits element-level text-index records (a plain <text qname> with no named <field> yields a map with only "element"); those aren't named, field:(...)-queryable fields, so the catalog drops maps without a "field" key. dedup now surfaces per-field analyzer VARIANCE as an array (a shared field indexed with different analyzers on different elements — e.g. site-content StandardAnalyzer vs WordDelimiter — is reported as both, not hidden). Validated on a 2-app + bundled-apps bed: cross-collection union works (site-content elements [a,b]); analyzer variance shows both analyzers; FLS differentiates guest (public site-* only) from authenticated (also sees function-name/secret-notes) from dba (all). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
eXist-db/exist#6459 (d724759) addressed both integration findings: ft:fields now aggregates across every collection in scope, and every record carries field + kind (field/facet/vector). So fields:catalog collapses to a single ft:fields($scope) call — removing the descendant-collection union walk and the [exists(?field)] filter. Verified on the 2-app + bundled-apps bed: ft:fields("/db/apps") unions across collections (site-content elements [a,b]), analyzer variance still surfaces as an array, and FLS differentiates guest (public site-* only) / authenticated (+ function-name, secret-notes, and the test-embedding vector field) / dba. Confirmed for the core session: the previously field-less records were the vector case (e.g. test-embedding, now kind:"vector"), not bare element-text indexes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the field-discovery handler (fields:list) into the API: - modules/api.xq: import the fields module so function-lookup resolves the "fields:list" operationId (same mechanism as search:query). - modules/api.json: add the GET /api/search/fields path — scope (default /db/apps) + optional field params; documented response envelope (scope/user/total/fields[] with field/kind/elements/analyzer/type/returnable) and an example. - src/test/cypress/e2e/search-fields.cy.js: self-contained suite (seeds a fixture collection with a public site-content field + a non-public field) asserting the envelope, the per-field contract, the field= filter, and that an authenticated caller sees non-public fields. Validated at the handler level on the ft:fields bed (fields:list over synthetic roaster $request maps): guest sees public site-* only; authenticated sees the non-public fields too; field= narrows to one; default scope applied. Depends on ft:fields (eXist-db/exist#6459): the route and the Cypress suite require an eXist that ships ft:fields, so this is branch work until that lands in a release (CI uses the stock image). Not for merge until then. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…FLS test End-to-end HTTP testing (existdb-openapi installed on an ft:fields-enabled eXist) surfaced a bug the handler-level test missed: the scope fallback `($request?parameters?scope[. ne ""], $fields:default-scope)` always appended the default, so a provided scope echoed twice (and was passed doubled to ft:fields). Use an if/else so a provided scope (one or more) is used as-is and the default applies only when none is given. The HTTP test also confirmed the route admits unauthenticated callers (identity resolves to guest), so the guest "public-only" FLS tier is reachable over HTTP. Added a Cypress assertion for it: a guest sees the public site-* fields but not the non-public secret-notes; a dba sees both. Verified on a full PoC bed (producers snapshot + the ft:fields lucene jar): GET /api/search/fields returns, for the real corpus, site-content unioned across 6 elements with mixed analyzers [StandardAnalyzer, SimpleAnalyzer]; admin sees 12 fields (field/facet/vector), guest sees 7 (public site-* only). All five Cypress scenarios pass against the live route. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…earch Implements the oxygen field-scoped-search contract (existdb-openapi#55): - `field` (optional): restrict the query to one named field (a value from GET /api/search/fields) instead of the default shared site-content/site-title query. Built on standard ft:query (a field-qualified query string), so it works on a stock eXist — NOT gated on #6455/#6459. Only discovery needs ft:fields. - `scope` (optional, repeatable): collection path(s) to search under, recursive; same semantics as /api/search/fields. Defaults to the sitewide /db/apps. - Field-level security: a field the caller may not see is not queryable — returns 403, enforced by the same policy /api/search/fields uses. - Response shape unchanged (query/total/offset/limit/facets/results), so the plugin's parser is untouched; the deferred ~10-line plugin wiring can now land. To avoid a regression, the FLS policy (public/restricted/visible) is extracted to a new field-policy.xqm with NO ft:fields dependency, imported by both search.xqm and fields.xqm. Previously search would have transitively pulled ft:fields via fields.xqm and failed to compile on a stock eXist (XPST0017); verified fixed — /api/search compiles and field-scoped search works on stock beta3 (ft:fields absent). 7 self-contained Cypress tests (field isolation, scope, guest-403, public-200, dba override, stable default); all green. Verified on the trio instance (:19110). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…st-db#55 / oxygen c) Adds &facet=<dimension>:<value> to /api/search (repeatable; same dimension -> OR, different dimensions -> AND), implementing the oxygen facet drill-down design (c). ES post_filter semantics: selecting a facet value narrows the returned HITS but NOT the bucket counts — counts are computed on the base query (q + scope) so they stay stable as the user drills (the "blog (12)" still shows after filtering to docs). Implemented as: one base query for the facets map + ft:score ranking; a second drill-down query only when a facet is selected, to narrow the hits. - The app/section params become shortcuts for facet=site-app:… / facet=site-section:… (generalized into the one mechanism). - facet (and scope, also documented repeatable) are declared array-typed in api.json so roaster accepts repetition; the handler unwraps roaster's array(*) to a sequence. Values grouped by dimension with explicit for/where (NOT a ?key-in-predicate, which eXist mis-handles as XPTY0004 for >1 item). Self-contained Cypress suite (5): bucket counts, drill narrows hits, post_filter count stability, multi-value OR, app-shortcut equivalence. search.cy.js (9) and search-field-scope.cy.js (7) stay green (no regression from moving facet counts onto the base query). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oxygen d) Discovery-driven kNN: GET /api/search?vector=<field>&similar=<text>&k=<n>. The client sends only the vector field + query text; the server resolves the field's embedding model from its ft:fields record (eXist-db/exist#6459), embeds the text with that model, and runs ft:query-field-vector over the scoped collection. Hits return in the existing ES-shaped envelope plus field/model/max-score. Notes: - ft:query-field-vector is context-scoped, so it is called as collection($scope)/ft:query-field-vector(...); its k is a candidate-pool hint, not a hard limit, so k is enforced via ft:score ordering + subsequence. - static vector:/ft: calls (this targets the vector-capable integration build, not a stock eXist). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- total now reflects the number of results returned (= k, or fewer if the corpus is smaller), instead of the kNN's over-returned candidate pool (the pool is the symptom of the eXist-core ft:query-field-vector k bug; the server-side re-rank + subsequence already returns exactly k). - k is clamped to [1, 100] (default 10) — a kNN result count, not paging. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch the vector branch from collection($scope)/ft:query-field-vector($field, …) to ft:query-vector(collection($scope)//<element>, …, k), targeting the vector field's indexed element (read from ft:fields discovery alongside the model). The field form is evaluated per-document in a path step (a 1-doc kNN per node, unioned) — so it ignores k AND scales poorly; the node form does one true cross-document top-k with k authoritative server-side (root-cause + fix handed off by exist-strategy; the field-targeted core fix is tracked separately with Duncan). Client contract unchanged (still ?vector=<field>, discovery-driven). The order-by-ft:score + subsequence is kept (explicit ranking, robust). Caveat noted in-code: an element with >1 vector field targets the first; revisit when the core ft:query-field-vector fix lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
joewiz
force-pushed
the
feat/api-search-vector
branch
from
June 12, 2026 03:36
fa1280d to
5fb283d
Compare
… / oxygen c) The facets-option ft:query (Lucene drill-down) doesn't collect per-match offsets, so its result nodes can't drive ft:highlight-field-matches/KWIC — the faceted path returned a full-body snippet with no <mark> and empty highlights. Intersect the drill set with $base-hits (which carry the match data) by node identity, so the returned hits are the match-bearing nodes narrowed to the facet selection. Preserves post_filter narrowing + the base-query facet counts. Reproduced + fix verified on :19110: faceted hit went from 0 to 2 highlight matches; counts unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
|
This PR uses |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[This PR was co-authored with Claude Code. -Joe]
Summary
Adds vector-similarity ("Similar to…") search to
/api/search— the Oxygen plugin's design option (d). The client sends a vector field + query text (+ optionalk/scope); the server resolves the field's embedding model fromft:fields, embeds the text, runs a kNN, and returns the nearest documents ranked by similarity in the same ES-shaped envelope as keyword search. The client sends only field + text — the model is resolved server-side.Contract
GET /api/search?vector=<field>&similar=<text>&k=<n>&scope=<collection>vector— a vector field (afieldfromGET /api/search/fieldswhosekindisvector).similar— the query text (embedded server-side with the field's model).k— nearest-neighbor count (default 10, clamped to 1–100).scope— collection path(s), recursive, repeatable; default/db/apps(same semantics as keyword search).Response:
{ query, field, model, k, total, max-score, results: [{ uri, path, title, app, url, score, snippet }] }.total= number of results returned.snippetis a plain<span>excerpt — no<mark>(semantic, not keyword-matched). FLS-enforced: 403 if the caller can't see the field, identical to keyword/field search.Notes
ft:fieldsrecord ([feature] ft:fields: introspect configured Lucene fields/facets in a scope exist#6459 exposes a vector field'smodel/dimension/similarity), so the client never sends the model.ft:query-vector(collection($scope)//<element>, …, k)—<element>is the vector field's indexed element (read fromft:fieldsdiscovery alongside the model) — which does one true cross-document top-k withkauthoritative server-side. This avoids theft:query-field-vectorfield form, which eXist evaluates per-document in a path step (a 1-doc kNN per node, unioned) so it both ignoreskand scales poorly (root-caused by exist-strategy; the field-targeted core fix is tracked separately with Duncan, who designed the vector feature).order by ft:score descending+subsequenceis kept for explicit, robust ranking. Caveat: an element carrying more than one vector field targets the first, not necessarily$field— fine for one-vector-field-per-element corpora; revisit when the core fix lands.Gating
Requires a vector-capable eXist: the
vector:embedextension,ft:query-vector, andft:fieldsexposing the vector field's model (#6459). Built and verified on the trio bed (existdb-combined-vector:beta4).Stacking — how to read this PR
Stacked on #58 (search field-scope/facet/discovery), which is not yet merged — so until it lands, this PR's diff against
developalso carries #58's commits. The net-new work to review is thefeat(search): vector-similarity …commits. Rebases to a clean diff once #58 merges. (Posted from a fork branch, so GitHub can't render a literalbase: #58stack — hence the bundled diff.) Draft until #58 lands.Testing
search-vector.cy.js(self-contained vector-field fixture): ranks by similarity, resolves the model server-side,klimits, missingsimilar→ 400, unknown field → 404. Verified live on :19110 (beta4): "how do I speed up queries" → top hitperformance-tuning.xml@ ~0.74,model: all-MiniLM-L6-v2resolved server-side.