Skip to content

feat(gfql/cypher): searchAny cross-column search predicate — inspector semantics, native on 4 engines#1680

Merged
lmeyerov merged 10 commits into
masterfrom
feat/gfql-search-any
Jul 5, 2026
Merged

feat(gfql/cypher): searchAny cross-column search predicate — inspector semantics, native on 4 engines#1680
lmeyerov merged 10 commits into
masterfrom
feat/gfql-search-any

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Implements searchAny(entity, term[, opts]) — the streamgl-viz inspector's cross-column table search as a composable openCypher-style WHERE predicate, native on all four engines (pandas/cuDF/polars/polars-gpu), plus Python twins g.search_nodes() / g.search_edges(). Viz-filter L2 (frozen semantics: panel-algebra D2; naming/options via a 4-persona design pass — artifacts under plans/viz-filter-pipeline/research/searchany-persona-pass.md). Stacked on #1679.

Semantics (dtype gate is SEMANTICS, not optimization)

  • OR across the entity's columns; case-insensitive substring by default (case-folded, never regex — the common viz call avoids every engine regex limit); regex: true opt-in obeying the same per-engine decline rules as =~ (now enforced on cuDF too: prep + casefold-or-decline before delegating, instead of libcudf crashes or \D\d inversion).
  • Dtype gate: string columns always; integer columns iff the term is a numeric literal (the inspector's /^[0-9.-]+$/ gate); floats/dates/booleans only via the explicit columns: list. Explicit non-string columns are served where stringification provably matches the pandas oracle: bool everywhere (polars canonicalizes 'True'/'False'), float/date on pandas only — polars and cuDF decline them honestly (dgx-probed: pandas str(1e16)='1e+16' vs polars '1e16' vs cuDF '1.0e+16', and cuDF truncates mantissas — 0.1+0.2 → '0.3').
  • Options map {caseSensitive, regex, columns} strict-validated: unknown keys, non-boolean flags, malformed column lists, and unbound aliases all error clearly; missing explicit columns raise the same E108 on pandas AND polars (pinned to beat the dtype gate in precedence); $param terms decline with a clear message; null cells never match (incl. astype(str)'s 'nan'/'<NA>' artifacts, masked back out).
  • searchAny(...) spelled inside a string literal is data, not a call — the lowering lift matches against a literal/comment-masked copy of the WHERE text (with rescan-after-skip so quote-spanning pseudo-matches can't swallow real calls).
  • Nodes and edges independently searchable; composes through AND/OR/NOT (lowered like the pattern-predicate markers); two searchAny in one WHERE compose (pinned).
MATCH (n) WHERE searchAny(n, 'acme') RETURN n
MATCH (n) WHERE searchAny(n, 'alpha') AND searchAny(n, '7') RETURN n

Engineering

Per-column matching reuses the parity-hardened Contains predicate on pandas/cuDF (every cuDF workaround + honest decline from the #1675 review waves inherited) and a fold/any_horizontal lowering on polars (Rust-regex guard on the regex path; module-per-family polars/search.py). Safelisted row op with schema effects; conformance-ledger entries; polars coverage baseline gains a search.py floor (79.0; dgx-measured 86.96 on the CI-verbatim lane). One deliberate base change: the call executor's NIE pass-through extends from polars-only to row-pipeline calls on any engine (is_row_pipeline_call), so kernel-level per-engine declines surface as honest NotImplementedError instead of a wrapped E303 — non-row-pipeline NIEs (e.g. fa2_layout without GPU) still wrap, pinned.

Review + verification

5 adversarial findings-mode waves to convergence (waves 4+5 consecutive clean; artifacts plans/viz-filter-pipeline/waves/pr1680/). Waves fixed 2 BLOCKERs (in-literal lift rewrite; cuDF regex guard bypass) and 5+1 IMPORTANTs (null-cell stringify match, polars E108, $param, polars bool 'true' silent divergence, float repr divergence — each now pinned). dgx GB10 (--gpus all, rapids 26.02): targeted 5-file lane 1703 passed at the converged tip; CI green incl. the polars coverage audit. ruff + mypy clean.

Boundaries (clear errors, all pinned)

searchAny in RETURN/WITH projections (WHERE only); edge-alias searchAny(r, …) declines on polars (pending multi-entity binding-row support); explicit columns beyond string/int/bool decline on polars + cuDF; polars-frame Python twins decline (use the cypher op); regex declines mirror =~.

🤖 Generated with Claude Code

from .row_pipeline import _active_table, _rewrap


def search_any_polars(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensure cross-platform not just polars, and cross-test

Comment thread graphistry/compute/gfql/lazy/engine/polars/search.py
else:
exprs.append(base.str.to_lowercase().str.contains(term.lower(), literal=True))
marked = left.with_columns(pl.any_horizontal(exprs).fill_null(False).alias(out_col))
return _rewrap(g, marked)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should audit our tests:

  • what happens when nested structures
  • are we appropriatley short circuiting, eg, skipping bool cols and numeric cols for non-numeric str
  • ... what happens when pretty printer adds "." and "," to ui? should we search those too?
  • true/false variants (True, False, 0, 1, ...) can probably be converted to bool vals for when searching bool cols, what's normal here?

@lmeyerov lmeyerov force-pushed the feat/gfql-cypher-exists branch from c0a8c1f to 08c1e01 Compare July 5, 2026 22:11
@lmeyerov lmeyerov force-pushed the feat/gfql-search-any branch from c0634f1 to 9af5808 Compare July 5, 2026 22:11
"search_edges columns= includes a column absent from the edges table",
field="columns", value=columns,
suggestion="List only columns present on the edges table.")
return self.edges(df[mask])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DRY

@lmeyerov lmeyerov force-pushed the feat/gfql-cypher-exists branch from 08c1e01 to 889f59e Compare July 5, 2026 22:28
@lmeyerov lmeyerov force-pushed the feat/gfql-search-any branch 2 times, most recently from dbf67ae to 5b692fe Compare July 5, 2026 22:39
Base automatically changed from feat/gfql-cypher-exists to master July 5, 2026 23:11
lmeyerov and others added 10 commits July 5, 2026 16:15
…-filter L2-a)

Kernel (gfql/search_any.py): OR-across-columns, case-insensitive substring default,
regex opt-in, dtype gate AS SEMANTICS (string cols always; int cols iff numeric-
literal term per the inspector /^[0-9.-]+$/ gate; floats/dates/bools via explicit
columns=); per-column mask = the parity-hardened Contains predicate (all cuDF
workarounds + honest declines inherited). Row op w/ alias-prefix scoping + safelist
+ builder; polars native (schema dtype gate; lower()-fold literal default — the
common viz call never touches regex; rust-guard on regex path; any_horizontal);
ComputeMixin twins search_nodes/search_edges (pandas/cuDF; polars frames honest-NIE
v1). Matrix: 5-case oracle pins + 4-engine parity-or-NIE + twins test; ledger
calls-axis waiver (count_table pattern) + rowops exercised. dgx: 1996 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cypher surface: searchAny(alias, 'term'[, {caseSensitive, regex, columns}]) lifts
at lowering assembly into a search_any pre-filter + fresh marker column (the
pattern-predicate marker mechanism) — composes through AND/OR/NOT; strict opts
validation (unknown keys/bad bools/bad lists/unbound alias -> clear E108);
oracle-pinned cypher-surface cases + 4-engine parity-or-NIE incl. NOT/AND
composition; docs + CHANGELOG.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…uDF =~-parity regex guards, null-cell mask, polars E108, $param clear-decline

B1: _lift_search_any_from_row_where now finds calls against a literal/comment-
masked copy of the WHERE text, so searchAny(...) inside a string literal is data
(was: bare .sub rewrote literal contents — silent wrong answer / spurious
unbound-alias error). I5: any residual searchAny( outside literals after the
lift (e.g. $param term) raises a clear E108 instead of leaking downstream.
B2: the cuDF kernel path applies the same decline rules as =~ (_cudf_regex_prep
+ _cudf_casefold_or_decline) before delegating to Contains — NIE on lookaround/
backrefs/inline-flags/unsound folds instead of libcudf crashes or \D->\d
inversion. I1: null cells never match — explicit non-string columns mask
isna() back out after astype(str) ('nan'/'<NA>' artifacts). I2: polars
missing-explicit-column now raises the same E108 as pandas (the comment claimed
a fallback that doesn't exist). Pins for each in the conformance matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e + explicit-dtype gate, discriminating float pin, polars E108 pin, docs caveats, rescan-after-skip lift

W2-3: polars explicit-column search now canonicalizes Boolean rendering to
pandas' 'True'/'False' (cast gave 'true' — a SILENT caseSensitive divergence)
with nulls kept null, and declines explicit columns beyond string/int/float/
bool (temporal/categorical/nested stringification is engine-divergent) — NIE,
never silent. W2-1: float auto-gate pin made discriminating (7.25 moved to a
row with no int '7'). W2-2: polars-engine missing-explicit-column E108 pinned
directly. W2-4: docs/CHANGELOG caveat the edge-alias polars NIE + dtype gate.
S1: lift rescans from m.start()+1 after an in-literal skip so a quote-spanning
pseudo-match can't swallow a real call. S4: two-searchAny composition pin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dgx-measured 86.96% on the CI-verbatim CPU polars lane; floor 79.0 per the
baseline's ~8-below convention (also absorbs the wave-2 canonicalize/dtype-
gate lines, whose decline/parity branches are pinned in the matrix).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…umns (repr divergence), precedence pin, wording

Floats leave the polars _stringify_ok gate: pandas str(1e16)='1e+16' vs the
Rust formatter's '1e16' is a silent exponent-regime divergence — the SAME
decision the polars toString lowering already made (row_pipeline.py declines
Float 'repr diverges'). Pinned NIE for explicit float columns on polars;
pandas oracle pin unchanged. Missing-column-beats-dtype-gate precedence pinned
(E108, not NIE). Docs/CHANGELOG: float added to the polars decline list;
'folded via lowercase' -> 'case-folded' (pandas upper-folds internally).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… stringify diverges from pandas (dgx-probed)

Probe (rapids 26.02): cudf.Series.astype(str) renders 0.1+0.2 as '0.3' (pandas
'0.30000000000000004'), 1e16 as '1.0e+16' (pandas '1e+16'), and truncates long
mantissas ('123456789.1') — the explicit-columns float path silently diverged
from the pandas oracle. cuDF now declines explicit non-string/int/bool columns
(NIE), symmetric with the polars gate; string/int/bool stay native (bool
parity pinned). Kernel pin added; docs/CHANGELOG updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…date decline (wave-4 S-w4a)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…FrameT/SeriesT (no Any)

Use the codebase's cross-engine aliases (compute/typing.py, as predicates/str.py
already does): frames are DataFrameT, masks/series are SeriesT; dtype params are
"object" (stricter than Any — only passed to the pandas dtype API). Annotation-
only; mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o gate (inspector 2a)

Match the streamgl-viz inspector (sortAndFilterRowsByQuery.js shouldSearch): the
auto gate only fires on string dtype. A pandas OBJECT column can hold lists/dicts;
we included it as 'string' then Contains-over-lists silently never-matched (+ a
fillna FutureWarning). Now _has_string_content inspects object columns via
infer_dtype and skips non-string content — clean skip, not silent no-match. Real
string / StringDtype / cuDF-str columns still searched; polars/cudf already used
typed columns (no object ambiguity). Pandas-path fix; pin verifies pandas skip +
polars parity. Reference-parity spec: research/searchany-inspector-parity.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lmeyerov lmeyerov force-pushed the feat/gfql-search-any branch from 48d529b to 9388be7 Compare July 5, 2026 23:16
@lmeyerov lmeyerov merged commit b9bdceb into master Jul 5, 2026
77 checks passed
@lmeyerov lmeyerov deleted the feat/gfql-search-any branch July 5, 2026 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant