Skip to content

(13) feat(cascade): in-memory LSM match-action store + real-shape ACL tests#1567

Draft
daniel-noland wants to merge 8 commits into
mainfrom
pr/daniel-noland/acl-pat-design
Draft

(13) feat(cascade): in-memory LSM match-action store + real-shape ACL tests#1567
daniel-noland wants to merge 8 commits into
mainfrom
pr/daniel-noland/acl-pat-design

Conversation

@daniel-noland

@daniel-noland daniel-noland commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Stack (13), this PR (retargeted). Base: pr/daniel-noland/acl-dpdk.

In-memory LSM-shaped match-action storage primitive, plus tests that drive it
with a real ACL-shaped consumer.

  • feat(cascade): in-memory LSM-shaped match-action storage primitive.
  • test(cascade): real-shape ACL consumer + bolero + subscribe tests.

Review stack (merge bottom -> top):

@daniel-noland daniel-noland changed the base branch from main to pr/daniel-noland/threading-rewrite May 26, 2026 00:39
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch 7 times, most recently from 45262b6 to df3cbd7 Compare May 28, 2026 05:53
@daniel-noland daniel-noland self-assigned this May 28, 2026
@daniel-noland daniel-noland requested a review from Copilot May 28, 2026 06:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

The PR title explicitly says "(ignore for now) acl design test" — this is a large design-exploration PR introducing a new match-action / ACL framework spanning many new workspace crates, plus supporting changes to existing crates.

Changes:

  • Introduces several new workspace crates: fixed-size, lookup, match-action, match-action-derive, cascade, acl, dpdk-test-macros, and a scratch ACL module under dpdk.
  • Adds a MatchKey derive (with #[exact]/#[prefix]/#[mask]/#[range] field attributes), parallel <Name>Rule struct emission, and lowering into a reference and DPDK rte_acl backend, plus extensive property/differential tests and Criterion benches.
  • Adds infrastructure: a dpdk::test_support EAL bootstrap + #[with_eal] attribute macro, a cascade crate (LSM-style match-action storage with Upsert/MergeInto/drain subscription), a bench-builder Nix target, and a compute_min_input_size const fn on AclBuildConfig.

Reviewed changes

Copilot reviewed 66 out of 67 changed files in this pull request and generated no comments.

Show a summary per file
File Description
Cargo.toml Registers new workspace members, deps, and per-package miri/wasm metadata
fixed-size/** New no_std crate defining FixedSize trait with primitive/IpAddr impls
lookup/** New crate with Lookup/Projection traits and BTreeMap/HashMap impls
match-action/, match-action-derive/ New backend-agnostic match-key vocabulary + #[derive(MatchKey)]
net/src/fixed_size.rs, net/src/lib.rs, net/Cargo.toml FixedSize impls for TcpPort/UdpPort/UnicastIpv4Addr/Vni
cascade/** New LSM-style cascade primitive with Upsert/MergeInto/drain subscription
dpdk/Cargo.toml, dpdk/src/lib.rs, dpdk/src/test_support.rs Optional test feature exposing shared EAL bootstrap + #[with_eal] macro
dpdk-test-macros/** New proc-macro crate providing the #[with_eal] attribute
dpdk/src/acl/config.rs Makes compute_min_input_size a public const fn; doc/spelling cleanup
dpdk/src/acl/scratch.rs Scratch/sketch file with non-compiling design notes
acl/** New crate: DPDK rte_acl backend, reference linear-scan oracle, tests, benches
default.nix, justfile New benches Nix target and just bench recipe to build/run benches

@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch 5 times, most recently from 1d88968 to 5109431 Compare May 28, 2026 22:03
@daniel-noland daniel-noland changed the title (ignore for now) acl design test acl design work May 28, 2026
@daniel-noland daniel-noland requested a review from Copilot May 28, 2026 22:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 64 out of 65 changed files in this pull request and generated no new comments.

@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch 2 times, most recently from 137d9d9 to af41ab0 Compare May 28, 2026 22:23
Comment thread match-action/src/lib.rs Outdated
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch 3 times, most recently from c347061 to 592cd9d Compare May 29, 2026 01:59
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/threading-rewrite branch 3 times, most recently from de5a2c1 to fd80413 Compare May 31, 2026 02:03
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch from 70afa5a to ba7f579 Compare June 3, 2026 19:50
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch from 16e2188 to 01d98b2 Compare June 3, 2026 19:50
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch from ba7f579 to c9e5eb3 Compare June 15, 2026 20:18
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch from 01d98b2 to ddbc607 Compare June 15, 2026 20:18
@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 23556064-ef74-4732-91d6-3366b0cc2753

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch from ddbc607 to 19ca1a1 Compare June 15, 2026 20:23
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch from c9e5eb3 to 0683aff Compare June 15, 2026 20:23
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch from 19ca1a1 to 29232af Compare June 15, 2026 22:08
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch from 0683aff to 057f155 Compare June 15, 2026 22:08
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch from 29232af to fbcf630 Compare June 18, 2026 19:34
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch 2 times, most recently from 4915824 to e5d8a01 Compare June 18, 2026 19:56
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch from c795ed9 to 19de724 Compare June 18, 2026 20:16
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-dpdk branch 7 times, most recently from ca1cc66 to a445674 Compare June 30, 2026 04:15
Base automatically changed from pr/daniel-noland/acl-dpdk to main June 30, 2026 10:36
Introduces dataplane-cascade.  Models a small in-memory LSM: writes
land in a concurrent multi-writer head, periodically frozen into
immutable intermediate layers, eventually compacted into an immutable
tail.  Readers walk head -> frozen[] -> tail and stop at the first
definitive answer.

The same primitive serves three problems via one Upsert trait:
match-action table updates (atomic publish under load), hardware
offload programming (drain output feeds the HW backend), and
active-active state replication (serialized drain output ships to
peer dataplanes).  freeze / fuse / compact / merge are all the same
Upsert operation applied to different operands.

Layout:

- lib.rs publishes the surface: Cascade / DrainEvent / FrozenEntry /
  Snapshot, the Upsert / MergeInto / MutableHead traits, Generation,
  and re-exports Lookup / Projection from the lookup crate.
- cascade.rs is the central type plus the rotate / drain / publish
  state machine.  Cascade::subscribe is feature-gated under
  `subscribe` (tokio broadcast).
- head.rs / merge.rs / upsert.rs are the trait definitions plus
  LastWriteWins as a stock blanket-friendly Upsert impl.
- generation.rs carries the monotonic Generation counter that orders
  layers within a cascade.
- property_tests.rs is a reusable bolero harness (under the bolero
  feature) consumer crates use to verify their own Upsert impls
  against the cascade's algebraic laws.  Exercised by a self-test
  in the next PR.

tests/smoke.rs is the trait-shape end-to-end with a trivial concrete
implementation: head shadows sealed shadows tail, tombstones in the
head suppress lower-layer hits, rotate seals and publishes.  Uses
the shared tests/common/mod.rs helpers that subsequent tests also
share.

Cascade is independent of every other PR in this stack: no acl dep,
no match-action dep.  Real-shape consumer pressure plus
bolero / subscribe integration tests land in the next PR.

just fmt; cargo check --workspace --all-targets passes.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Three integration tests that round out the cascade's test surface --
each exercises a different part of the design pressure that shaped
the trait surface in the previous PR.

tests/acl_consumer.rs is the first real-shaped consumer: a minimal
ACL classifier built on top of the cascade with no upfront ACL
crate dep -- the toy ACL is built inline using only Cascade /
MergeInto / MutableHead.

Purpose: surface design pressure on the trait surface against a use
case that is NOT exact-match-keyed.  ACL classification looks up
packets by header match expressions, not by a single key, and rules
carry their own identity (priority) separate from the lookup input.
If the cascade trait shape works for ACL it almost certainly works
for anything simpler.

Scope intentionally tight: rules install-only (no shadow-rule
removal yet), match expressions src/dst IPv4 with optional single
port, head Lookup always returns None (writes visible only after a
rotation seals the head).  Comment block at the end captures the
open question about removal under cascade semantics.

tests/upsert_properties.rs is the self-test of the property harness
landed in the previous PR.  Exercises check_upsert_order_independent
against the provided LastWriteWins -- known correct by construction.
If it fails, the harness itself is broken; if it passes, it's a
useful black-box check for downstream Upsert impls.

tests/subscribe.rs is the drain-subscription integration under tokio.
Exercises Cascade::subscribe end-to-end -- gated by the cascade
`subscribe` feature (enabled in dev-deps via the self-path override
on Cargo.toml).

just fmt; cargo check --workspace --all-targets passes.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@daniel-noland daniel-noland force-pushed the pr/daniel-noland/acl-pat-design branch from e5d8a01 to b84e168 Compare June 30, 2026 23:45
I updated Cargo.lock to keep the other diffs as clean as possible.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Pin every third-party `uses:` reference (marketplace actions and
composite actions) to a specific commit SHA, with the resolved tag
kept as a trailing comment for readability. Managed going forward
via `pinact run` (config in .pinact.yaml).

pinact does not resolve SHAs for cross-repo reusable workflow calls,
so `githedgehog/fabricator/.github/workflows/run-vlab.yaml@master` is
pinned by hand instead and exempted in .pinact.yaml so future
`pinact run` invocations don't flag or fight the manual pin.

Also adds `pinact` to the nix devshell so it can be run locally.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Trim GITHUB_TOKEN permissions to what each job actually consumes,
traced through the composite actions and justfile recipes:

- bump.yml / version-bump.yml: the blanket contents/id-token/
  pull-requests grants had no consumer -- the actual git push and PR
  creation go through a separately-minted, already-scoped GitHub App
  token, not GITHUB_TOKEN. Reduced to `contents: read` (needed for
  checkout on this private repo).
- dev.yml: same story for the shared `check-perms` block (check,
  sanitize, test_each, concurrency, miri, wasm, cross) -- none of
  those jobs build or push containers, so checks/pull-requests/
  packages/id-token were dead weight. `build` and `publish` keep
  `packages: write` since they push images via `just push-container`.
  `check_changes` keeps `pull-requests: read` for dorny/paths-filter's
  API-based file-list lookup on PR events.

Also scope the two `create-github-app-token` steps (dev.yml,
version-bump.yml) to `permission-contents`/`permission-pull-requests`
instead of inheriting the full GitHub App installation's permissions,
and set `persist-credentials: false` on checkout steps that never
push with the default token (bump.yml, dev.yml, fossa.yml,
lint-validate-dependabot.yml), plus add zizmor to the nix devshell
used to audit this.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
- Name the previously-anonymous cargo-upgrades, version, publish,
  commitlint, and validate jobs (zizmor anonymous-definition).
- Add a concurrency group to mergeability.yml and
  lint-validate-dependabot.yml (zizmor concurrency-limits); the
  former falls back to github.run_id for merge_group events so an
  in-progress merge-queue check is never cancelled.
- Document, via `zizmor: ignore[superfluous-actions]` comments, the
  decision to keep peter-evans/create-pull-request (diff-detection,
  draft, sign-commits) over hand-rolling `gh pr create` in the three
  workflows that open automated PRs.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Give each linter its own just recipe (clippy, opengrep, pinact,
zizmor) instead of burying cargo clippy alone in `lint`, then compose
them: `lint: (clippy) opengrep (pinact "-check") zizmor`. This gives
local parity with the individual CI lint jobs and a single place to
tune each tool's flags.

dev.yml's `check` job now calls the `clippy` recipe directly (it
already ran individual recipes like fmt/test/doctest rather than the
aggregate `lint`, so this just keeps that step's naming honest).
lint-opengrep.yml now delegates to `just opengrep` instead of
duplicating the scan invocation inline.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
New required PR checks, matching the shape of the existing lint-*
workflows (lab runner, nix-shell composite, shallow checkout with
persist-credentials: false, skipped-but-present in merge_group so the
required status check still resolves):

- lint-zizmor.yml audits our own GitHub Actions workflows and
  composite actions for the supply-chain/permission issues zizmor
  checks for (excessive permissions, unpinned refs, credential
  persistence, etc.), gating on the pedantic persona since the repo
  is already clean at that level.
- lint-pinact.yml verifies every action/reusable-workflow reference
  stays pinned to a commit SHA with an accurate version comment
  (`pinact run --check --verify`), so a future PR can't silently
  reintroduce a floating tag or a stale version comment.

.github/zizmor.yml is an empty rules stub for future per-repo
overrides (e.g. targeted ignores), picked up automatically by zizmor.

Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants