diff --git a/.agents/skills/edit-ponder-sync/SKILL.md b/.agents/skills/edit-ponder-sync/SKILL.md new file mode 100644 index 0000000000..7646f3f913 --- /dev/null +++ b/.agents/skills/edit-ponder-sync/SKILL.md @@ -0,0 +1,206 @@ +--- +name: edit-ponder-sync +description: > + Inspect and surgically edit Ponder's RPC sync cache (the `ponder_sync` Postgres schema): + understand what's cached, diagnose stale/contaminated cache (e.g. a restarted ens-test-env + devnet reusing chain id 31337), and clip/purge a block range so Ponder re-fetches it cleanly + without re-syncing the whole chain. Use when a handler throws RecordNotFoundError on an entity + that should exist, when cached block hashes diverge from the live chain, or when splicing one + chain's sync data into another (upstream chain → a fork of it). +--- + +# Editing Ponder Sync (`ponder_sync`) + +`ponder_sync` is Ponder's **RPC cache** — blocks, logs, transactions, receipts, traces, and the +per-filter "what ranges have I synced" bookkeeping. It is separate from the **entity** schema +(`ENSINDEXER_SCHEMA_NAME`, e.g. `ensindexer_temp_dev`) which holds the indexed app data (`domains`, +`registries`, …) plus Ponder's `_ponder_checkpoint` / `_ponder_meta`. + +Editing `ponder_sync` is the right tool when the cache is **wrong or contaminated** and you want +Ponder to re-fetch a range from the RPC. It is reversible (anything deleted is re-fetchable), but +you must keep the `intervals` table consistent with the row tables or Ponder will trust a hole. + +Local Postgres in this repo: `postgresql://postgres@localhost:5432/ponder`. Always scope edits by `chain_id`. + +## Schema cheat-sheet + +``` +ponder_sync.blocks -- one row per CACHED block. KEY COLUMN: number (NOT block_number). + -- columns: number, hash, parent_hash, timestamp, chain_id, … +ponder_sync.logs -- block_number, log_index, address, topic0..3, block_hash, data, … +ponder_sync.transactions -- block_number, hash, from, to, … +ponder_sync.transaction_receipts -- block_number, transaction_hash, status, … (often empty: Ponder + -- only fetches receipts when a handler/config needs them) +ponder_sync.traces -- block_number, trace_index, … (often empty) +ponder_sync.rpc_request_results -- block_number (nullable), request_hash, result -- cached eth_call etc. +ponder_sync.intervals -- THE SOURCE OF TRUTH for "synced". fragment_id (PK), chain_id, + -- blocks (nummultirange). +ponder_sync.factories / -- factory-pattern (CREATE2 child) address discovery. Children +ponder_sync.factory_addresses -- discovered above a fork are valid current-chain data. +``` + +Gotchas that will bite you: + +- **`address`, `hash`, `topic*` are stored as TEXT** (`0x…` lowercase strings), **not bytea**. + `WHERE address = '\x0000…'` silently matches nothing and returns count 0. Use + `WHERE address = '0x00000000000c2e074ec69a0dfb2997ba6c7d2e1e'`. +- **`blocks` is sparse.** Ponder only caches blocks that contain a matching log (plus interval + boundaries / finalized blocks). Most heights have no row — a missing block in `blocks` is normal, + not a gap. Don't infer sync coverage from `blocks`; infer it from `intervals`. +- **`blocks` uses `number`; every other table uses `block_number`.** +- A row existing in `logs`/`blocks` does **not** mean Ponder will use it. Ponder decides what to + fetch/replay from `intervals`. If an interval says a range is synced, Ponder trusts the cache and + will **not** re-fetch — even if the cached blocks are stale. + +## How `intervals` works + +One row per **filter fragment** — a specific (chain, address, topic0, …) combination Ponder syncs. +`fragment_id` looks like: + +``` +log_______ +``` + +`blocks` is a Postgres `nummultirange` of the synced ranges, e.g. +`{[3702728,10770756],[10887910,10888672]}`. A **gap** between segments = unsynced → Ponder +re-fetches it. A continuous range = fully trusted cache. + +**Bounds must be inclusive `[a,b]`.** `nummultirange` is over `numeric`, which (unlike +`int8range`) does **not** canonicalize `[)` → `[]`. If you intersect with a default-bound range you +get a half-open `[a,b)` that Ponder mis-reads. Always force the inclusive upper bound: + +```sql +-- WRONG: leaves an exclusive upper bound [3702728,10770757) +blocks * nummultirange(numrange(NULL, 10770757)) +-- RIGHT: yields inclusive [3702728,10770756] +blocks * nummultirange(numrange(NULL, 10770756, '(]')) +``` + +Sanity check after any interval edit — this must return 0: + +```sql +SELECT count(*) FROM ponder_sync.intervals +WHERE chain_id = AND blocks::text LIKE '%)%'; +``` + +## Recipe: purge a chain wholesale (devnet restart) + +The ens-test-env devnet (anvil, chain id `31337`) is a fresh chain on every restart — but the chain +id stays the same, so `ponder_sync` keeps trusting cache from the previous run and Ponder serves +stale blocks instead of re-fetching. There is no clean prefix worth keeping (the chain is tiny and +free to re-sync), so purge everything for the chain: + +```sql +BEGIN; +DELETE FROM ponder_sync.intervals WHERE chain_id = 31337; +DELETE FROM ponder_sync.logs WHERE chain_id = 31337; +DELETE FROM ponder_sync.transaction_receipts WHERE chain_id = 31337; +DELETE FROM ponder_sync.transactions WHERE chain_id = 31337; +DELETE FROM ponder_sync.traces WHERE chain_id = 31337; +DELETE FROM ponder_sync.rpc_request_results WHERE chain_id = 31337; +DELETE FROM ponder_sync.blocks WHERE chain_id = 31337; +COMMIT; +``` + +(Deterministic deploys may reproduce identical early blocks across runs — don't rely on it.) + +## Recipe: clip the cache to block N and re-fetch the tail + +Use when cache above some block N is stale/contaminated and you want Ponder to re-fetch `(N, head]` +from the RPC while keeping the (expensive) deep history below N. Pick N = the **last block whose +cached hash matches the live chain** (see diagnosis below). + +```sql +BEGIN; +-- 1. clip every synced interval to [.., N] inclusive +UPDATE ponder_sync.intervals +SET blocks = blocks * nummultirange(numrange(NULL, , '(]')) +WHERE chain_id = ; + +-- 2. delete cached rows above N so they re-fetch clean (note: blocks uses `number`) +DELETE FROM ponder_sync.logs WHERE chain_id= AND block_number > ; +DELETE FROM ponder_sync.transaction_receipts WHERE chain_id= AND block_number > ; +DELETE FROM ponder_sync.transactions WHERE chain_id= AND block_number > ; +DELETE FROM ponder_sync.traces WHERE chain_id= AND block_number > ; +DELETE FROM ponder_sync.rpc_request_results WHERE chain_id= AND block_number > ; +DELETE FROM ponder_sync.blocks WHERE chain_id= AND number > ; +COMMIT; +``` + +Then restart the indexer. If the chain has **no** `_ponder_checkpoint` row and no entity rows, Ponder +indexes it from scratch over the (now-correct) cache + re-fetched tail. If it has a checkpoint, it +resumes from there. + +Always do interval clip + row deletes in **one transaction**, and keep them consistent: an interval +that claims a range is synced while the rows are gone makes Ponder skip real events; rows present +under a clipped interval are harmless (they get overwritten on re-fetch). + +## Diagnosing a stale / contaminated cache + +Symptom: a handler throws `RecordNotFoundError: No existing record found in table 'domains'` on a +`db.update` (e.g. `ENSv1Registry:Transfer`) for an entity whose creating event (`NewOwner`) is +earlier on-chain. The creating event was never indexed because the cache around it is stale. + +The usual culprit in this repo is a **restarted ens-test-env devnet** (chain id `31337`): the new +run is a different chain under the same chain id, so every cached block from the old run diverges — +purge the chain wholesale (recipe above). The same pattern appears with re-created forked testnets: +re-forking gives the chain a **new fork block**, so cached blocks from the old incarnation diverge +from the live chain, and clearing the cache only _above_ your resume point leaves the old-fork +window _below_ it intact. The diagnosis below finds the clip point for that partial-divergence case. + +Steps: + +1. **Identify the failing node and its creating tx.** From the error, get `node`, `tx hash`, + `block`. Pull the receipt via the chain's RPC (`RPC_URL_` in `apps/ensindexer/.env.local`) + to see the real event ordering (`NewOwner` creates, `Transfer`/`NewResolver` mutate). +2. **Compare cached block hashes to the live chain** at the failing block and the creating block: + + ```sql + SELECT number, hash FROM ponder_sync.blocks WHERE chain_id= AND number IN (,); + ``` + + vs `eth_getBlockByNumber` on the live RPC. A mismatch at the creating block but a match at the + failing block = the cache below your resume point is from a different fork. + +3. **Find the true fork block** (last block where live testnet == real upstream chain) by binary + search over `eth_getBlockByNumber` hashes (testnet RPC vs a real upstream RPC, e.g. a public + node for the upstream chain). Test activity lives strictly **above** this block. (For the + ens-test-env devnet there is no upstream — skip the search and purge the chain wholesale.) +4. **Find where the cache goes stale** (last cached block whose hash matches the real upstream + chain). Sample cached block numbers ascending and compare hashes; binary-search the boundary. + That boundary is the **old** fork — your clip point N. +5. Clip + purge with the recipe above, using N = last clean block. + +Helper to compare a live RPC hash to the cache: + +```bash +RPC=$(grep RPC_URL_ apps/ensindexer/.env.local | cut -d= -f2-) +rpchash(){ curl -s -X POST "$1" -H 'content-type: application/json' \ + --data "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_getBlockByNumber\",\"params\":[\"$(printf 0x%x $2)\",false]}" \ + | grep -o '"hash":"0x[0-9a-f]*"' | head -1 | sed 's/"hash":"//;s/"//'; } +cachehash(){ psql "postgresql://postgres@localhost:5432/ponder" -tA \ + -c "select hash from ponder_sync.blocks where chain_id= and number=$1;"; } +``` + +## Splicing one chain's sync into a forked chain + +A forked testnet _is_ the upstream chain below its fork block, so you can avoid re-syncing millions +of blocks through the fork's (often rate-limited) RPC by reusing real-upstream sync data: + +- For blocks `≤ forkBlock`: forked testnet ≡ upstream chain. Cache built against the upstream chain + (with `chain_id` rewritten to the testnet's id) is valid. +- For blocks `> forkBlock`: only the testnet has the data; must come from the testnet RPC. + +The clip-and-refetch recipe handles the common case: keep valid cache below the (true, current) +fork, let Ponder fetch the small tail above it from the fork's RPC. Only hand-splice upstream rows if even +that tail is too large for the testnet RPC. When splicing, rewrite `chain_id` on every table **and** +`intervals`, and keep `intervals` bounds inclusive. + +## Don't + +- Don't propose a Drizzle/Ponder schema migration — `ponder_sync` is Ponder-owned; edit it directly + in SQL, never via app migrations. +- Don't delete rows without clipping the matching `intervals` (Ponder will trust the now-empty range + and skip events). +- Don't trust the `blocks` table for coverage (it's sparse); trust `intervals`. +- Don't use `\x…` bytea literals against the TEXT `address`/`hash`/`topic*` columns. diff --git a/.agents/skills/ensindexer-perf-testing/SKILL.md b/.agents/skills/ensindexer-perf-testing/SKILL.md new file mode 100644 index 0000000000..c6fcfa4ab2 --- /dev/null +++ b/.agents/skills/ensindexer-perf-testing/SKILL.md @@ -0,0 +1,286 @@ +--- +name: ensindexer-perf-testing +description: > + Run perf tests on the ENSIndexer (Ponder app under apps/ensindexer). Establishes a clean, + reproducible baseline by disabling ENSRainbow healing on the hot path + (DISABLE_ENSRAINBOW_HEAL), isolating the schema, wiping prometheus between runs, and + snapshotting per-handler metrics at realtime. Use when the user + asks to compare branches/configs, build a perf baseline, or measure a plugin combination. +--- + +# ENSIndexer perf testing + +Captures wall-clock + per-handler timings for an ENSIndexer run. Always works in the same shape: +configure → disable heal → wipe → start → wait → snapshot → restore. + +## Required state before starting + +- Postgres at `postgresql://postgres@localhost:5432/ponder` is up. +- ENSRainbow data at `apps/ensrainbow/data/data-subgraph_0/` is hydrated. (The `data-ens-test-env_0/` dir is for ETE; it lacks subgraph labels.) +- Docker daemon is running (for prometheus/grafana sidecars). +- Working tree is clean (perf numbers must come from the branch as committed — no stray edits). + +If any of these aren't true, fix that first — don't paper over it. + +## Step 1 — start prometheus + grafana + +```bash +cd packages/ensindexer-perf-testing && pnpm start +# prometheus :9090, grafana :3001 (anonymous admin) +``` + +Confirm scrape target is reachable: + +```bash +curl -s http://localhost:9090/api/v1/targets?state=active | jq -r '.data.activeTargets[].health' +# should be 'up' once indexer starts +``` + +## Step 2 — start ENSRainbow (subgraph label set) + +ENSIndexer performs an explicit health + ready check against ENSRainbow at startup, so the +service must be running even when healing is disabled. Use the `subgraph` data dir for mainnet +runs: + +```bash +cd apps/ensrainbow && \ + DATA_DIR=$PWD/data/data-subgraph_0 PORT=3223 LOG_LEVEL=warn \ + pnpm tsx src/cli.ts serve > /tmp/perf-test/ensrainbow.log 2>&1 & +# wait ~5s, then verify +curl -s http://localhost:3223/health +``` + +For ens-test-env runs, swap `DATA_DIR=…/data-ens-test-env_0` and set +`LABEL_SET_ID=ens-test-env` in `.env.local`. + +## Step 3 — disable ENSRainbow healing + +Set the undocumented env var `DISABLE_ENSRAINBOW_HEAL=true` (add it to `.env.local` in Step 4, or +export it before launching). `labelByLabelHash` (`src/lib/graphnode-helpers.ts`) early-returns +`null` — every label is treated as unhealable — giving an apples-to-apples baseline without +ENSRainbow round-trips. No code edit needed. + +This changes indexing output: never set it outside perf runs, and always remove it from +`.env.local` at end of run. + +## Step 4 — configure `.env.local` + +Pick a unique schema per run so concurrent/sequential runs don't pollute each other: + +```env +NAMESPACE=mainnet +PLUGINS=ensv2,protocol-acceleration # or whatever combo +ENSINDEXER_SCHEMA_NAME=ensindexer_perf_