docs(operators): role-based operator docs with guided setup#24219
docs(operators): role-based operator docs with guided setup#24219y3v63n wants to merge 1 commit into
Conversation
y3v63n
force-pushed
the
docs/operator-docs-role-based-restructure
branch
from
b47d7e4 to
3dc06ab
Compare
|
Follow-up note for the v5 changelog (not blocking this PR) From v5 onward, each changelog entry under
A flat breaking-changes list doesn't convey the pre-flight/blocking steps or the ordering, which is exactly what operators currently reconstruct from Discord and Telegram. Suggested section shape: pre-flight/backup → ordered steps → breaking-changes table → env var renames/removals → verification → rollback note → link to the canonical migration guide. The breaking-changes table above should always include three standard subsections, because these are the changes that silently break operator tooling and currently force operators to diff the codebase to find them:
The "Updates, alerts, rewards" pages already link operators to this changelog for upgrade announcements, so standardizing the structure here is the single place that closes the loop. No template work in this PR; capturing the intent so the v5 entry adopts it. |
|
|
||
| ### The spec is per sequencer | ||
|
|
||
| The CPU and RAM above are sized for one attesting sequencer. Stake scales in units of 200,000 AZTEC, and each unit is a separate attester identity. |
There was a problem hiding this comment.
I wouldn't say this is accurate. The load is fairly identical whether you run one or 100 sequencers.
|
|
||
| - Each delegated stake position you operate is a separate attester identity. | ||
| - For high availability, run one publisher plus one or more redundant nodes with a coordinated shared keystore. | ||
| - A provider running 10 attester positions across 2 HA-paired nodes is operating roughly 2x the spec above. |
There was a problem hiding this comment.
If you meant that the 2 nodes is whats leading to 2x spec then fine, but 10, 100, 1000 attesters should be roughly the same load.
| | Role | Type | Purpose | Slashable? | Rotatable? | | ||
| |---|---|---|---|---| | ||
| | **Attester** | L1 address plus a BLS key pair | Signs attestations and proposals. Identifies the validator on-chain. | Yes (offenses listed in [Slashing](./slashing)) | No. The attester address and its BLS key are bound permanently the first time they register. If the attester exits or is ejected, that address cannot return to sequencer duties; generate a new attester identity and stake again. | | ||
| | **Publisher** | L1 EOA | Pays L1 gas to submit checkpoints, proposals, and attestations. Holds ETH only. | No directly, but a publisher that runs out of ETH causes missed proposals which slash the attester. | Yes (edit publisher config and restart) | |
There was a problem hiding this comment.
As of now, there is no slashing for missed checkpoints
|
|
||
| - **Block rewards** (per checkpoint you propose) go to your **coinbase**. | ||
| - **Stake** is held against the **attester address**; only the **withdrawer** can move it. When stake leaves, it does not return to the attester: a voluntary withdrawal goes to the recipient the withdrawer chooses, and an ejection (stake slashed below the threshold) creates an exit that returns the stake to the withdrawer. | ||
| - **Future transaction fees** will route to the **fee recipient** (currently unused). |
There was a problem hiding this comment.
wrong, tx fees go to coinbase address
|
|
||
| - Slashing is live on mainnet. | ||
| - Each slash deducts **2,000 AZTEC** from your bonded stake (1% of the 200,000 AZTEC activation stake). | ||
| - Your stake is ejected from the active set if it falls below **190,000 AZTEC**, and is no longer recoverable without a full exit if it falls below **100,000 AZTEC**. |
There was a problem hiding this comment.
stake is no longer recoverable without a full exit if it falls below 190k (and not necessarily 100k).
| LOG_LEVEL=info | ||
| ETHEREUM_HOSTS=[your L1 execution endpoint] | ||
| L1_CONSENSUS_HOST_URLS=[your L1 consensus endpoint] | ||
| ETHEREUM_DEBUG_HOSTS=[your trace capable L1 execution endpoint] |
There was a problem hiding this comment.
I don't think you need both hosts (debug) but not 100% sure
| /usr/src/yarn-project/aztec/dest/bin/index.js | ||
| start | ||
| --node | ||
| --archiver |
There was a problem hiding this comment.
note: deprecated for v5
|
|
||
| The current canonical address is always whatever `getCanonicalRollup()` returns on the Aztec Registry: `0x35b22e09ee0390539439e24f06da43d83f90e298` on mainnet, `0xa0bfb1b494fb49041e5c6e8c2c1be09cd171c6ba` on testnet. A testnet relaunch deploys a new Rollup, so confirm against the Registry if you are unsure which is current. | ||
|
|
||
| Note: prover reward claiming is currently disabled on testnet (`isRewardsClaimable()` returns false there); rewards accrue but cannot yet be claimed. |
There was a problem hiding this comment.
not true with the current v5 testnet. Rewards are claimable on testnet today.
y3v63n
force-pushed
the
docs/operator-docs-role-based-restructure
branch
from
3dc06ab to
cb1df13
Compare
|
The branch is now rebased onto current New since the PR opened:
Corrections to review points:
Reconciled with intervening upstream changes: kept |
There was a problem hiding this comment.
don't think this file is adding much value
| } | ||
| ``` | ||
|
|
||
| 4. Restart the node so the new keystore loads: |
There was a problem hiding this comment.
I prefer to document calling reloadKeystore() via the admin RPC instead.
| - **[Aztec staking dashboard](https://stake.aztec.network/my-position)** — aggregates pending rewards across every sequencer you operate, claims in one wallet flow. Easiest for most operators. | ||
| - **CLI claim** — see the [Claiming Rewards guide](/operate/operators/sequencer-management/claiming-rewards) for the full reference (pending-rewards check, `claimSequencerRewards` cast send, keystore-file and hardware-wallet variants, gas-limit troubleshooting). | ||
|
|
||
|
|
There was a problem hiding this comment.
To be accurate, there's actually two functions to call. One distribute() which sends all funds to the warehouse contract. And then there's a second function call on the warehouse contract.
| ```bash | ||
| # View logs | ||
| docker compose logs -f aztec-sequencer | ||
|
|
||
| # Restart the node | ||
| docker compose restart aztec-sequencer | ||
|
|
||
| # Stop the node | ||
| docker compose down | ||
|
|
||
| # Update to latest version | ||
| docker compose pull && docker compose up -d | ||
|
|
||
| # Check disk usage (run inside ~/aztec-sequencer/) | ||
| du -sh ./data | ||
| ``` |
There was a problem hiding this comment.
I don't think these are useful.
| - **Storage:** 1 TB SSD, NVMe recommended (more if you co-host an L1 node; see [Hardware spec](/operate/operators/concepts/hardware#co-hosting-an-l1-node)) | ||
| - **Network:** 25 Mbps or faster, with a static public IP | ||
|
|
||
| The CPU must be Skylake or newer: the prover binary is built for that instruction set and crashes at startup on older chips (some 2015 CPUs are pre-Skylake). See [Hardware spec](/operate/operators/concepts/hardware). |
There was a problem hiding this comment.
I've never heard this before. Double checking
| Then restart the node: | ||
|
|
||
| ```bash | ||
| docker compose restart aztec-sequencer |
There was a problem hiding this comment.
use the admin api to update keystore without restart.
|
|
||
| ### 2. Add sequencer keystores to your queue | ||
|
|
||
| Your provider queue holds the attester identities that get activated when delegators stake. Use the [following helper script](https://gist.github.com/koenmtb1/1b665d055fbc22581c288f90cdc60d88) to build the `addKeysToProvider` call from your `keyN_staker_output.json` (the file generated by `aztec validator-keys new --staker-output`): |
There was a problem hiding this comment.
Not a fan of linking community scripts in the official docs.
|
|
||
| **Severity: Investigate.** As the slot's proposer, you did not collect two-thirds of the committee's attestations in time, so that slot was missed. `X/Y` is how many attestations you collected versus how many were needed. The usual cause is that not enough of the committee's attesters were online that round. | ||
|
|
||
| **Action:** an occasional timeout is normal. Check your own peer count and connectivity first. If your node is healthy and this is happening across many slots, it reflects a wider network liveness condition rather than a problem you can fix locally. |
There was a problem hiding this comment.
an occasional timeout is normal.
Not really, they miss on rewards
| <IfChoice axis="network" is="testnet"> | ||
|
|
||
| :::warning Testnet blob requirement | ||
| On testnet your beacon client must serve all blob data. Run it with `--subscribe-all-data-subnets` (or as a supernode), or the sequencer will fail with blob sidecar errors. See the [FAQ](/operate/operators/operator-faq) for details. |
There was a problem hiding this comment.
whyis this just for testnet?
|
|
||
| **Full attester report.** When a status is anything other than `1` and you need to see why, download [`check-attester.sh`](pathname:///scripts/check-attester.sh). It checks one or more attesters (status, effective balance, exit state, withdrawer) and prints a labeled, color-coded report with a health verdict per attester: | ||
|
|
||
| <ConfigCode language="bash">{`curl -o check-attester.sh https://docs.aztec.network/scripts/check-attester.sh |
There was a problem hiding this comment.
this is a good example of where scripts should live
Restructure the operator documentation around what you run (full node, solo sequencer, staking provider, prover), with a role-picker landing page, guided per-role setup paths, concept essentials, and a reference section. Adds interactive helpers (an operator config pane that fills commands from your values, an L1 publisher funding calculator, and a prover-rewards claim generator), corrects metric names, slashing thresholds, reward-distribution mechanics, and the L1 blob and trace requirements against the current protocol, and hosts operator helper scripts in the docs.
y3v63n
force-pushed
the
docs/operator-docs-role-based-restructure
branch
from
cb1df13 to
084c407
Compare
|
Thanks for the review. I've pushed an update addressing the comments; the branch was also rebased onto current Reward distribution & provider flow
Hardware & CPU
Accuracy fixes
Scripts & content
Also replaced the role-picker emoji with line icons on the landing page. |
2 participants
Summary
Restructures the operator docs (
docs/docs-operate/operators/) into a role-based guided setup: an operator picks what they want to run (full node, solo sequencer, staking provider, or prover) and follows a path sized to that role, instead of navigating a flat list of setup pages.Rationale. The current operator docs are hard to navigate: new operators don't know where to start, and existing operators can't find answers to recurring questions. The top-level "Build / Operate / Participate" navigation is abstract and overlaps for anyone who doesn't already know the network's vocabulary, and the operator landing page is theory-heavy with no obvious "what do I do next."
This redesign is driven by two data sources:
#mn-sequencer-technical,#new-sequencer-technical,#new-prover-technical), analyzed for recurring complaints as of 2026-05-15.It follows two principles: (1) the first page an operator sees should answer "what can I do here and where do I start," not "what is a sequencer"; and (2) if a question has been asked dozens of times in Discord, the answer should be one or two clicks from the landing page.
What's actually new in these pages. The content below didn't exist in the operator docs before; each page was written from on-chain verification and the operator-reported pain points:
Identity model (new Essentials page) — the full address-to-role table (attester, publisher, coinbase, fee recipient, withdrawer, plus provider admin and rewards-recipient) with slashable/rotatable columns, a "which address gets which reward" model, where each address lives in the keystore, how to check each on-chain, and the four most common confusions as an FAQ.
L1 RPC requirements (new Essentials page) — the three endpoints a node needs (
ETHEREUM_HOSTS,L1_CONSENSUS_HOST_URLS,ETHEREUM_DEBUG_HOSTS); the post-Fusaka blob-serving requirement with concrete supernode/semi-supernode versions (Lighthouse v8.0.0+, Prysm v7.10+ and the flags); thedebug_traceTransactionhistory requirement (~12h / ~3,600 blocks); measured RPC call volume (~170,000/day/node floor); polling-interval tuning; and a common-errors table. None of this was canonically documented before.Hardware spec (new Essentials page) — corrected canonical numbers (8 core / 8 GB / 1 TB SSD, NVMe recommended, 50,000+ IOPS, 25 Mbps floor), an explicit per-sequencer note (the spec is per attester and committee overlap drives real load), co-hosted L1 disk math by network (~2 TB testnet / ~4 TB mainnet), provider per-attester multiplication, and prover broker/agent scaling (the 1→4 agent table and the
aztec.proving_queue.sizemetric).Slashing (new Essentials page) — the eight offenses with their penalty parameters and which actually fire today, the flat 2,000-AZTEC penalty and the V5 tiered-penalty forward note, the three thresholds (200K / 190K / 100K) and the five-slashes-of-headroom math, the veto council and how to request a veto, how to detect a slash, and the full recovery path after ejection including the genesis-sequencer case and the zombie initiate/finalize-unstake flow on the dashboard.
Claiming rewards (new Essentials page) — reframed around the staking dashboard as the primary path. Distinguishes self-staker claiming (per-sequencer Claim on
/my-position) from provider claiming (Operator Commission tab on/operator), with the two coinbase scenarios (separate address vs coinbase=attester) that explain why the dashboard total can read zero, and the Manage-Addresses tracking-list steps. Screenshots included.Claiming prover rewards (new Prover page) — the previously-undocumented claim flow: how prover rewards accrue per epoch to
PROVER_ID, theclaimProverRewards(coinbase, [epochs])call, the fact that any wallet can submit it (the prover key never has to sign the claim), the manualcastfallback, and the mainnet + testnet Rollup addresses with the Registry as source of truth.Reading your logs (new Reference page) — a catalogue of the most common log messages labeled ignore / investigate / urgent, covering the keystore ZodError (staker-output file in the keystore dir), libp2p churn, attestation timeouts, the archiver debug-trace warning, and the prover payload-size error. Each entry verified against the v4.3.1 source.
Monitoring (new Essentials page) — the handful of metrics that actually matter (block height, publisher balance, peer count, mempool) with concrete alert thresholds, why publisher-balance is the key slashing proxy, the OTEL→Prometheus naming, and how to wire up a stack or use the community dashboards.
Run a Node landing + Popular questions + Operator tooling index — the role-picker entry point, the recurring-question shortcuts, and a curated index of Foundation vs community tools (clearly labeled as third-party).
A set of loopholes is intentionally staged, not yet in this PR (tracked for follow-up): jailed/genesis-sequencer recovery framing, a "Sequencer troubleshooting" decision tree, "Managing delegations," "Where to stay informed," migration guides for HA topology changes, and the per-version changelog template extensions (RPC schema / metric-name / env-var subsections).
What's new
/operate/...for backwards compatibility./operate/operators: four cards (full node, solo sequencer, staking provider, prover), each with the role's purpose, key requirements (stake, hardware, what you earn), what's distinctive, and a CTA into the matching setup path. Operators decide what they're setting up before touching any commands.Notable changes
Three interactive helpers were added to remove the most error-prone manual steps. All are client-side React components (no backend); they read public on-chain data via the operator's own RPC and never handle private keys.
OperatorConfig) — on the guided playbooks, the operator pastes their values (network, RPC, addresses, sequencer count) once via a track-picker, and every code block on the page fills in with those values. Removes copy-paste-and-edit errors across a multi-page walkthrough.propose()activity, derives the per-propose gas and the gas-price percentiles actually paid (including the proposal blob fee), and converts a top-up amount into a runway estimate at chosen and elevated gas prices. Falls back to a measured baseline before a scan.PROVER_ID; the component runs read-only calls to find every recent epoch with an unclaimed reward and assembles the exactclaimProverRewards(coinbase, [epochs])command with the epoch list filled in. Solves the "which epochs do I claim, and how" problem that previously forced provers into Etherscan.Supporting plumbing:
src/components/(OperatorConfig,RolePicker,PopularQuestions), registered insrc/theme/MDXComponents.js.--aztec-card-bgadded tosrc/css/custom.css(used by the role cards).Verification
Builds clean against
nextwithonBrokenLinks: "throw"(yarn preprocess && yarn docusaurus build→[SUCCESS] Generated static files).