[draft] Feature branch: add sled-agent-scrimlet-reconcilers#10441
Draft
jgallagher wants to merge 12 commits into
Draft
[draft] Feature branch: add sled-agent-scrimlet-reconcilers#10441jgallagher wants to merge 12 commits into
sled-agent-scrimlet-reconcilers#10441jgallagher wants to merge 12 commits into
Conversation
This is groundwork for #10167, and introduces the skeleton of network config reconcilers for use within sled-agent. None of this is wired up yet and all the service-specific reconcilers are placeholders, but it does have the real setup for how these tasks get started and how they report status. The PR is pretty big but hopefully not too bad to review; more than half the code falls into either "tests", "status type definitions", or "placeholder/dummy reconcilers". A tentative suggestion for review order is: 1. The crate-level docs in `lib.rs`; these are written assuming #10167 is complete, not based on the current state of the crate. 2. `handle.rs`, particularly `ScrimletReconcilers` - this is the entry point for sled-agent. It will hold a `ScrimletReconcilers` in its set of long-running tasks. 3. `reconciler_task.rs` - this implements the common control flow for all of the service-specific reconcilers in the crate; handling periodic reactivation, activation when the config changes, transitioning to inert if we stop being a scrimlet because the sidecar goes away at runtime, and transitioning out of inert if it comes back. ~~The only production-affecting change here is that the `ThisSledSwitchZoneUnderlayIpAddr` type moved out of sled-agent and into this crate, so sled-agent depends on this crate just for that type.~~ Edit: As of #10340, `ThisSledSwitchZoneUnderlayIpAddr` has moved to `sled-agent-types`, so now this PR uses it from there and makes no changes to sled-agent proper.
This is almost all new code (and about 60/40 "real code" / "tests"). Today NAT entry reconciliation is inverted from what this PR implements: dpd fetches NAT entries from Nexus and does reconciliation internally. In this PR, sled-agent does the reconciliation. This requires fetching _all_ NAT entries from dpd (to decide what to remove, if anything). My biggest concerns here are: 1. Is it correct? 2. Can we ship this, or do we need to block on oxidecomputer/dendrite#255 or something like it? This is targeted at the feature / integration branch that's building up the complete `sled-agent-scrimlet-reconcilers` crate.
This PR looks big but over 60% of it is tests; the production code that needs careful review is the new `sled-agent/scrimlet-reconcilers/src/dpd_reconciler/port_reconciler.rs` module. It contains a port of what `sync_switch_reconciler` does to configure QSFP ports, with a couple changes: 1. We read the desired settings from `RackNetworkConfig` instead of the db 2. We compute diffs via `daft` --------- Co-authored-by: Levon Tarver <levon@oxide.computer>
Fleshes out the next sub-reconciler. As with the dpd PR on which this is stacked (#10449), this ports over the behavior from `sync_switch_configuration` but changes the mechanics (read from `RackNetworkConfig` and compute diffs via `daft`). Also as before, the majority of this PR (~2/3) is tests; the actual new prod code here is more like ~600 lines.
…nt-scrimlet-reconcilers
…nt-scrimlet-reconcilers
This was added to early networking / Nexus as a part of #10472; we need to keep pace with it until this project lands. --------- Co-authored-by: Levon Tarver <11586085+internet-diglett@users.noreply.github.com>
This is a significant departure from the `sync_switch_configuration` bg task's implementation of BGP config reconciliation. While adding proptests (which tend to hit unusual cases - here, it was "changing the ASN that owns a given peer"), we discovered: 1. mgd's `bgp_apply` endpoint is a little buggy (the specific thing I hit is fixed by oxidecomputer/maghemite#773) 2. The structure of the `bgp_apply` endpoint, because it takes the config for a single ASN, doesn't really allow for easy "remove the BGP config" or "change the config from ASN x to ASN y". Based on oxidecomputer/maghemite#773 (comment), this PR doesn't use the `bgp_apply` endpoint, and instead makes multiple smaller calls to the various REST endpoints that comprise the BGP config. This would be much racier in Nexus, where different Nexus instances could be competing (potentially with out of date information if the config just changed), but here the one and only client is the local sled-agent, so we don't have to worry about that problem.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature branch to build up the initial implementation of the
sled-agent-scrimlet-reconcilerscrate. Contains these PRs:sled-agent-scrimlet-reconcilerscrate #10313