feat(anc): gate check-hotfix on enable_provisioning_hotfix contract field by Devinwong · Pull Request #8717 · Azure/AgentBaker

Devinwong · 2026-06-16T00:22:03Z

2.1d - gate check-hotfix on the enable_provisioning_hotfix contract field

POC / M1 draft. AgentBaker / Node SIG side only.

This is the final layer of the provisioning-hotfix stack. It makes the AKSNodeConfig
contract field the single source of truth for whether aks-node-controller check-hotfix
does any work, and relaxes the env gate added in 2.1c.

What changed

Proto contract field: add bool enable_provisioning_hotfix = 45; to
aksnodeconfig/v1/config.proto (next free tag after cse_timeout = 44) and regenerate
the Go bindings.
Go gate: check-hotfix reads the field at the very top of checkHotfix() via
App.provisioningHotfixEnabled() (reads the node-config JSON that is already on disk and
calls GetEnableProvisioningHotfix()). When the field is not true (false, unset, or the
config cannot be read/parsed) it returns the new telemetry outcome disabled and exits 0
WITHOUT any remote hotfix call. Fail-open everywhere.
Wrapper relaxation: aks-node-controller-wrapper.sh now calls check-hotfix
UNCONDITIONALLY (still wrapped defensively so it can never block provisioning). The
Go binary self-gates on the contract field.

Read channel

The hotfix-pointer READ CHANNEL is moving from a kube-system ConfigMap (read with a bootstrap
token) to the live-patching-service (LPS) IMDS-attested endpoint, validated reachable pre-kubelet
in e2e. That fetch/auth rewrite lives in #8696; this PR is channel-agnostic. The
enable_provisioning_hotfix contract field and the Go self-gate decide WHETHER check-hotfix
runs at all, independent of which channel it then uses, so the proto field is intentionally
channel-neutral and unchanged by the pivot.

Supersedes the env-delivery approach

An earlier revision of this PR delivered the toggle as an env var via a cse_cmd.sh
template var plus a systemd drop-in (Environment="ENABLE_PROVISIONING_HOTFIX=...") on
aks-node-controller.service, mirroring the IMDS-restriction pattern. That approach was
dropped because:

check-hotfix already parses the AKSNodeConfig for its own connection details, so a real
contract field is available to the binary with zero new plumbing -
no template var, no drop-in, no env var.
In the self-provisioning path the wrapper and the drop-in writer are the same service, so an
env/drop-in written during provisioning would only take effect on the NEXT boot. Reading the
contract field directly avoids that activation-timing problem - it works on the same boot
because the config JSON is on disk before the service starts.

This also means absvc sets ONE field (the contract bool), not an env var plus a field.

Relaxes the 2.1c env gate

This PR relaxes the ENABLE_PROVISIONING_HOTFIX env gate introduced in #8715 (2.1c); gating
now lives in the Go binary via the enable_provisioning_hotfix contract field - single source
of truth, so absvc sets ONE field, not an env var plus a field. The 2.1c env gate is
intentionally added-then-relaxed across the stack so each PR stays reviewable on its own.

Default-off and fail-open

When enable_provisioning_hotfix is false or unset, behavior is byte-identical to before this
stack: check-hotfix makes no remote call and provisioning proceeds unchanged. Any read or
parse error is treated as off. This preserves the 6-month VHD support window in both directions
(older VHD + newer config, and newer VHD + older binary are both safe).

Before / after

field false or unset -> check-hotfix returns outcome=disabled, no remote call, exit 0
field true -> check-hotfix reads the hotfix pointer from the LPS (live-patching-service)
endpoint and stages it (the read channel itself is owned by feat(anc): add check-hotfix subcommand to read hotfix pointer from LPS #8696)

Stack

main
 \- #8694  2.1a  base->version hotfix map (Go)
     \- #8696  2.1b  check-hotfix LPS endpoint reader (Go)
         \- #8715  2.1c  wire check-hotfix into wrapper (shell)
             \- #8717  2.1d  enable_provisioning_hotfix contract field + Go self-gate   <- this PR

The aks-rp region toggle that sets the field is in a different repo and is the only remaining
out-of-repo piece. With the field settable on a node, the on-node PoC e2e tests (fail-open and
multi-base) become runnable.

Tests

go test ./... in aks-node-controller: all check-hotfix tests pass, including new gate
tests (disabled -> outcome=disabled and the injected fetcher is never called; enabled ->
fetch path runs). Pre-existing Windows-only failures (CRLF goldens, file locks, os-release
message text) are unrelated and also fail on the base branch.
Wrapper shellspec updated for unconditional check-hotfix: 7 examples, 0 failures.
shellcheck clean on the wrapper.

github-actions · 2026-06-16T00:22:31Z

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Jul 10, 2026, 8:12 PM

Devinwong · 2026-06-16T02:26:19Z

Acknowledged - no action needed. This is the automated Buf CI status, and it reports Build, Format, Lint, and Breaking all passing for the additive optional field enable_provisioning_hotfix = 45. That matches the local buf verification (lint STANDARD clean, WIRE_JSON breaking clean against the 2.1c base, format clean), so the proto change is confirmed compatible and no change is warranted.

Address review feedback: sourcing the feature-flag file via . executed arbitrary shell, so a malformed file could run commands or �xit and abort the wrapper - undermining fail-open. Parse KEY=VALUE lines instead and never execute the file; blank lines, comments, and non-identifier keys are skipped. The file is fully producer-controlled, so any valid identifier=value is accepted (not a fixed key list). Producer contract is unchanged (boothook writes the literal unquoted line ENABLE_PROVISIONING_HOTFIX=true). Adds a shellspec case proving arbitrary lines (exit, touch) are never executed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Address Copilot review feedback: expand the FEATURES_PATH comment to document the on-node delivery mechanism explicitly - the cloud-init boothook (producer, #8717) is the sole trusted writer, running as root, and the file lands 0644 root-owned under /opt/azure/containers. Clarifies for reviewers/operators how ENABLE_PROVISIONING_HOTFIX is actually set, since there is no systemd env-var delivery path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The KEY=VALUE parse loop dropped the last line when the file had no trailing newline (read returns non-zero at EOF). Add the standard '|| [ -n "$_key" ]' guard so the final flag is honored, and add a shellspec regression covering a file with no trailing newline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR extends the AKSNodeConfig contract with a new boolean field intended to control whether aks-node-controller check-hotfix performs provisioning-time hotfix pointer refresh work, and updates supporting code/tests around the wrapper/bootstrapping path.

Changes:

Add enable_provisioning_hotfix to the aksnodeconfig/v1 proto contract and regenerate Go bindings.
Update nodeconfigutils.CustomData() boothook generation to optionally write /opt/azure/containers/enabled_features.sh when the field is enabled.
Adjust ShellSpec coverage for the wrapper’s feature-flag behavior.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
spec/parts/linux/cloud-init/artifacts/aks_node_controller_wrapper_spec.sh	Updates wrapper ShellSpec expectations/coverage around feature-flag parsing behavior.
aks-node-controller/proto/aksnodeconfig/v1/config.proto	Adds the `enable_provisioning_hotfix` contract field (tag 45) and documents intended semantics.
aks-node-controller/pkg/nodeconfigutils/utils.go	Adds enabled-features file path + boothook emission of an enabled-features write block.
aks-node-controller/pkg/nodeconfigutils/utils_test.go	Adds unit tests asserting boothook emission/omission behavior for enabled-features output.
aks-node-controller/pkg/gen/aksnodeconfig/v1/config.pb.go	Regenerated protobuf Go output including the new field + accessor.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/config.pb.go: Generated file

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/config.pb.go: Generated file

+  // Enables provisioning-time hotfix checking. When true, the cloud-boothook writes
+  // an enabled_features.sh file that the aks-node-controller wrapper parses to gate
+  // the check-hotfix subcommand; when false or unset no file is written and the
+  // wrapper skips check-hotfix entirely. Default-off, fail-open.


+	require.Contains(t, on, "\nENABLE_PROVISIONING_HOTFIX=true\n")
+	require.Contains(t, on, "chmod 0644 '/opt/azure/containers/enabled_features.sh'")
+
+	// The features file must be written BEFORE the service is started, so the wrapper can source it.


+	// This path is a shared contract with the aks-node-controller wrapper's FEATURES_PATH default.
+	// If it changes here it must change there too, or the wrapper will never source the file.
+	require.Equal(t, "/opt/azure/containers/enabled_features.sh", EnabledFeaturesFilePath)


Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/config.pb.go: Generated file

+  // Enables provisioning-time hotfix checking. When true, the cloud-boothook writes
+  // an enabled_features.sh file that the aks-node-controller wrapper parses to gate
+  // the check-hotfix subcommand; when false or unset no file is written and the
+  // wrapper skips check-hotfix entirely. Default-off, fail-open.
+  bool enable_provisioning_hotfix = 45;


+	// The features file must be written BEFORE the service is started, so the wrapper can source it.
+	featuresIdx := strings.Index(on, "enabled_features.sh")
+	startIdx := strings.Index(on, "systemctl start --no-block aks-node-controller.service")
+	require.NotEqual(t, -1, featuresIdx)
+	require.NotEqual(t, -1, startIdx)
+	require.Less(t, featuresIdx, startIdx, "enabled_features.sh must be written before the service starts")


+func TestEnabledFeaturesFilePathMatchesWrapperContract(t *testing.T) {
+	// This path is a shared contract with the aks-node-controller wrapper's FEATURES_PATH default.
+	// If it changes here it must change there too, or the wrapper will never source the file.
+	require.Equal(t, "/opt/azure/containers/enabled_features.sh", EnabledFeaturesFilePath)
+}


Adds the producer half of the provisioning-hotfix gate. The AKSNodeConfig contract field `enable_provisioning_hotfix` (tag 45) now drives an on-node feature-flag file: when true, the cloud-boothook writes /opt/azure/containers/enabled_features.sh containing the literal line `ENABLE_PROVISIONING_HOTFIX=true` (quoted heredoc, chmod 0600) before the aks-node-controller service is started. The 2.1c wrapper sources that file and keeps the existing gate, so this file is the single delivery channel. When the field is false/unset, nothing is written and the generated custom data is byte-identical to before this feature existed (6-month VHD backward-compat). Flatcar/Ignition path does not write enabled_features.sh in M1; provisioning- hotfix is a no-op on Flatcar (follow-up if product needs it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 6 comments.

Files not reviewed (1)

aks-node-controller/pkg/gen/aksnodeconfig/v1/config.pb.go: Generated file

+# flags like ENABLE_PROVISIONING_HOTFIX (there is no systemd environment-variable delivery).
+# Writer: the cloud-init boothook (producer side, PR #8717), running as root at provision time,
+# writes it ONLY when the corresponding aks-rp toggle is on. It lands under /opt/azure/containers
+# (0644, root-owned) like the other provisioning artifacts, so only root can populate it and the


+# $HOTFIX_JSON) that download-hotfix reads below, so it must run first. Gated default-off
+# behind ENABLE_PROVISIONING_HOTFIX (only the literal "true" enables it) - the on-node
+# terminal of the EnableProvisioningHotfix aks-rp region toggle. Wrapped defensively: it is
+# fail-open, but an older ANC binary predating the subcommand exits non-zero.
+if [ "${ENABLE_PROVISIONING_HOTFIX:-}" = "true" ]; then


+	return fmt.Sprintf(`cat <<'EOF' >%[1]s
+ENABLE_PROVISIONING_HOTFIX=true
+EOF
+chmod 0600 %[1]s
+`, EnabledFeaturesFilePath)


        The variable thirdArg should eq ""
    End
+
+    It 'does not call check-hotfix when ENABLE_PROVISIONING_HOTFIX is unset'


+	require.Contains(t, on, "\nENABLE_PROVISIONING_HOTFIX=true\n")
+	require.Contains(t, on, "chmod 0600 /opt/azure/containers/enabled_features.sh")
+
+	// The features file must be written BEFORE the service is started, so the wrapper can source it.


+
+func TestEnabledFeaturesFilePathMatchesWrapperContract(t *testing.T) {
+	// This path is a shared contract with the aks-node-controller wrapper's FEATURES_PATH default.
+	// If it changes here it must change there too, or the wrapper will never source the file.


Devinwong · 2026-07-10T22:16:21Z

Closing 2.1d as no-longer-needed — enablement delivery has been re-architected.

Decision

Per the design review on the 2.1c wrapper session, the on-node gate for check-hotfix needs only the feature-flag file /opt/azure/containers/enabled_features.sh parsed by the wrapper (aks-node-controller-wrapper.sh, merged in #8715/2.1c). The enable_provisioning_hotfix proto field this PR added is an unnecessary cross-component signal: enablement will be delivered by aks-rp writing the feature-flag file directly via native cloud-init, gated by the region toggle — not via an AgentBaker contract field or boothook.

Why this PR's approach didn't reach production anyway

This PR's boothook write lived in aks-node-controller/pkg/nodeconfigutils/utils.go (CustomData()), which is referenced only by e2e/vmss.go and the README — not the production provisioning path. Production scriptless custom data is assembled by getScriptlessNBCCustomData → its own boothookTemplate in pkg/agent/baker.go. So the flag written here would have reached only e2e nodes. Rather than re-home the write into baker.go, the cleaner architecture is aks-rp-side native cloud-init delivery (no AgentBaker contract field, single source of truth at the region toggle).

Disposition

No AgentBaker change is required for enablement; the only AgentBaker-side piece (the wrapper gate) already lives in feat(anc): wire check-hotfix into node wrapper behind ENABLE_PROVISIONING_HOTFIX #8715/2.1c.
Enablement is now tracked in the aks-rp enable-provisioning-hotfix toggle work, which will inject the cloud-init part writing enabled_features.sh (literal ENABLE_PROVISIONING_HOTFIX=true) before the wrapper runs.
Branch and commit are preserved (tag backup-2.1d-featuresfile-9102831736) in case any part is revived.

Closing.

aks-node-assistant · 2026-07-10T23:34:16Z

AgentBaker Linux gate detective

Run: 171764413
Failed job/stage/task: e2e / Run AgentBaker E2E / Test_Ubuntu2204_DisableKubeletServingCertificateRotationWithTags_CustomKubeletConfig/default
Detective summary: E2E failed because VMSS creation exceeded WestUS3 standardDDSv5Family quota. This is infrastructure capacity pressure, not the enable_provisioning_hotfix contract change.

Likely cause/signature: VMSS create returned ARM 409 Conflict / OperationNotAllowed: WestUS3 standardDDSv5Family cores quota was fully consumed (1300/1300) and the test needed 2 additional cores.
Confidence: High
Recommended owner/action: AgentBaker E2E / Node Lifecycle infra owner should increase/free WestUS3 standardDDSv5Family quota or reduce concurrent DDSv5 E2E usage; PR author can rerun after capacity is available.
Strongest alternative: The provisioning-hotfix contract change broke ANC/provisioning behavior, but less likely because VMSS allocation failed before provisioning logic could run and unrelated PRs hit the same quota signature.

Evidence: build metadata, timeline, failed E2E log 538, failed test results, and same-window quota failures across unrelated PRs.
Wiki signature: linux-vhd-gate-westus3-standarddsv5-quota-exceeded

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from 2d4b37d to 59b7cef Compare June 16, 2026 01:33

Devinwong changed the title ~~feat(anc): deliver ENABLE_PROVISIONING_HOTFIX to node-controller via contract field + systemd drop-in~~ feat(anc): gate check-hotfix on enable_provisioning_hotfix contract field Jun 16, 2026

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from 59b7cef to d80ae7d Compare June 16, 2026 02:21

This was referenced Jun 16, 2026

feat(anc): wire check-hotfix into node wrapper behind ENABLE_PROVISIONING_HOTFIX #8715

Merged

feat(anc): add check-hotfix subcommand to read hotfix pointer from LPS #8696

Merged

Devinwong marked this pull request as ready for review June 16, 2026 02:27

Devinwong requested review from AbelHu, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright and zachary-bailey as code owners June 16, 2026 02:27

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from d80ae7d to 6854cfa Compare June 16, 2026 16:59

Devinwong force-pushed the devinwong/anc-wire-check-hotfix-wrapper branch from f842590 to 3ebabf0 Compare June 16, 2026 18:11

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch 2 times, most recently from 297282e to d552a0a Compare June 19, 2026 21:05

Devinwong requested a review from xuexu6666 as a code owner June 19, 2026 21:05

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch 2 times, most recently from 66c6968 to a928ad4 Compare July 9, 2026 19:16

Devinwong force-pushed the devinwong/anc-wire-check-hotfix-wrapper branch from 673ebf0 to d5bedcf Compare July 9, 2026 19:21

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from a928ad4 to 980ed53 Compare July 9, 2026 19:22

Devinwong and others added 2 commits July 9, 2026 12:40

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from 980ed53 to 7fcf159 Compare July 9, 2026 19:50

Base automatically changed from devinwong/anc-wire-check-hotfix-wrapper to main July 10, 2026 01:05

Copilot AI review requested due to automatic review settings July 10, 2026 19:25

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from 7fcf159 to 45bf399 Compare July 10, 2026 19:25

Copilot started reviewing on behalf of Devinwong July 10, 2026 19:26 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 20:04

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from 45bf399 to 3d1a949 Compare July 10, 2026 20:04

Copilot started reviewing on behalf of Devinwong July 10, 2026 20:04 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 20:11

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from 3d1a949 to c9f66d4 Compare July 10, 2026 20:11

Copilot started reviewing on behalf of Devinwong July 10, 2026 20:12 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 21:40

Devinwong force-pushed the devinwong/anc-hotfix-env-delivery branch from c9f66d4 to 9102831 Compare July 10, 2026 21:40

Copilot started reviewing on behalf of Devinwong July 10, 2026 21:41 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Devinwong closed this Jul 10, 2026

Devinwong mentioned this pull request Jul 13, 2026

feat(anc): deliver node toggles via generic enabled_features map (2.1d) #8917

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(anc): gate check-hotfix on enable_provisioning_hotfix contract field#8717

feat(anc): gate check-hotfix on enable_provisioning_hotfix contract field#8717
Devinwong wants to merge 11 commits into
mainfrom
devinwong/anc-hotfix-env-delivery

Devinwong commented Jun 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

Devinwong commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Devinwong commented Jul 10, 2026

Uh oh!

aks-node-assistant Bot commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Devinwong commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

2.1d - gate check-hotfix on the enable_provisioning_hotfix contract field

What changed

Read channel

Supersedes the env-delivery approach

Relaxes the 2.1c env gate

Default-off and fail-open

Before / after

Stack

Tests

Uh oh!

github-actions Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Devinwong commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Devinwong commented Jul 10, 2026

Decision

Why this PR's approach didn't reach production anyway

Disposition

Uh oh!

aks-node-assistant Bot commented Jul 10, 2026

AgentBaker Linux gate detective

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Devinwong commented Jun 16, 2026 •

edited

Loading

github-actions Bot commented Jun 16, 2026 •

edited

Loading