Skip to content

CMP-4478: Fix ProfileBundle reversion on operator restart using Subscription patch#79

Open
yuumasato wants to merge 1 commit into
ComplianceAsCode:mainfrom
yuumasato:fix-profilebundle-race-subscription
Open

CMP-4478: Fix ProfileBundle reversion on operator restart using Subscription patch#79
yuumasato wants to merge 1 commit into
ComplianceAsCode:mainfrom
yuumasato:fix-profilebundle-race-subscription

Conversation

@yuumasato

@yuumasato yuumasato commented Jul 3, 2026

Copy link
Copy Markdown
Member

Problem

ProfileBundles with custom content images are reverted to default images when the compliance-operator restarts (e.g., node reboots, pod crashes).

Root cause: The operator's ensureDefaultProfileBundles() runs at startup and unconditionally patches existing ProfileBundles with the image from the RELATED_IMAGE_PROFILE env var, which defaults to the CSV's default image.

Impact: Tests using custom content images, from PRs for example, failed intermittently when the operator pod restarted during test execution.

Solution

Add RELATED_IMAGE_PROFILE to Subscription.spec.config.env before creating the Subscription, so the operator uses the custom content image from startup.

Why add to Subscription before creation:

  • Subscription.spec.config.env is the designed OLM mechanism for env var overrides
  • Simpler than create-then-patch approach
  • Subscription config persists across OLM reconciliation and operator upgrades
  • Subscription env vars take precedence over CSV defaults (documented OLM behavior)
  • Solves both initial setup AND restart/reboot cases

Changes

  • Modified ensureSubscriptionExists() to add RELATED_IMAGE_PROFILE env var to Subscription before creating it (if using custom image)
  • Added addSubscriptionEnvVar() helper to safely add env vars to spec.config.env
  • Removed ensureTestProfileBundles() (no longer needed - 74 lines removed)
  • Simplified Setup() flow: install → wait for valid ProfileBundles

Testing

Run this script to reproduce the original problem and verify the fix:

#!/bin/bash
# Reproduce the original problem: ProfileBundle with custom image gets reverted on operator restart

# Setup with default image (no custom image specified)
go test -v -timeout 15m . -run='^$' -install-operator=true

# Check ProfileBundle has default image
oc get profilebundle ocp4 -n openshift-compliance -o jsonpath='{.spec.contentImage}'
# Output: ghcr.io/complianceascode/k8scontent:latest

# Manually patch ProfileBundle to custom image (simulating old test approach)
oc patch profilebundle ocp4 -n openshift-compliance --type=merge \
  -p '{"spec":{"contentImage":"quay.io/wsato/ocp4-content:latest"}}'

oc get profilebundle ocp4 -n openshift-compliance -o jsonpath='{.spec.contentImage}'
# Output: quay.io/wsato/ocp4-content:latest

# Restart operator (simulates node reboot)
oc delete pod -l name=compliance-operator -n openshift-compliance
oc wait --for=condition=Ready pod -l name=compliance-operator -n openshift-compliance --timeout=120s

# Check ProfileBundle - OLD APPROACH: reverted to default ❌
oc get profilebundle ocp4 -n openshift-compliance -o jsonpath='{.spec.contentImage}'
# Output: ghcr.io/complianceascode/k8scontent:latest (REVERTED!)

echo ""
echo "=== Now test the FIX ==="
oc delete compliancesuite,compliancescan,profilebundle --all -n openshift-compliance --wait=true && oc delete namespace openshift-compliance && oc delete catalogsource compliance-operator -n openshift-marketplace

# Setup with custom image using NEW approach (adds to Subscription before creation)
go test -v -timeout 15m . -run='^$' -install-operator=true \
  -content-image=quay.io/wsato/ocp4-content:latest

# Check ProfileBundle has custom image
oc get profilebundle ocp4 -n openshift-compliance -o jsonpath='{.spec.contentImage}'
# Output: quay.io/wsato/ocp4-content:latest

# Restart operator (simulates node reboot)
oc delete pod -l name=compliance-operator -n openshift-compliance
oc wait --for=condition=Ready pod -l name=compliance-operator -n openshift-compliance --timeout=120s

# Check ProfileBundle - NEW APPROACH: still has custom image ✅
oc get profilebundle ocp4 -n openshift-compliance -o jsonpath='{.spec.contentImage}'
# Output: quay.io/wsato/ocp4-content:latest (PERSISTS!)

@yuumasato yuumasato changed the title Fix ProfileBundle reversion on operator restart using Subscription patch CMP-4478: Fix ProfileBundle reversion on operator restart using Subscription patch Jul 3, 2026
ProfileBundles with custom content images are reverted to default images
when the compliance-operator restarts (e.g., node reboots, pod crashes).
This happened because the operator's ensureDefaultProfileBundles() runs
at startup and unconditionally patches existing ProfileBundles with the
image from the RELATED_IMAGE_PROFILE env var, which defaults to the CSV's
default image.

Solution: Add RELATED_IMAGE_PROFILE to Subscription.spec.config.env before
creating the Subscription, so the operator uses the custom content image
from startup.

Changes:
- Modified ensureSubscriptionExists() to add RELATED_IMAGE_PROFILE env var
  to Subscription before creating it (if using custom image)
- Added addSubscriptionEnvVar() helper to safely add env vars to spec.config.env
- Removed ensureTestProfileBundles()
- Simplified Setup() flow: install → wait for valid ProfileBundles

How it works:
1. ensureSubscriptionExists() reads Subscription YAML template
2. If custom image specified, adds RELATED_IMAGE_PROFILE to spec.config.env
3. Creates Subscription with correct configuration
4. OLM merges Subscription env vars into Deployment
5. Operator starts with custom env var and creates ProfileBundles with custom image
6. Changes survive operator restarts (node reboots, pod crashes)

Why add to Subscription before creation:
- Subscription.spec.config.env is the designed OLM mechanism for env overrides
- Subscription config persists across OLM reconciliation and operator upgrades
- Subscription env vars take precedence over CSV defaults
- Solves both initial setup AND restart/reboot cases

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@yuumasato yuumasato force-pushed the fix-profilebundle-race-subscription branch from fa30229 to 02de6a2 Compare July 6, 2026 08:02
@yuumasato

Copy link
Copy Markdown
Member Author

/retest

@openshift-ci

openshift-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

@yuumasato: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ocp4-stig 02de6a2 link true /test e2e-aws-ocp4-stig
ci/prow/e2e-aws-rhcos4-moderate 02de6a2 link true /test e2e-aws-rhcos4-moderate
ci/prow/e2e-aws-ocp4-cis 02de6a2 link true /test e2e-aws-ocp4-cis
ci/prow/e2e-aws-openshift-node-compliance 02de6a2 link true /test e2e-aws-openshift-node-compliance

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@vickeybrown

Copy link
Copy Markdown

/lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants