SPLAT-2637: Promoting gate AWSServiceLBNetworkSecurityGroup to GA by mtulio · Pull Request #2717 · openshift/api

mtulio · 2026-02-18T20:23:54Z

PR blocked by:

tests SPLAT-2220: tests/ote: introduce openshift-only e2e to validate feature by feature gate cloud-provider-aws#126
EP SPLAT-2137: Support Security Group on NLB for Default router on AWS enhancements#1802

Is blocking GA on Hypershift: TPNU ref openshift/hypershift#7460

Testscloud-provider-aws-e2e-openshift have been added to this feature, as long with the cloud-provider-aws-e2e exercising CCM though OTE.

openshift-ci-robot · 2026-02-18T20:23:57Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-02-18T20:23:59Z

@mtulio: This pull request references SPLAT-2637 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

PR blocked by:
tests SPLAT-2220: tests/ote: introduce openshift-only e2e to validate feature by feature gate cloud-provider-aws#126
EP SPLAT-2137: Support Security Group on NLB for Default router on AWS enhancements#1802
Is blocking GA on Hypershift: TPNU ref openshift/hypershift#7460
Feature OCPSTRAT-1553 / SPLAT-2553

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-02-18T20:24:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 0dc5047f-aadc-4d37-be11-69b04fbcda95

📥 Commits

Reviewing files that changed from the base of the PR and between e188bac and 98fa4f3.

📒 Files selected for processing (4)

features.md
features/features.go
payload-manifests/featuregates/featureGate-4-10-SelfManagedHA-Default.yaml
payload-manifests/featuregates/featureGate-4-10-SelfManagedHA-OKD.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

features.md

📝 Walkthrough

Walkthrough

The AWSServiceLBNetworkSecurityGroup feature gate was refactored to support cluster-profile-aware configuration. The feature gate declaration was updated to include two separate enable branches: one targeting SelfManaged profiles and another targeting Hypershift profiles. Correspondingly, AWSServiceLBNetworkSecurityGroup was transitioned from disabled to enabled status in multiple feature gate manifests, including the Default and OKD configurations. Documentation in features.md was updated to reflect these configuration changes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: promoting the AWSServiceLBNetworkSecurityGroup feature gate to GA, which aligns with all file modifications.
Description check	✅ Passed	The description is related to the changeset, providing context about the feature promotion, blocked dependencies, test coverage, and verification status.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Stable And Deterministic Test Names	✅ Passed	PR does not modify any Ginkgo test files; changes are limited to feature gate configuration, documentation, and YAML manifests.
Test Structure And Quality	✅ Passed	This PR does not introduce or modify any Ginkgo test code. The only test file uses the standard Go testing framework, not Ginkgo patterns.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.5.0)

Error: build linters: unable to load custom analyzer "kubeapilinter": tools/_output/bin/kube-api-linter.so, plugin: not implemented
The command is terminated due to an error: build linters: unable to load custom analyzer "kubeapilinter": tools/_output/bin/kube-api-linter.so, plugin: not implemented

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

mtulio · 2026-02-18T20:24:27Z

/test ?

qodo-code-review · 2026-02-18T20:24:32Z

PR-Agent: could not fine a component named ? in a supported language in this PR.

openshift-ci · 2026-02-18T20:25:15Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2026-02-18T20:25:53Z

Hello @mtulio! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

mtulio · 2026-02-23T22:41:48Z

/test all

qodo-code-review · 2026-02-23T22:41:51Z

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan

ⓘ Paying users. Check that your Qodo account is linked with this Git user account

mtulio · 2026-02-24T00:06:39Z

/test verify-feature-promotion
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-techpreview

qodo-code-review · 2026-02-24T00:06:43Z

ⓘ Your monthly quota for Qodo has expired. Upgrade your plan

ⓘ Paying users. Check that your Qodo account is linked with this Git user account

openshift-ci-robot · 2026-02-24T01:34:54Z

@mtulio: This pull request references SPLAT-2637 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

PR blocked by:
tests SPLAT-2220: tests/ote: introduce openshift-only e2e to validate feature by feature gate cloud-provider-aws#126
EP SPLAT-2137: Support Security Group on NLB for Default router on AWS enhancements#1802
Is blocking GA on Hypershift: TPNU ref openshift/hypershift#7460
Feature OCPSTRAT-1553 / SPLAT-2553
Testscloud-provider-aws-e2e-openshift have been added to this feature, as long with the cloud-provider-aws-e2e exercising CCM though OTE.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

mtulio · 2026-02-24T13:15:00Z

e2e-aws-ovn-hypershift-conformance is legitm, I just created the skip openshift/release#75211 while we are waiting for the feature PR in hypershift openshift/hypershift#7460

mtulio · 2026-02-24T14:28:58Z

verify-feature-promotion is also failing locally.

$ make verify-feature-promotion hack/verify-promoted-features-pass-tests.sh
comparing against master
...
Query sippy for all test run results for pattern "FeatureGate:AWSServiceLBNetworkSecurityGroup]" on variant main.JobVariant{Cloud:"aws", Architecture:"amd64", Topology:"single", NetworkStack:""}
Querying sippy release 4.22 for test run results
F0224 11:24:48.988679 2287144 root.go:64] Error running codegen: Get "https://sippy.dptools.openshift.org/api/tests?filter=%7....erator%22%3A%22and%22%7D&period=default&release=4.22":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
make: *** [Makefile:95: verify-feature-promotion] Error 255

mtulio · 2026-03-03T16:59:58Z

/test e2e-aws-ovn-techpreview

qodo-code-review · 2026-03-03T17:00:05Z

PR-Agent: could not fine a component named e2e-aws-ovn-techpreview in a supported language in this PR.

mtulio · 2026-03-03T17:03:57Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ccm-techpreview

openshift-ci · 2026-03-03T17:04:43Z

@mtulio: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ccm-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f8192380-1722-11f1-9762-ff925bc266de-0

mtulio · 2026-03-05T14:51:08Z

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ccm-techpreview

openshift-ci · 2026-03-05T14:51:12Z

@mtulio: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ccm-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/beb86480-18a2-11f1-8079-b2880f72d5dc-0

mtulio · 2026-03-05T14:56:07Z

/test all

This change promotes the feature gate AWSServiceLBNetworkSecurityGroup from TechPreviewNoUpgrade to Default feature set on self-managed. Also the feature AWSServiceLBNetworkSecurityGroup is kept TP and DP on Hypershift while the implementation is finished on Hypershift. The tests covering this feature is added by openshift/cloud-provider-aws#129.

mtulio · 2026-03-09T15:03:01Z

/pipeline required

openshift-ci-robot · 2026-03-09T15:03:07Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-techpreview
/test e2e-aws-serial-1of2
/test e2e-aws-serial-2of2
/test e2e-aws-serial-techpreview-1of2
/test e2e-aws-serial-techpreview-2of2
/test e2e-azure
/test e2e-gcp
/test e2e-upgrade
/test e2e-upgrade-out-of-change
/test minor-e2e-upgrade-minor

mtulio · 2026-03-09T17:57:23Z

e2e-aws-ovn-hypershift: Issue is unrelated with the change:

/test e2e-aws-ovn-hypershift

e2e-aws-ovn-techpreview: unrelated issue, all six tests cloud-provider-aws-e2e-openshift are passing:

/test e2e-aws-ovn-techpreview

/test e2e-aws-serial-1of2

mtulio · 2026-03-09T18:38:45Z

e2e-azure failed on destroy, failing to delete RG. Unrelated to this change, since it's not required and feature not related to the platform, I will skip the re-run.

mtulio · 2026-03-09T23:23:43Z

e2e-aws-serial-1of2 and e2e-azure are failing for unrelated reasons:

e2e-aws-serial-1of2: monitor test on disruption/metrics-api
e2e-azure: destroy flow

mtulio · 2026-03-09T23:24:04Z

Hi Joel, would you mind reviewing it please?
/assign @JoelSpeed

JoelSpeed · 2026-03-10T10:32:45Z

/lgtm
/verified by CI

openshift-ci-robot · 2026-03-10T10:32:59Z

@JoelSpeed: This PR has been marked as verified by CI.

Details

In response to this:

/lgtm
/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-03-10T10:33:15Z

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

openshift-ci · 2026-03-10T10:33:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcpowermac, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mtulio · 2026-03-10T11:51:02Z

/hold cancel

mtulio · 2026-03-10T13:57:37Z

Investigation of job e2e-aws-serial-1of2 failure by Claude 🤖

PR Analysis by Claude 🤖

Analyzing test failures for job `pull-ci-openshift-api-master-e2e-aws-serial-1of2` (ID: `2031076482759004160`)

Failed Tests

[Monitor:metrics-api-availability][sig-instrumentation] disruption/metrics-api connection/new should be available throughout the test
[Monitor:metrics-api-availability][sig-instrumentation] disruption/metrics-api connection/reused should be available throughout the test

Both tests failed with 31 seconds of disruption against an allowed threshold of 6 seconds, reporting 503 Service Unavailable errors starting at Mar 09 19:52:11.

Root Cause

The failures are caused by a test framework interaction between two concurrent tests:

Disruptive Test (running in foreground):

[sig-node] NoExecuteTaintManager Multiple Pods [Serial]
only evicts pods without tolerations from tainted nodes

Started: ~19:51:08
Completed: 19:53:14
Duration: 2m6s
Purpose: Validates pod eviction behavior when nodes are tainted

Monitor Test (running in background):

[Monitor:metrics-api-availability][sig-instrumentation]
disruption/metrics-api connection/new|reused

Runs continuously throughout the test suite
Measures API availability and allowed disruption windows

Event Timeline

19:51:08 NoExecuteTaintManager test begins
├─ Adds NoExecute taints to worker nodes
│
19:52:09 TaintManager triggers mass pod eviction
├─ 23 pods evicted from 7 namespaces
├─ Including: thanos-querier (serves metrics-api)
├─ prometheus-k8s-1
├─ metrics-server instances
└─ alertmanager, kube-state-metrics, etc.
│
19:52:10 Replacement pods created and scheduled
├─ thanos-querier-f9488dbcd-f4pzt
└─ metrics-server, kube-state-metrics (new instances)
│
19:52:11 metrics-api becomes unavailable
├─ Old thanos-querier terminating
├─ New thanos-querier starting (pulling image, initializing)
└─ Monitor tests detect: 503 Service Unavailable
│
19:52:42 New monitoring pods become Ready
├─ metrics-api service restored
│
19:53:14 NoExecuteTaintManager test completes
└─ Taints removed from nodes

Technical Details

Event Breakdown (19:52:09 - 19:52:30):

TaintManagerEviction: 23 events
Killing: 46 events
SuccessfulCreate: 15 events
Started: 12 events

Affected Namespaces:

openshift-monitoring (11 pods)
openshift-cluster-storage-operator
openshift-dns
openshift-image-registry
openshift-ingress-canary
openshift-insights
openshift-network-console

Critical Impact:
The thanos-querier pod, which serves the metrics-api endpoint, was evicted and rescheduled. During the ~30-second gap between the old pod terminating and the new pod becoming Ready, the metrics-api endpoint returned 503 errors.

Disruption Calculation:

Historical P99: 0s (new connections) / 680ms (reused connections)
Rounded up: 1s
Grace period: 5s
Total allowed: 6s
Actual disruption: 31s
Result: Failed by 25s

Test Framework Issue

This represents a known conflict in the test suite design:

Serial disruptive tests (like NoExecuteTaintManager) intentionally cause node-level disruptions to validate cluster behavior
Background monitor tests continuously measure API availability across all operations
Result: Legitimate test-induced disruptions are flagged as failures by the monitors

The NoExecuteTaintManager test is working correctly - it successfully validated that pods without tolerations are evicted when nodes receive NoExecute taints. The monitoring stack pods (thanos-querier, prometheus, etc.) correctly do not tolerate arbitrary test taints and were properly evicted.

References

Job: pull-ci-openshift-api-master-e2e-aws-serial-1of2/2031076482759004160
JUnit: e2e-monitor-tests__20260309-193528.xml
Events: events.json
Test history: Job History

Analysis Artifacts

Full analysis available in the investigation artifacts:

Event correlation and pod eviction details
Node status verification (all nodes Ready, no persistent taints)
Monitoring pod lifecycle during the disruption window

mtulio · 2026-03-10T13:59:34Z

/retest-required

mtulio · 2026-03-10T19:28:23Z

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

^ verify-feature-promotion while calling to Sippy.

/test verify-feature-promotion

openshift-ci · 2026-03-10T19:34:17Z

@mtulio: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Feb 18, 2026

openshift-ciBot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 18, 2026

openshift-ciBot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 18, 2026

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2026

mtulio force-pushed the SPLAT-2637-ga-aws-ccm-nlb-sg branch from d29a368 to 8a2811fCompare February 23, 2026 22:41

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2026

mtulio force-pushed the SPLAT-2637-ga-aws-ccm-nlb-sg branch from 8a2811f to a02121cCompare March 5, 2026 14:50

openshift-ciBot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 5, 2026

mtulio force-pushed the SPLAT-2637-ga-aws-ccm-nlb-sg branch from e188bac to 98fa4f3Compare March 9, 2026 14:10

openshift-ciBot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 9, 2026

openshift-ciBot assigned JoelSpeed Mar 9, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Mar 10, 2026

openshift-ciBot added the lgtm Indicates that a PR is ready to be merged. label Mar 10, 2026

openshift-ciBot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 10, 2026

openshift-ciBot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 10, 2026

openshift-merge-botBot merged commit 1f950f7 into openshift:masterMar 10, 2026
28 checks passed

mtulio deleted the SPLAT-2637-ga-aws-ccm-nlb-sg branch March 12, 2026 21:13

xueqzhan mentioned this pull request Mar 16, 2026
TRT-2518: Add variant OS and JobTier to promotion check #2763
Merged

coderabbitaiBot mentioned this pull request May 12, 2026
SPLAT-2588: Promoting AWSServiceLBNetworkSecurityGroup on hypershift #2838
Merged

gcs278 mentioned this pull request Jun 25, 2026
NE-2386: IngressController API for AWS NLB security group selection openshift/enhancements#2037
Merged

gcs278 mentioned this pull request Jul 8, 2026
NE-2387: IngressController AWS NLB Security Group Selection #2914
Open

Uh oh!

Conversation

mtulio commented Feb 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Feb 18, 2026

Uh oh!

openshift-ci-robot commented Feb 18, 2026• edited by openshift-ci BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitaiBot commented Feb 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

mtulio commented Feb 18, 2026

Uh oh!

qodo-code-reviewBot commented Feb 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ciBot commented Feb 18, 2026

Uh oh!

openshift-ciBot commented Feb 18, 2026

Uh oh!

mtulio commented Feb 23, 2026

Uh oh!

qodo-code-reviewBot commented Feb 23, 2026

Uh oh!

mtulio commented Feb 24, 2026

Uh oh!

qodo-code-reviewBot commented Feb 24, 2026

Uh oh!

openshift-ci-robot commented Feb 24, 2026• edited by openshift-ci BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtulio commented Feb 24, 2026

Uh oh!

mtulio commented Feb 24, 2026

Uh oh!

mtulio commented Mar 3, 2026

Uh oh!

qodo-code-reviewBot commented Mar 3, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtulio commented Mar 3, 2026

Uh oh!

openshift-ciBot commented Mar 3, 2026

Uh oh!

mtulio commented Mar 5, 2026

Uh oh!

openshift-ciBot commented Mar 5, 2026

Uh oh!

mtulio commented Mar 5, 2026

Uh oh!

mtulio commented Mar 9, 2026

Uh oh!

openshift-ci-robot commented Mar 9, 2026

Uh oh!

mtulio commented Mar 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtulio commented Mar 9, 2026

Uh oh!

mtulio commented Mar 9, 2026

Uh oh!

mtulio commented Mar 9, 2026

Uh oh!

JoelSpeed commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

openshift-ci-robot commented Mar 10, 2026

Uh oh!

openshift-ciBot commented Mar 10, 2026

Uh oh!

mtulio commented Mar 10, 2026

Uh oh!

mtulio commented Mar 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Analysis by Claude 🤖

Analyzing test failures for job pull-ci-openshift-api-master-e2e-aws-serial-1of2 (ID: 2031076482759004160)

mtulio commented Feb 18, 2026•
edited
Loading

openshift-ci-robot commented Feb 18, 2026•
edited by openshift-ci Bot
Loading

coderabbitaiBot commented Feb 18, 2026•
edited
Loading

qodo-code-reviewBot commented Feb 18, 2026•
edited
Loading

openshift-ci-robot commented Feb 24, 2026•
edited by openshift-ci Bot
Loading

qodo-code-reviewBot commented Mar 3, 2026•
edited
Loading

mtulio commented Mar 9, 2026•
edited
Loading

mtulio commented Mar 10, 2026•
edited
Loading

Analyzing test failures for job `pull-ci-openshift-api-master-e2e-aws-serial-1of2` (ID: `2031076482759004160`)