Add CPMS e2e-presubmit test stage role for shiftstack-qa

[tusharjadhav3302]

Summary

Add a new cpms_test stage role that runs the upstream cluster-control-plane-machine-set-operator e2e-presubmit tests, and include the cpms_replace_attrs day2ops procedure (unwired, for future use).

What this PR includes

New cpms_test stage role (collection/stages/roles/cpms_test/):

• Clones the upstream operator repo at the matching release-X.Y branch
• Runs make e2e-presubmit (rolls 1 master, ~30 min) with OPENSHIFT_CI=true for JUnit output
• Post-processes results via the shared post_openshift_tests.yml (XML tag modification, HTML conversion, copy to report directory)
• Defaults to e2e-presubmit only via cpms_test_suites variable — e2e-periodic can be enabled by overriding this list in a dedicated job

New cpms_replace_attrs day2ops procedure (unwired):

• Ported from openshift-ir-plugin — patches CPMS with modified failure domains, networks, and security groups
• Validates reconciliation and restores original state
• Not wired into any job definition in this PR — will be enabled in a follow-up with a dedicated periodic job

Wiring:

• cpms_test stage added to osp_verification.yaml (runs in the existing Zuul periodic job)
• New sub-play playbooks/plays/cpms_test.yaml and import in ocp_testing.yaml

What this PR does NOT include

• e2e-periodic is not enabled by default (5h timeout, rolls all 3 masters — exceeds the 8h Zuul job budget)
• cpms_replace_attrs is not wired into any day2ops_procedures list (no cpms_replacements defaults — those create real OpenStack resources and require explicit configuration)
• Both will be enabled in a dedicated periodic job in a follow-up PR

Test plan

• Manual run on serval70 — automation executed end-to-end, JUnit XML produced for both suites
• Zuul periodic job pass with e2e-presubmit completing within the 8h budget
• Collect timing numbers for presubmit on a clean environment

[imatza-rh]

Good port from the IR-plugin. Nice improvements: OpenStack resource cleanup in always, must-gather on test failure. Nit: PR description says "branch fallback" but prepare_openshift_tests.yml has no fallback - same as all other roles, works fine.

Please run ./gate.sh (ansible-lint + pre-commit inside the shiftstack-client container) - no CI checks are configured on GitHub PRs. yamllint passes locally, but ansible-lint needs the container.

[imatza-rh]

Three concerns before merging:

1. Timing budget - The osp_verification Zuul job currently runs 7-8.6h against an 8h timeout (recent SUCCESS builds). The CPMS e2e-periodic target has a 5h ginkgo timeout and rolls all 3 masters. Adding both presubmit + periodic would push the total well past the timeout. Consider running only e2e-presubmit initially, or running CPMS tests in a separate periodic job. Has this been tested on a live environment? Duration numbers would help.

2. Suggest splitting the binary download rewrite - The tools_get_openshift_release rewrite (commits bda356e-8548bec, merged from the use-oc-adm-release-extract-for-binaries branch) is independent of the CPMS test role. It changes a shared role used by all job definitions and introduces a hard dependency on rhoso_kubeconfig for pull secret extraction - the old code just did an HTTP GET with no cluster access needed. Splitting into a separate PR would let the CPMS test merge independently and give the shared role change its own review cycle.

3. cpms_replace_attrs wiring - The procedure is added but not wired into any day2ops_procedures list (it was removed from 4.17_ovnkubernetes_ipi.yaml per previous review). The cpms_replacements variable definition was also removed in that commit. Worth adding defaults and wiring it in osp_verification.yaml if it's intended to run, or noting in the description that it'll be wired separately.

[tusharjadhav3302]

Three concerns before merging:

1. Timing budget - The osp_verification Zuul job currently runs 7-8.6h against an 8h timeout (recent SUCCESS builds). The CPMS e2e-periodic target has a 5h ginkgo timeout and rolls all 3 masters. Adding both presubmit + periodic would push the total well past the timeout. Consider running only e2e-presubmit initially, or running CPMS tests in a separate periodic job. Has this been tested on a live environment? Duration numbers would help.

2. Suggest splitting the binary download rewrite - The tools_get_openshift_release rewrite (commits bda356e-8548bec, merged from the use-oc-adm-release-extract-for-binaries branch) is independent of the CPMS test role. It changes a shared role used by all job definitions and introduces a hard dependency on rhoso_kubeconfig for pull secret extraction - the old code just did an HTTP GET with no cluster access needed. Splitting into a separate PR would let the CPMS test merge independently and give the shared role change its own review cycle.

3. cpms_replace_attrs wiring - The procedure is added but not wired into any day2ops_procedures list (it was removed from 4.17_ovnkubernetes_ipi.yaml per previous review). The cpms_replacements variable definition was also removed in that commit. Worth adding defaults and wiring it in osp_verification.yaml if it's intended to run, or noting in the description that it'll be wired separately.

@imatza-rh
For the timing budget: I'm thinking of defaulting cpms_test_suites to ["e2e-presubmit"] only in defaults/main.yml, which rolls 1 master (~30 min). Since the role defaults are what the Zuul job picks up (no variable overrides are passed from ci-framework), this ensures only presubmit runs in the periodic job. This should fit within the 8h budget.

For e2e-periodic: it's available in the code but won't run unless someone explicitly adds it to cpms_test_suites in a job definition. We can enable it in a dedicated periodic job in a follow-up once we have confirmed timing numbers.

For cpms_replace_attrs: both roll all 3 masters (~1-2h) and has no dependency on presubmit — just needs a settled cluster (Progressing=False). Should I keep the procedure file in this PR (as code-only, unwired, with a note that it'll be enabled in a follow-up)? Or would you prefer I remove it entirely and submit it as a separate PR when we're ready to wire it with a dedicated job? Let me know which is cleaner for review.

For the binary download rewrite: use-oc-adm-release-extract-for-binaries has already been merged to main via PR #15. I've rebased this branch on top of current main and force-pushed — those commits are no longer part of this PR's diff.

So this PR would ship:

• e2e-presubmit wired into osp_verification.yaml via role default (runs in the existing Zuul periodic job)
• e2e-periodic in the code but not enabled by default
• cpms_replace_attrs — pending your preference above

Does this approach work?

[imatza-rh]

Presubmit-only default sounds right. Worth noting main.yml still hardcodes both targets - might want to add the cpms_test_suites gating before merging.

Keeping cpms_replace_attrs unwired is fine. Probably better without defaults for cpms_replacements since those create real OpenStack resources.

Binary rewrite confirmed out of this diff.

[tusharjadhav3302]

Update: CPMS e2e-presubmit validated end-to-end on serval71

Manual run completed successfully — 76 tests, 0 failures, 0 errors (~4.5h on constrained environment). All artifacts generated correctly (JUnit XML, HTML, Polarion report directory).

Latest commit (8c04179) reverts master RAM to 16GB and switches to soft-anti-affinity — required to fit within the virtualized compute capacity on serval70/71 (3 computes × 46.5GB each). Without this, Nova fails to schedule the 3rd master.

Zuul end-to-end is blocked on the CAPO webhook fix (PR openshift/cluster-api-provider-openstack#423) landing in a nightly. Once that merges and 4-stable picks it up, a retrigger of the testproject MR should give us a clean automated run.

Timing note: e2e-presubmit took ~4.5h on this hardware. On serval70/71 with the osp_verification job (7-8h baseline), this would exceed the 8h Zuul timeout.

@imatza-rh Should have have a separate periodic job for cpms e2e-presubmit as well and not run it during the periodic ci run, given the timeout issue it could have?

[tusharjadhav3302]

@imatza-rh for now I have removed cpms_test from osp_verification.yaml stages. The e2e-presubmit takes ~4.5h on our virtualized environment which would exceed the 8h Zuul job timeout when combined with the other stages. The role code, sub-play, and ocp_testing.yaml wiring remain in place — CPMS tests can still be run manually or enabled in a dedicated periodic job (follow-up PR).

[ekuris-redhat]

LGTM , Also tested and it worked