Adapt openstack_test stage to OTE (drop legacy make openstack-tests)

[tusharjadhav3302]

Summary

• After openshift/openstack-test#303 (OSASINFRA-4375, merged 2026-08-04) dropped the legacy make openstack-tests target on release-4.20+, shiftstack-qa’s openstack_test stage fails immediately with No rule to make target 'openstack-tests'.
• For discovered_openshift_release >= 4.20, build with make extension, install bin/openstack-test-tests-ext, list via list --suite openstack-test/all -o names, and run via run-suite (or run-test + stdin when allow/block lists are set).
• Keep the legacy make openstack-tests / run openshift/openstack path for < 4.20 so existing 4.16/4.17 job defs still work.

CLI / behavior trade-offs

Legacy	OTE (>= 4.20)
make openstack-tests	make extension
run openshift/openstack --dry-run	list --suite … -o names
run … -f list --junit-dir	Unfiltered: run-suite … --junit-path=junit_e2e_openstack.xml; filtered: run-test via stdin (OTE has no -f / --junit-path on run-test) + synthesize junit XML from JSON for post_openshift_tests
Log lines passed: / failed:	JSON "result" field; rescue counters updated accordingly

Env vars unchanged: OS_CLOUD, KUBECONFIG, RHOSO_KUBECONFIG, SHIFTSTACK_PASS_FILE.

Test plan

• On shiftstackclient (serval70, guest OCP 4.23.0-0.nightly-2026-08-05-200523): clone release-4.23, confirm make openstack-tests fails and make extension + list --suite openstack-test/all -o names succeeds.
• Run one real test via OTE run-test against the live guest cluster (bz_2022627) — PASSED ("result": "passed"). See PR comment for full evidence.
• Full Ansible stage-only run against an existing cluster (stages: [openstack_test]) — blocked in this env by undefined hypervisor (cifmw-dnsmasq delegate); OTE CLI path above still validates the migration.
• Optional: exercise allowlist path via lb_tests (filtered run-test + synthesized junit).
• Note: verification-only jobs that omit openstack_test will not exercise this change.

Links

• Upstream: OSASINFRA-4375: Drop Legacy openshift-test entry point: releases 4.23+ openshift/openstack-test#303 (backports: #306, #307, #308)
• Jira: https://redhat.atlassian.net/browse/OSASINFRA-4375

[tusharjadhav3302]

Manual validation evidence (serval70 / shiftstackclient)

Env

• Host: serval70 → oc rsh -n openstack shiftstackclient-shiftstack
• Guest OCP: 4.23.0-0.nightly-2026-08-05-200523 (AVAILABLE)
• Clone: ~/artifacts/openstack-test-repro @ release-4.23

1. Legacy path broken (root cause reproduced)

make openstack-tests
make: *** No rule to make target 'openstack-tests'. Stop.

2. OTE build succeeds

$ make extension
Building OTE test extension binary...
GOTOOLCHAIN=auto GOSUMDB=sum.golang.org go build -mod=vendor -o bin/openstack-test-tests-ext ./cmd/extension
go: downloading go1.23.4 (linux/amd64)
Extension binary built: bin/openstack-test-tests-ext
$ test -x bin/openstack-test-tests-ext && echo "binary OK"
binary OK

3. List suite (replaces --dry-run)

$ ./bin/openstack-test-tests-ext list --suite openstack-test/all -o names | head
[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2022627: Machine should report all openstack instance addresses
[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2073398: [Serial] MachineSet scale-in does not leak OpenStack ports
...

4. Run one real test against the live cluster — PASSED

$ ./bin/openstack-test-tests-ext run-test \
'[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2022627: Machine should report all openstack instance addresses'
Will run 1 of 1 specs
...
Ran 1 of 1 Specs in 6.713 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[
{
"name": "[OTP][sig-installer][Suite:openshift/openstack] Bugfix bz_2022627: Machine should report all openstack instance addresses",
"result": "passed",
"duration": 6713,
...
}
]

Notes

• This validates the OTE build/list/run path that this PR wires into the openstack_test role for discovered_openshift_release >= 4.20.
• A full ansible-navigator stage-only run was attempted but failed early on hypervisor undefined (cifmw-dnsmasq delegate_to); ~/artifacts/openstack-test-results/ stayed empty because the suite never started. Re-run with --extra-vars '{"hypervisor":"serval70",...}' if full role wiring needs a check later.

[danchild]

Thanks for catching this @tusharjadhav3302

For documentation purposes, the behavior you're seeing is due to these PR's as well:

openshift/openstack-test#308
openshift/openstack-test#307
openshift/openstack-test#306

[tusharjadhav3302]

@danchild Done.

[imatza-rh]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from imatza-rh. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tusharjadhav3302]

Follow-up: OTE false-failure counting (pushed)

run-suite was marking all outer JSON results as `failed` due to `Deserializaion Error: invalid character 'I'` (klog on stderr). That made rescue counters 0/0 and hard-failed with "no tests run", and junit showed 51 failures.

Fix (this PR): `files/ote_resolve_results.py` resolves true outcomes from nested STDOUT JSON / ginkgo `SUCCESS!`|`FAIL!`, used to:

• count passed/failed in rescue
• rewrite `junit_e2e_openstack.xml` after `run-suite` / filtered `run-test`

Validated against the 4.23 TP log: 24 passed / 20 skipped / 7 real failed (not 51).

[tusharjadhav3302]

Pushed register fix: skipped legacy count tasks no longer wipe OTE resolved pass/fail counts (5781579).

[tusharjadhav3302]

Manual validation (serval71 / live RHOSO + guest OCP 4.23)

Validated on shiftstackclient against an existing cluster (no full TP recheck yet). Collection installed from this PR / combine #34 tip including 5781579 (register-count fix).

What we validated

1. OTE path — make extension, run-suite openstack-test/all, junit rewrite via files/ote_resolve_results.py
2. False “51 failed” / “no tests where run!” — outer OTE still reports Error: 51 tests failed (klog deserialize noise), but rescue now resolves real counts and does not hard-fail
3. Register wipe — distinct OTE vs legacy registers + normalize set_fact; skipped legacy count tasks no longer clear OTE counts
4. Stage behavior — suite failure → UNSTABLE → must-gather + post (XML/HTML) → dnsmasq restore → play ends failed=0, rescued=1

Evidence (manual stages: ["openstack_test"])

• Hypervisor dnsmasq delegate to serval71.lab.eng.tlv2.redhat.com OK
• Resolver offline on prior log: passed=24 failed=7 skipped=20
• Live suite run printed: openstack-test counts: passed=20, failed=11
• Fail task “no tests where run!” was skipped
• PLAY RECAP: ok=74 … failed=0 … rescued=1

The ~11 real Amphora/PVC (etc.) failures are suite/product noise on this env; they are correctly treated as UNSTABLE, not as an automation hard fail.

Not claimed

• Full osp_verification green / zero openstack-test failures
• TP !2584 recheck (pending when ready; pin is DNM: TP combine #32 + #33 #34)

[tusharjadhav3302]

Manual validation (serval71 / live RHOSO + guest OCP 4.23)

Validated on shiftstackclient against an existing cluster (no full TP recheck yet). Collection installed from this PR / combine #34 tip including 5781579 (register-count fix).

What we validated

1. OTE path — make extension, run-suite openstack-test/all, junit rewrite via files/ote_resolve_results.py
2. False “51 failed” / “no tests where run!” — outer OTE still reports Error: 51 tests failed (klog deserialize noise), but rescue now resolves real counts and does not hard-fail
3. Register wipe — distinct OTE vs legacy registers + normalize set_fact; skipped legacy count tasks no longer clear OTE counts
4. Stage behavior — suite failure → UNSTABLE → must-gather + post (XML/HTML) → dnsmasq restore → play ends failed=0, rescued=1

Evidence (manual stages: ["openstack_test"])

• Hypervisor dnsmasq delegate to serval71.lab.eng.tlv2.redhat.com OK
• Resolver offline on prior log: passed=24 failed=7 skipped=20
• Live suite run printed: openstack-test counts: passed=20, failed=11
• Fail task “no tests where run!” was skipped
• PLAY RECAP: ok=74 … failed=0 … rescued=1

The ~11 real Amphora/PVC (etc.) failures are suite/product noise on this env; they are correctly treated as UNSTABLE, not as an automation hard fail.

Not claimed

• Full osp_verification green / zero openstack-test failures
• TP !2584 recheck (pending when ready; pin is DNM: TP combine #32 + #33 #34)

[tusharjadhav3302]

Follow-up: lb_tests / egressip allowlist vs OTE list

Manual `lb_tests` on serval71 failed with allowlist `".[lb]."` not found in `openstack_tests.txt`.

Cause: OTE `list -o names` writes unquoted names (and a leading klog `I0810 ...` line on stdout). Allowlist conversion still emits quoted patterns meant for legacy dry-run, so `filter_tests_list` fullmatch failed. `sed` on the artifact did not help because the playbook regenerates `allowlist.txt` each run.

Fix (this PR):

• `filter_tests_list`: strip optional surrounding quotes; ignore klog lines
• OTE list task: drop klog from captured names (`grep -v` / stderr redirect)

Re-fetch/reinstall collection before re-running `lb_tests` / `egressip_tests`.

[imatza-rh]

/lgtm

[tusharjadhav3302]

Manual validation complete (serval71) — still targeting Zuul green

Validated on shiftstackclient against live RHOSO + guest OCP 4.23 with collection from combine #34 (includes this PR through allowlist/OTE list fix 3259e02 + prior resolve/register commits).

Stage	Result
openstack_test	UNSTABLE with resolved counts (e.g. passed=20 failed=11); nono tests where run!; must-gather/post OK
lb_tests	Allowlist/OTE list OK (22 [lb] tests); run-test UNSTABLE; resolver e.g. passed=3 failed=4 skipped=15
egressip_tests	Pass — PLAY RECAP failed=0 rescued=0; post XML/HTML OK

Fixes covered in this PR that were exercised: OTE migration, result resolve, register-count wipe, allowlist quote/klog matching.

We will still run Zuul via testproject !2584 pinned to #34 for an official green before merge. Manual work was to de-risk; not a substitute for the check job.

[tusharjadhav3302]

Zuul green (ocp423 TP)

Official check passed with throwaway combine #34 (includes this PR — OTE migration, result resolve, register-count, allowlist/klog fixes):

• Job: shiftstack-rhel9-rhoso18.0-ocp423-test (serval71)
• Pin: refs/pull/34/head
• Zuul: https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/zuul/t/components-integration/build/8b4280d907be4e39ab4212e1798777f7
• TP MR: https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/2584 (Zuul-CI::ok)

No No rule to make target 'openstack-tests' / no tests where run! hard-fail on the check job. Ready to merge from a CI perspective once reviews are in; close DNM #34 after landing.