:seedling: OPRUN-3956: Add experimental e2e tests by tmshort · Pull Request #2064 · operator-framework/operator-controller

tmshort · 2025-06-29T01:13:20Z

Fix#2016

This adds manifests/experimental-e2e.yaml, and uses it to run the experimental-e2e.

The experimental-e2e is run via make test-experimental-e2e and a CI job is added to run it.

A "no-op" experimental-e2e test has been added.

This is part of the feature-gated API functionality.

Description

Reviewer Checklist

API Go Documentation
Tests: Unit Tests (and E2E Tests, if appropriate)
Comprehensive Commit Messages
Links to related GitHub Issue(s)

This adds `manifests/experimental-e2e.yaml`, and uses it to run the experimental-e2e. The experimental-e2e is run via `make test-experimental-e2e` and a CI job is added to run it. A "no-op" experimental-e2e test has been added. This is part of the feature-gated API functionality. Signed-off-by: Todd Short <tshort@redhat.com>

netlify · 2025-06-29T01:16:47Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`1a42acf`
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/6860bf64aae3650008143766
😎 Deploy Preview	https://deploy-preview-2064--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

/lgtm cancel

camilamacedo86 · 2025-06-29T01:47:43Z

/approved cancel
sorry mistake

codecov · 2025-06-29T01:55:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.89%. Comparing base (f050355) to head (1a42acf).
Report is 1 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2064 +/- ##
==========================================
+ Coverage 73.82% 73.89% +0.06% 
==========================================
Files 81 81 Lines 7365 7365 ==========================================
+ Hits 5437 5442 +5 + Misses 1588 1585 -3 + Partials 340 338 -2

Flag	Coverage Δ
e2e	`44.10% <ø> (+0.04%)`	⬆️
experimental-e2e	`20.14% <ø> (?)`
unit	`60.24% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

camilamacedo86 · 2025-06-29T02:02:10Z

@@ -0,0 +1,1984 @@
+apiVersion: v1


Quick question — I noticed that before we didn’t commit the install.yaml file.
Are we planning to commit both files now? Just curious what’s changed and why we’re doing that now, if so.

What install.yaml file are you referring to?
This does not change the existing release mechanism; those files are still created via the quickstart Makefile target.

But we generated it when we run target, why do we need to commit?
Can we not add those to gitignore?

I assume you're talking about manifest/experimental-e2e.yaml?
We are now checking in all the manifests for standard, experimental and e2e (look in the manifest directory). This is done on purpose (it's mentioned in the RFC, and is a continuation of #2025). It allows people (and reviewers) to easily identify the effects of any changes to the kustomize config, which can usually be hard to figure out.
Having these files committed made it very easy to do all the earlier kustomizae changes (#2063, #2061, #2060, #2059, #2057, #2056, #2055 and #2025), because I could easily see that there were no changes to the resulting manifests.

tmshort · 2025-06-29T03:35:16Z

/approve

openshift-ci · 2025-06-29T03:35:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [camilamacedo86,tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Todd Short <tshort@redhat.com>

camilamacedo86 · 2025-06-29T07:24:29Z

 codecov:
 notify:
- after_n_builds: 2
+ after_n_builds: 3


Is this change related ?

Yes, as the experimental-e2e pushes code coverage. There are three builds that do code-coverage:
e2e
unit
experimental-e2e
This is supposed to make codecov wait until it gets three build pushes (the above).
See: https://app.codecov.io/gh/operator-framework/operator-controller/pull/2064/flags

camilamacedo86 · 2025-06-29T09:27:35Z

+ go-version-file: go.mod
+
+ - name: Run e2e tests
+ run: ARTIFACT_PATH=/tmp/artifacts make test-experimental-e2e


For curiosity, what was the issue that was not finding the path?

The CollectTestArtifacts() function was not being called; one of the things it does is create the /tmp/artifacts directory.

tmshort · 2025-06-29T11:47:49Z

+
+func TestNoop(t *testing.T) {
+	t.Log("Running experimental-e2e tests")
+	defer utils.CollectTestArtifacts(t, artifactName, c, cfg)


@camilamacedo86 as a side effect, this creates the /tmp/artifacts directory.

camilamacedo86 · 2025-06-29T22:20:24Z

/lgtm

Great work 🎉

Thank you for all explanations

tmshort requested a review from a team as a code owner June 29, 2025 01:13

openshift-ciBot requested review from gallettilance and oceanc80 June 29, 2025 01:13

This comment was marked as outdated.
Sign in to view

openshift-ciBot assigned camilamacedo86 Jun 29, 2025

openshift-ciBot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jun 29, 2025

camilamacedo86 removed lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jun 29, 2025

openshift-ciBot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 29, 2025

camilamacedo86 removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 29, 2025

operator-framework deleted a comment from openshift-ciBotJun 29, 2025

camilamacedo86 self-requested a review June 29, 2025 01:47

camilamacedo86 reviewed Jun 29, 2025
View reviewed changes

openshift-ciBot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 29, 2025

tmshort added 2 commits June 29, 2025 00:07

fixup! Add experimental e2e tests
8eff28d
Signed-off-by: Todd Short <tshort@redhat.com>

fixup! Add experimental e2e tests
1a42acf
Signed-off-by: Todd Short <tshort@redhat.com>

camilamacedo86 reviewed Jun 29, 2025
View reviewed changes

tmshort commented Jun 29, 2025
View reviewed changes

openshift-ciBot added the lgtm Indicates that a PR is ready to be merged. label Jun 29, 2025

openshift-merge-botBot merged commit 6465e63 into operator-framework:mainJun 29, 2025

tmshort deleted the exp-e2e branch June 30, 2025 03:12

tmshort mentioned this pull request Jun 30, 2025
[epic] Add capability to run feature gated e2e tests #1890
Closed

tmshort changed the title ~~🌱 Add experimental e2e tests~~🌱 OPRUN-3956: Add experimental e2e testsJun 30, 2025

Uh oh!

Conversation

tmshort commented Jun 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer Checklist

Uh oh!

netlifyBot commented Jun 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for olmv1 ready!

Uh oh!

This comment was marked as outdated.

Uh oh!

camilamacedo86 commented Jun 29, 2025

Uh oh!

codecovBot commented Jun 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

camilamacedo86Jun 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshortJun 29, 2025

Choose a reason for hiding this comment

Uh oh!

camilamacedo86Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

tmshortJun 29, 2025

Choose a reason for hiding this comment

Uh oh!

tmshort commented Jun 29, 2025

Uh oh!

openshift-ciBot commented Jun 29, 2025

Uh oh!

camilamacedo86Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

tmshortJun 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

camilamacedo86Jun 29, 2025

Choose a reason for hiding this comment

Uh oh!

tmshortJun 29, 2025

Choose a reason for hiding this comment

Uh oh!

tmshortJun 29, 2025

Choose a reason for hiding this comment

Uh oh!

camilamacedo86 commented Jun 29, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tmshort commented Jun 29, 2025•
edited
Loading

netlifyBot commented Jun 29, 2025•
edited
Loading

codecovBot commented Jun 29, 2025•
edited
Loading

camilamacedo86Jun 29, 2025•
edited
Loading

tmshortJun 29, 2025•
edited
Loading

camilamacedo86 commented Jun 29, 2025•
edited
Loading