🌱 Migrate e2e prometheus from custom chart to kube-prometheus-stack

[pedjak]

Description

Replace the hand-rolled prometheus-operator install script (hack/test/install-prometheus.sh) and custom Helm chart (helm/prometheus/) with the official kube-prometheus-stack community chart (v86.2.2), installed from OCI registry.

Motivation: The custom chart required maintaining 12 templates, a separate install script, kustomize tooling, and conftest policies. The official chart provides the same functionality with less maintenance burden.

What changed

• Prometheus deployment: Single helm install from oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack replaces the kustomize-based operator install + custom chart template pipeline
• Unused components disabled: grafana, alertmanager, node-exporter, kube-state-metrics, default rules, admission webhooks, operator TLS
• Kubelet ServiceMonitor: Uses chart's built-in kubelet support with kubelet: false (only cAdvisor), honorTimestamps: false, trackTimestampsStaleness: false, and cAdvisorRelabelings: [] to match old behavior
• Custom ServiceMonitors: operator-controller and catalogd added via additionalServiceMonitors using bearerTokenFile (projected SA token) instead of the legacy prometheus-metrics-token Secret
• PrometheusRules: Split into controller-panic-alerts and controller-resource-alerts map keys so the experimental override only replaces the resource-usage group (reducing duplication)
• NetworkPolicies: Managed by the chart via prometheus.networkPolicy and prometheusOperator.networkPolicy settings
• Install logic: Inlined into the Makefile prometheus target — no separate script
• Conftest: Removed prometheus-networkpolicies.rego policy (NetworkPolicy now managed by the chart)
• Kustomize: Removed unused bingo tooling (was only used by the deleted install script)

Reviewer Checklist

• API Go Documentation
• Tests: Unit Tests (and E2E Tests, if appropriate)
• Comprehensive Commit Messages
• Links to related GitHub Issue(s)

[Copilot]

Pull request overview

This PR migrates the e2e Prometheus setup from a bespoke Helm chart + install script to the upstream kube-prometheus-stack chart (installed from the OCI registry), aiming to reduce maintenance overhead while preserving the existing e2e monitoring/alerting behavior.

Changes:

• Added kube-prometheus-stack values files under testdata/prometheus/ (including an experimental override).
• Updated the prometheus Makefile target to install kube-prometheus-stack directly via Helm and removed Prometheus-specific helm+conftest linting.
• Removed the custom Prometheus Helm chart, the install script, and the Prometheus NetworkPolicy conftest policy.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
testdata/prometheus/values.yaml	New baseline kube-prometheus-stack values for e2e (disables unused components, adds ServiceMonitors and rules).
testdata/prometheus/values-experimental.yaml	New experimental override values to adjust alert thresholds.
Makefile	Replaces the Prometheus install script with an inline Helm install from OCI; updates linting and experimental values path.
hack/conftest/policy/README.md	Updates documentation to reflect removal of Prometheus-specific conftest policies and lint flow.
AGENTS.md	Updates repository layout documentation to reference the new testdata/prometheus/ location.
helm/prometheus/Chart.yaml (deleted)	Removes the custom Prometheus Helm chart definition.
helm/prometheus/values.yaml (deleted)	Removes custom chart values (thresholds/namespaces).
helm/prometheus/templates/servicemonitor-operator-controller-controller-manager-metrics-monitor.yml (deleted)	Removes custom ServiceMonitor for operator-controller metrics.
helm/prometheus/templates/servicemonitor-catalogd-controller-manager-metrics-monitor.yml (deleted)	Removes custom ServiceMonitor for catalogd metrics.
helm/prometheus/templates/servicemonitor-kubelet.yml (deleted)	Removes custom kubelet ServiceMonitor.
helm/prometheus/templates/prometheusrule-controller-alerts.yml (deleted)	Removes custom PrometheusRule definitions (now supplied via upstream chart values).
helm/prometheus/templates/prometheus-prometheus.yml (deleted)	Removes custom Prometheus CR manifest.
helm/prometheus/templates/serviceaccount-prometheus.yml (deleted)	Removes custom Prometheus ServiceAccount manifest.
helm/prometheus/templates/secret-prometheus-metrics-token.yml (deleted)	Removes the legacy service-account-token Secret approach for scraping.
helm/prometheus/templates/service-prometheus-service.yml (deleted)	Removes the custom NodePort Service manifest.
helm/prometheus/templates/networkpolicy-prometheus.yml (deleted)	Removes custom Prometheus NetworkPolicy manifest.
helm/prometheus/templates/networkpolicy-prometheus-operator.yml (deleted)	Removes custom Prometheus Operator NetworkPolicy manifest.
helm/prometheus/templates/clusterrole-prometheus.yml (deleted)	Removes custom Prometheus ClusterRole manifest.
helm/prometheus/templates/clusterrolebinding-prometheus.yml (deleted)	Removes custom Prometheus ClusterRoleBinding manifest.
helm/prom_experimental.yaml (deleted)	Removes the legacy experimental Prometheus values file (replaced by testdata/prometheus/values-experimental.yaml).
hack/test/install-prometheus.sh (deleted)	Removes the bespoke Prometheus install script (replaced by Makefile logic).
hack/conftest/policy/prometheus-networkpolicies.rego (deleted)	Removes Prometheus NetworkPolicy conftest policy (now chart-managed).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[netlify]

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	a0e1d24
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/6a2a734392b4ec00072bc10c
😎 Deploy Preview	https://deploy-preview-2757--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.82%. Comparing base (23b7e52) to head (a0e1d24).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #2757 +/- ##
==========================================
- Coverage 66.84% 66.82% -0.02% 
==========================================
Files 149 149 Lines 11382 11382 ==========================================
- Hits 7608 7606 -2 - Misses 3218 3219 +1 - Partials 556 557 +1 

Flag	Coverage Δ
e2e	35.09% <ø> (-0.05%)	⬇️
experimental-e2e	52.34% <ø> (ø)
unit	52.17% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

[dtfranz]

This rule can probably move to values.yaml as well since it doesn't need to change.

[pedjak]

Good point. However, the oom-events rule is part of the resource-usage group which is a YAML array — Helm replaces arrays entirely when merging -f files, so we can't selectively keep individual rules from the base. Moving it to a third map key (e.g. controller-oom-alerts) would work but adds another PrometheusRule resource. Happy to do it if you prefer.

[dtfranz]

That's fine then, we can leave it like this

[dtfranz]

/lgtm

[rashmigottipati]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: rashmigottipati

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment