Skip to content

🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567) - #2169

Merged
openshift-merge-bot[bot] merged 1 commit into
operator-framework:mainfrom
camilamacedo86:fix-preflight
Sep 3, 2025
Merged

🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)#2169
openshift-merge-bot[bot] merged 1 commit into
operator-framework:mainfrom
camilamacedo86:fix-preflight

Conversation

@camilamacedo86

@camilamacedo86camilamacedo86 commented Aug 27, 2025

Copy link
Copy Markdown
Contributor

When upgrading operators, CRD validation errors can be very large (50KB+). Kubernetes rejects status updates over 32KB with "Too long: may not be more than 32768 bytes". This causes ClusterExtension upgrades to fail and get stuck.

Messages keep important info at the start and add "... [message truncated]" suffix. Now upgrades complete successfully even with large CRD validation errors.

Added unit tests for truncation logic and CRD error scenarios.

Reviewer Checklist

  • [N/A] API Go Documentation
  • Tests: Unit Tests (and E2E Tests, if appropriate)
  • Comprehensive Commit Messages
  • [N/A ] Links to related GitHub Issue(s)

@camilamacedo86
camilamacedo86 requested a review from a team as a code ownerAugust 27, 2025 13:18
@netlify

netlifyBot commented Aug 27, 2025

Copy link
Copy Markdown

Deploy Preview for olmv1 ready!

NameLink
🔨 Latest commit87fb307
🔍 Latest deploy loghttps://app.netlify.com/projects/olmv1/deploys/68b7cf45327ed30008791c72
😎 Deploy Previewhttps://deploy-preview-2169--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@camilamacedo86camilamacedo86 changed the title 🐛 Fix: Truncate large error messages in status conditions🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)Aug 27, 2025
@joelanford

Copy link
Copy Markdown
Member

Can you include some details of the messages that are too long? I feel like arbitrarily truncating the message is sort of papering over the underlying issue, which is that 30k-byte messages in conditions are a poor UX, and the real solution would be to make the message shorter to begin with.

/hold

@openshift-ciopenshift-ciBot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 27, 2025
Comment threadinternal/operator-controller/controllers/common_controller.go Outdated
Status: metav1.ConditionFalse,
Reason: string(reason),
Message: message,
Message: truncateMessage(message),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's a limit imposed on all condition messages, it seems like we need to make sure that we truncate all condition messages.

This is one of many places where we set condition messages, right?

We may need to implement a wrapper around the meta.SetCondition() that:

  1. truncates messages
  2. everything throughout our project uses.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think wrapper will be better as well +1

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@camilamacedo86camilamacedo86 changed the title 🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)WIP 🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)Aug 27, 2025
@openshift-ciopenshift-ciBot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 27, 2025
@camilamacedo86

This comment was marked as outdated.

@camilamacedo86camilamacedo86 changed the title WIP 🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)Aug 29, 2025
@openshift-ciopenshift-ciBot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 29, 2025
@camilamacedo86
camilamacedo86force-pushed the fix-preflight branch 2 times, most recently from aec2345 to b30dc47CompareAugust 29, 2025 20:47
@codecov

codecovBot commented Aug 29, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.77%. Comparing base (c56a811) to head (87fb307).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #2169 +/- ##
==========================================
+ Coverage 72.73% 72.77% +0.03% 
==========================================
Files 79 79 Lines 7384 7391 +7 ==========================================
+ Hits 5371 5379 +8 + Misses 1666 1665 -1 
Partials 347 347 
FlagCoverage Δ
e2e44.18% <88.23%> (-0.02%)⬇️
experimental-e2e56.17% <88.23%> (-0.03%)⬇️
unit58.28% <100.00%> (+0.05%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@camilamacedo86camilamacedo86 changed the title 🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)WIP 🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)Aug 29, 2025
@openshift-ciopenshift-ciBot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 29, 2025
@joelanford

Copy link
Copy Markdown
Member

/hold cancel

Thanks Camila! +1 on solving this both ways. Truncate long messages AND try to avoid long messages in the first place!

@openshift-ciopenshift-ciBot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 29, 2025
@camilamacedo86camilamacedo86 changed the title WIP 🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)🐛 Fix: Truncate large error messages in status conditions (OCPBUGS-59518, OCPBUGS-38567)Aug 30, 2025
@openshift-ciopenshift-ciBot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 30, 2025
Comment threadinternal/operator-controller/controllers/common_controller.go Outdated
@joelanford

Copy link
Copy Markdown
Member

CRD validation errors can be very large (50KB+)

I've noticed this as well. Not only are they large, but the information density is low. IIRC, the CRD upgrade check output can contain a "diff-style" output that show context around the diff, which isn't really useful.

I think we should revisit our message formatting specifically in the case of the CRD upgrade check output to make it easier to understand:

  1. Which field(s) is/are problematic?
  2. Which changes were considered breaking?
  3. (no other information)

When upgrading operators, CRD validation errors can be very large (50KB+).
Kubernetes rejects status updates over 32KB with "Too long: may not be more than 32768 bytes".
This causes ClusterExtension upgrades to fail and get stuck.
Assisted-by: Cursor
@camilamacedo86

Copy link
Copy Markdown
ContributorAuthor

Hi @joelanford@perdasilva

I think it is good to get merged. I think we should truncate anyway.
However, the real fix for the issue it seems for me to be: #2179

I updated the last comment in : https://issues.redhat.com/browse/OCPBUGS-59518

The problem here is v1alpha1: ^.status.resources: unhandled: unhandled changes found :

@openshift-ciopenshift-ciBot added the lgtm Indicates that a PR is ready to be merged. label Sep 3, 2025
@openshift-ci

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ciopenshift-ciBot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 3, 2025
@openshift-merge-bot
openshift-merge-botBot merged commit 81be2e9 into operator-framework:mainSep 3, 2025
24 checks passed
@camilamacedo86
camilamacedo86 deleted the fix-preflight branch September 3, 2025 13:21
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@camilamacedo86@joelanford@perdasilva