:seedling: Use Patch instead of Update for finalizer operations by tmshort · Pull Request #2328 · operator-framework/operator-controller

tmshort · 2025-11-12T15:54:47Z

Refactor all controllers to use Patch() instead of Update()
when adding or removing finalizers to improve performance, and to avoid
removing non-cached fields erroneously.

This is necesary because we no longer cache the last-applied-configuration
annotation, so when we add/remove the finalizers, we are removing that field
from the metadata. This causes issues with clients when they don't see that
annotation (e.g. apply the same ClusterExtension twice).

Update ClusterCatalog and ClusterExtension controllers to use Patch-based
finalizer management (ClusterExtensionRevision already uses Patch())

This also restores the non-caching of "last-applied-config".

Description

Reviewer Checklist

API Go Documentation
Tests: Unit Tests (and E2E Tests, if appropriate)
Comprehensive Commit Messages
Links to related GitHub Issue(s)

netlify · 2025-11-12T15:54:54Z

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	`5d23bfe`
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/692db2312898dc0008c4f859
😎 Deploy Preview	https://deploy-preview-2328--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

tmshort · 2025-11-12T15:56:27Z

The ClusterExtension and ClusterCatalog finalizer code used Update, the ClusterExtensionRevision used Apply, so the CE and CE finalizer code was rewritten to be like CER. There is common finalizer code between them now.

tmshort · 2025-11-12T15:56:43Z

Ping @joelanford

tmshort · 2025-11-12T16:25:20Z

catalogd pod is panicing... it only fails due to the summary generated, it's not "noticeable" during a regular run!

pedjak · 2025-11-12T16:36:35Z

This is necesary because we no longer cache thelast-applied-configuration annotation, so when we add/remove the finalizers, we are removing that field from the metadata. This causes issues with clients when they don't see that annotation (e.g. apply the same ClusterExtension twice).

If we want to remove conflicts, we should use patch, but adding/removing finalizer we do not touch removed fields (including the annotation). Controller-runtime cache is write-through cache, so an update or patch operation does not perform any direct modification on the cache - all changes are made by the informer.

In what situations did we see conflicts, given that only single controller is responsible for a resource type?

tmshort · 2025-11-12T17:32:10Z

If we want to remove conflicts, we should use patch, but adding/removing finalizer we do not touch removed fields (including the annotation). Controller-runtime cache is write-through cache, so an update or patch operation does not perform any direct modification on the cache - all changes are made by the informer.
In what situations did we see conflicts, given that only single controller is responsible for a resource type?

This is not a conflict issue. The problem was that we are using Update(), which uses the cached version of the resource to make the changes. The cached version no longer includes the last-applied-configuration annotation, so that annotation is removed on the Update().

This causes a problem with e.g. kubectl applying a ClusterExtension a second time. If you apply a CE twice without this code (i.e. current main branch), you will see the following:

tshort@cube:~/.../config/samples (main %=)$ kubectl apply -f ce.yaml
clusterextension.olm.operatorframework.io/argocd created
tshort@cube:~/.../config/samples (main %=)$ oc get clusterextension -o yaml
apiVersion: v1
items:
- apiVersion: olm.operatorframework.io/v1
kind: ClusterExtension
metadata:
creationTimestamp: "2025-11-12T15:22:25Z"
finalizers:
- olm.operatorframework.io/cleanup-unpack-cache
- olm.operatorframework.io/cleanup-contentmanager-cache
generation: 1
name: argocd
resourceVersion: "1296"
uid: b096377d-577e-4d25-8f3d-b0e0b12fe894
...
tshort@cube:~/.../config/samples (main %=)$ kubectl apply -f ce.yaml
Warning: resource clusterextensions/argocd is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterextension.olm.operatorframework.io/argocd configured
tshort@cube:~/.../config/samples (main %=)$

The problem is that the last-applied-configuration annotation is not present. Thus, causing the second kubectl apply command to complain. The expected result is that the second kubectl apply states clusterextension.olm.operatorframework.io/argocd unchanged

codecov · 2025-11-12T19:30:24Z

Codecov Report

❌ Patch coverage is 89.28571% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.57%. Comparing base (91b03c7) to head (5d23bfe).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/operator-controller/applier/boxcutter.go	33.33%	1 Missing and 1 partial ⚠️
internal/shared/util/cache/transform.go	92.00%	1 Missing and 1 partial ⚠️
internal/shared/util/k8s/k8s.go	87.50%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #2328 +/- ##
==========================================
+ Coverage 74.48% 74.57% +0.09% 
==========================================
Files 93 95 +2 Lines 7333 7375 +42 ==========================================
+ Hits 5462 5500 +38 - Misses 1436 1438 +2 - Partials 435 437 +2

Flag	Coverage Δ
e2e	`44.64% <71.42%> (+0.12%)`	⬆️
experimental-e2e	`48.94% <80.35%> (+0.18%)`	⬆️
unit	`58.45% <33.92%> (-0.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joelanford · 2025-11-12T20:55:06Z

+ return false, fmt.Errorf("marshalling patch to add finalizer: %w", err)
+	}
+	// Note: Patch will update obj with the server response, including ResourceVersion
+	if err := c.Patch(ctx, obj, client.RawPatch(types.MergePatchType, patchJSON)); err != nil {


Should we use SSA here?

that could be a good improvement, that would simplify the reconcile logic a lot, i.e. we do not need to care about appending/sorting finalizers at all.

pedjak · 2025-11-13T11:28:57Z

This is not a conflict issue. The problem was that we are using Update(), which uses the cached version of the resource to make the changes. The cached version no longer includes the last-applied-configuration annotation, so that annotation is removed on the Update().

Thanks for explaining it, make sense. Given that I think we should update PR description to reflect that, currently we have:

Refactor all controllers to use client.Patch() instead of Update() when adding or removing finalizers to reduce conflicts and improve performance.

Given the explanation above, we are not reducing conflict or improving performances - we are ensuring correctness, i.e. not removing the annotation.

pedjak · 2025-11-14T12:16:51Z

 	Applier Applier
 	RevisionStatesGetter RevisionStatesGetter
-Finalizers crfinalizer.Finalizers
+FinalizerHandlersmap[string]FinalizerHandler


a controller should set single finalizer, more than one is strange. I would suggest reverting the change.

I think there is more than one for CEs right now because there are multiple finalizer concerns:
Cleaning up the unpacked bundle image directory
Shutting down the content manager informers (helm-only)
I'm not sure how the current helm-to-boxcutter migration deals with the content manager informer shutdown. Do those informers get cleaned up when boxcutter takes over or when the CE is deleted (potentially much later). Ideally we'd shutdown the helm-only content manager informers as soon as boxcutter takes over.

(but that's getting a bit outside the scope of this PR)

In this case, there ARE two pre-existing finalizers on the ClusterExtension

@joelanford
I'm not sure how the current helm-to-boxcutter migration deals with the content manager informer shutdown. Do those informers get cleaned up when boxcutter takes over or when the CE is deleted (potentially much later). Ideally we'd shutdown the helm-only content manager informers as soon as boxcutter takes over.
In order to switch from the Helm applier to the Boxcutter applier, the pod has to restart, so everything is effectively shutdown. The content-manager finalizer is a no-op under Boxcutter, but the finalizer still needs to be removed.

pedjak · 2025-11-14T12:29:15Z

+	u.SetGroupVersionKind(gvk)
+	u.SetName(obj.GetName())
+	u.SetNamespace(obj.GetNamespace())
+	u.SetFinalizers(newFinalizers)


it has to be u.SetFinalizers(finalizers)

No. this is based on the assumption that the arguments to EnsureFinalizers represent the only finalizers we want in the list. This is not the intent.
The EnsureFinalizers API ensures that the given finalizers are present. The intent is not to set the finalizers to only be those given, and is not intended to remove finalizers.
Because this is a list, we can't patch individual items into the list, we need to specify the whole list (at least that's how the SSA API is working). I verified this in the code by calling EnsureFinalizers() separately for each finalizer string, rather than in bulk. Only the last finalizer showed up when it was u.SetFinalizers(finalizers). Both finalizers showed up when the code was as above.
Yes, I realize the name EnsureFinalizers is a bit confusing, and should probably be AddFinalizers, but I was following the naming scheme (and behavior) of the removed code, which was inconsistent.

Because this is a list, we can't patch individual items into the list, we need to specify the whole list (at least that's how the SSA API is working).
I need to disagree here, because Finalizer is defined as a set with merge patch stratefy:
https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go#L256-L259
Therefore, we do not patch item by item, we set all finalizers we need to own and make SSA requests.
BTW, my understanding here is that finalizers argument contains all finalizers we would like to set, not just one.

Therefore, we do not patch item by item, we set all finalizers we need to own and make SSA requests.
Agreed, that's how SSA works, but that's not how the API layer above it has to work.
The function as defined adds finalizers, that's the intent. The intent is not to make the finalizer list equal to the arguments.
I'm renaming the function to indicate the desired (existing) behavior. It is confusing to have the mechanism to remove finalizers as "ensuring nothing".
If we wanted the behavior as you suggest, then SetFinalizers() would be more appropriate. But it would be at the cost of flexibility, and potential issues if other controllers add their own finalizers.

pedjak · 2025-11-14T12:30:17Z

+	}
+
+	// Update the passed object with the new finalizers
+	obj.SetFinalizers(newFinalizers)


it has to get `obj.SetFinalizers(u.GetFinalizers())

pedjak · 2025-11-14T12:31:53Z

+
+// RemoveFinalizer removes one or more finalizers from the object using server-side apply.
+// If none of the finalizers exist, this is a no-op.
+func RemoveFinalizer(ctx context.Context, c client.Client, obj client.Object, finalizers ...string) error {


no need to have this function because EnsureFinalizer(ctx, c, obj) is going to remove previously added finalizers that we owned.

The intent of EnsureFinalizers is to only add finalizers to the list of finalizers, not to remove them. The RemoveFinalizers removes finalizers individually.

in SSA approach, you are not remove things by omitting them in request, if you owned them previously. Hence, if we had somewhere in the code:
EnsureFinalizers(ctx, client, ce, "foo", "bar")
to remove bar, we can perform:
EnsureFinalizers(ctx, client, ce, "foo")

But that is not the intent of the API; it's to add items individually; not to remove. Just because the underlying mechanism is SSA, doesn't mean the layered API has to behave that way.
IMHO, It's confusing to have "Ensure" remove items, "Ensure" means to check that something is true or correct. In this case "Ensure" is confusing, so I'm renaming the function.

tmshort · 2025-11-14T22:08:32Z

After digging into the solutions (Add/Ensure/Remove/Update/Set), none of them are optimal. Specifically, if another controller adds a finalizer to our resources, we see it, and that messes with our finalizer management. This means we hit the API server with a SSA more than we should.

If a single-function mechanism (e.g. Set/Ensure) is used, then on every reconcile we'll see the extra finalizer and attempt to set the finalizers to the input list (which may be empty) which attempts to remove the extra finalizer, and because we don't own it, it will never be removed. So any checks to ensure finalizers are set as we expect will fail, forcing an API server call.

I prefer the explicit add and remove mechanism, as in this current PR. It explicitly checks the current list of finalizers and only cares about the ones that are input. It will use the existing list of finalizers to add/remove, and will only issue a SSA if changes are needed; it will ignore the extra finalizer. However, in the unlikely case where the cache is out-of-sync, then we will hit the API server with a SSA until the cache is synced. It's better than every reconcile as in the single-function case.

EDIT: All our finalizers use the same prefix, other controllers' finalizers should use a different prefix. So, now that I thought about it some more, I think a single Ensure function will work. However, sorting is still required to check if the finalizers are in the right state, such that we reduce the number of Server API calls.

pedjak · 2025-11-17T13:36:18Z

@tmshort apologize I did not mention it earlier: moving to SSA requires also to migrate the information stored under .metadata.managedField - otherwise field removal would not be possible, see for example:

kubernetes/kubernetes#99003

In our case, it is easy to reproduce:

deploy OLM from main branch, create a ClusterExtension. Under .metadata.managedField we have now:

- apiVersion: olm.operatorframework.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata:
f:finalizers: .: {} v:"olm.operatorframework.io/cleanup-contentmanager-cache": {} v:"olm.operatorframework.io/cleanup-unpack-cache": {} manager: operator-controller operation: Update

This is expected, because we have added finalizers via Update call.

Now, build and deploy OLM using this PR and try to remove the existing ClusterExtension. The operation is not going to be successful, even if there is no errors to be see in the logs. The controller is not able to remove finalizers, because we now use operation Apply and the combination of manager and operation determines the field owner. Clearly, given that we do not provide finalizers, we are not able to remove it because we are not the owners.

To fix that we would need to migrate the ownership under .metadata.managedFields. However, it looks like a way more work for the cause that needs to be fixed: removal of last-applied-configuration annotation to save a bit of memory.

Instead of going to through SSA migration, I would suggest that we revert removal of that annotation as the quick fix, and then introduce SSA if needed + detailed testing.

tmshort · 2025-11-17T14:16:13Z

/hold
Yes, this does not take into account migration of pre-existing resources. I could switch it back to using Patch(), as moving to SAA with existing fields will give us some trouble.

That being said, there is currently a failure in the test-experimental-e2e, which seems to be triggered by this change; since I'm not seeing the issue on main. I would need to fix that before this could be merged.

I will point out that managed fields is not a "bit of memory", but can double the size of the cache. The memory savings was significant.

pedjak · 2025-11-17T14:45:37Z

I will point out that managed fields is not a "bit of memory", but can double the size of the cache. The memory savings was significant.

I agree, but managed fields are readded in the cache with #2318

tmshort · 2025-11-18T16:44:50Z

/unhold

tmshort · 2025-11-24T20:31:14Z

Reworked (i.e. rebased to main) to use Patch only for the CE and CC controllers, since CER already used Patch. This is a much smaller change than before, as I didn't attempt to create a common finalizer infrastructure.

pedjak · 2025-11-25T10:17:18Z

+ // Use patch to update finalizers on the server
+ patch := map[string]any{
+ "metadata": map[string]any{
+ "finalizers": finalizers,
+ },
+ }
+ patchJSON, err := json.Marshal(patch)
+ if err != nil {
+ reconcileErr = errors.Join(reconcileErr, fmt.Errorf("marshalling patch to ensure finalizers: %w", err))
+ } else if err := r.Patch(ctx, reconciledCatsrc, client.RawPatch(types.MergePatchType, patchJSON)); err != nil {


How about that we use already available controller-runtime's CreateOrPatch?

Possibly overkill (looking at the size of the function), but I can look.

Possibly overkill (looking at the size of the function), but I can look.
I would create RawPatch, only if really needed. CreateOrPatch() covers us both in updating finalizers and status and could simplify the code a lot, i.e. we do not need to check if statuses/finalizers are changed during the reconcile, that function does it all. The whole updateFinalizers { ... } block could be replaced with something like:
if_, err:=controllerutil.CreateOrPatch(ctx, r.Client, &existingCatsrc, func() error { existingCatsrc.Finalizers=reconciledCatsrc.Finalizersreturnnil }); err!=nil { return ctrl.Result{}, err }
and we could also go beyond that (slightly out of the PR scope, could be done separately) and also to drop if updateStatus {...} block with:
if_, err:=controllerutil.CreateOrPatch(ctx, r.Client, &existingCatsrc, func() error { existingCatsrc.Finalizers=reconciledCatsrc.FinalizersexistingCatsrc.Status=reconciledCatsrc.Statusreturnnil }); err!=nil { return ctrl.Result{}, err }

Possibly overkill (looking at the size of the function), but I can look.
As in the size of CreateOrPatch(), that is...
Dammit, I forgot to push my changes last night,...

pedjak · 2025-11-25T10:18:19Z

+ // Use patch to update finalizers on the server
+ patch := map[string]any{
+ "metadata": map[string]any{
+ "finalizers": finalizers,
+ },
+ }
+ patchJSON, err := json.Marshal(patch)
+ if err != nil {
+ reconcileErr = errors.Join(reconcileErr, fmt.Errorf("marshalling patch to ensure finalizers: %w", err))
+ } else if err := r.Patch(ctx, reconciledExt, client.RawPatch(types.MergePatchType, patchJSON)); err != nil {


same question as above.

tmshort · 2025-11-26T13:54:16Z

Not a fan of this package name, but I couldn't think of anything better? Suggestions welcome.

pedjak · 2025-11-26T15:04:30Z

+
+// GetSpec returns the Spec field of the ClusterExtension.
+// This method is provided to satisfy generic interfaces that require a GetSpec() method.
+func (c *ClusterExtension) GetSpec() ClusterExtensionSpec {
+	return c.Spec
+}
+
+// GetSpec returns the Spec field of the ClusterCatalog.
+// This method is provided to satisfy generic interfaces that require a GetSpec() method.
+func (c *ClusterCatalog) GetSpec() ClusterCatalogSpec {
+	return c.Spec
+}
+
+// GetSpec returns the Spec field of the ClusterExtensionRevision.
+// This method is provided to satisfy generic interfaces that require a GetSpec() method.
+func (c *ClusterExtensionRevision) GetSpec() ClusterExtensionRevisionSpec {
+	return c.Spec
+}


It is recommended to avoid adding helper functions:
https://github.com/openshift/enhancements/blob/master/dev-guide/api-conventions.md#no-functions

pedjak · 2025-11-27T12:14:34Z

+// Type parameter:
+// - O: The object type that implements metav1.Object and has a Spec field


Given that O is interface, no need to declare a generic function, we can do it like:
funcCheckForUnexpectedFieldChange(a, b metav1.Object) bool {

Refactor all controllers to use Patch() instead of Update() when adding or removing finalizers to improve performance, and to avoid removing non-cached fields erroneously. This is necesary because we no longer cache the last-applied-configuration annotation, so when we add/remove the finalizers, we are removing that field from the metadata. This causes issues with clients when they don't see that annotation (e.g. apply the same ClusterExtension twice). Update ClusterCatalog and ClusterExtension controllers to use Patch-based funalizer management (ClusterExtensionRevision already uses Patch()) Signed-off-by: Todd Short <tshort@redhat.com>

…mework#2338)" This reverts commit 39cbdbe.

pedjak

/lgtm

rashmigottipati

looks great overall!

not adding the label to leave an opportunity for other folks to review

tmshort · 2025-12-02T21:44:46Z

/approve
There are two LGTMs on it.

openshift-ci · 2025-12-02T21:44:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pedjak, rashmigottipati, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tmshort requested a review from a team as a code owner November 12, 2025 15:54

openshift-ciBot requested review from OchiengEd and grokspawn November 12, 2025 15:54

tmshort force-pushed the apply-finalizers branch from 21e90d9 to 0608a72Compare November 12, 2025 19:15

joelanford reviewed Nov 12, 2025
View reviewed changes

Comment threadinternal/shared/util/finalizer/finalizer.go Outdated

joelanford reviewed Nov 12, 2025
View reviewed changes

Comment threadinternal/shared/util/finalizer/finalizer.go Outdated

joelanford reviewed Nov 12, 2025
View reviewed changes

Comment threadinternal/shared/util/finalizer/finalizer.go Outdated

joelanford reviewed Nov 12, 2025
View reviewed changes

Comment threadinternal/shared/util/finalizer/finalizer.go Outdated

tmshort force-pushed the apply-finalizers branch 2 times, most recently from cf27d6e to bfb7a16Compare November 13, 2025 16:40

tmshort changed the title ~~🐛 Use Patch instead of Update for finalizer operations~~🐛 Use SSA instead of Update for finalizer operationsNov 13, 2025

tmshort requested review from joelanford and pedjak November 13, 2025 16:41

pedjak requested changes Nov 14, 2025
View reviewed changes

openshift-ciBot assigned pedjak Nov 14, 2025

tmshort force-pushed the apply-finalizers branch from bfa9322 to 9dea235Compare November 16, 2025 18:58

openshift-ciBot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 17, 2025

openshift-ciBot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 18, 2025

openshift-ciBot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 18, 2025

tmshort force-pushed the apply-finalizers branch 3 times, most recently from 9fc4525 to 6e95c7bCompare November 19, 2025 21:07

tmshort changed the title ~~WIP: Use SSA instead of Update for finalizer operations~~🌱 Use Patch instead of Update for finalizer operationsNov 19, 2025

openshift-ciBot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 19, 2025

tmshort force-pushed the apply-finalizers branch 2 times, most recently from 6c820a5 to 8fba829Compare November 24, 2025 20:29

tmshort force-pushed the apply-finalizers branch from 8fba829 to e2b3cdaCompare November 24, 2025 20:51

pedjak reviewed Nov 25, 2025
View reviewed changes

tmshort force-pushed the apply-finalizers branch from e2b3cda to 97d5a33Compare November 26, 2025 13:52

tmshort commented Nov 26, 2025
View reviewed changes

pedjak reviewed Nov 26, 2025
View reviewed changes

tmshort force-pushed the apply-finalizers branch from 97d5a33 to 2582233Compare November 26, 2025 19:29

pedjak reviewed Nov 27, 2025
View reviewed changes

Comment threadinternal/shared/util/k8s/k8s.go

pedjak reviewed Nov 27, 2025
View reviewed changes

tmshort added 2 commits December 1, 2025 09:37

Revert "Restore last-applied-config annotation in cache (operator-fra…
5d23bfe
…mework#2338)" This reverts commit 39cbdbe.

tmshort force-pushed the apply-finalizers branch from 2582233 to 5d23bfeCompare December 1, 2025 15:20

pedjak approved these changes Dec 1, 2025
View reviewed changes

openshift-ciBot added the lgtm Indicates that a PR is ready to be merged. label Dec 1, 2025

rashmigottipati approved these changes Dec 1, 2025
View reviewed changes

openshift-ciBot assigned rashmigottipati Dec 1, 2025

openshift-ciBot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 2, 2025

openshift-merge-botBot merged commit 4e349e6 into operator-framework:mainDec 2, 2025
27 checks passed

		// Type parameter:
		// - O: The object type that implements metav1.Object and has a Spec field

Uh oh!

Conversation

tmshort commented Nov 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer Checklist

Uh oh!

netlifyBot commented Nov 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for olmv1 ready!

Uh oh!

tmshort commented Nov 12, 2025

Uh oh!

tmshort commented Nov 12, 2025

Uh oh!

tmshort commented Nov 12, 2025

Uh oh!

pedjak commented Nov 12, 2025

Uh oh!

tmshort commented Nov 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecovBot commented Nov 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pedjak commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshortNov 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshortNov 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshortNov 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshortNov 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmshort commented Nov 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmshort commented Nov 12, 2025•
edited
Loading

netlifyBot commented Nov 12, 2025•
edited
Loading

tmshort commented Nov 12, 2025•
edited
Loading

codecovBot commented Nov 12, 2025•
edited
Loading

tmshortNov 14, 2025•
edited
Loading

tmshortNov 14, 2025•
edited
Loading

tmshortNov 14, 2025•
edited
Loading

tmshortNov 14, 2025•
edited
Loading

tmshort commented Nov 14, 2025•
edited
Loading

tmshortNov 26, 2025•
edited
Loading

pedjakNov 27, 2025•
edited
Loading