Resolution fails when manually upgrading through legacy replaces chain beyond n+1

[everettraven]

When attempting to manually step through upgrading versions of a package using the ClusterExtension using the replaces chain in the FBC channel, moving beyond the first replaced version results in the following status condition:

 - lastTransitionTime: "2024-07-03T19:44:12Z"
 message: 'error upgrading from currently installed version "0.1.0": no package
 "argocd-operator" matching version "0.3.0" in channel "alpha" found'
 observedGeneration: 4
 reason: ResolutionFailed
 status: "False"
 type: Resolved

In this case, I had followed the replaces chain from v0.1.0 --> v0.2.0 successfully. When attempting to go from v0.2.0 --> v0.3.0 the upgrade failed and contained the above resolution failure.

To verify this is a valid upgrade path, you can see the channel and upgrade edges with:

docker run --rm -it quay.io/operator-framework/opm:latest render quay.io/operatorhubio/catalog:latest | jq -s '.[] | select( .schema == "olm.channel" ) | select( .package == "argocd-operator")'

Output

{
 "schema": "olm.channel",
 "name": "alpha",
 "package": "argocd-operator",
 "entries": [
 {
 "name": "argocd-operator.v0.0.11",
 "replaces": "argocd-operator.v0.0.9"
 },
 {
 "name": "argocd-operator.v0.0.12",
 "replaces": "argocd-operator.v0.0.11"
 },
 {
 "name": "argocd-operator.v0.0.13",
 "replaces": "argocd-operator.v0.0.12"
 },
 {
 "name": "argocd-operator.v0.0.14",
 "replaces": "argocd-operator.v0.0.13"
 },
 {
 "name": "argocd-operator.v0.0.15",
 "replaces": "argocd-operator.v0.0.14"
 },
 {
 "name": "argocd-operator.v0.0.2"
 },
 {
 "name": "argocd-operator.v0.0.3",
 "replaces": "argocd-operator.v0.0.2"
 },
 {
 "name": "argocd-operator.v0.0.4",
 "replaces": "argocd-operator.v0.0.3"
 },
 {
 "name": "argocd-operator.v0.0.5",
 "replaces": "argocd-operator.v0.0.4"
 },
 {
 "name": "argocd-operator.v0.0.6",
 "replaces": "argocd-operator.v0.0.5"
 },
 {
 "name": "argocd-operator.v0.0.8",
 "replaces": "argocd-operator.v0.0.6"
 },
 {
 "name": "argocd-operator.v0.0.9",
 "replaces": "argocd-operator.v0.0.8"
 },
 {
 "name": "argocd-operator.v0.1.0",
 "replaces": "argocd-operator.v0.0.15"
 },
 {
 "name": "argocd-operator.v0.10.0",
 "replaces": "argocd-operator.v0.9.2"
 },
 {
 "name": "argocd-operator.v0.10.1",
 "replaces": "argocd-operator.v0.10.0"
 },
 {
 "name": "argocd-operator.v0.2.0",
 "replaces": "argocd-operator.v0.1.0"
 },
 {
 "name": "argocd-operator.v0.2.1",
 "replaces": "argocd-operator.v0.2.0"
 },
 {
 "name": "argocd-operator.v0.3.0",
 "replaces": "argocd-operator.v0.2.1"
 },
 {
 "name": "argocd-operator.v0.4.0",
 "replaces": "argocd-operator.v0.3.0"
 },
 {
 "name": "argocd-operator.v0.5.0",
 "replaces": "argocd-operator.v0.4.0"
 },
 {
 "name": "argocd-operator.v0.6.0",
 "replaces": "argocd-operator.v0.5.0"
 },
 {
 "name": "argocd-operator.v0.7.0",
 "replaces": "argocd-operator.v0.6.0"
 },
 {
 "name": "argocd-operator.v0.8.0",
 "replaces": "argocd-operator.v0.7.0"
 },
 {
 "name": "argocd-operator.v0.9.0",
 "replaces": "argocd-operator.v0.8.0"
 },
 {
 "name": "argocd-operator.v0.9.1",
 "replaces": "argocd-operator.v0.9.0"
 },
 {
 "name": "argocd-operator.v0.9.2",
 "replaces": "argocd-operator.v0.9.1"
 }
 ]
}

Reproduction Steps

• Create a ClusterCatalog that references the operatorhub.io catalog image:

apiVersion: catalogd.operatorframework.io/v1alpha1
kind: ClusterCatalog
metadata:
 name: operatorhubio
spec:
 source:
 type: image
 image:
 ref: quay.io/operatorhubio/catalog:latest
 pollInterval: 24h

• Create a ClusterExtension that installs the argocd-operator at v0.1.0:

apiVersion: olm.operatorframework.io/v1alpha1
kind: ClusterExtension
metadata:
 name: argocd
spec:
 packageName: argocd-operator
 installNamespace: default
 channel: alpha
 version: 0.1.0

• Once the previously created ClusterExtension is successfully installed, manually update the version to v0.2.0 (note you will have to disable the recently added CRD Upgrade Safety check for this to work due to unknown changes to argo's CRDs):

apiVersion: olm.operatorframework.io/v1alpha1
kind: ClusterExtension
metadata:
 name: argocd
spec:
 packageName: argocd-operator
 installNamespace: default
 channel: alpha
 version: 0.2.0
 preflight:
 crdUpgradeSafety:
 disabled: true

• Once the first manual upgrade is successfully installed, manually upgrade again to v0.3.0:

apiVersion: olm.operatorframework.io/v1alpha1
kind: ClusterExtension
metadata:
 name: argocd
spec:
 packageName: argocd-operator
 installNamespace: default
 channel: alpha
 version: 0.3.0
 preflight:
 crdUpgradeSafety:
 disabled: true

You should now be able to see the same resolution failure.

Full ClusterExtension output YAML

apiVersion: olm.operatorframework.io/v1alpha1
kind: ClusterExtension
metadata:
 annotations:
 kubectl.kubernetes.io/last-applied-configuration: |
 {"apiVersion":"olm.operatorframework.io/v1alpha1","kind":"ClusterExtension","metadata":{"annotations":{},"name":"argocd"},"spec":{"channel":"alpha","installNamespace":"default","packageName":"argocd-operator","preflight":{"crdUpgradeSafety":{"disabled":true}},"version":"0.3.0"}}
 creationTimestamp: "2024-07-03T19:41:50Z"
 finalizers:
 - olm.operatorframework.io/cleanup-unpack-cache
 - olm.operatorframework.io/delete-cached-bundle
 generation: 4
 name: argocd
 resourceVersion: "1979"
 uid: 86f0b4a0-d6a7-4856-8aa2-e2d39e40b2a9
spec:
 channel: alpha
 installNamespace: default
 packageName: argocd-operator
 preflight:
 crdUpgradeSafety:
 disabled: true
 upgradeConstraintPolicy: Enforce
 version: 0.3.0
status:
 conditions:
 - lastTransitionTime: "2024-07-03T19:41:52Z"
 message: ""
 observedGeneration: 3
 reason: Deprecated
 status: "False"
 type: Deprecated
 - lastTransitionTime: "2024-07-03T19:41:52Z"
 message: ""
 observedGeneration: 3
 reason: Deprecated
 status: "False"
 type: PackageDeprecated
 - lastTransitionTime: "2024-07-03T19:41:52Z"
 message: ""
 observedGeneration: 3
 reason: Deprecated
 status: "False"
 type: ChannelDeprecated
 - lastTransitionTime: "2024-07-03T19:41:52Z"
 message: ""
 observedGeneration: 3
 reason: Deprecated
 status: "False"
 type: BundleDeprecated
 - lastTransitionTime: "2024-07-03T19:44:12Z"
 message: 'error upgrading from currently installed version "0.1.0": no package
 "argocd-operator" matching version "0.3.0" in channel "alpha" found'
 observedGeneration: 4
 reason: ResolutionFailed
 status: "False"
 type: Resolved
 - lastTransitionTime: "2024-07-03T19:41:54Z"
 message: 'unpack successful: '
 observedGeneration: 3
 reason: UnpackSuccess
 status: "True"
 type: Unpacked
 - lastTransitionTime: "2024-07-03T19:43:53Z"
 message: Instantiated bundle argocd successfully
 observedGeneration: 3
 reason: Success
 status: "True"
 type: Installed

Just to note, this was originally found by OpenShift QE. I verified the bug was reproducible and used a different package for installation.

[joelanford]

Definitely a blocker, IMO. Any idea why this is happening? My initial suspicion is that we somehow have an incorrect understanding of the currently installed version?

[everettraven]

Definitely a blocker, IMO. Any idea why this is happening? My initial suspicion is that we somehow have an incorrect understanding of the currently installed version?

That is my suspicion as well. It seems like

operator-controller/internal/controllers/clusterextension_controller.go

Line 426 in 8bf5cf4

installedBundle, err := r.InstalledBundleGetter.GetInstalledBundle(ctx, &ext)

may be returning only the initially installed version

[kevinrizza]

It's happening because the helm release for the upgraded bundle still has the first version label. When we install we set the version label:

operator-controller/internal/controllers/clusterextension_controller.go

Lines 355 to 372 in 7cc9872

	rel, err = ac.Install(ext.GetName(), ext.Spec.InstallNamespace, chrt, values, func(install *action.Install) error {
	install.CreateNamespace = false
	install.Labels = map[string]string{labels.BundleNameKey: bundle.Name, labels.PackageNameKey: bundle.Package, labels.BundleVersionKey: bundleVersion.String()}
	return nil
	}, helmclient.AppendInstallPostRenderer(post))
	if err != nil {
	setInstalledStatusConditionFailed(ext, fmt.Sprintf("%s:%v", ocv1alpha1.ReasonInstallationFailed, err))
	return ctrl.Result{}, err
	}
	case stateNeedsUpgrade:
	rel, err = ac.Upgrade(ext.GetName(), ext.Spec.InstallNamespace, chrt, values, helmclient.AppendUpgradePostRenderer(post))
	if err != nil {
	setInstalledStatusConditionFailed(ext, fmt.Sprintf("%s:%v", ocv1alpha1.ReasonUpgradeFailed, err))
	return ctrl.Result{}, err
	}
	case stateUnchanged:
	if err := ac.Reconcile(rel); err != nil {
	setInstalledStatusConditionFailed(ext, fmt.Sprintf("%s:%v", ocv1alpha1.ReasonResolutionFailed, err))

but we're not doing the same for the upgrade case.