DNR: Wip: Poc/catsrc v2

[varshaprasad96]

This PR attempts to integrate the Catalogd project (which introduces catsrcv1) to Operator controller. It also removes existing support with OLMv0 catsrc CRDs. The following action items are required:

• Introduce new EntitySource that converts v1beta bundles to entities.
  • Write the logic implementing the entity source methods.
  • Modify deppy interface to return an error when bundles are fetched from cluster - Modify the function signature here to return an error along with an entity. ([EntitySource] Modify entity source interface deppy#90)
  • Write unit tests for each of these methods.
• After a catlogd release is out, pin that particular release in go.mod. Right now Go workspaces are being used. To be specific, this commit is being taken into consideration.

Follow up:

• Remove support for v0 catsrc and write e2e tests
• The following code would end up being unused. Remove the following files
  - [ ] Entity to bundle implementation which is already re-written for catsrc v1beta - here.
  - [ ] ListEntities implementation for v0 - here.
  - [ ] Catsrc reconciler implementation. I'd suggest to even remove the reconciler logic and leave it as is, so that we can come back and implement optimization on it later. Having a file of unused code to be implemented in future should be fine.

[joelanford]

Is this part of a deppy interface that we needed to implement? If so, it seems like that interface needs to be updated to accomadate an error being returned.

[varshaprasad96]

yes, this is a deppy interface. The Get signature of EntitySource interface needs to be modified in here (https://github.com/operator-framework/deppy/blob/dc02e928470f892984255923c1cc449fe4a574e8/pkg/deppy/input/entity_source.go#L27) to return an error

[openshift-merge-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[joelanford]

Since catalogd currently uses CRDs for Package and BundleMetadata, we get caching for free if we use the default controller-runtime client. Also, given there is more planned work on the catalogd APIs, I propose the following:

1. Revert the OLMv0 catalog source support. (This ensures we don't have to think about how to handle the presence of both APIs simulatenously)
2. Create an EntitySource implementation that uses a caching controller-runtime client to generate deppy entities on the fly via c-r Client Get and/or List calls for the catalogd Package and BundleMetadata APIS. I think we can get away with not even having a controller that watches the catalogd CatalogSource API. Anytime we reconcile Operator objects, we would just go directly to this c-r client based implementation, fetch the underlying objects directly, and convert them to entities on the fly.

Does this seem like a reasonable path?

[everettraven]

Since catalogd currently uses CRDs for Package and BundleMetadata, we get caching for free if we use the default controller-runtime client. Also, given there is more planned work on the catalogd APIs, I propose the following:

1. Revert the OLMv0 catalog source support. (This ensures we don't have to think about how to handle the presence of both APIs simulatenously)

2. Create an EntitySource implementation that uses a caching controller-runtime client to generate deppy entities on the fly via c-r Client Get and/or List calls for the catalogd Package and BundleMetadata APIS. I think we can get away with not even having a controller that watches the catalogd CatalogSource API. Anytime we reconcile Operator objects, we would just go directly to this c-r client based implementation, fetch the underlying objects directly, and convert them to entities on the fly.

Does this seem like a reasonable path?

I think that sounds reasonable.

If we are getting caching for free that's nice, but does the default caching implementation end up caching all instances of those resources? For example, if I do a Get for a specific BundleMetadata resource, does the cache end up creating a new watch for all BundleMetadata resources? For the time being, are we concerned about large amounts of memory consumption due to caching so many resources (for BundleMetadata we are talking in terms of 1000s)? Are we comfortable saying that the operator-controller could run into OOM issues and it is an optimization we have to make in the future?

[joelanford]

does the cache end up creating a new watch for all BundleMetadata resources

yes

are we concerned about large amounts of memory consumption due to caching so many resources

If it makes operator-controller unusable, that would be a problem, but I think its fine to ignore the memory concerns for now. We can optimize later, IMO.

[anik120]

1 .Revert the OLMv0 catalog source support. (This ensures we don't have to think about how to handle the presence of both APIs simulatenously)
2. Create an EntitySource implementation that uses a caching controller-runtime client to generate deppy entities on the fly via c-r Client Get and/or List calls for the catalogd Package and BundleMetadata APIS. I think we can get away with not even having a controller that watches the catalogd CatalogSource API. Anytime we reconcile Operator objects, we would just go directly to this c-r client based implementation, fetch the underlying objects directly, and convert them to entities on the fly.

#182

[joelanford]

@varshaprasad96, should this be closed now that #182 merged?

[varshaprasad96]

Closing in favour of #182.