Limit objects cached in webhooks manager

[amisevsk]

What does this PR do?

Similarly to #652, limit pods cached by the webhook server to be only those with the controller.devfile.io/devworkspace_id label. This prevents the webhook server from mirroring all pods on the cluster to memory, causing memory usage to increase when there is a large number of pods in the cluster.

What issues does this PR fix or reference?

Closes#888

Is it tested? How?

1. Verify that memory usage does not increase when creating non-DevWorkspace pods in the cluster
2. Verify that webhooks continue to work as expected:
   • pods/exec is permitted except for workspaces that have the restricted-access annotation, in which case only the creator can access them
   • mutating/validating webhooks continue to work as before

PR Checklist

• E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
  • v8-devworkspace-operator-e2e: DevWorkspace e2e test
  • v8-che-happy-path: Happy path for verification integration with Che

[ibuziuk]

/retest

[ibuziuk]

/retest

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amisevsk, ibuziuk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [amisevsk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

New changes are detected. LGTM label has been removed.

[amisevsk]

CI actually caught an error for once -- earlier this PR was filtering to pods that had the controller.devfile.io/restricted-access label, which is incorrect because it's an annotation (i.e. no pods have it on the cluster). I've force-pushed a change to only filter for pods with the devworkspace_id label.

I've verified this PR in the following way:

1. Start a small cluster
2. Create 100 completed pods in 10 namespaces using the pod template in Webhooks server memory usage depends on number of pods in cluster. #888
3. Check webhooks server memory usage for both the current next image and an image built from this PR

In the next image, I see the webhook server pod consuming ~100Mi, which isn't an OOM situation yet could easily get there (extrapolating 100 pods -> 70Mi memory usage increase, we'd expect to hit OOM at around 380-400 completed pods).

Testing with image quay.io/amisevsk/devworkspace-controller:webhooks-filtering (built from this PR), memory usage in the same situation is around 30Mi.

N.B. if you are testing this, it's necessary to run oc exec into a pod at least once after the webhook server restarts. The watches used internally in controller-runtime are initialized on-demand, so pods aren't mirrored to the internal cache until they are requested by the webhook server. This doesn't occur until the webhook server is asked to verify a pods/exec request.

[amisevsk]

Found another minor issue (unrelated to this PR but a simple fix): it's possible to update the image for containers/initContainers in a pod, and this change is apparently not reverted by replicasets. I've added a check to the webhook server for restricted-access workspaces.