Incompatible changes introduced with Diego release 2.109.0

[iaftab-alam]

Current behavior

When upgrading Cloud Foundry deployment from version v45.0.0 to v45.1.0+ or deploying CF v45.1.0, all the applications are crashing because of the breaking changes introduced in Diego release version 2.109. This regression occurs under a specific conditions thats why it went through the normal cf validation pipeline, for instance, if your foundation is offering two stacks(e.g, fs3 and fs4).

! The affected Diego versions are 2.109.0 to 2.112.0.

Steps to reproduce

1. Deploy CF v45.1.0+ with two stacks (cflinuxfs3 and cflinuxfs4).
2. Run CATS against it
3. CATS fail and all the apps crashes.

You will observe that all the applications, start failing with the following error:

  2024-12-26T22:17:15.47+0000 [PROXY/0] ERR /etc/cf-assets/envoy/envoy: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /etc/cf-assets/envoy/envoy)
   2024-12-26T22:17:15.47+0000 [PROXY/0] ERR /etc/cf-assets/envoy/envoy: /lib/x86_64-linux-gnu/libpthread.so.0: version `GLIBC_2.30' not found (required by /etc/cf-assets/envoy/envoy)
   2024-12-26T22:17:15.47+0000 [PROXY/0] ERR /etc/cf-assets/envoy/envoy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /etc/cf-assets/envoy/envoy)
   2024-12-26T22:17:15.56+0000 [PROXY/0] OUT Exit status 1
   2024-12-26T22:17:15.57+0000 [CELL/SSHD/0] OUT Exit status 0
   2024-12-26T22:17:15.57+0000 [APP/PROC/WEB/0] OUT Exit status 143
   2024-12-26T22:17:18.23+0000 [CELL/SSHD/0] OUT Exit status 0
   2024-12-26T22:17:18.24+0000 [APP/PROC/WEB/0] OUT Exit status 143 

When the envoy was bumped?

With Diego release v2.107.0 and v2.108.0, it was still using envoy version 1.28.7 and works well:

https://github.com/cloudfoundry/diego-release/blob/v2.107.0/config/blobs.yml
proxy/envoy-4848d6d548438b50d6d78187a7cfbe61e02d8b85-1.28.7.tgz

However, with Diego release v2.109, envoy was bumped to version 1.32.2 and we observed such behavior:

https://github.com/cloudfoundry/diego-release/blob/v2.109.0/config/blobs.yml
proxy/envoy-a0504e87c5a246cb097b37049b1e4dc7706c2a90-1.32.2.tgz:

Describe how the problem can be fixed?

There has been a detailed explanation provide by Diego Fidalgo: https://cloudfoundry.slack.com/archives/C02FM2BPE/p1735293118448499?thread_ts=1734709809.366679&cid=C02FM2BPE. Why such behavior is observed.

As follow-up, we can confirm that after creating a diego dev release with the code changes suggested. The issue is gone and CATS are passing despite being using the latest envoy version. The following code was patched here

stacks := make([]string, 0, len(rootFSes))
	for stack := range rootFSes {
			stacks = append(stacks, stack)
	}

	sort.Strings(stacks)

	if len(stacks) > 0 {
		mostRecentStack := stacks[len(stacks)-1]
		gardenHealthcheckRootFS = rootFSes[mostRecentStack]
	}

Desired behavior

After the upgrade or deploying CF v45.1.0+, the applications should not crash for a foundation with more then one stack offering.

Quick follow-up: We need to have a special release note that should be added to all the affected CF releases.

Affected Version

2.109.0,2.110.0, 2.111.0,2.112.0

[PlamenDoychev]

There are two aspects of this issue buildpack based apps and docker apps

1. Buildpack apps
   In this case the application developer is allowed to assign stack via the stack property in the app manifest I.e.

   ...
   stack: cflinuxfs4

   Seems like although a given stack is selected this is not exactly true, as sidecar processes like envoy and healthcheck may start using another stack available on the platform (i.e. fs3). This is a bit misleading behaviour from my point of view given i have stack selected.

2. Docker apps
   In this case the stack property is ignored for the main process but it seems like even if the platform operator has specified fs4 as default one the sidecar processes like envoy and healthcheck may pick up fs3 and run on issues. This is also not correct behaviour from my point if view given we have default stack configuration set by the platform operator.

[PlamenDoychev]

Is it an option to downgrade envoy to unblock diego updates and have consistent cf-deployment until we come up with a permanent solution for this case?

[jochenehret]

Hi Plamen,
downgrading a release in cf-deployment is not an option. The update and validation processes and pipelines are designed to integrate new versions only. Pinning an existing version can be done if you know that the next version could have issues (we did this for the uaa-release some time ago).

[PlamenDoychev]

Hi @jochenehret,

I meant is it an option to downgrade envoy in future diego-release. In this case we shall not hit this limitation in the cf-d, because we'll have a new diego-release with downgraded envoy

[jochenehret]

Yes, that would of course be possible.

[jossy]

If I understand this issue correctly, it means that people running CF with multiple stacks can't upgrade their platform. Is there a timeline for a fix or a workaround?

[oliver-heinrich]

Hi @PlamenDoychev, any updates on a fix?

[PlamenDoychev]

Hi @oliver-heinrich, nothing yet to discuss.
We started progress on the issue but lets see when something like PoC will appear.

[ivo1116]

Hello, here is a proposal, the PRs contain a change to the way the FSes are stored, from a map to an array, so that the order can be preserved. They should be listed in an ascending order.

[winkingturtle-vmw]

cflinuxfs3 has been deprecated for nearly 2 years. Why are you proposing this change when the stack is not receiving any updates?

Additionally, this proposal is turning rootfses spec property to an ordered list that I don't think it'd be the intention of it.

[ivo1116]

Hello @winkingturtle-vmw,

Yes, the FS3 is indeed deprecated, but we are still using it in our production environments, because we have customer workloads that have not yet migrated to FS4. According to our corporate product standards we are not allowed to either stop or migrate FS3 running containers to FS4 on behalf of our customers. This must be a decision that they take by themselves.

Yes, we are turning it into an ordered list and that is the point. First, we are supporting multiple stacks and we want to always use the latest available FS for the sidecar processes like healthcheck and envoy. By using an ordered list we ensure that we get the latest one consistently, which does not happen at the moment. At the moment the sidecar processes pick a FS on a random principle, which causes issues - like the latest version of envoy being strictly coupled to FS4

[winkingturtle-vmw]

At the moment the sidecar processes pick a FS on a random principle

@ivo1116 Is it random or is it the first one? It looks these PRs will reverse and pick the last one instead.