HostModel: Retry ResourceUpdate on Win32 error

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.

[swaroop-sridhar]

This change attempts to fix a non-deterministic customer reported failure.

Several customers have observed failure during resource update when the HostModel updates the AppHost (to transfer resources from the managed app). The failure is not detereminisitc, not reproducible on our machines, and depends on specific computers/software running.
This indicates interference by other software while the HostWriter is updating the AppHost.

The current implementation retries the resource update if an update because the device or drive is locked (say by an antivurus) HRESULT 0x21 and 0x6C. However, the failures reported have errors 0x5 (Access violation) and 0x6# (Open failed). Windows/Defender team said that file-locking with these error-codes is not expected. However, different AVs work differently about examining files.

We believe that the correct fix for this issue is to complete:

• To implement Use cross-platform code to update apphost win32 resources, open apphost only once for writing #3828 and Provide offset to placeholder in apphost pack #3829
• Ship AppHost with an extension/permissions not indicating an executable.

However the above is a fairly large change for servicing .net core 3.1.
So, this change implements a simpler fix intended for servicing 3.1 branch: Always retry the resource-update on Win32 error. While this may cause unnecessary retries on legitimate failures (ex: file missing) such cases are rare, because the SDK supplies the apphost, and the HostModel itself creates the file to update.

Fixes#3832

[brcaswell]

Can you provide some insight on how you came up with this list of codes, and whether or not you anticipate more codes to be added upon further review?

[swaroop-sridhar]

I wrote down a list of error code by inspecting the windows error codes, scanning for errors that cannot succeed on retry. The list is conservative; that is, there could be more failures that shouldn't be retried. But I do not anticipate adding more codes to this list. I believe tripping into one of these cases is rare, since the file to be accessed is created by the HostModel library itself.

[stimpy77]

When does this propagate to the wild? I have updated all the NuGet packages and have latest Visual Studio 2019 and it happens to me till I clean and sometimes re-clean and sometimes restart VS, and we also rely on Azure DevOps Pipelines (we haven't switched to .NET Core 3.x in the pipeline but won't till this is fixed).
EDIT: Disabling Trend Micro (antivirus) actually fixed this locally, but it isn't tenable as TM doesn't allow us to wildcard the folders for the exclusion list.

[swaroop-sridhar]

@Anipik when is this release expected to be available? Thanks.

[glasssd]

Is there a beta patch available? Disabling virus protection on our on-prem Azure DevOps Agent workspace folder did not help.