Build the shim assemblies as part of libs.sfx

[ViktorHofer]

• Unblocks Enable Nullable compilation on the Options source gen #88973
• Removes long stand prebuilds in sourcebuild (System.Drawing.Common/7.0.0 and System.Data.SqlClient/4.8.5), cc @MichaelSimons@mthalman
• Adds APICompat build protection to the .NET Framework and .NET Standard shim assemblies

The shims build was intentionally kept out of the libs.sfx subset as some shims reference out-of-band assemblies that also need to be built.

Since that change was made, the shim assemblies now don't reference all out-of-band assemblies by default anymore and instead use fine grained dependencies. Because of that, the number of out-of-band projects referenced and built as part of the shims build is much smaller (~10 projects).

This fixes issues with ouf-of-band source generators not being able to target the live targeting pack. These source generators need the shims (more precisely the netstandard.dll shims) as some of the referenced Microsoft.CodeAnalysis packages don't provide a .NETCoreApp assembly.

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unblocks #88973

The shims build was intentionally kept out of the libs.sfx subset as some shims reference out-of-band assemblies that also need to be built.

Since that change was made, the shim assemblies now don't reference all out-of-band assemblies by default anymore and instead use fine grained dependencies. Because of that, the number of out-of-band projects referenced and built as part of the shims build is much smaller (~10 projects).

This fixes issues with ouf-of-band source generators not being able to target the live targeting pack. These source generators need the shims (more precisely the netstandard.dll shims) as some of the referenced Microsoft.CodeAnalysis packages don't provide a .NETCoreApp assembly.

Author:	ViktorHofer
Assignees:	ViktorHofer
Labels:	area-Infrastructure-libraries
Milestone:	-

[ViktorHofer]

cc @MichalStrehovsky as we chatted about this in the past. You can now build libs.sfx and then immediately build tests as the shim assemblies are now included in that subset build. The only downside is that the subset now builds the few out-of-band assemblies (less than 10) that are referenced by the shims.

[ericstj]

Love to see us adding this infra. Have a few suggestions.

[ericstj]

This is good and my feedback is not blocking. Remaining feedback is naming and testing suggestions.

[ViktorHofer]

Here are the diffs for the impacted shim assemblies. I found a few issues that I fixed with the latest commit:

1. Type forwards for nested classes were missing (thanks for the feedback @ericstj)
2. A few types were located in the wrong assembly destination.

APICompat doesn't seem to protect these type forwards as the left side is missing the destination assemblies. We could protect this path by adding PackageReferences to apicompat.proj for the ~19 missing out-of-band assemblies on the left hand side (baseline).

Assembly	Link
mscorlib	https://www.diffchecker.com/F8KC0BaO/
System.Configuration	https://www.diffchecker.com/DXYwL6A1/
System.Core	https://www.diffchecker.com/LAkoB6ud/
System.Data	https://www.diffchecker.com/w3qtLVAT/
System	https://www.diffchecker.com/FTsmZHKl/
System.Drawing	https://www.diffchecker.com/JJGanOdj/
System.Net	https://www.diffchecker.com/oCadspIw/
System.Runtime.Serialization	https://www.diffchecker.com/5acRzSLc/
System.Security (fixed the assembly reference change)	https://www.diffchecker.com/7fS43tpW/
System.ServiceModel.Web	https://www.diffchecker.com/YoIfxfZv/
System.ServiceProcess	https://www.diffchecker.com/uUe8VKTP/
System.Transactions	https://www.diffchecker.com/77PuXZhY/
WindowsBase	https://www.diffchecker.com/JhpegEzg/

[ericstj]

It looks like you got this all fixed up now, but one thing you might consider is using a project to do an actual restore of the old Microsoft.Windows.Compatibility package and all assets for a particular target framework. That might avoid juggling all the PackageDownload items.

[elinor-fung]

@ViktorHofer the coreclr / mono runtime tests builds appear to be broken with this change. Those legs weren't run in this PR.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=343619&view=logs&jobId=c92f2c34-43c3-5f9c-356c-2e863ce9eb4e

CSC : error CS8785: Generator 'XUnitWrapperGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'NullReferenceException' with message 'Object reference not set to an instance of an object.' [/__w/1/s/src/tests/Interop/COM/ComWrappers/API/ComWrappersTests.csproj] [/__w/1/s/src/tests/build.proj]
/__w/1/s/src/tests/Interop/COM/ComWrappers/API/Program.cs(931,10): error CS0012: The type 'Attribute' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. [/__w/1/s/src/tests/Interop/COM/ComWrappers/API/ComWrappersTests.csproj] [/__w/1/s/src/tests/build.proj]
/__w/1/s/src/tests/Interop/COM/ComWrappers/API/Program.cs(931,10): error CS0012: The type 'Enum' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. [/__w/1/s/src/tests/Interop/COM/ComWrappers/API/ComWrappersTests.csproj] [/__w/1/s/src/tests/build.proj]
/__w/1/s/src/tests/Interop/COM/ComWrappers/API/Program.cs(931,10): error CS0012: The type 'Type' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. [/__w/1/s/src/tests/Interop/COM/ComWrappers/API/ComWrappersTests.csproj] [/__w/1/s/src/tests/build.proj]

Example PRs:
#89091
#88941
#89061

[ViktorHofer]

Taking a look

[ViktorHofer]

Give me 30min and otherwise let's revert my change. Here's my attempt to fix it: #89115

[MichalStrehovsky]

cc @MichalStrehovsky as we chatted about this in the past. You can now build libs.sfx and then immediately build tests as the shim assemblies are now included in that subset build. The only downside is that the subset now builds the few out-of-band assemblies (less than 10) that are referenced by the shims.

This is awesome! Thank you! Submitted #89153 to save money on CI.