Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName by steveharter · Pull Request #59092 · dotnet/runtime

steveharter · 2021-09-14T15:07:46Z

Change remaining source generator locations that use Compilation.GetTypeByMetadataName to an extension that addresses conflict cases. The guidance and extension implementation is from the Roslyn team.

See #57349

This is a simple 1:1 method replacement; has been previously tested in System.Text.Json (which has a unit test) and by Roslyn.

This may be ported to 6.0.

ghost · 2021-09-14T15:08:02Z

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

Change remaining source generator locations that use Compilation.GetTypeByMetadataName to an extension that address conflict cases. The guidance and extension implementation is from the Roslyn team.

See #57349

This is a simple 1:1 method replacement; has been previously tested in System.Text.Json (which has a unit test) and by Roslyn.

This may be ported to 6.0.

Author:	steveharter
Assignees:	steveharter
Labels:	`area-System.Text.Json`
Milestone:	6.0.0

steveharter · 2021-09-20T19:26:46Z

WASM issue likely caused by #51961

binlog section (crash on publish/ProxyProjectForAOTOnHelix.proj):

/datadisks/disk1/work/AF0109AA/p/build/wasm/WasmApp.Native.targets(306,5): Failed to compile /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -> /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.o
"/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpDfyD7v.tmp
clang version 13.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 5852582532b3eb3ea8da51a1e272d8d017bd36c9)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin
(in-process)
"/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -main-file-name Microsoft.CodeAnalysis.CSharp.dll.bc -mrelocation-model static -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility hidden -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish -resource-dir /datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/lib/clang/13.0.0 -Oz -Werror=implicit-function-declaration -fdebug-compilation-dir=/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish -ferror-limit 19 -fgnuc-version=4.2.1 -fno-inline-functions -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/tmpDfyD7v.tmp -x ir /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc
clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu
emcc: error: '/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpDfyD7v.tmp' failed (received SIGKILL (-9)) [took 356.790s] [/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish/ProxyProjectForAOTOnHelix.proj]

console log:

/datadisks/disk1/work/AAE90973/p/build/wasm/WasmApp.Native.targets(306,5): error : Failed to compile /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -> /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.o [/datadisks/disk1/work/AAE90973/w/B4AD09E0/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/AAE90973/p/build/wasm/WasmApp.Native.targets(306,5): error : "/datadisks/disk1/work/AAE90973/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AAE90973/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpyZM8kY.tmp [/datadisks/disk1/work/AAE90973/w/B4AD09E0/e/publish/ProxyProjectForAOTOnHelix.proj]

lewing · 2021-09-20T19:38:53Z

cc @vargaz

lewing · 2021-09-20T19:41:09Z

Something is killing the processes here, we need to figure out what and why before we can work on solving it.

steveharter · 2021-09-21T18:03:12Z

@lewing unless there is an imminent fix for the crash, can you supply the build condition to avoid the crash? I tried this but the crash is still occurring. Thanks

lewing · 2021-09-21T18:16:47Z

@steveharter have you checked against main after #59352 landed?

lewing · 2021-09-21T18:31:12Z

you shouldn't need a new group for this, the exclusions 4 lines above should be enough but it looks like the paths are incorrect? I see
./Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj ./Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.csproj

The previous section (Condition="'$(TargetsMobile)' == 'true' and '$(TargetOS)' != 'Browser'") also excludes the same tests.
I obtained the conditions from #54833 which is per #51961 (but for System.Text.Json).
The paths appear correct to me and are consistent with other references to them.

shouldn't the path start with Microsoft.Extensions.Logging.Abstractions\tests not Microsoft.Extensions.Logging\tests ?

Yes thanks. Missed that.
@eerhardt it looks like the test exclusion added in 9035d94 has the wrong path. I left it as-is in this PR.

since it's pointing at the wrong path and passing those old exclusions can probably be removed. I'll do it in a separate pr

FWIW Eric fixed that in #59608

steveharter · 2021-09-21T21:45:16Z

@steveharter have you checked against main after #59352 landed?

I just did a rebase and the same crash is occuring.

ericstj · 2021-09-28T21:33:18Z

Looks like test failure is in regex source generator on checked coreclr. May not be related to this change, see #10680. cc @stephentoub

https://dev.azure.com/dnceng/public/_build/results?buildId=1388390&view=ms.vss-test-web.build-test-results-tab&runId=40412074&resultId=192547&paneView=debug

 Timed out at 9/27/2021 8:13:44 PM after 60000ms waiting for remote process.
Wrote mini dump to: C:\h\w\B3E009D5\w\9C35086E\uploads\3232.igz15kp2.52b.dmp
Process ID: 3232
Handle: 1756
Name: dotnet
MainModule: C:\h\w\B3E009D5\p\dotnet.exe
StartTime: 9/27/2021 8:12:44 PM
TotalProcessorTime: 00:01:02.7812500
Threads:
Thread #1 (OS 0x16D0) [MTA]
[PrestubMethodFrame] (Microsoft.CodeAnalysis.GeneratorRunResult.get_GeneratedSources)
Microsoft.CodeAnalysis.GeneratorDriverRunResult+<>c.<get_GeneratedTrees>b__9_0(Microsoft.CodeAnalysis.GeneratorRunResult)
System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[Microsoft.CodeAnalysis.GeneratorRunResult, Microsoft.CodeAnalysis],[System.__Canon, System.Private.CoreLib]].ToArray()
System.Linq.Enumerable.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
System.Collections.Immutable.ImmutableArray.CreateRange[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
Microsoft.CodeAnalysis.GeneratorDriverRunResult.get_GeneratedTrees()
System.Text.RegularExpressions.Tests.RegexGeneratorHelper+<SourceGenRegexAsync>d__6.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(System.String, System.Nullable`1<System.Text.RegularExpressions.RegexOptions>, System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
System.Text.RegularExpressions.Tests.RegexHelpers+<GetRegexAsync>d__8.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(System.Text.RegularExpressions.Tests.RegexEngine, System.String, System.Nullable`1<System.Text.RegularExpressions.RegexOptions>, System.Nullable`1<System.TimeSpan>)
System.Text.RegularExpressions.Tests.RegexMatchTests+<>c+<<Match_ExcessPrefix>b__29_0>d.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
System.Text.RegularExpressions.Tests.RegexMatchTests+<>c.<Match_ExcessPrefix>b__29_0(System.String)
[DebuggerU2MCatchHandlerFrame]
[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
Thread #2 (OS 0x224) [Finalizer] [Background] [MTA]
[DebuggerU2MCatchHandlerFrame]
Thread #3 (OS 0x640) [Background] [MTA]
[DebuggerU2MCatchHandlerFrame]
Thread #5 (OS 0x1434) [Thread pool worker] [Background] [MTA]
[HelperMethodFrame] (System.Threading.WaitHandle.WaitOneCore)
System.Threading.WaitHandle.WaitOneNoCheck(Int32)
System.Threading.WaitHandle.WaitOne()
System.Threading.PortableThreadPool+GateThread.GateThreadStart()
System.Threading.Thread.StartCallback()
[DebuggerU2MCatchHandlerFrame]

ericstj · 2021-09-28T21:34:47Z

I see @stephentoub disabled that test here: 5d71e32

stephentoub · 2021-09-28T21:51:22Z

I see @stephentoub disabled that test here: 5d71e32

Yeah, the actual issue is the test is attributed with a SkipOnCoreClr attribute in order to avoid running on checked runtimes but the test isn't being skipped, as the attribute is being processed in a way different from what was expected. I disabled it while the attribute gets fixed.

ericstj · 2021-09-28T21:53:25Z

@stephentoub while you're here, this PR might be relevant for your RegEx source generator:

runtime/src/libraries/System.Text.RegularExpressions/gen/RegexGenerator.Parser.cs

Lines 49 to 51 in 5bfae1c

// TODO: Use https://github.com/dotnet/runtime/pull/59092
INamedTypeSymbol?regexSymbol=compilation.GetTypeByMetadataName(RegexName);
INamedTypeSymbol?regexGeneratorAttributeSymbol=compilation.GetTypeByMetadataName(RegexGeneratorAttributeName);

edit, looks like I should expand my search by a couple lines.

ericstj · 2021-09-29T15:56:39Z

/azp run runtime

azure-pipelines · 2021-09-29T15:57:13Z

Azure Pipelines successfully started running 1 pipeline(s).

steveharter · 2021-09-29T18:41:49Z

Build failure in Build windows x64 Release SingleFile caused by timeout and not related to this PR.

steveharter · 2021-09-29T18:44:49Z

/backport to release/6.0

github-actions · 2021-09-29T18:45:01Z

Started backporting to release/6.0: https://github.com/dotnet/runtime/actions/runs/1288124568

github-actions · 2021-09-29T18:46:34Z

@steveharter backporting to release/6.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch
Applying: Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
Using index info to reconstruct a base tree...
M	src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
Falling back to patching base and 3-way merge...
Auto-merging src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
CONFLICT (content): Merge conflict in src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

…me (dotnet#59092)

…me (#59092) (#59774)

steveharter added this to the 6.0.0 milestone Sep 14, 2021

steveharter requested review from benaadams, layomia, maryamariyan and sharwell September 14, 2021 15:07

steveharter requested a review from eiriktsarpalis as a code owner September 14, 2021 15:07

steveharter self-assigned this Sep 14, 2021

ghost added the area-System.Text.Json label Sep 14, 2021

steveharter force-pushed the SrcGenGetBestType branch from 6e95673 to 2f8b5cdCompare September 14, 2021 15:11

dotnet deleted a comment from azure-pipelinesBotSep 14, 2021

steveharter removed the area-System.Text.Json label Sep 14, 2021

layomia reviewed Sep 17, 2021
View reviewed changes

Comment thread...ons/tests/Microsoft.Extensions.Logging.Generators.Tests/LoggerMessageGeneratorParserTests.cs Outdated
Comment threadsrc/libraries/System.Text.Json/gen/Reflection/RoslynExtensions.cs Outdated

steveharter force-pushed the SrcGenGetBestType branch 2 times, most recently from db8fb31 to 129b716Compare September 17, 2021 20:50

steveharter mentioned this pull request Sep 20, 2021
[wasm][aot] Link step failing for Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests - wasm-opt getting killed #51961
Closed

steveharter force-pushed the SrcGenGetBestType branch from c06550c to 4dda364Compare September 20, 2021 20:07

lewing reviewed Sep 21, 2021
View reviewed changes

steveharter force-pushed the SrcGenGetBestType branch from f6cf633 to e08eb3dCompare September 21, 2021 19:21

lewing reviewed Sep 23, 2021
View reviewed changes

Comment threadsrc/libraries/tests.proj Outdated

steveharter force-pushed the SrcGenGetBestType branch from 57b2ae9 to 8cf4af3Compare September 23, 2021 16:51

runfoappBot mentioned this pull request Sep 23, 2021
System.Text.RegularExpressions.Tests.RegexMatchTests.Match_ExcessPrefix Failing in CI #59541
Closed

steveharter force-pushed the SrcGenGetBestType branch from 8cf4af3 to 9b20c61Compare September 24, 2021 16:52

eerhardt mentioned this pull request Sep 25, 2021
Remove invalid test exclusions. #59608
Merged

steveharter added 10 commits September 27, 2021 09:35

Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
28db0c6

Fix merge issue
25dcab8

Add unit test to Logging generator
ff67850

Misc test cleanup
a3e6eb1

Attempt to address wasm test failure
9c89de0

temp: remove ref to new code
46b3905

temp: remove ref to new code (attempt 2)
051ed1b

Add back intended code
4d6c7a3

Add test.proj ProjectExclusions
7a4eee9

Fix test exclusion path
8e2ca47

steveharter force-pushed the SrcGenGetBestType branch from 32c3b4b to 8e2ca47Compare September 27, 2021 14:37

Disambiguate Runtime-provided from Roslyn-provided helpers
e1ac8d8

ericstj approved these changes Sep 28, 2021
View reviewed changes

steveharter merged commit 04b6c41 into dotnet:mainSep 29, 2021

steveharter deleted the SrcGenGetBestType branch September 29, 2021 18:42

steveharter added a commit to steveharter/runtime that referenced this pull request Sep 29, 2021
Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataNa…
307b2e4
…me (dotnet#59092)

steveharter mentioned this pull request Sep 29, 2021
[release/6.0] Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName #59774
Merged

danmoseley pushed a commit that referenced this pull request Sep 30, 2021
Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataNa…
9f66ef7
…me (#59092) (#59774)

ghost locked as resolved and limited conversation to collaborators Nov 3, 2021

Uh oh!

Conversation

steveharter commented Sep 14, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Sep 14, 2021

Uh oh!

Uh oh!

Uh oh!

steveharter commented Sep 20, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lewing commented Sep 20, 2021

Uh oh!

lewing commented Sep 20, 2021

Uh oh!

steveharter commented Sep 21, 2021

Uh oh!

lewing commented Sep 21, 2021

Uh oh!

lewingSep 21, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveharterSep 21, 2021

Choose a reason for hiding this comment

Uh oh!

lewingSep 22, 2021

Choose a reason for hiding this comment

Uh oh!

steveharterSep 23, 2021

Choose a reason for hiding this comment

Uh oh!

lewingSep 23, 2021

Choose a reason for hiding this comment

Uh oh!

steveharterSep 29, 2021

Choose a reason for hiding this comment

Uh oh!

steveharter commented Sep 21, 2021

Uh oh!

Uh oh!

ericstj commented Sep 28, 2021

Uh oh!

ericstj commented Sep 28, 2021

Uh oh!

stephentoub commented Sep 28, 2021

Uh oh!

ericstj commented Sep 28, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericstj commented Sep 29, 2021

Uh oh!

azure-pipelinesBot commented Sep 29, 2021

Uh oh!

steveharter commented Sep 29, 2021

Uh oh!

steveharter commented Sep 29, 2021

Uh oh!

github-actionsBot commented Sep 29, 2021

Uh oh!

github-actionsBot commented Sep 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

steveharter commented Sep 14, 2021•
edited
Loading

steveharter commented Sep 20, 2021•
edited
Loading

lewingSep 21, 2021•
edited
Loading

ericstj commented Sep 28, 2021•
edited
Loading