tracing/eventpipe/providervalidation/providervalidation/providervalidation.cmd test failure

[ViktorHofer]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=661154
Build error leg or test failing: tracing/eventpipe/providervalidation/providervalidation/providervalidation.cmd
Pull request: #101715

Error Message

Fill the error message using step by step known issues guidance.

{
 "ErrorMessage": "./nptl/futex-internal.c: No such file or directory",
 "ErrorPattern": "",
 "BuildRetry": true,
 "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=661154
Error message validated: [./nptl/futex-internal.c: No such file or directory]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 5/1/2024 8:59:40 AM UTC

Report

Build	Definition	Test	Pull Request
738661	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
738193	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
737434	dotnet/runtime	System.Net.Quic.Functional.Tests.WorkItemExecution	#101624
737604	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#104736
737059	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
736529	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
735640	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
734970	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
734140	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
733547	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
732752	dotnet/runtime	System.Net.WebSockets.Client.Tests.WorkItemExecution	#104543
732589	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
732431	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
732378	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104445
732173	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104445
732179	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
732101	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	#104445
732064	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
732053	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104445
731832	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
731745	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104445
731713	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#104195
731599	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
731168	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
731069	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104445
730972	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
730950	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104445
730474	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
730299	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#102655
729862	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
728987	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
728228	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104309
728178	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
727239	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
726540	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
725778	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
725502	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
2485541	dotnet-runtime	x64.micro_mono.net9.0.Partition9.WorkItemExecution
725288	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
725271	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#103837
725108	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
724846	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
724500	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
724452	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104049
724271	dotnet/runtime	System.Text.Json.SourceGeneration.Roslyn4.4.Tests.WorkItemExecution	#104069
723841	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#104104
723821	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
723303	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#103757
723092	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
723260	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
722988	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#104115
722226	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
721655	dotnet/runtime	System.Numerics.Vectors.Tests.WorkItemExecution	#104049
721660	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
720992	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#103412
720779	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
720689	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#104009
720314	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
720283	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#103757
719372	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
719184	dotnet/runtime	System.Runtime.CompilerServices.Unsafe.Tests.WorkItemExecution	#103915
718997	dotnet/runtime	System.Runtime.CompilerServices.Unsafe.Tests.WorkItemExecution	#103915
719043	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#103757
718755	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
718708	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#103372
717715	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
717491	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
717200	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
717011	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
716673	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
716357	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
716194	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
715648	dotnet/runtime	Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution	#103635
715312	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
715268	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#103617
714740	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
714490	dotnet/runtime	Microsoft.Extensions.Logging.EventSource.Tests.WorkItemExecution	#103655
713808	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
713359	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
711960	dotnet/runtime	Microsoft.Extensions.Logging.EventSource.Tests.WorkItemExecution	#103655
712515	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
712474	dotnet/runtime	System.Private.Uri.Functional.Tests.WorkItemExecution	#102144
712361	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#102655
711823	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#103502
711514	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#103635
711308	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
710961	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#103608
710799	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
710377	dotnet/runtime	Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution	#103565
709764	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
709579	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
709287	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
709132	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
708778	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
708366	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
707642	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
707186	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#103308
707135	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
707003	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#103436
706379	dotnet/runtime	System.Data.Odbc.Tests.WorkItemExecution
Displaying 100 of 104 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	23	104

[tommcdon]

Test output from failing test in case it is useful

Raw output file: /datadisks/disk1/work/9EDF091C/w/A6940942/uploads/eventpipe/providervalidation/providervalidation/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/9EDF091C/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true providervalidation.dll ''
 0.0s: ==TEST STARTING==
 0.2s: Connecting to EventPipe...
 0.2s: Started sending sentinel events...
 0.3s: Creating EventPipeEventSource...
 0.3s: EventPipeEventSource created
 0.3s: Dynamic.All callback registered
 0.3s: Starting stream processing...
 0.4s: Saw new provider 'Microsoft-DotNETCore-SampleProfiler'
 0.4s: Saw sentinel event
 0.4s: Stopped sending sentinel events
 0.4s: Starting event generating action...
 0.4s: Fired MyEvent 0/100,000 times...
 0.5s: Fired MyEvent 10,000/100,000 times...
 0.5s: Saw new provider 'MyEventSource'
 0.5s: Fired MyEvent 20,000/100,000 times...
 0.6s: Fired MyEvent 30,000/100,000 times...
 0.6s: Fired MyEvent 40,000/100,000 times...
 0.7s: Fired MyEvent 50,000/100,000 times...
 0.7s: Fired MyEvent 60,000/100,000 times...
 0.8s: Fired MyEvent 70,000/100,000 times...
 0.8s: Fired MyEvent 80,000/100,000 times...
 0.9s: Fired MyEvent 90,000/100,000 times...
 0.9s: Stopping event generating action
 0.9s: Sending StopTracing command...
 1.0s: Saw new provider 'Microsoft-DotNETCore-EventPipe'
 1.0s: Saw new provider 'Microsoft-Windows-DotNETRuntimeRundown'

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x76e90ccf248f - Unknown
	0x76e90cc9447e - Unknown
	0x76e90cbf87f1 - Unknown
	0x76e90dc42520 - Unknown
	0x41f904f6 - Unknown
 1.1s: Finished StopTracing command

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x41f904f6):0x41f904e6 f9 ff 49 8b c6 49 83 c6 04 49 8b cd 49 83 c5 04 ..I..I...I..I...
0x41f904f6 48 63 09 89 08 41 ff cf 45 85 ff 7f cd 4c 8b 2c Hc...A..E....L.,
0x41f90506 24 4c 8b 74 24 08 4c 8b 7c 24 10 48 83 c4 18 c3 $L.t$.L.|$.H....
0x41f90516 00 00 00 00 00 00 00 00 00 00 48 83 ec 28 4c 89 ..........H..(L.

=================================================================
	Managed Stacktrace:
=================================================================
	 at Microsoft.Diagnostics.Tracing.TraceEvent:CopyBlob <0x00066>
	 at Microsoft.Diagnostics.Tracing.TraceEvent:Clone <0x002e3>
	 at Microsoft.Diagnostics.Tracing.Parsers.ExternalTraceEventParser:<.ctor>b__0_0 <0x000a1>
	 at Microsoft.Diagnostics.Tracing.TraceEventDispatcher:Lookup <0x00355>
	 at Microsoft.Diagnostics.Tracing.EventPipeEventSource:DispatchEventRecord <0x00047>
	 at Microsoft.Diagnostics.Tracing.EventPipeEventSource:EventCache_OnEvent <0x000ef>
	 at Microsoft.Diagnostics.Tracing.EventPipe.EventCache:SortAndDispatch <0x0030a>
	 at Microsoft.Diagnostics.Tracing.EventPipe.EventCache:ProcessSequencePointBlock <0x000bf>
	 at Microsoft.Diagnostics.Tracing.EventPipeSequencePointBlock:ReadBlockContents <0x00083>
	 at Microsoft.Diagnostics.Tracing.EventPipeBlock:FromStream <0x00122>
	 at FastSerialization.Deserializer:ReadObjectDefinition <0x00303>
	 at FastSerialization.Deserializer:ReadObject <0x002df>
	 at Microsoft.Diagnostics.Tracing.EventPipeEventSource:Process <0x000cb>
	 at <>c__DisplayClass9_0:<Validate>b__1 <0x00444>
	 at System.Threading.Tasks.Task:InnerInvoke <0x0006d>
	 at <>c:<.cctor>b__292_0 <0x0002f>
	 at System.Threading.ExecutionContext:RunFromThreadPoolDispatchLoop <0x00086>
	 at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x002d7>
	 at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000a3>
	 at System.Threading.Tasks.Task:ExecuteFromThreadPool <0x00033>
	 at System.Threading.ThreadPoolWorkQueue:Dispatch <0x003ae>
	 at WorkerThread:WorkerThreadStart <0x001bb>
	 at System.Threading.Thread:StartCallback <0x000f0>
	 at System.Object:runtime_invoke_void__this__ <0x00091>
=================================================================

[mdh1418]

The error message regex pattern ./nptl/futex-internal.c: No such file or directory is matching failed builds on CI not related to the providervalidation eventpipe tests.

I ran providervalidation locally, and wasn't able to immediately repro the error Tom provided.

[mdh1418]

From looking at the past 9 build console logs, it looks like the futex-internal.c: No such file or directory is a product of something very similar to what @lambdageek mentioned in #87444 (comment).

All of the 9 logs' thread 1 stacks call ./nptl/pthread_cond_wait.c:627 stemming from src/native/libs/System.Native/pal_threading.c:152/src/mono/mono/mini/interp/interp.c:2368.

So it seems mono specific. Is it possible to highlight the actual reason for the crash from the External Debugger Dump section for mono instead of the misleading futex-internal.c: No such file or directory messages @thaystg @lambdageek?

@carlossanlop did you create a different tracking issue for #87444, or should we just create error message matching at the Managed stacktrace level?

[mdh1418]

Closing for the same reason as #87444, futex-internal.c: No such file or directory is a generic error message.

Instead, issues for CI failures should try to match part of the managed stack trace instead if the Native stacktrace is unknown and the External Debugger dump points towards "futex-internal"