Re-enable a benchmark was disabled earlier due to an issue by radical · Pull Request #2590 · dotnet/performance

radical · 2022-09-02T01:54:29Z

radical · 2022-09-02T01:55:18Z

CI is not very usable right now, so I can't build this on dotnet/runtime.

radical · 2022-09-02T02:11:23Z

@adamsitnik I'm curious about this from the original issue #2575:

// Benchmark: Perf_BigInteger.ModPow: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [arguments=1024,1024,64 bits]
// *** Execute ***
// Launch: 1 / 1
// Execute: /home/helixbot/.jsvu/v8 --expose_wasm test-main.js -- --run 95a3fee8-6d92-4c8b-8059-369b73cd756f.dll --benchmarkName "System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits)" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0" --benchmarkId 67 in /datadisks/disk1/work/A6C10953/w/B67009A6/e/performance/artifacts/bin/for-running/MicroBenchmarks/95a3fee8-6d92-4c8b-8059-369b73cd756f/bin/net7.0/browser-wasm/AppBundle
Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
Incoming arguments: --run 95a3fee8-6d92-4c8b-8059-369b73cd756f.dll --benchmarkName System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits) --job PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0 --benchmarkId 67
Application arguments: --run 95a3fee8-6d92-4c8b-8059-369b73cd756f.dll --benchmarkName System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits) --job PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0 --benchmarkId 67
console.debug: mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
console.info: Initializing.....
// BeforeAnythingElse
// Benchmark Process Environment Information:
// Runtime=.NET Core (Mono) 8.0.0-ci, Wasm NativeAOT
// GC=Non-concurrent Workstation
// HardwareIntrinsics=
// Job: Job-WFMUPX(PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)

The run starts, and it throws:

// BeforeActualRun
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.ArgumentException: Destination is too short. (Parameter 'destination')
at System.Numerics.BigIntegerCalculator.PowCore(Span`1 , Int32 , ReadOnlySpan`1 , ReadOnlySpan`1 , Span`1 , Span`1 )
at System.Numerics.BigIntegerCalculator.Pow(ReadOnlySpan`1 , ReadOnlySpan`1 , ReadOnlySpan`1 , Span`1 )
at System.Numerics.BigInteger.ModPow(BigInteger , BigInteger , BigInteger )
at System.Numerics.Tests.Perf_BigInteger.ModPow(BigIntegers arguments)
at BenchmarkDotNet.Autogenerated.Runnable_67.WorkloadActionUnroll(Int64 invokeCount)
at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
at BenchmarkDotNet.Engines.EngineStage.RunIteration(IterationMode mode, IterationStage stage, Int32 index, Int64 invokeCount, Int32 unrollFactor)
at BenchmarkDotNet.Engines.EngineActualStage.RunSpecific(Int64 invokeCount, IterationMode iterationMode, Int32 iterationCount, Int32 unrollFactor)
at BenchmarkDotNet.Engines.EngineActualStage.Run(Int64 invokeCount, IterationMode iterationMode, Boolean runAuto, Int32 unrollFactor, Boolean forceSpecific)
at BenchmarkDotNet.Engines.EngineActualStage.RunWorkload(Int64 invokeCount, Int32 unrollFactor, Boolean forceSpecific)
at BenchmarkDotNet.Engines.Engine.Run()
at BenchmarkDotNet.Autogenerated.Runnable_67.Run(IHost host, String benchmarkName)
at System.Reflection.MethodInvoker.InterpretedInvoke(Object , Span`1 , BindingFlags )
--- End of inner exception stack trace ---
at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo )
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args)
// AfterAll
{"name":"ExitStatus","message":"Program terminated with exit(-1)","status":-1}
No Workload Results were obtained from the run.
// Benchmark Process 4941 has exited with code 255.
// ** Remained 60 (46.9%) benchmark(s) to run. Estimated finish 2022-08-24 19:33 (0h 14m from now) **
// ***** BenchmarkRunner: Finish *****
// * Export *
datadisks/disk1/work/A6C10953/w/B67009A6/e/artifacts/BenchmarkDotNet.Artifacts/results/System.Numerics.Tests.Perf_BigInteger-report-github.md
datadisks/disk1/work/A6C10953/w/B67009A6/e/artifacts/BenchmarkDotNet.Artifacts/results/System.Numerics.Tests.Perf_BigInteger-report-full.json
// * Detailed results *
Perf_BigInteger.ToByteArray: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [numberString=-2147483648]
Runtime = .NET Core (Mono) 8.0.0-ci, Wasm NativeAOT; GC = Non-concurrent Workstation
Mean = 406.000 us, StdErr = 0.000 us (0.00%), N = 1, StdDev = 0.000 us
Min = 406.000 us, Q1 = 406.000 us, Median = 406.000 us, Q3 = 406.000 us, Max = 406.000 us
IQR = 0.000 us, LowerFence = 406.000 us, UpperFence = 406.000 us
ConfidenceInterval = [NaN us; NaN us] (CI 99.9%), Margin = NaN us (NaN% of Mean)
Skewness = NaN, Kurtosis = NaN, MValue = 2
-------------------- Histogram --------------------
[406.000 us ; 406.000 us) | @
---------------------------------------------------
Perf_BigInteger.ModPow: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [arguments=1024,1024,64 bits]
Runtime = .NET Core (Mono) 8.0.0-ci, Wasm NativeAOT; GC = Non-concurrent Workstation
There are not any results runs
// * Summary *
BenchmarkDotNet=v0.13.1.1847-nightly, OS=ubuntu 18.04
AMD EPYC 7452, 1 CPU, 2 logical cores and 1 physical core
.NET SDK=7.0.100-rc.2.22423.10
[Host] : .NET 7.0.0 (7.0.22.42212), X64 RyuJIT AVX2
Job-BJXMHI : .NET Core (Mono) 8.0.0-ci, Wasm NativeAOT
PowerPlanMode=00000000-0000-0000-0000-000000000000 Runtime=Wasm Toolchain=Wasm
InvocationCount=1 IterationCount=1 IterationTime=250.0000 ms
MaxIterationCount=20 MinIterationCount=15 RunStrategy=ColdStart
UnrollFactor=1 WarmupCount=0

This shows NA for various columns for ModPow. Does it mean that the benchmark run will get recorded as having data from this run, and have a value of NA, or 0?

| Method | numberString | arguments | Mean | Error | Median | Min | Max | Allocated ||------------ |------------- |------------------ |---------:|------:|---------:|---------:|---------:|----------:|| ToByteArray | -2147483648 | ? | 406.0 us | NA | 406.0 us | 406.0 us | 406.0 us | 104 B || ModPow | ? | 1024,1024,64 bits | NA | NA | NA | NA | NA | - |

Benchmarks with issues:
Perf_BigInteger.ModPow: Job-BJXMHI(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=Wasm, Toolchain=Wasm, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [arguments=1024,1024,64 bits]

The reason I'm looking at this is because of dotnet/perf-autofiling-issues#7979 which says the value improved to 0! And this failure was around the same time.

adamsitnik · 2022-09-02T09:08:31Z

The reason I'm looking at this is because of dotnet/perf-autofiling-issues#7979 which says the value improved to 0! And this failure was around the same time.

@radical I've re-run the benchmark locally and the exported JSON file contains no statistics, but a single measurement (the jitting time). From BDN perspective I think it's a valid output.

@DrewScoggins how the Reporting System handles such cases?

{
"DisplayInfo":"Perf_BigInteger.ModPow: Job-QRRJIW(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [arguments=1024,1024,64 bits]",
"Namespace":"System.Numerics.Tests",
"Type":"Perf_BigInteger",
"Method":"ModPow",
"MethodTitle":"ModPow",
"Parameters":"arguments=1024,1024,64 bits",
"FullName":"System.Numerics.Tests.Perf_BigInteger.ModPow(arguments: 1024,1024,64 bits)",
"HardwareIntrinsics":"AVX2,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256",
"Statistics":null,
"Memory":{
"Gen0Collections":0,
"Gen1Collections":0,
"Gen2Collections":0,
"TotalOperations":0,
"BytesAllocatedPerOperation":0
},
"Measurements":[
{
"IterationMode":"Overhead",
"IterationStage":"Jitting",
"LaunchIndex":1,
"IterationIndex":1,
"Operations":1,
"Nanoseconds":364600
}
]
}

radical · 2022-09-06T19:55:53Z

The issue doesn't seem to have been fixed by dotnet/runtime#74585 . Still getting the same failure in performance-ci on windows - here, and here.

DrewScoggins · 2022-09-06T21:10:35Z

I took a look at the code and the way that we handle this is here,

performance/src/harness/BenchmarkDotNet.Extensions/PerfLabExporter.cs

Lines 33 to 122 in 5f63e8d

foreach(varreportinsummary.Reports)
{
// Skip individual reports with errors
if(report.HasAnyErrors())
{
continue;
}
vartest=newTest();
test.Name=FullNameProvider.GetBenchmarkName(report.BenchmarkCase);
test.Categories=report.BenchmarkCase.Descriptor.Categories;
if(hasCriticalErrors)
{
test.AdditionalData["criticalErrors"]="true";
}
varresults=fromresultinreport.AllMeasurements
whereresult.IterationMode==Engines.IterationMode.Workload&&result.IterationStage==Engines.IterationStage.Result
 orderby result.LaunchIndex,result.IterationIndex
selectnew{result.Nanoseconds,result.Operations};
varoverheadResults=fromresultinreport.AllMeasurements
whereresult.IsOverhead()&&result.IterationStage!=Engines.IterationStage.Jitting
 orderby result.LaunchIndex,result.IterationIndex
selectnew{result.Nanoseconds,result.Operations};
test.Counters.Add(newCounter
{
Name="Duration of single invocation",
TopCounter=true,
DefaultCounter=true,
HigherIsBetter=false,
MetricName="ns",
Results=(fromresultinresults
selectresult.Nanoseconds/result.Operations).ToList()
});
test.Counters.Add(newCounter
{
Name="Overhead invocation",
TopCounter=false,
DefaultCounter=false,
HigherIsBetter=false,
MetricName="ns",
Results=(fromresultinoverheadResults
selectresult.Nanoseconds/result.Operations).ToList()
});
test.Counters.Add(newCounter
{
Name="Duration",
TopCounter=false,
DefaultCounter=false,
HigherIsBetter=false,
MetricName="ms",
Results=(fromresultinresults
selectresult.Nanoseconds).ToList()
});
test.Counters.Add(newCounter
{
Name="Operations",
TopCounter=false,
DefaultCounter=false,
HigherIsBetter=true,
MetricName="Count",
Results=(fromresultinresults
select(double)result.Operations).ToList()
});
foreach(varmetricinreport.Metrics.Keys)
{
varm=report.Metrics[metric];
test.Counters.Add(newCounter
{
Name=m.Descriptor.DisplayName,
TopCounter=false,
DefaultCounter=false,
HigherIsBetter=m.Descriptor.TheGreaterTheBetter,
MetricName=m.Descriptor.Unit,
Results=new[]{m.Value}
});
}
if(disassemblyDiagnoser!=null&&disassemblyDiagnoser.Results.TryGetValue(report.BenchmarkCase,outvardisassemblyResult))
{
stringdisassembly=DiffableDisassemblyExporter.BuildDisassemblyString(disassemblyResult,disassemblyDiagnoser.Config);
test.AdditionalData["disasm"]=disassembly;
}
reporter.AddTest(test);

The gist is that we inspect the BDN object model and pull out results from the Reports property in the Summary object.

var results = from result in report.AllMeasurements
where result.IterationMode == Engines.IterationMode.Workload && result.IterationStage == Engines.IterationStage.Result
orderby result.LaunchIndex, result.IterationIndex
select new { result.Nanoseconds, result.Operations};

adamsitnik · 2022-09-07T06:11:07Z

@DrewScoggins thanks! I've created new issue #2600

DrewScoggins · 2022-09-07T16:56:22Z

The issue doesn't seem to have been fixed by dotnet/runtime#74585 . Still getting the same failure in performance-ci on windows - here, and here.

That is because we have not had an update of the runtime into the installer branch in 16 days, and the fix for this went in 12 days ago. I assume it is so broken because most of the focus is on the release branches these days, but I am not fully sure.

LoopedBard3 · 2022-10-04T20:24:48Z

/azp run

azure-pipelines · 2022-10-04T20:24:59Z

Azure Pipelines successfully started running 1 pipeline(s).

LoopedBard3

LGTM assuming generally clear run

adamsitnik

LGTM, thank you @radical !

Re-enable a benchmark was disabled earlier due to an issue
37e4da0
Issue: dotnet#2575

radical requested a review from adamsitnik September 2, 2022 01:54

radical requested review from LoopedBard3 and lewing September 2, 2022 01:55

radical mentioned this pull request Sep 6, 2022
[perf pipeline] Perf_BigInteger.ModPow failing with System.ArgumentException: Destination is too short. (Parameter 'destination')#2575
Closed

adamsitnik mentioned this pull request Sep 7, 2022
Benchmark that failed to execute should not export any metrics #2600
Open

Merge branch 'dotnet:main' into enable-big-int
40f93c4

LoopedBard3 approved these changes Oct 4, 2022
View reviewed changes

adamsitnik approved these changes Oct 5, 2022
View reviewed changes

adamsitnik merged commit 10d532f into dotnet:mainOct 5, 2022

radical deleted the enable-big-int branch October 5, 2022 09:37

Navigation Menu

Uh oh!

Re-enable a benchmark was disabled earlier due to an issue - #2590

Re-enable a benchmark was disabled earlier due to an issue#2590
adamsitnik merged 2 commits into
dotnet:mainfrom
radical:enable-big-int

radical commented Sep 2, 2022

Uh oh!

radical commented Sep 2, 2022

Uh oh!

radical commented Sep 2, 2022

Uh oh!

adamsitnik commented Sep 2, 2022

Uh oh!

radical commented Sep 6, 2022

Uh oh!

DrewScoggins commented Sep 6, 2022

Uh oh!

adamsitnik commented Sep 7, 2022

Uh oh!

DrewScoggins commented Sep 7, 2022

Uh oh!

LoopedBard3 commented Oct 4, 2022

Uh oh!

azure-pipelinesBot commented Oct 4, 2022

Uh oh!

LoopedBard3 left a comment

Uh oh!

adamsitnik left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

radical commented Sep 2, 2022

Uh oh!

radical commented Sep 2, 2022

Uh oh!

radical commented Sep 2, 2022

Uh oh!

adamsitnik commented Sep 2, 2022

Uh oh!

radical commented Sep 6, 2022

Uh oh!

DrewScoggins commented Sep 6, 2022

Uh oh!

adamsitnik commented Sep 7, 2022

Uh oh!

DrewScoggins commented Sep 7, 2022

Uh oh!

LoopedBard3 commented Oct 4, 2022

Uh oh!

azure-pipelinesBot commented Oct 4, 2022

Uh oh!

LoopedBard3 left a comment

Choose a reason for hiding this comment

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants