.NET 5 performance regression (~10%) from .NET Core 3.1

[Sergio0694]

Description

I have a .NET Standard 2.0 library implementing a high performance interpreter for the brainf*ck language. At the end of the day it is essentially just a Turing machine, so all the operations being performed are basically accesses to memory, incrementing values, looking up tables, etc. It's all "self-contained", with no external APIs being invoked, IO operations or anything.

I've run some benchmarks on .NET Core 2.1, .NET Core 3.1 and .NET 5 (Preview 4) and noticed that .NET 5 is consistently slower than .NET Core 3.1 in almost all cases. Especially in the mandlebrot test case, which is the most intensive, .NET 5 is almost one second slower, so about 10% slower.

I find this very surprising, as .NET 5 is supposed to be much more optimized than .NET Core 3.1.

NOTE: the solution is not open source for now, but I'd be more than happy to give access to the repo to anyone in the team wanting to take a look, just ping me and I'll send you the invite.

Pinging @EgorBo as you seemed interested in this when I shared some older benchmarks about this on Twitter (here) back when .NET 5 preview 1 had just been released, and cc. @tannergooding.

Configuration

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19619
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.4.20258.7
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT
.NET Core 3.1 : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.22101), X64 RyuJIT
.NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT

Regression?

This is a regression in speed from .NET Core 3.1 to .NET 5 Preview 4, on the same machine.

Data

Runtime	Name	Mean	Error	StdDev
.NET Core 2.1	Division	56.82 us	0.810 us	0.758 us
.NET Core 3.1	Division	41.06 us	0.588 us	0.521 us
.NET Core 5.0	Division	42.95 us	0.511 us	0.453 us
.NET Core 2.1	Fibonacci	21,298.13 us	398.437 us	372.698 us
.NET Core 3.1	Fibonacci	13,197.50 us	65.613 us	58.164 us
.NET Core 5.0	Fibonacci	15,302.14 us	83.934 us	78.512 us
.NET Core 2.1	HelloWorld	17.61 us	0.296 us	0.341 us
.NET Core 3.1	HelloWorld	16.09 us	0.311 us	0.394 us
.NET Core 5.0	HelloWorld	15.68 us	0.305 us	0.510 us
.NET Core 2.1	Mandelbrot	10,221,974.39 us	131,372.509 us	116,458.307 us
.NET Core 3.1	Mandelbrot	7,799,742.74 us	65,822.677 us	58,350.088 us
.NET Core 5.0	Mandelbrot	8,544,272.25 us	8,136.804 us	7,611.172 us
.NET Core 2.1	Multiply	1,204.57 us	14.566 us	12.912 us
.NET Core 3.1	Multiply	792.31 us	5.899 us	4.926 us
.NET Core 5.0	Multiply	790.11 us	1.301 us	1.087 us
.NET Core 2.1	Sum	52.86 us	0.590 us	0.493 us
.NET Core 3.1	Sum	39.53 us	0.620 us	0.580 us
.NET Core 5.0	Sum	37.39 us	0.470 us	0.417 us

category:cq
theme:jit-block-layout
skill-level:expert
cost:large

[Lxiamail]

@Sergio0694 thank you for reporting the issue. From your reported results, besides looks like Division, Fibonacci, and Mandelbrot may have potential regression on .NET5, the other cases have improvements. We can repro the Fibonacci regression on .NET5 and will investigate the root cause. For the other cases, we don't have repro showing regressions on .NET5. Can you share your microbenchmark tests with me so that we can do further investigation?

[Sergio0694]

Hey @Lxiamail - thank you for chiming in!

I'm very confused by what you said here:

"We can repo the Fibonacci regression on .NET5 and will investigate the root cause"

What do you mean with "we can repo the Fibonacci regression"?
All these tests were done specifically with my interpreter for that language as I mentioned, that "Fibonacci" test was not just the classic Fibonacci method written in C# 🤔

Anyway, I've sent you an invite to be added as a collaborato to my private repo Brainf_ckSharp.
To repro the issue, you need to switch to the repro/net5-regression branch.
There you'll find the Brainf_ckSharp.Profiler project (in the "Profiling") folder in the solution.
It's already configured to run with both .NET Core 3.1 and .NET 5.

Let me know if there's anything else I can help you with, thanks again! 😊

[AndyAyersMS]

"Fibonacci" test was not just the classic Fibonacci method written in C#

We got our Fibonaccis crossed. We have an internal benchmark with that name that looks like it might have regressed.

Let me know if there's anything else I can help you with

Have you done any profiling to try and pin down where things got slower?

You can add me as a collaborator.

[Sergio0694]

Ah, that explains it, sorry for the confusion! 😄

@AndyAyersMS I've added you to a collaborator as well!
As mentioned before, the repro is in the repro/net5-regression branch, in the Brainf_ckSharp.Profiler project (under the "Profiling" folder).

I can say that 99.8% of the CPU time is spent in this function:

Brainf_ckSharp.Brainf_ckInterpreter.Release.Run<TExecutionContext>(
 ref TExecutionContext executionContext,
 ref Brainf_ckOperation opcodes,
 ref int jumpTable,
 ref Range functions,
 ref ushort definitions,
 ref StackFrame stackFrames,
 ref int depth,
 ref int totalOperations,
 ref int totalFunctions,
 ref StdinBuffer stdin,
 StdoutBuffer stdout,
 CancellationToken executionToken)
 where TExecutionContext : struct, IMachineStateExecutionContext
{ }

It's in the main Brainf_ckSharp project in the solution.
Hope this helps, let me know if there's anything else I can do on my end!

Thank you again for your time to look into this!

[Lxiamail]

What do you mean with "we can repo the Fibonacci regression"?

Sorry for the typo. I meant repro. :)

[AndyAyersMS]

Here's what I get locally for preview3 vs 3.1.4.

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.836 (1909/November2018Update/19H2)
Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-preview.3.20216.6
 [Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.21406, CoreFX 5.0.20.21406), X64 RyuJIT
 .NET Core 3.1 : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.21406), X64 RyuJIT
 .NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.21406, CoreFX 5.0.20.21406), X64 RyuJIT


| Method | Job | Runtime | Name | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------- |-------------- |-------------- |----------- |----------------:|---------------:|---------------:|----------------:|-------:|-------:|------:|----------:|
| Release | .NET Core 3.1 | .NET Core 3.1 | Division | 46.72 us | 1.221 us | 3.505 us | 45.69 us | 0.4272 | 0.0610 | - | 1.98 KB |
| Release | .NET Core 5.0 | .NET Core 5.0 | Division | 47.10 us | 0.932 us | 1.818 us | 46.78 us | 0.4272 | 0.0610 | - | 1.98 KB |
| | | | | | | | | | | | |
| Release | .NET Core 3.1 | .NET Core 3.1 | Fibonacci | 15,390.92 us | 278.227 us | 320.406 us | 15,340.11 us | - | - | - | 1.88 KB |
| Release | .NET Core 5.0 | .NET Core 5.0 | Fibonacci | 16,290.62 us | 304.622 us | 687.583 us | 16,217.32 us | - | - | - | 1.88 KB |
| | | | | | | | | | | | |
| Release | .NET Core 3.1 | .NET Core 3.1 | Mandelbrot | 9,198,119.60 us | 178,346.549 us | 505,939.475 us | 9,036,080.90 us | - | - | - | 34.92 KB |
| Release | .NET Core 5.0 | .NET Core 5.0 | Mandelbrot | 9,107,098.79 us | 85,771.875 us | 91,774.945 us | 9,107,802.05 us | - | - | - | 56.3 KB |

So Fibonacci seems to have a clear regression; the other two are less definite.

I'll also try this on my coffee lake machine with preview 4.

[Sergio0694]

It's really interesting that the result seems to be different compared than my notebook with the i7 8750H, and between tests as well. The code is pretty much always the same, the only difference is just the script being executed. Very curious to know what the cause of this regression is! 😊

Also if it helps, I've run the benchmark on my desktop as well:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.264 (2004/?/20H1)
AMD Ryzen 7 2700X, 1 CPU, 16 logical and 8 physical cores
.NET Core SDK=5.0.100-preview.4.20258.7
 [Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT
 .NET Core 3.1 : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.22101), X64 RyuJIT
 .NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT


| Method | Job | Runtime | Name | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------- |-------------- |-------------- |----------- |-----------------:|--------------:|--------------:|-------:|-------:|------:|----------:|
| Release | .NET Core 3.1 | .NET Core 3.1 | Division | 58.02 us | 0.425 us | 0.377 us | 0.4272 | 0.0610 | - | 1.98 KB |
| Release | .NET Core 5.0 | .NET Core 5.0 | Division | 57.44 us | 0.218 us | 0.193 us | 0.4272 | 0.0610 | - | 1.98 KB |
| | | | | | | | | | | |
| Release | .NET Core 3.1 | .NET Core 3.1 | Fibonacci | 18,769.40 us | 188.645 us | 176.458 us | - | - | - | 1.88 KB |
| Release | .NET Core 5.0 | .NET Core 5.0 | Fibonacci | 19,512.48 us | 65.521 us | 54.713 us | - | - | - | 1.88 KB |
| | | | | | | | | | | |
| Release | .NET Core 3.1 | .NET Core 3.1 | Mandelbrot | 10,193,671.39 us | 56,002.394 us | 49,644.663 us | - | - | - | 34.92 KB |
| Release | .NET Core 5.0 | .NET Core 5.0 | Mandelbrot | 10,567,216.99 us | 55,330.619 us | 51,756.296 us | - | - | - | 34.92 KB |

Here it looks like both Fibonacci and Mandelbrot have a regression on .NET 5 🤔

[Lxiamail]

v5.0 Preview4 on my desktop machine:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.836 (1909/November2018Update/19H2)
Intel Xeon CPU E5-1650 v3 3.50GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.4.20258.7
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT
.NET Core 3.1 : .NET Core 3.1.4 (CoreCLR 4.700.20.20201, CoreFX 4.700.20.22101), X64 RyuJIT
.NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT

Method	Job	Runtime	Name	Mean	Error	StdDev	Median	Gen 0	Gen 1	Gen 2	Allocated
Release	.NET Core 3.1	.NET Core 3.1	Division	56.76 us	1.134 us	2.781 us	56.35 us	0.2441	-	-	1.98 KB
Release	.NET Core 5.0	.NET Core 5.0	Division	59.53 us	1.188 us	3.067 us	59.58 us	0.2441	0.0610	-	1.98 KB
Release	.NET Core 3.1	.NET Core 3.1	Fibonacci	18,712.15 us	361.648 us	816.299 us	18,670.31 us	-	-	-	1.88 KB
Release	.NET Core 5.0	.NET Core 5.0	Fibonacci	20,263.92 us	405.230 us	799.885 us	20,200.51 us	-	-	-	1.88 KB
Release	.NET Core 3.1	.NET Core 3.1	Mandelbrot	9,068,163.27 us	179,311.793 us	389,809.080 us	8,912,338.50 us	-	-	-	34.92 KB
Release	.NET Core 5.0	.NET Core 5.0	Mandelbrot	8,844,831.10 us	112,542.738 us	93,978.283 us	8,821,120.60 us	-	-	-	34.92 KB

[Lxiamail]

v5.0 preview4 on my laptop:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.836 (1909/November2018Update/19H2)
Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-preview.4.20258.7
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT
.NET Core 3.1 : .NET Core 3.1.2 (CoreCLR 4.700.20.6602, CoreFX 4.700.20.6702), X64 RyuJIT
.NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.25106, CoreFX 5.0.20.25106), X64 RyuJIT

Method	Job	Runtime	Name	Mean	Error	StdDev	Median	Gen 0	Gen 1	Gen 2	Allocated
Release	.NET Core 3.1	.NET Core 3.1	Division	53.78 us	1.883 us	5.524 us	54.77 us	0.4272	0.0610	-	1.98 KB
Release	.NET Core 5.0	.NET Core 5.0	Division	54.75 us	2.043 us	5.861 us	54.89 us	0.4272	0.0610	-	1.98 KB
Release	.NET Core 3.1	.NET Core 3.1	Fibonacci	17,393.43 us	484.839 us	1,383.273 us	18,087.78 us	-	-	-	1.88 KB
Release	.NET Core 5.0	.NET Core 5.0	Fibonacci	19,879.41 us	659.548 us	1,783.126 us	20,369.69 us	-	-	-	1.88 KB
Release	.NET Core 3.1	.NET Core 3.1	Mandelbrot	9,702,039.52 us	320,848.461 us	910,193.682 us	9,478,173.00 us	-	-	-	34.92 KB
Release	.NET Core 5.0	.NET Core 5.0	Mandelbrot	12,128,826.70 us	454,169.554 us	1,310,383.007 us	11,903,840.25 us	-	-	-	34.92 KB

[AndyAyersMS]

On a lot of these run the noise level is fairly high so getting a clear picture of what is going on may be a challenge. My kaby lake seems to have low noise levels so I'm going to start drilling in there.