.NET 7.0 GA Microbenchmarks Performance Study Report

[dakersnar]

Data

This time we have covered following configurations comparing .NET 7.0 vs. .NET 6.0:

• architectures: arm64, x64, arm
• Unix: raspbian 11, ubuntu 18.04, macOS 13.0, macOS Monterey 12.6
• Windows: 10 & 11

Details

Operating System	Bit	Processor Name
Windows 11	Arm64	Microsoft SQ1 3.0 GHz
macOS 13.0	Arm64	Apple M1
Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Windows 10	X64	Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
raspbian 11	Arm	ARMv7 Processor rev 3 (v7l)
macOS Monterey 12.6	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @adamsitnik, @jozkee, @janvorli, @jeffhandley, @mrsharm, and @cincuranet who contributed their results and time.

The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871.

The RC2 vs 6.0 report can be found here. As that report almost acted as a release-to-release comparison, and contains a different spread of configs, it continues to be a valuable resource when evaluating .NET 7 performance.

Improvements

There are too many improvements to individually call out (what a great problem to have!). Instead, I collected the top 300 in an excel document (7.0-GA-Report-Top-300.zip). All high-level analysis of this data should be taken with a grain of salt, and I highly encourage readers to examine the raw data for specifics on individual benchmarks (start at the bottom for improvements). Additionally, the previous report was run with a different set of machines, so it may be worth looking at that data as well.

The most improved grouping of benchmarks was System.Text.RegularExpressions.Tests.Perf_Regex*, making up 103 of the top 300, and 56 of the top 100. Feel free to sort the table A to Z in excel to see other groupings of tests.

Just for fun: the winner of the "most improved benchmark" award is System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None), with a staggering 400x improvement.

Details

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)

Result	Ratio	Alloc Delta	Operating System	Bit	Processor Name	Modality
Faster	400.11	-29756	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Faster	433.10	-29646	macOS 13.0	Arm64	Apple M1
Faster	463.42	-29656	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	464.66	-29756	Windows 10	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	several?
Faster	447.08	-29756	Windows 10	X64	Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Faster	525.32	-29656	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Faster	420.86	-29608	Windows 11	X64	11th Gen Intel Core i9-11900 2.50GHz
Faster	418.30	-29756	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	232.79	-18278	raspbian 11	Arm	ARMv7 Processor rev 3 (v7l)	bimodal
Faster	403.98	-29756	macOS Monterey 12.6	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)

For a more in-depth look at .NET 7 performance improvements, check out Stephen Toub's writeup here: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/. To quote his TL;DR:

.NET 7 is fast. Really fast. A thousand performance-impacting PRs went into runtime and core libraries this release, never mind all the improvements in ASP.NET Core and Windows Forms and Entity Framework and beyond. It’s the fastest .NET ever. If your manager asks you why your project should upgrade to .NET 7, you can say “in addition to all the new functionality in the release, .NET 7 is super fast.”

Thank you to all contributors for making 7.0 the fastest .NET yet!

Regressions

For regressions, this report focused on what was not called out already by the RC2 report. Consider the combination of the two reports to be a total view of .NET 7's regressions compared to .NET 6.

Fixed since RC2

As part of writing this report, I double checked all regressions called out in the previous RC2 report. Assume all issues in the previous report are unresolved unless called out here:

• Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty
  • No longer shown as regressing
  • Added comment: [Perf] Linux/x64: 16 Regressions on 9/22/2022 3:16:12 AM #77900 (comment)
• System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024]
  • No longer shown as regressing
  • Added comment [Perf] Windows/x64: 188 Regressions from GC changes #74014 (comment)

Investigation In Progress

• System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite*
  • These benchmarks were recently added here: Add a JsonNode roundtrip benchmark performance#2714
  • They were added after report from a user: .net 7 cycles Performance slow #78089
  • The regression was fixed and backported here: Fix expensive typo in JsonNode #78130
  • Added comment: Fix expensive typo in JsonNode #78130 (comment)
• System.Numerics.Tests.Perf_Matrix4x4.AddOperatorBenchmark
  • Added comment: System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191 (comment)
• BenchmarksGame.SpectralNorm_3.RunBench
  • Detected here, added comment: Regression in BenchmarksGame.SpectralNorm_3.RunBench #79178 (comment)
  • I believe this slipped past the perf team.
• System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True), System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)
  • The data does not indicate this has been solved, as opposed to what we stated last time
  • Added comment: GC.AllocateUninitializedArray has regressed on macOS #65198 (comment)
• System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic), System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
  • Seems to still not be fixed on MacOS Arm64
  • Added update comment: [Perf] Regressions in System.Text.Perf_Utf8Encoding #68405 (comment)

By Design

• System.Diagnostics.Perf_Process.GetProcessById
  • Reported in the Preview 3 report: .NET 7.0 Preview 3 Microbenchmarks Performance Study Report #67976
  • Explained as "by design" here: System.Diagnostics.Perf_Process.GetProcessById has regressed 50% on Windows #67847 (comment)
• System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: True, chunkedResponse: True, responseLength: 100000),
  System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: True, responseLength: 100000),
  System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: False, responseLength: 100000)
  • By design, reported and explained here: Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest #66343 (comment)
  • Added comment: Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest #66343 (comment)

Noise, Flaky or Multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
• System.Numerics.Tests.Constructor.ConstructorBenchmark_SByte
• System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)
  • Might be bimodal, but seems to have been resolved regardless
• Perf_VectorOf*
  • Had a lot of regressions on the Linux devices we tested. However, there are not enough data points to make a conclusion as these benchmarks are historically noisy.

Statistics

Total: 45809
Same: 57.48 %
Slower: 5.99 %
Faster: 24.58 %
Noise: 11.91 %
Unknown: 0.04 %

Statistics per Architecture

Architecture	Same	Slower	Faster	Noise	Unknown
Arm64	49.81 %	3.62 %	35.66 %	10.88 %	0.03 %
Arm	51.72 %	9.15 %	35.27 %	3.79 %	0.07 %
X64	60.45 %	6.20 %	19.95 %	13.36 %	0.04 %

Statistics per Operating System

Operating System	Same	Slower	Faster	Noise	Unknown
macOS 13.0	45.38 %	2.30 %	39.90 %	12.36 %	0.07 %
Windows 11	60.21 %	4.65 %	23.37 %	11.75 %	0.03 %
raspbian 11	51.72 %	9.15 %	35.27 %	3.79 %	0.07 %
macOS Monterey 12.6	60.96 %	3.45 %	20.07 %	15.43 %	0.09 %
ubuntu 18.04	59.23 %	13.35 %	18.99 %	8.38 %	0.04 %
Windows 10	58.81 %	5.86 %	20.69 %	14.62 %	0.03 %

Statistics per Namespace

Details

Namespace	Same	Slower	Faster	Noise	Unknown
Benchmark	12.50 %	0.00 %	87.50 %	0.00 %	0.00 %
BenchmarksGame	70.48 %	9.05 %	20.48 %	0.00 %	0.00 %
Benchstone.BenchF	76.50 %	8.50 %	15.00 %	0.00 %	0.00 %
Benchstone.BenchI	62.08 %	10.42 %	27.50 %	0.00 %	0.00 %
Benchstone.MDBenchF	38.00 %	2.00 %	60.00 %	0.00 %	0.00 %
Benchstone.MDBenchI	21.00 %	12.00 %	67.00 %	0.00 %	0.00 %
Devirtualization	82.00 %	2.00 %	0.00 %	16.00 %	0.00 %
Exceptions	72.78 %	2.78 %	24.44 %	0.00 %	0.00 %
FractalPerf	0.00 %	90.00 %	10.00 %	0.00 %	0.00 %
GuardedDevirtualization	84.76 %	0.00 %	0.00 %	15.24 %	0.00 %
IfStatements	53.00 %	0.00 %	47.00 %	0.00 %	0.00 %
Inlining	36.67 %	0.00 %	10.00 %	53.33 %	0.00 %
Interop	96.67 %	0.00 %	3.33 %	0.00 %	0.00 %
JetStream	35.00 %	5.00 %	60.00 %	0.00 %	0.00 %
Layout	90.00 %	0.00 %	10.00 %	0.00 %	0.00 %
Lowering	90.00 %	0.00 %	0.00 %	10.00 %	0.00 %
MicroBenchmarks.Serializers	67.02 %	2.82 %	30.15 %	0.00 %	0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance	77.50 %	7.50 %	15.00 %	0.00 %	0.00 %
Microsoft.Extensions.Caching.Memory.Tests	14.55 %	0.00 %	85.45 %	0.00 %	0.00 %
Microsoft.Extensions.Configuration	0.00 %	0.00 %	100.00 %	0.00 %	0.00 %
Microsoft.Extensions.Configuration.Xml	77.50 %	0.00 %	22.50 %	0.00 %	0.00 %
Microsoft.Extensions.DependencyInjection	34.19 %	0.97 %	62.26 %	2.58 %	0.00 %
Microsoft.Extensions.Logging	69.43 %	2.57 %	24.57 %	3.43 %	0.00 %
Microsoft.Extensions.Primitives.Performance	47.50 %	2.50 %	3.75 %	46.25 %	0.00 %
Microsoft.Extensions.Primitives	38.89 %	2.78 %	21.67 %	36.67 %	0.00 %
PerfLabTests	59.75 %	9.63 %	22.88 %	7.75 %	0.00 %
PerfLabTests.CastingPerf2	54.67 %	8.00 %	37.33 %	0.00 %	0.00 %
SciMark2	56.00 %	2.00 %	42.00 %	0.00 %	0.00 %
SIMD	88.89 %	0.00 %	11.11 %	0.00 %	0.00 %
Span	82.80 %	1.60 %	15.60 %	0.00 %	0.00 %
StoreBlock	54.00 %	0.33 %	2.67 %	43.00 %	0.00 %
System.Buffers.Binary.Tests	50.00 %	3.75 %	20.00 %	26.25 %	0.00 %
System.Buffers.Tests	56.90 %	6.47 %	31.47 %	5.17 %	0.00 %
System.Buffers.Text.Tests	54.41 %	4.41 %	21.18 %	20.00 %	0.00 %
System.Collections	68.91 %	4.71 %	24.86 %	1.51 %	0.00 %
System.Collections.Concurrent	74.20 %	8.24 %	7.45 %	10.11 %	0.00 %
System.Collections.Tests	55.61 %	4.49 %	26.17 %	13.74 %	0.00 %
System.ComponentModel.Tests	57.27 %	0.00 %	42.73 %	0.00 %	0.00 %
System.ConsoleTests	88.33 %	8.33 %	3.33 %	0.00 %	0.00 %
System.Diagnostics	48.50 %	7.00 %	44.50 %	0.00 %	0.00 %
System.Diagnostics.Tracing	85.00 %	2.50 %	12.50 %	0.00 %	0.00 %
System.Formats.Cbor.Tests	72.31 %	0.77 %	26.92 %	0.00 %	0.00 %
System.Globalization.Tests	78.24 %	7.63 %	12.29 %	1.83 %	0.00 %
System	70.00 %	1.67 %	25.56 %	2.78 %	0.00 %
System.IO.Compression	85.69 %	3.47 %	10.83 %	0.00 %	0.00 %
System.IO.MemoryMappedFiles.Tests	32.76 %	3.45 %	63.79 %	0.00 %	0.00 %
System.IO.Pipelines.Tests	75.00 %	2.50 %	22.50 %	0.00 %	0.00 %
System.IO.Pipes.Tests	80.00 %	13.33 %	6.67 %	0.00 %	0.00 %
System.IO.Tests	66.65 %	7.08 %	23.76 %	2.51 %	0.00 %
System.Linq.Tests	77.52 %	7.82 %	13.86 %	0.79 %	0.00 %
System.MathBenchmarks	85.94 %	3.91 %	8.91 %	1.25 %	0.00 %
System.Memory	47.78 %	4.23 %	21.01 %	26.98 %	0.00 %
System.Net.Http.Tests	45.83 %	40.83 %	13.33 %	0.00 %	0.00 %
System.Net.NetworkInformation.Tests	60.00 %	5.00 %	35.00 %	0.00 %	0.00 %
System.Net.Primitives.Tests	63.18 %	5.00 %	17.27 %	14.55 %	0.00 %
System.Net.Security.Tests	80.14 %	4.79 %	15.07 %	0.00 %	0.00 %
System.Net.Sockets.Tests	75.51 %	6.12 %	18.37 %	0.00 %	0.00 %
System.Net.Tests	86.67 %	3.33 %	10.00 %	0.00 %	0.00 %
System.Net.WebSockets.Tests	65.00 %	25.00 %	10.00 %	0.00 %	0.00 %
System.Numerics.Tests	39.67 %	8.64 %	11.08 %	40.61 %	0.00 %
System.Reflection	21.85 %	3.57 %	71.43 %	3.15 %	0.00 %
System.Runtime.InteropServices.Tests	85.00 %	0.00 %	15.00 %	0.00 %	0.00 %
System.Runtime.Serialization.Formatters.Tests	70.00 %	20.00 %	10.00 %	0.00 %	0.00 %
System.Security.Cryptography.Primitives.Tests.Performance	87.50 %	7.50 %	5.00 %	0.00 %	0.00 %
System.Security.Cryptography.Tests	68.89 %	5.56 %	25.56 %	0.00 %	0.00 %
System.Security.Cryptography.X509Certificates.Tests	83.33 %	10.00 %	6.67 %	0.00 %	0.00 %
System.Tests	55.60 %	5.46 %	27.84 %	11.10 %	0.00 %
System.Text.Encodings.Web.Tests	76.67 %	1.67 %	21.67 %	0.00 %	0.00 %
System.Text.Json.Document.Tests	63.00 %	0.40 %	36.60 %	0.00 %	0.00 %
System.Text.Json.Node.Tests	0.00 %	100.00 %	0.00 %	0.00 %	0.00 %
System.Text.Json.Reader.Tests	57.50 %	0.00 %	42.50 %	0.00 %	0.00 %
System.Text.Json.Serialization.Tests	72.03 %	1.46 %	26.51 %	0.00 %	0.00 %
System.Text.Json.Tests	71.71 %	2.54 %	25.49 %	0.26 %	0.00 %
System.Text	78.67 %	14.00 %	7.33 %	0.00 %	0.00 %
System.Text.RegularExpressions.Tests	15.25 %	3.46 %	80.31 %	0.00 %	0.97 %
System.Text.Tests	66.90 %	11.67 %	18.81 %	2.62 %	0.00 %
System.Threading.Channels.Tests	67.50 %	8.33 %	24.17 %	0.00 %	0.00 %
System.Threading.Tasks.Tests	43.33 %	16.67 %	33.33 %	6.67 %	0.00 %
System.Threading.Tasks	75.00 %	2.14 %	20.71 %	2.14 %	0.00 %
System.Threading.Tests	72.98 %	2.34 %	15.11 %	9.57 %	0.00 %
System.Xml.Linq	58.95 %	2.63 %	28.95 %	9.47 %	0.00 %
System.Xml.Tests	0.00 %	0.00 %	100.00 %	0.00 %	0.00 %
V8.Crypto	90.00 %	10.00 %	0.00 %	0.00 %	0.00 %
V8.Richards	80.00 %	0.00 %	20.00 %	0.00 %	0.00 %
XmlDocumentTests.XmlDocumentTests	63.33 %	6.67 %	13.33 %	16.67 %	0.00 %
XmlDocumentTests.XmlNodeListTests	95.00 %	5.00 %	0.00 %	0.00 %	0.00 %
XmlDocumentTests.XmlNodeTests	30.00 %	0.00 %	0.00 %	70.00 %	0.00 %
System.Drawing.Tests	84.17 %	0.83 %	10.00 %	5.00 %	0.00 %
HardwareIntrinsics.RayTracer	85.71 %	14.29 %	0.00 %	0.00 %	0.00 %

Big thanks to everyone involved!

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Data

This time we have covered following configurations comparing .NET 7.0 vs. .NET 6.0:

• architectures: arm64, x64, arm
• Unix: raspbian 11, ubuntu 18.04, macOS 13.0, macOS Monterey 12.6
• Windows: 10 & 11

Details

Operating System	Bit	Processor Name
Windows 11	Arm64	Microsoft SQ1 3.0 GHz
macOS 13.0	Arm64	Apple M1
Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Windows 10	X64	Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
raspbian 11	Arm	ARMv7 Processor rev 3 (v7l)
macOS Monterey 12.6	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @adamsitnik, @jozkee, @janvorli, @jeffhandley, @mrsharm, and @cincuranet who contributed their results and time.

The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871.

The RC2 vs 6.0 report can be found here. As that report almost acted as a release-to-release comparison, and contains a different spread of configs, it continues to be a valuable resource when evaluating .NET 7 performance.

Improvements

There are too many improvements to individually call out (what a great problem to have!). Instead, I collected the top 300 in an excel document (7.0-GA-Report-Top-300.zip). All high-level analysis of this data should be taken with a grain of salt, and I highly encourage readers to examine the raw data for specifics on individual benchmarks (start at the bottom for improvements). Additionally, the previous report was run with a different set of machines, so it may be worth looking at that data as well.

The most improved grouping of benchmarks was System.Text.RegularExpressions.Tests.Perf_Regex*, making up 103 of the top 300, and 56 of the top 100. Feel free to sort the table A to Z in excel to see other groupings of tests.

Just for fun: the winner of the "most improved benchmark" award is System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None), with a staggering 400x improvement.

Details

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)

Result	Ratio	Alloc Delta	Operating System	Bit	Processor Name	Modality
Faster	400.11	-29756	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Faster	433.10	-29646	macOS 13.0	Arm64	Apple M1
Faster	463.42	-29656	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	464.66	-29756	Windows 10	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	several?
Faster	447.08	-29756	Windows 10	X64	Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Faster	525.32	-29656	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Faster	420.86	-29608	Windows 11	X64	11th Gen Intel Core i9-11900 2.50GHz
Faster	418.30	-29756	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	232.79	-18278	raspbian 11	Arm	ARMv7 Processor rev 3 (v7l)	bimodal
Faster	403.98	-29756	macOS Monterey 12.6	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)

For a more in-depth look at .NET 7 performance improvements, check out Stephen Toub's writeup here: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/. To quote his TL;DR:

.NET 7 is fast. Really fast. A thousand performance-impacting PRs went into runtime and core libraries this release, never mind all the improvements in ASP.NET Core and Windows Forms and Entity Framework and beyond. It’s the fastest .NET ever. If your manager asks you why your project should upgrade to .NET 7, you can say “in addition to all the new functionality in the release, .NET 7 is super fast.”

Thank you to all contributors for making 7.0 the fastest .NET yet!

Regressions

For regressions, this report focused on what was not called out already by the RC2 report. Consider the combination of the two reports to be a total view of .NET 7's regressions compared to .NET 6.

Fixed since RC2

As part of writing this report, I double checked all regressions called out in the previous RC2 report. Assume all issues in the previous report are unresolved unless called out here:

• Benchmark.GetChildKeysTests.AddChainedConfigurationEmpty
  • No longer shown as regressing
  • Added comment: [Perf] Linux/x64: 16 Regressions on 9/22/2022 3:16:12 AM #77900 (comment)
• System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0]),
  System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024]
  • No longer shown as regressing
  • Added comment [Perf] Windows/x64: 188 Regressions from GC changes #74014 (comment)

Investigation In Progress

• System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite*
  • These benchmarks were recently added here: Add a JsonNode roundtrip benchmark performance#2714
  • They were added after report from a user: .net 7 cycles Performance slow #78089
  • The regression was fixed and backported here: Fix expensive typo in JsonNode #78130
  • Added comment: Fix expensive typo in JsonNode #78130 (comment)
• System.Numerics.Tests.Perf_Matrix4x4.AddOperatorBenchmark
  • Added comment: System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191 (comment)
• BenchmarksGame.SpectralNorm_3.RunBench
  • Detected here, added comment: Regression in BenchmarksGame.SpectralNorm_3.RunBench #79178 (comment)
  • I believe this slipped past the perf team.
• System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True), System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)
  • The data does not indicate this has been solved, as opposed to what we stated last time
  • Added comment: GC.AllocateUninitializedArray has regressed on macOS #65198 (comment)
• System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic), System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
  • Seems to still not be fixed on MacOS Arm64
  • Added update comment: [Perf] Regressions in System.Text.Perf_Utf8Encoding #68405 (comment)

By Design

• System.Diagnostics.Perf_Process.GetProcessById
  • Reported in the Preview 3 report: .NET 7.0 Preview 3 Microbenchmarks Performance Study Report #67976
  • Explained as "by design" here: System.Diagnostics.Perf_Process.GetProcessById has regressed 50% on Windows #67847 (comment)
• System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: True, chunkedResponse: True, responseLength: 100000),
  System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: True, responseLength: 100000),
  System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: False, responseLength: 100000)
  • By design, reported and explained here: Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest #66343 (comment)
  • Added comment: Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest #66343 (comment)

Noise, Flaky or Multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
• System.Numerics.Tests.Constructor.ConstructorBenchmark_SByte
• System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)
  • Might be bimodal, but seems to have been resolved regardless
• Perf_VectorOf*
  • Had a lot of regressions on the Linux devices we tested. However, there are not enough data points to make a conclusion as these benchmarks are historically noisy.

Statistics

Total: 45809
Same: 57.48 %
Slower: 5.99 %
Faster: 24.58 %
Noise: 11.91 %
Unknown: 0.04 %

Statistics per Architecture

Architecture	Same	Slower	Faster	Noise	Unknown
Arm64	49.81 %	3.62 %	35.66 %	10.88 %	0.03 %
Arm	51.72 %	9.15 %	35.27 %	3.79 %	0.07 %
X64	60.45 %	6.20 %	19.95 %	13.36 %	0.04 %

Statistics per Operating System

Operating System	Same	Slower	Faster	Noise	Unknown
macOS 13.0	45.38 %	2.30 %	39.90 %	12.36 %	0.07 %
Windows 11	60.21 %	4.65 %	23.37 %	11.75 %	0.03 %
raspbian 11	51.72 %	9.15 %	35.27 %	3.79 %	0.07 %
macOS Monterey 12.6	60.96 %	3.45 %	20.07 %	15.43 %	0.09 %
ubuntu 18.04	59.23 %	13.35 %	18.99 %	8.38 %	0.04 %
Windows 10	58.81 %	5.86 %	20.69 %	14.62 %	0.03 %

Statistics per Namespace

Details

Namespace	Same	Slower	Faster	Noise	Unknown
Benchmark	12.50 %	0.00 %	87.50 %	0.00 %	0.00 %
BenchmarksGame	70.48 %	9.05 %	20.48 %	0.00 %	0.00 %
Benchstone.BenchF	76.50 %	8.50 %	15.00 %	0.00 %	0.00 %
Benchstone.BenchI	62.08 %	10.42 %	27.50 %	0.00 %	0.00 %
Benchstone.MDBenchF	38.00 %	2.00 %	60.00 %	0.00 %	0.00 %
Benchstone.MDBenchI	21.00 %	12.00 %	67.00 %	0.00 %	0.00 %
Devirtualization	82.00 %	2.00 %	0.00 %	16.00 %	0.00 %
Exceptions	72.78 %	2.78 %	24.44 %	0.00 %	0.00 %
FractalPerf	0.00 %	90.00 %	10.00 %	0.00 %	0.00 %
GuardedDevirtualization	84.76 %	0.00 %	0.00 %	15.24 %	0.00 %
IfStatements	53.00 %	0.00 %	47.00 %	0.00 %	0.00 %
Inlining	36.67 %	0.00 %	10.00 %	53.33 %	0.00 %
Interop	96.67 %	0.00 %	3.33 %	0.00 %	0.00 %
JetStream	35.00 %	5.00 %	60.00 %	0.00 %	0.00 %
Layout	90.00 %	0.00 %	10.00 %	0.00 %	0.00 %
Lowering	90.00 %	0.00 %	0.00 %	10.00 %	0.00 %
MicroBenchmarks.Serializers	67.02 %	2.82 %	30.15 %	0.00 %	0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance	77.50 %	7.50 %	15.00 %	0.00 %	0.00 %
Microsoft.Extensions.Caching.Memory.Tests	14.55 %	0.00 %	85.45 %	0.00 %	0.00 %
Microsoft.Extensions.Configuration	0.00 %	0.00 %	100.00 %	0.00 %	0.00 %
Microsoft.Extensions.Configuration.Xml	77.50 %	0.00 %	22.50 %	0.00 %	0.00 %
Microsoft.Extensions.DependencyInjection	34.19 %	0.97 %	62.26 %	2.58 %	0.00 %
Microsoft.Extensions.Logging	69.43 %	2.57 %	24.57 %	3.43 %	0.00 %
Microsoft.Extensions.Primitives.Performance	47.50 %	2.50 %	3.75 %	46.25 %	0.00 %
Microsoft.Extensions.Primitives	38.89 %	2.78 %	21.67 %	36.67 %	0.00 %
PerfLabTests	59.75 %	9.63 %	22.88 %	7.75 %	0.00 %
PerfLabTests.CastingPerf2	54.67 %	8.00 %	37.33 %	0.00 %	0.00 %
SciMark2	56.00 %	2.00 %	42.00 %	0.00 %	0.00 %
SIMD	88.89 %	0.00 %	11.11 %	0.00 %	0.00 %
Span	82.80 %	1.60 %	15.60 %	0.00 %	0.00 %
StoreBlock	54.00 %	0.33 %	2.67 %	43.00 %	0.00 %
System.Buffers.Binary.Tests	50.00 %	3.75 %	20.00 %	26.25 %	0.00 %
System.Buffers.Tests	56.90 %	6.47 %	31.47 %	5.17 %	0.00 %
System.Buffers.Text.Tests	54.41 %	4.41 %	21.18 %	20.00 %	0.00 %
System.Collections	68.91 %	4.71 %	24.86 %	1.51 %	0.00 %
System.Collections.Concurrent	74.20 %	8.24 %	7.45 %	10.11 %	0.00 %
System.Collections.Tests	55.61 %	4.49 %	26.17 %	13.74 %	0.00 %
System.ComponentModel.Tests	57.27 %	0.00 %	42.73 %	0.00 %	0.00 %
System.ConsoleTests	88.33 %	8.33 %	3.33 %	0.00 %	0.00 %
System.Diagnostics	48.50 %	7.00 %	44.50 %	0.00 %	0.00 %
System.Diagnostics.Tracing	85.00 %	2.50 %	12.50 %	0.00 %	0.00 %
System.Formats.Cbor.Tests	72.31 %	0.77 %	26.92 %	0.00 %	0.00 %
System.Globalization.Tests	78.24 %	7.63 %	12.29 %	1.83 %	0.00 %
System	70.00 %	1.67 %	25.56 %	2.78 %	0.00 %
System.IO.Compression	85.69 %	3.47 %	10.83 %	0.00 %	0.00 %
System.IO.MemoryMappedFiles.Tests	32.76 %	3.45 %	63.79 %	0.00 %	0.00 %
System.IO.Pipelines.Tests	75.00 %	2.50 %	22.50 %	0.00 %	0.00 %
System.IO.Pipes.Tests	80.00 %	13.33 %	6.67 %	0.00 %	0.00 %
System.IO.Tests	66.65 %	7.08 %	23.76 %	2.51 %	0.00 %
System.Linq.Tests	77.52 %	7.82 %	13.86 %	0.79 %	0.00 %
System.MathBenchmarks	85.94 %	3.91 %	8.91 %	1.25 %	0.00 %
System.Memory	47.78 %	4.23 %	21.01 %	26.98 %	0.00 %
System.Net.Http.Tests	45.83 %	40.83 %	13.33 %	0.00 %	0.00 %
System.Net.NetworkInformation.Tests	60.00 %	5.00 %	35.00 %	0.00 %	0.00 %
System.Net.Primitives.Tests	63.18 %	5.00 %	17.27 %	14.55 %	0.00 %
System.Net.Security.Tests	80.14 %	4.79 %	15.07 %	0.00 %	0.00 %
System.Net.Sockets.Tests	75.51 %	6.12 %	18.37 %	0.00 %	0.00 %
System.Net.Tests	86.67 %	3.33 %	10.00 %	0.00 %	0.00 %
System.Net.WebSockets.Tests	65.00 %	25.00 %	10.00 %	0.00 %	0.00 %
System.Numerics.Tests	39.67 %	8.64 %	11.08 %	40.61 %	0.00 %
System.Reflection	21.85 %	3.57 %	71.43 %	3.15 %	0.00 %
System.Runtime.InteropServices.Tests	85.00 %	0.00 %	15.00 %	0.00 %	0.00 %
System.Runtime.Serialization.Formatters.Tests	70.00 %	20.00 %	10.00 %	0.00 %	0.00 %
System.Security.Cryptography.Primitives.Tests.Performance	87.50 %	7.50 %	5.00 %	0.00 %	0.00 %
System.Security.Cryptography.Tests	68.89 %	5.56 %	25.56 %	0.00 %	0.00 %
System.Security.Cryptography.X509Certificates.Tests	83.33 %	10.00 %	6.67 %	0.00 %	0.00 %
System.Tests	55.60 %	5.46 %	27.84 %	11.10 %	0.00 %
System.Text.Encodings.Web.Tests	76.67 %	1.67 %	21.67 %	0.00 %	0.00 %
System.Text.Json.Document.Tests	63.00 %	0.40 %	36.60 %	0.00 %	0.00 %
System.Text.Json.Node.Tests	0.00 %	100.00 %	0.00 %	0.00 %	0.00 %
System.Text.Json.Reader.Tests	57.50 %	0.00 %	42.50 %	0.00 %	0.00 %
System.Text.Json.Serialization.Tests	72.03 %	1.46 %	26.51 %	0.00 %	0.00 %
System.Text.Json.Tests	71.71 %	2.54 %	25.49 %	0.26 %	0.00 %
System.Text	78.67 %	14.00 %	7.33 %	0.00 %	0.00 %
System.Text.RegularExpressions.Tests	15.25 %	3.46 %	80.31 %	0.00 %	0.97 %
System.Text.Tests	66.90 %	11.67 %	18.81 %	2.62 %	0.00 %
System.Threading.Channels.Tests	67.50 %	8.33 %	24.17 %	0.00 %	0.00 %
System.Threading.Tasks.Tests	43.33 %	16.67 %	33.33 %	6.67 %	0.00 %
System.Threading.Tasks	75.00 %	2.14 %	20.71 %	2.14 %	0.00 %
System.Threading.Tests	72.98 %	2.34 %	15.11 %	9.57 %	0.00 %
System.Xml.Linq	58.95 %	2.63 %	28.95 %	9.47 %	0.00 %
System.Xml.Tests	0.00 %	0.00 %	100.00 %	0.00 %	0.00 %
V8.Crypto	90.00 %	10.00 %	0.00 %	0.00 %	0.00 %
V8.Richards	80.00 %	0.00 %	20.00 %	0.00 %	0.00 %
XmlDocumentTests.XmlDocumentTests	63.33 %	6.67 %	13.33 %	16.67 %	0.00 %
XmlDocumentTests.XmlNodeListTests	95.00 %	5.00 %	0.00 %	0.00 %	0.00 %
XmlDocumentTests.XmlNodeTests	30.00 %	0.00 %	0.00 %	70.00 %	0.00 %
System.Drawing.Tests	84.17 %	0.83 %	10.00 %	5.00 %	0.00 %
HardwareIntrinsics.RayTracer	85.71 %	14.29 %	0.00 %	0.00 %	0.00 %

Big thanks to everyone involved!

Author:	dakersnar
Assignees:	-
Labels:	area-Meta, tenet-performance, tenet-performance-benchmarks, tracking
Milestone:	-

[jeffhandley]

Fantastic work on these reports, @dakersnar. And congratulations to the whole team for the exciting performance improvements across the whole product.