Skip to content

.NET 7.0 Preview 3 Microbenchmarks Performance Study Report #67976

Description

@adamsitnik

Data

This time we have covered following configs:

  • architectures: x64, x86, arm64, arm
  • Unix: Ubuntu 18.04, Ubuntu 20.04, pop 20.04, Alpine 3.13, CentOS 7, Debian 11, macOS Monterey
  • Windows: 7, 10 & 11
Details
Operating SystemBitProcessor Name
alpine 3.12Arm64Unknown processor
ubuntu 18.04Arm64Unknown processor
Windows 10Arm64Microsoft SQ1 3.0 GHz
Windows 11Arm64Unknown processor
Windows 11Arm64Microsoft SQ1 3.0 GHz
macOS Monterey 12.3Arm64Apple M1 Max
Windows 10X64Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 11X64AMD Ryzen Threadripper PRO 3945WX 12-Cores
Windows 11X64Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 11X64Intel Core i9-9900T CPU 2.10GHz
alpine 3.13X64Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
centos 7X64Intel Xeon CPU E5530 2.40GHz
debian 11X64Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
opensuse-leap 15.3X64Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
ubuntu 18.04X64Intel Xeon CPU E5-1650 v4 3.60GHz
ubuntu 18.04X64Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Windows 10ArmMicrosoft SQ1 3.0 GHz
Windows 11ArmUnknown processor
ubuntu 18.04ArmARMv7 Processor rev 3 (v7l)
Windows 10X86Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 11X86AMD Ryzen Threadripper PRO 3945WX 12-Cores
Windows 11X86Intel Core i7-10510U CPU 1.80GHz
Windows 7 SP1X86Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
macOS Monterey 12.2.1X64Intel Core i7-5557U CPU 3.10GHz (Broadwell)
macOS Monterey 12.3.1X64Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @AndyAyersMS@carlossanlop@danmoseley@dakersnar@janvorli@jeffhandley@kunalspathak and @sblom who contributed their results and time.

The full report generated by the tool is available here. The full report contains also improvements, so if you read it from the end you can see the biggest perf improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview2 report can be found here.

Stress bugs

Regressions

By design

Investigation in progress

Already solved

  • System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)
  • System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateGetPositionTenSegments
  • System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated*, System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get*
    • a lot of socket benchmarks have regressed on Windows, but have been already solved

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

  • System.Collections.Concurrent.Count<String>.Dictionary(Size: 512)
  • System.Numerics.Tests.Perf_VectorOf<*>.*
  • System.Collections.TryAddDefaultSize<String>.ConcurrentDictionary(Count: 512)
    • whether it is slower or faster correlates with the alloc delta. Very noisy.
  • PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

Statistics

  • Statistical Test threshold: 10%, the noise filter: 2 ns
  • Result is conclusion: Slower|Faster|Same|Noise|Unknown. Noise means that the difference was larger than 10% but not 2 ns.

Total: 105070
Same: 74.24 %
Slower: 4.64 %
Faster: 10.13 %
Noise: 10.95 %
Unknown: 0.04 %

Statistics per Architecture

ArchitectureSameSlowerFasterNoiseUnknown
Arm6479.78 %3.72 %6.13 %10.35 %0.02 %
Arm73.79 %5.46 %14.75 %5.96 %0.04 %
X6471.73 %4.47 %11.05 %12.71 %0.04 %
X8674.81 %6.22 %9.64 %9.29 %0.04 %

Statistics per Operating System

Operating SystemSameSlowerFasterNoiseUnknown
alpine 3.1287.67 %2.02 %2.38 %7.93 %0.00 %
debian 1175.36 %2.93 %8.66 %13.04 %0.00 %
macOS Monterey 12.375.89 %0.64 %12.30 %11.17 %0.00 %
ubuntu 18.0466.00 %3.53 %21.29 %9.07 %0.11 %
ubuntu 20.0481.05 %6.23 %6.73 %5.99 %0.00 %
Windows 1074.50 %5.62 %8.36 %11.51 %0.01 %
Windows 1174.32 %6.18 %8.88 %10.59 %0.02 %
alpine 3.1367.48 %11.22 %4.63 %16.68 %0.00 %
centos 772.34 %3.79 %12.99 %10.79 %0.09 %
macOS Monterey 12.2.180.04 %2.13 %4.71 %13.03 %0.09 %
macOS Monterey 12.3.180.02 %1.24 %3.43 %15.31 %0.00 %
opensuse-leap 15.380.02 %2.61 %4.09 %13.28 %0.00 %
pop 20.0484.93 %4.87 %6.50 %3.69 %0.00 %
Windows 7 SP185.61 %2.86 %3.80 %7.73 %0.00 %

Statistics per Namespace

Details
NamespaceSameSlowerFasterNoiseUnknown
BenchmarksGame79.91 %15.77 %4.33 %0.00 %0.00 %
Benchstone.BenchF85.54 %8.77 %5.69 %0.00 %0.00 %
Benchstone.BenchI81.93 %5.94 %12.14 %0.00 %0.00 %
Benchstone.MDBenchF92.26 %5.16 %2.58 %0.00 %0.00 %
Benchstone.MDBenchI88.85 %4.18 %6.97 %0.00 %0.00 %
Devirtualization73.79 %5.52 %3.45 %17.24 %0.00 %
Exceptions89.38 %7.43 %3.19 %0.00 %0.00 %
FractalPerf91.18 %5.88 %2.94 %0.00 %0.00 %
GuardedDevirtualization89.54 %0.19 %0.09 %10.18 %0.00 %
Inlining36.08 %0.00 %5.15 %58.76 %0.00 %
Interop89.90 %6.06 %4.04 %0.00 %0.00 %
JetStream75.00 %8.33 %16.67 %0.00 %0.00 %
Layout84.85 %9.09 %6.06 %0.00 %0.00 %
Lowering92.59 %0.00 %0.00 %7.41 %0.00 %
MicroBenchmarks.Serializers78.51 %0.54 %20.95 %0.00 %0.00 %
Microsoft.Extensions.Logging75.19 %1.84 %22.21 %0.77 %0.00 %
PerfLabTests64.33 %9.46 %20.95 %5.27 %0.00 %
PerfLabTests.CastingPerf275.06 %4.20 %20.74 %0.00 %0.00 %
SciMark269.09 %2.42 %28.48 %0.00 %0.00 %
SIMD86.20 %2.36 %11.45 %0.00 %0.00 %
Span91.52 %2.55 %5.94 %0.00 %0.00 %
StoreBlock81.23 %0.62 %0.62 %17.54 %0.00 %
System.Buffers.Tests81.16 %2.57 %12.78 %3.49 %0.00 %
System.Buffers.Text.Tests77.85 %1.72 %7.66 %12.77 %0.00 %
System.Collections84.57 %3.78 %11.08 %0.57 %0.00 %
System.Collections.Concurrent79.23 %3.08 %7.56 %10.13 %0.00 %
System.Collections.Tests79.08 %4.06 %10.41 %6.45 %0.00 %
System.ConsoleTests80.25 %12.96 %6.79 %0.00 %0.00 %
System.Diagnostics78.33 %11.30 %10.37 %0.00 %0.00 %
System.Diagnostics.Tracing90.00 %5.00 %5.00 %0.00 %0.00 %
System.Formats.Cbor.Tests84.92 %0.00 %15.08 %0.00 %0.00 %
System.Globalization.Tests84.99 %4.37 %8.86 %1.78 %0.00 %
System84.01 %4.71 %9.26 %2.02 %0.00 %
System.IO.Compression91.62 %0.41 %7.97 %0.00 %0.00 %
System.IO.Pipelines.Tests85.19 %1.85 %12.96 %0.00 %0.00 %
System.IO.Pipes.Tests90.00 %5.33 %4.67 %0.00 %0.00 %
System.IO.Tests79.41 %8.05 %11.10 %1.44 %0.00 %
System.Linq.Tests80.26 %8.11 %10.96 %0.67 %0.00 %
System.MathBenchmarks84.68 %4.74 %9.77 %0.81 %0.00 %
System.Memory66.17 %2.99 %8.40 %22.44 %0.00 %
System.Net.Http.Tests60.33 %34.96 %4.71 %0.00 %0.00 %
System.Net.Primitives.Tests82.60 %3.04 %8.56 %5.80 %0.00 %
System.Net.Sockets.Tests87.50 %9.82 %2.68 %0.00 %0.00 %
System.Net.Tests81.48 %10.49 %8.02 %0.00 %0.00 %
System.Numerics.Tests50.81 %3.87 %5.38 %39.94 %0.00 %
System.Reflection84.50 %1.11 %12.55 %1.85 %0.00 %
System.Runtime.InteropServices.Tests79.41 %5.88 %11.76 %2.94 %0.00 %
System.Security.Cryptography.Tests92.52 %0.68 %6.80 %0.00 %0.00 %
System.Tests75.29 %5.32 %10.64 %8.76 %0.00 %
System.Text.Encodings.Web.Tests87.50 %3.26 %8.70 %0.54 %0.00 %
System.Text.Json.Document.Tests88.95 %2.70 %8.36 %0.00 %0.00 %
System.Text.Json.Reader.Tests83.70 %3.26 %13.04 %0.00 %0.00 %
System.Text.Json.Tests84.97 %1.35 %13.67 %0.00 %0.00 %
System.Text87.41 %2.72 %9.88 %0.00 %0.00 %
System.Text.Tests83.93 %6.05 %8.04 %1.98 %0.00 %
System.Threading.Tasks.Tests81.82 %4.55 %13.64 %0.00 %0.00 %
System.Threading.Tasks75.00 %6.17 %17.86 %0.97 %0.00 %
System.Threading.Tests82.13 %3.80 %8.43 %5.65 %0.00 %
System.Xml.Linq79.07 %0.60 %12.27 %8.05 %0.00 %
System.Xml.Tests14.17 %0.00 %85.83 %0.00 %0.00 %
V8.Crypto80.00 %6.67 %13.33 %0.00 %0.00 %
V8.Richards80.00 %3.33 %16.67 %0.00 %0.00 %
XmlDocumentTests.XmlDocumentTests86.36 %6.06 %4.55 %3.03 %0.00 %
XmlDocumentTests.XmlNodeListTests84.09 %2.27 %13.64 %0.00 %0.00 %
XmlDocumentTests.XmlNodeTests50.00 %4.55 %2.27 %43.18 %0.00 %
System.Net.Security.Tests86.13 %9.68 %4.19 %0.00 %0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance84.09 %6.82 %9.09 %0.00 %0.00 %
Microsoft.Extensions.Caching.Memory.Tests74.38 %5.79 %19.83 %0.00 %0.00 %
Microsoft.Extensions.Configuration.Xml90.91 %0.00 %9.09 %0.00 %0.00 %
Microsoft.Extensions.DependencyInjection72.73 %0.88 %24.63 %1.76 %0.00 %
Microsoft.Extensions.Primitives.Performance50.00 %0.00 %0.00 %50.00 %0.00 %
Microsoft.Extensions.Primitives70.20 %2.02 %14.14 %13.64 %0.00 %
System.Buffers.Binary.Tests67.05 %10.23 %13.64 %9.09 %0.00 %
System.ComponentModel.Tests90.91 %1.65 %7.44 %0.00 %0.00 %
System.IO.MemoryMappedFiles.Tests84.85 %11.36 %3.79 %0.00 %0.00 %
System.Net.NetworkInformation.Tests100.00 %0.00 %0.00 %0.00 %0.00 %
System.Net.WebSockets.Tests81.82 %9.09 %9.09 %0.00 %0.00 %
System.Runtime.Serialization.Formatters.Tests54.55 %45.45 %0.00 %0.00 %0.00 %
System.Security.Cryptography.Primitives.Tests.Performance77.27 %10.23 %12.50 %0.00 %0.00 %
System.Security.Cryptography.X509Certificates.Tests84.85 %0.00 %15.15 %0.00 %0.00 %
System.Text.Json.Serialization.Tests80.54 %5.02 %14.43 %0.00 %0.00 %
System.Text.RegularExpressions.Tests67.28 %6.24 %25.16 %0.00 %1.33 %
System.Threading.Channels.Tests89.17 %0.00 %10.83 %0.00 %0.00 %
System.Drawing.Tests83.86 %2.76 %8.66 %4.72 %0.00 %
HardwareIntrinsics.RayTracer88.24 %0.00 %0.00 %11.76 %0.00 %

Big thanks to everyone involved!

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-Metatenet-performancePerformance related issuetrackingThis issue is tracking the completion of other related issues.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions