Project with cascading files benchmarks - compare effect on signature files on incremental build by T-Gro · Pull Request #14136 · dotnet/fsharp

T-Gro · 2022-10-18T11:49:36Z

This set of benchmarks validates that the existence of .fsi files does have multi-fold impact on the speed of incremental build.
That is, especially if the file being modified happens to be between beginning and middle of the project.

The benchmark was running in 2^3= configurations corresponding to arguments of FsCheck, and the fact whether .fsi files were added to the compilation or not.
enablePartialTypeChecking = this.PartialCheck,
enableParallelCheckingWithSignatureFiles = this.ParaChecking

If you focus on a specific example, let's say **ParseProjectWithChangingMiddleFile ** which changes the middle file of a 128-files large project, you can see the differences across these 3 boolean configs.
In this case, the impact on the speed of incremental build can be up to 2-3x in terms of speed gains.

BenchmarkDotNet=v0.13.2, OS=Windows 11 (10.0.22621.674)
11th Gen Intel Core i9-11950H 2.60GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=7.0.100-rc.2.22477.23
[Host] : .NET 7.0.0 (7.0.22.47203), X64 RyuJIT AVX2 DEBUG
ShortRun : .NET 7.0.0 (7.0.22.47203), X64 RyuJIT AVX2
Job=ShortRun IterationCount=3 LaunchCount=1 WarmupCount=3

Method	PartialCheck	ParaChecking	GenerateFSI	Mean	Error	StdDev	Ratio	RatioSD	Rank	Gen0	Gen1	Gen2	Allocated	Alloc Ratio
ParseAndCheckLastFileProjectAsIs	False	False	False	11.027 ms	10.202 ms	0.5592 ms	0.01	0.00	I	-	-	-	120.92 KB	0.000
ParseProjectWithChangingPenultimateFile	False	False	False	25.373 ms	17.441 ms	0.9560 ms	0.02	0.00	II	656.2500	281.2500	-	8402.02 KB	0.019
ParseProjectWithChangingMiddleFile	False	False	False	364.452 ms	894.278 ms	49.0184 ms	0.35	0.07	III	14000.0000	2000.0000	-	179193.09 KB	0.416
ParseProjectWithChangingFirstFile	False	False	False	642.130 ms	804.596 ms	44.1026 ms	0.62	0.05	IV	28000.0000	3000.0000	-	350922.82 KB	0.814
ParseProjectWithFullCacheClear	False	False	False	1,046.582 ms	1,394.643 ms	76.4451 ms	1.00	0.00	V	35000.0000	9000.0000	2000.0000	430979.57 KB	1.000

ParseAndCheckLastFileProjectAsIs	False	False	True	21.822 ms	38.597 ms	2.1156 ms	0.02	0.00	I	-	-	-	228.33 KB	0.001
ParseProjectWithChangingPenultimateFile	False	False	True	43.916 ms	36.434 ms	1.9971 ms	0.05	0.00	II	692.3077	230.7692	-	9320.81 KB	0.023
ParseProjectWithChangingMiddleFile	False	False	True	408.622 ms	861.017 ms	47.1952 ms	0.44	0.05	III	13000.0000	2000.0000	-	167302.96 KB	0.412
ParseProjectWithChangingFirstFile	False	False	True	735.510 ms	1,964.457 ms	107.6785 ms	0.79	0.12	IV	26000.0000	5000.0000	-	325652.77 KB	0.802
ParseProjectWithFullCacheClear	False	False	True	929.975 ms	81.306 ms	4.4567 ms	1.00	0.00	V	33000.0000	10000.0000	2000.0000	406068.07 KB	1.000

ParseAndCheckLastFileProjectAsIs	False	True	False	9.807 ms	7.528 ms	0.4126 ms	0.01	0.00	I	-	-	-	118.53 KB	0.000
ParseProjectWithChangingPenultimateFile	False	True	False	22.698 ms	3.642 ms	0.1996 ms	0.02	0.00	II	656.2500	281.2500	-	8453.61 KB	0.020
ParseProjectWithChangingMiddleFile	False	True	False	338.556 ms	334.933 ms	18.3588 ms	0.37	0.01	III	14000.0000	2000.0000	-	179077.58 KB	0.415
ParseProjectWithChangingFirstFile	False	True	False	612.335 ms	1,075.664 ms	58.9607 ms	0.67	0.02	IV	28000.0000	3000.0000	-	351421.38 KB	0.815
ParseProjectWithFullCacheClear	False	True	False	915.377 ms	1,150.000 ms	63.0354 ms	1.00	0.00	V	35000.0000	9000.0000	2000.0000	431008.76 KB	1.000

ParseAndCheckLastFileProjectAsIs	False	True	True	19.375 ms	15.346 ms	0.8412 ms	0.02	0.00	I	-	-	-	230.86 KB	0.001
ParseProjectWithChangingPenultimateFile	False	True	True	35.793 ms	12.569 ms	0.6890 ms	0.04	0.00	II	714.2857	214.2857	-	9303.83 KB	0.023
ParseProjectWithChangingMiddleFile	False	True	True	340.513 ms	157.150 ms	8.6139 ms	0.36	0.02	III	13000.0000	2000.0000	-	167654.45 KB	0.413
ParseProjectWithChangingFirstFile	False	True	True	685.282 ms	339.027 ms	18.5832 ms	0.72	0.03	IV	26000.0000	5000.0000	-	325979.35 KB	0.803
ParseProjectWithFullCacheClear	False	True	True	958.442 ms	541.981 ms	29.7078 ms	1.00	0.00	V	33000.0000	10000.0000	2000.0000	406067.13 KB	1.000

ParseAndCheckLastFileProjectAsIs	True	False	False	8.724 ms	1.675 ms	0.0918 ms	0.01	0.00	I	-	-	-	119.38 KB	0.000
ParseProjectWithChangingPenultimateFile	True	False	False	23.033 ms	5.042 ms	0.2764 ms	0.03	0.00	II	656.2500	281.2500	-	8410.91 KB	0.019
ParseProjectWithChangingMiddleFile	True	False	False	312.968 ms	502.321 ms	27.5339 ms	0.41	0.05	III	14000.0000	2000.0000	-	177325.7 KB	0.403
ParseProjectWithChangingFirstFile	True	False	False	508.210 ms	123.797 ms	6.7857 ms	0.67	0.03	IV	28000.0000	3000.0000	-	347377.86 KB	0.789
ParseProjectWithFullCacheClear	True	False	False	764.298 ms	703.481 ms	38.5601 ms	1.00	0.00	V	36000.0000	11000.0000	4000.0000	440120.64 KB	1.000

ParseAndCheckLastFileProjectAsIs	True	False	True	17.545 ms	9.458 ms	0.5184 ms	0.04	0.00	I	-	-	-	133.31 KB	0.001
ParseProjectWithChangingPenultimateFile	True	False	True	28.262 ms	7.366 ms	0.4038 ms	0.07	0.00	II	312.5000	156.2500	-	4177.03 KB	0.028
ParseProjectWithChangingMiddleFile	True	False	True	115.049 ms	224.340 ms	12.2968 ms	0.28	0.03	III	2800.0000	800.0000	-	35153.53 KB	0.234
ParseProjectWithChangingFirstFile	True	False	True	192.335 ms	152.500 ms	8.3590 ms	0.46	0.01	IV	4666.6667	1000.0000	-	60968.41 KB	0.405
ParseProjectWithFullCacheClear	True	False	True	417.720 ms	190.919 ms	10.4649 ms	1.00	0.00	V	12000.0000	7000.0000	3000.0000	150548.34 KB	1.000

ParseAndCheckLastFileProjectAsIs	True	True	False	9.145 ms	9.866 ms	0.5408 ms	0.01	0.00	I	-	-	-	116.28 KB	0.000
ParseProjectWithChangingPenultimateFile	True	True	False	24.712 ms	18.141 ms	0.9944 ms	0.03	0.00	II	656.2500	281.2500	-	8406.78 KB	0.019
ParseProjectWithChangingMiddleFile	True	True	False	307.731 ms	474.400 ms	26.0035 ms	0.34	0.03	III	14000.0000	2000.0000	-	177520.01 KB	0.404
ParseProjectWithChangingFirstFile	True	True	False	524.612 ms	191.084 ms	10.4740 ms	0.58	0.02	IV	28000.0000	3000.0000	-	347610.17 KB	0.790
ParseProjectWithFullCacheClear	True	True	False	905.933 ms	191.779 ms	10.5121 ms	1.00	0.00	V	35000.0000	10000.0000	4000.0000	439927.7 KB	1.000

ParseAndCheckLastFileProjectAsIs	True	True	True	18.328 ms	2.036 ms	0.1116 ms	0.04	0.00	I	-	-	-	136.55 KB	0.001
ParseProjectWithChangingPenultimateFile	True	True	True	33.871 ms	20.096 ms	1.1015 ms	0.08	0.01	II	312.5000	156.2500	-	4195.5 KB	0.028
ParseProjectWithChangingMiddleFile	True	True	True	119.979 ms	96.769 ms	5.3043 ms	0.29	0.02	III	2750.0000	750.0000	-	35245.54 KB	0.234
ParseProjectWithChangingFirstFile	True	True	True	160.949 ms	109.366 ms	5.9947 ms	0.39	0.02	IV	4666.6667	1000.0000	-	60744.59 KB	0.403
ParseProjectWithFullCacheClear	True	True	True	409.830 ms	485.428 ms	26.6080 ms	1.00	0.00	V	12000.0000	7000.0000	3000.0000	150609.31 KB	1.000

…ascadeBenchmarks.fs Co-authored-by: Petr Pokorny <petrpokorny@microsoft.com>

psfinaki

Thanks for this. So I guess generally this means we should first speed up signature files before focusing on their convenient auto-generation?

T-Gro added 8 commits October 12, 2022 15:49

Generating cascading project
4e3137a

Benchmark attributes added
8a8499c

Merge branch 'main' into feature/project-with-cascading-files-benchmarks
9eea000

More agressive cache cleaning in benchmarks
1fb219a

Better naming, waiting for async, further benchmark case
5a7ad01

Expanding the benchmark
403f6ee

Benchmark for incremental compiling with or without .fsi files
29f4690

Merge branch 'main' into feature/project-with-cascading-files-benchmarks
57fa01f

T-Gro requested review from 0101, abonie, psfinaki and vzarytovskii October 18, 2022 11:53

0101 previously approved these changes Oct 18, 2022
View reviewed changes

Comment threadtests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/FileCascadeBenchmarks.fs Outdated
Comment threadtests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/FileCascadeBenchmarks.fs

Update tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/FileC…
510b4e8
…ascadeBenchmarks.fs Co-authored-by: Petr Pokorny <petrpokorny@microsoft.com>

T-Gro dismissed 0101’s stale review via 510b4e8October 18, 2022 15:15

New project added to assemblycheck ignore list
7126be9

T-Gro requested a review from 0101 October 18, 2022 16:08

0101 approved these changes Oct 18, 2022
View reviewed changes

psfinaki approved these changes Oct 18, 2022
View reviewed changes

T-Gro merged commit fbfc406 into mainOct 19, 2022

allisonchou mentioned this pull request Oct 24, 2022
[Automated] PRs inserted in VS build main-33024.23 #14165
Closed

T-Gro deleted the feature/project-with-cascading-files-benchmarks branch February 2, 2023 13:04

Navigation Menu

Uh oh!

Project with cascading files benchmarks - compare effect on signature files on incremental build - #14136

Project with cascading files benchmarks - compare effect on signature files on incremental build#14136
T-Gro merged 10 commits into
mainfrom
feature/project-with-cascading-files-benchmarks

T-Gro commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

psfinaki left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

T-Gro commented Oct 18, 2022

Uh oh!

Uh oh!

Uh oh!

psfinaki left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants