Fix and refactor FCSBenchmarks by safesparrow · Pull Request #13419 · dotnet/fsharp

safesparrow · 2022-06-29T23:13:25Z

Major cleanup and fixes of the benchmarks in tests/FCSBenchmarks projects.

Changes in all benchmarks

All benchmarks should now build and work
Use BDN's console API to make the benchmarks more flexible (end-user can tweak the config without having to change code and build)
General cleanup

Changes in historical benchmark (`BenchmarkComparison`)

Changes in the notebook (`runner.ipynb`)

Fix it
Fix the commit-based benchmark which should now work at least with recent versions (it might still fail when building an old codebase)
Replace multiple projects + String.Replace with a single project using MSBuild properties
Refactor/cleanup
Store results in JSON instead of CSV, put artifacts in separate directories (inside .artifacts/, read them at the end

Changes in the BDN benchmark (now renamed to `HistoricalBenchmark.fsproj`)

Refactor
Add it to VisualFSharp.sln
Add it to the InternalsVisibleTo list in FSharp.Compiler.Service

Changes in FSharp.Compiler.Benchmarks

Deduplicate some of the code by reusing code from HistoricalBenchmark
Now depends on HistoricalBenchmark mainly to reuse the SingleFileCompiler. This might be somewhat controversial - HistoricalBenchmark needs to support old versions of FCS to allow doing a comparison, which can be limiting. However, currently this isn't a problem - once it is, we could either duplicate the code again or extract the shared bits to a separate project.

* extract all types into separate files * deduplicate the shared source file * start creating a script for running all benchmarks

* Refactor all the benchmarks a bit

- use JSON reports instead of CSV - put results in separate directories so that they can be accessed after the benchmark - read the results after all benchmarks finish rather than keeping them in memory * Add `BenchmarkComparison/README.md`

* refactor runner.ipynb further * Clear notebook outputs * Rename entrypoint file to `Program.fs` * Finish the smoke test script

safesparrow · 2022-06-30T00:10:20Z

@@ -1,27 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


Replaced by a single project file with MSBuild properties

safesparrow · 2022-06-30T00:10:24Z

@@ -1,25 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


Replaced by a single project file with MSBuild properties

safesparrow · 2022-06-30T00:10:27Z

@@ -1,25 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


Replaced by a single project file with MSBuild properties

safesparrow · 2022-06-30T00:11:31Z

@@ -1,23 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


No need for an extra top-level project now

safesparrow · 2022-06-30T06:49:41Z

@vzarytovskii Would you be able to review? Thanks!

vzarytovskii · 2022-06-30T16:35:26Z

@vzarytovskii Would you be able to review? Thanks!

Thanks for changes, they look good will do more thorough review tomorrow, once at my laptop :)

One thing which comes to mind is to add benchmarks to FSharp.sln too (and maybe FSharp.Compiler.Service.sln, @auduchinok wdyt?). Some people are not using the VisulaFSharp.sln.

auduchinok · 2022-06-30T16:39:29Z

One thing which comes to mind is to add benchmarks to FSharp.sln too (and maybe FSharp.Compiler.Service.sln, @auduchinok wdyt?). Some people are not using the VisulaFSharp.sln`.

Yes, thanks. We only use the FSharp.Compiler.Service solution, so it would be useful.

…t benchmark files to solution folders

safesparrow · 2022-07-07T11:40:35Z

@vzarytovskii Let me know if this is something worth merging and any changes you'd like to see :)

… F# project `HistoricalBenchmark.Runner`, call that logic from `runner.ipynb` * Add a few customisations (mostly directories) * Reuse existing checkouts/builds/benchmark reports * Provide simple `Runner.runAll` method for doing everything * Include sample results with three historical comparisons

safesparrow · 2022-07-13T20:28:28Z

I did a few changes:

moved all the logic from BenchmarkComparison/runner.ipynb into a new F# project HistoricalBenchmark.Runner
added a few customisations
made the benchmarking more robust and easier to run (one line of code in the notebook)
added three sample results

@vzarytovskii @KevinRansom Would it be possible to get the changes reviewed? If this is not a change you are willing to consider at this time that's fine - but I would be keen to know whether there is any point spending more time on this👍

vzarytovskii · 2022-07-15T12:09:25Z

I did a few changes:
moved all the logic from BenchmarkComparison/runner.ipynb into a new F# project HistoricalBenchmark.Runner
added a few customisations
made the benchmarking more robust and easier to run (one line of code in the notebook)
added three sample results
@vzarytovskii Vlad Zarytovskii FTE @KevinRansom Kevin Ransom FTE Would it be possible to get the changes reviewed? If this is not a change you are willing to consider at this time that's fine - but I would be keen to know whether there is any point spending more time on this👍

It looks great, sorry it was taking long time, we were a bit preoccupied with some other stuff :)

safesparrow added 11 commits June 24, 2022 23:49

Fix benchmark source files
020c113

* cleanup benchmarks
f3b5ebd
* extract all types into separate files * deduplicate the shared source file * start creating a script for running all benchmarks

* Fix BenchmarkComparison notebook
cb4984b
* Refactor all the benchmarks a bit

Refactor BenchmarkComparison
d34fc2b

Cleanup benchmarks a bit more
e45c28c

More cleanup (WIP)
a247e82

changes
27ecbc3

More cleanup, rename a few files.
bedc09a

* Refactor runner.ipynb:
e1af655
- use JSON reports instead of CSV - put results in separate directories so that they can be accessed after the benchmark - read the results after all benchmarks finish rather than keeping them in memory * Add `BenchmarkComparison/README.md`

* Small cleanup
ed2872b
* refactor runner.ipynb further * Clear notebook outputs * Rename entrypoint file to `Program.fs` * Finish the smoke test script

Merge remote-tracking branch 'upstream/main' into benchmarks-cleanup
d9d7abb

safesparrow commented Jun 30, 2022
View reviewed changes

safesparrow marked this pull request as ready for review June 30, 2022 01:17

safesparrow added 3 commits June 30, 2022 18:40

Add benchmark projects to FSharp.Compiler.Service.sln, add non-projec…
5c049f6
…t benchmark files to solution folders

Add benchmark projects to FSharp.sln too
cffd7ac

Fix invalid filepath in CompilerServiceBenchmarks.fs
2d68e24

safesparrow added 5 commits July 11, 2022 01:46

Merge remote-tracking branch 'upstream/main' into benchmarks-cleanup
0baff42

Apply IL Cache Clearing in FSharp.Compiler.Benchmarks as before
5a16d42

Merge remote-tracking branch 'upstream/main' into benchmarks-cleanup
0d0a5d4

Merge main into benchmarks-cleanup
e332fae

Allow skipping local build, add some comments
ab274f8

vzarytovskii approved these changes Jul 15, 2022
View reviewed changes

vzarytovskii merged commit c19ebd5 into dotnet:mainJul 15, 2022

vzarytovskii mentioned this pull request Aug 1, 2022
Reliable FCS benchmarks which run on every release #13606
Open

This was referenced Jan 3, 2024
Facelifting FCS benchmarks #16484
Merged
Removing a dead bench notebook #16496
Merged

psfinaki mentioned this pull request Jan 19, 2024
Smoke test some benchmarks in CI #16554
Merged

Navigation Menu

Uh oh!

Fix and refactor FCSBenchmarks - #13419

Fix and refactor FCSBenchmarks#13419
vzarytovskii merged 20 commits into
dotnet:mainfrom
safesparrow:benchmarks-cleanup

safesparrow commented Jun 29, 2022•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

safesparrowJun 30, 2022

Uh oh!

safesparrowJun 30, 2022

Uh oh!

safesparrowJun 30, 2022

Uh oh!

Uh oh!

safesparrowJun 30, 2022

Uh oh!

safesparrow commented Jun 30, 2022

Uh oh!

vzarytovskii commented Jun 30, 2022•
edited
Loading

Uh oh!

auduchinok commented Jun 30, 2022•
edited
Loading

Uh oh!

safesparrow commented Jul 7, 2022

Uh oh!

safesparrow commented Jul 13, 2022•
edited
Loading

Uh oh!

vzarytovskii commented Jul 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

safesparrow commented Jun 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes in all benchmarks

Changes in historical benchmark (BenchmarkComparison)

Changes in the notebook (runner.ipynb)

Changes in the BDN benchmark (now renamed to HistoricalBenchmark.fsproj)

Changes in FSharp.Compiler.Benchmarks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

safesparrowJun 30, 2022

Choose a reason for hiding this comment

Uh oh!

safesparrowJun 30, 2022

Choose a reason for hiding this comment

Uh oh!

safesparrowJun 30, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

safesparrowJun 30, 2022

Choose a reason for hiding this comment

Uh oh!

safesparrow commented Jun 30, 2022

Uh oh!

vzarytovskii commented Jun 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

auduchinok commented Jun 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

safesparrow commented Jul 7, 2022

Uh oh!

safesparrow commented Jul 13, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vzarytovskii commented Jul 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

safesparrow commented Jun 29, 2022•
edited
Loading

Changes in historical benchmark (`BenchmarkComparison`)

Changes in the notebook (`runner.ipynb`)

Changes in the BDN benchmark (now renamed to `HistoricalBenchmark.fsproj`)

vzarytovskii commented Jun 30, 2022•
edited
Loading

auduchinok commented Jun 30, 2022•
edited
Loading

safesparrow commented Jul 13, 2022•
edited
Loading