Skip to content

Crossgen2 pipeline optimize - #51154

Merged
davidwrighton merged 3 commits into
dotnet:mainfrom
t-mustafin:crossgen2-pipeline-optimize
May 10, 2021
Merged

Crossgen2 pipeline optimize#51154
davidwrighton merged 3 commits into
dotnet:mainfrom
t-mustafin:crossgen2-pipeline-optimize

Conversation

@t-mustafin

@t-mustafint-mustafin commented Apr 13, 2021

Copy link
Copy Markdown
Contributor

This commit optimizes crossgen2 pipeline mode described in #37411. The optimization consists of loop-invariant-code-motion: move input dll loading out from loop.

Crossgen2 compilation of system dlls time compare

Architecturescriptpipelinepipeline optimize
x641m36.235s0m17.818s0m14.668s
armel13m30.493s3m48.829s3m36.691s
Time measurement details

Time is first real line of time command output.

Script
Command line to measure time:

time ../ngen_r2r.sh ./ true "-O"

ngen_r2r.sh first compiles System.Private.CoreLib.dll, then crossgen2/*.dll, then all other dlls.
Runtime state is 8fc5789.

Pipeline
Command line to measure time:

find ./ -name "*.ni.dll" -exec rm {} \;; time ./corerun ./crossgen2/crossgen2.dll -r:./*.dll -r:./crossgen2/*.dll --out-near-input ./crossgen2/*.dll ./*.dll --single-file-compilation -O

Runtime state as in script + Initial implementation of crossgen2 pipeline mode.

Pipeline optimize
Same cmdline as Pipeline.
Runtime state as in pipeline + [crossgen2] Optimize pipeline mode.

Crossgen2 compilation of system dlls time compare with options --inputbubble --compilebubblegenerics

Architecturescriptpipelinepipeline optimize
x641m43.536s0m30.380s0m25.035s
armel16m13.457s6m48.416s6m11.085s
Time measurement details

Script
Command line to measure time:

time ../ngen_r2r.sh ./ true "-O --inputbubble --compilebubblegenerics"

Script for armel compiles ILCompiler.ReadyToRun.dll and ILCompiler.TypeSystem.ReadyToRun.dll without --inputbubble --compilebubblegenerics due to issue #48466.

Pipeline
Command line to measure time:

find ./ -name "*.ni.dll" -exec rm {} \;; time ./corerun ./crossgen2/crossgen2.dll -r:./*.dll -r:./crossgen2/*.dll --out-near-input ./crossgen2/*.dll ./*.dll --single-file-compilation -O --inputbubble --compilebubblegenerics

Pipeline optimize
Same cmdline as Pipeline.

As part of work on pipeline mode we fixed a memory leak #49764, it allows us to finish compilation of all system dlls at one crossgen2 start and obtain time compare.

cc @jkotas@davidwrighton@MichalStrehovsky@gbalykov@alpencolt

gbalykovand others added 2 commits March 29, 2021 19:54
@gbalykov@t-mustafin
Initial implementation of crossgen2 pipeline mode
51457f7
- Add --out-near-input option, which adds .ni. suffix to input filepath
and stores resulting ni.dll near original dll. In this mode --out option can be skipped.
- Add --single-file-compilation mode, which allows to compile all input files separately.
@t-mustafin
[crossgen2] Optimize pipeline mode
f61635b
Load input and reference modules only once by crossgen2 start.
Signed-off-by: Timur Mustafin <t.mustafin@partner.samsung.com>
@ghostghost added the area-ReadyToRun label Apr 13, 2021
jkotas
jkotas reviewed Apr 13, 2021
View reviewed changes
Comment threadsrc/coreclr/tools/Common/TypeSystem/Ecma/EcmaModule.cs Outdated
@t-mustafin

t-mustafin commented Apr 13, 2021

Copy link
Copy Markdown
ContributorAuthor

Speedscope performance profiles of Pipeline and Pipeline optimized states.
speedscope_perf_graphs.zip
It collected on 6.0_preview branch due to symbol lack on main.

@t-mustafin

t-mustafin commented Apr 15, 2021•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Feedback time:

Architecturenoinputbubbleinputbubble
x640m14.884s0m25.321s
armel3m37.235s6m13.608s

MichalStrehovsky
MichalStrehovsky reviewed Apr 16, 2021
View reviewed changes
Comment threadsrc/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.cs Outdated
Comment threadsrc/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.cs Outdated
Comment threadsrc/coreclr/tools/Common/TypeSystem/Ecma/EcmaModule.cs Outdated
@t-mustafin
t-mustafinforce-pushed the crossgen2-pipeline-optimize branch from 0097096 to d70569aCompareApril 16, 2021 14:04
davidwrighton
davidwrighton reviewed Apr 19, 2021
View reviewed changes
Comment threadsrc/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.cs Outdated
davidwrighton
davidwrighton reviewed Apr 19, 2021
View reviewed changes
Comment threadsrc/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.cs Outdated
davidwrighton
davidwrighton reviewed Apr 19, 2021
View reviewed changes
Comment threadsrc/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.cs Outdated
MichalStrehovsky
MichalStrehovsky reviewed Apr 20, 2021
View reviewed changes
Comment threadsrc/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCompilerContext.cs Outdated
@t-mustafin
Feedback: make typeSystemContext with opened files
6ad1935
Construct new typeSystemContext with already opened PEReaders and PDBReaders.
Signed-off-by: Timur Mustafin <t.mustafin@partner.samsung.com>
@t-mustafin
t-mustafinforce-pushed the crossgen2-pipeline-optimize branch from d70569a to 6ad1935CompareApril 20, 2021 10:40
davidwrighton
davidwrighton approved these changes Apr 20, 2021
View reviewed changes

@davidwrightondavidwrighton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@gbalykovgbalykov mentioned this pull request May 10, 2021
Closed
@mangod9

mangod9 commented May 10, 2021

Copy link
Copy Markdown
Member

assume this is good to merge?

@davidwrighton
davidwrighton merged commit 4e2e990 into dotnet:mainMay 10, 2021
@karelzkarelz added this to the 6.0.0 milestone May 20, 2021
@ghostghost locked as resolved and limited conversation to collaborators Jun 19, 2021
@t-mustafin
t-mustafin deleted the crossgen2-pipeline-optimize branch August 10, 2023 14:27
@t-mustafin
t-mustafin restored the crossgen2-pipeline-optimize branch August 10, 2023 14:27
@t-mustafin
t-mustafin deleted the crossgen2-pipeline-optimize branch August 10, 2023 14:29
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@davidwrightondavidwrightondavidwrighton approved these changes
@jkotasjkotasAwaiting requested review from jkotas
@MichalStrehovskyMichalStrehovskyAwaiting requested review from MichalStrehovskyMichalStrehovsky is a code owner

Assignees

No one assigned

Labels

area-ReadyToRun

Projects

None yet

Milestone

6.0.0

Development

Successfully merging this pull request may close these issues.

7 participants

@t-mustafin@mangod9@jkotas@davidwrighton@MichalStrehovsky@karelz@gbalykov