Uh oh!
There was an error while loading. Please reload this page.
Run crossgen in parallel in crossgen_comparison.py - #33175
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
3c9c5c3 to
0752950Comparejashook
commented
Mar 25, 2020
I believe this is finally ready for review @echesakovMSFT @dotnet/runtime-infrastructure ptal |
jashook
commented
Mar 25, 2020
Will post runtime improvements once the run finishes |
| platform: Linux_arm | ||
| container: | ||
| image: ubuntu-16.04-cross-14.04-23cacb0-20200121150126 | ||
| image: ubuntu-18.04-cross-arm-16.04-09ec757-20200324125113 |
There was a problem hiding this comment.
Why did we need to upgrade the image here?
There was a problem hiding this comment.
The script needs python3.6+
There was a problem hiding this comment.
Its also generally not needed for us to target 14.04 anymore as 14.04 is eol
Uh oh!
There was an error while loading. Please reload this page.
| inputs: | ||
| scriptSource: 'filePath' | ||
| scriptPath: $(coreClrRepoRoot)/tests/scripts/crossgen_comparison.py | ||
| pythonInterpreter: /usr/bin/python3 |
There was a problem hiding this comment.
Do we need to add python3 to our linux requirements markdown file?
There was a problem hiding this comment.
We should be one python3, seeing that python2 is eol. We have not exactly mandated it yet.
jashook
commented
Mar 25, 2020
Time in seconds
|
jashook
commented
Mar 25, 2020
This will leave our cross-crossgen job with ~3 minutes of helix setup time (download, zip and docker) and ~2 minutes of execution time. Total time is reduced ~40% with this change. |
trylek
commented
Mar 25, 2020
Very nice. That matches the ratio between Crossgen / Crossgen2 framework compilation times in "build-test generatelayoutonly" (where there's a straight linear loop over the assemblies) and SuperIlc (which uses parallelization much like you do, just written in C#), something like 3:1, almost reaching the 4:1 ideal speedup on my 4-core box. |
trylek
left a comment
There was a problem hiding this comment.
Looks great, thanks for pulling off such a dramatic perf improvement!
This will hopefully reduce the runtime of crossgen_compairision.py and address #33172