Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); NativeAOT SuperPMI collections by TIHan · Pull Request #91037 · dotnet/runtime · GitHub
Skip to content

NativeAOT SuperPMI collections - #91037

Merged
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix
Sep 11, 2023
Merged

NativeAOT SuperPMI collections#91037
TIHan merged 37 commits into
dotnet:mainfrom
TIHan:nativeaot-collections-with-jakob-fix

Conversation

@TIHan

@TIHanTIHan commented Aug 23, 2023

Copy link
Copy Markdown
Contributor

This adds NativeAOT SuperPMI collections runs for the SmokeTests. Will work on windows+linux x64+arm64.

@ghostghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 23, 2023
@ghostghost assigned TIHanAug 23, 2023
@ghost

Copy link
Copy Markdown

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:TIHan
Assignees:-
Labels:

area-CodeGen-coreclr

Milestone:-

Comment threadsrc/coreclr/scripts/superpmi.py
@TIHan
TIHan marked this pull request as ready for review September 6, 2023 23:39
@TIHan

TIHan commented Sep 6, 2023

Copy link
Copy Markdown
ContributorAuthor

@dotnet/jit-contrib @BruceForstall this is ready.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

Comment threadeng/pipelines/coreclr/templates/run-superpmi-collect-job.yml Outdated
Comment threadsrc/coreclr/inc/jiteeversionguid.h Outdated
Comment threadsrc/coreclr/scripts/superpmi.py
@jakobbotsch

jakobbotsch commented Sep 7, 2023

Copy link
Copy Markdown
Member

@TIHan Can you please check the superpmi-collect runs you have been triggering for errors? E.g. I see all the "libraries pmi" runs failing with

Console log: '0' from job c74f489f-b165-4186-a8d9-e8f2f0f058df workitem 43264b0b-93c7-4f6b-b133-873825bb3b3b (ubuntu.2204.amd64.rt) executed on machine a0001QC running Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
chmod: cannot access '/datadisks/disk1/work/AC490993/w/A220090C/e/*.sh': No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/AC490993/p/superpmi/superpmi.py collect --clean -log_level DEBUG --pmi -pmi_location /datadisks/disk1/work/AC490993/p/superpmi/pmi.dll -pmi_path /datadisks/disk1/work/AC490993/p/superpmi/R2RDump /datadisks/disk1/work/AC490993/p/superpmi/R2RTest /datadisks/disk1/work/AC490993/p/superpmi/crossgen2 /datadisks/disk1/work/AC490993/p/superpmi/xunit -assemblies /datadisks/disk1/work/AC490993/w/A220090C/u/binaries -arch x64 -build_type checked -core_root /datadisks/disk1/work/AC490993/p/superpmi -output_mch_path /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.mch -log_file /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
[00:29:45] ================ Logging to /datadisks/disk1/work/AC490993/w/A220090C/uploads/libraries.pmi.0.linux.x64.checked.log
Specify `-ilc_rsps` if `--nativeaot` is given
+ export _commandExitCode=1
+ exit 1
['0' END OF WORK ITEM LOG: Command exited with 1]

(link)

@BruceForstall

Copy link
Copy Markdown
Contributor

I looked at a recent collection run and see a few concerning things:

2023-09-07T00:19:46.5692728Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5692958Z cp: cannot create regular file '/mnt/vss/_work/1/s/artifacts/obj/': Not a directory
2023-09-07T00:19:46.5693150Z /mnt/vss/_work/1/s/eng/native/version/copy_version_files.sh: line 29: /mnt/vss/_work/1/s/artifacts/obj/_version.c: No such file or directory

Why are these failing? What script/process is the invoker?

  1. Failures during collection
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/Managed/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/native/SharedLibrary.exports'.
...
Unhandled exception: System.IO.DirectoryNotFoundException: Could not find a part of the path '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/DynamicGenerics/rd.xml'.
...
Unhandled exception: System.CommandLine.CommandLineException: '/mnt/vss/_work/1/s/src/tests/nativeaot/SmokeTests/TrimmingBehaviors/NonEmbedded.ILLink.Descriptor.xml' doesn't exist

Are these files built when the smoke tests are built? Are they just not getting copied to Helix?

@TIHan

TIHan commented Sep 8, 2023

Copy link
Copy Markdown
ContributorAuthor

Why are these failing? What script/process is the invoker?

This one I am not sure.

Are these files built when the smoke tests are built?

Except for the .exports file, the .xml files are located where the source of the tests are. Which means we would have to do more work to get these files copied over.

Are they just not getting copied to Helix?

I updated it to at least copy over the .exports file, but I can't copy the .xml ones because they are not part of the output of the tests.

Seeing how there are multiple tests, when built, that will not include the non-DLL files they depend on in the output, it may make sense to do something similar to what @MichalStrehovsky suggested. I was hoping that it would be just the assemblies and the pinvokelist.

@BruceForstall What do you recommend we do? We get tens of thousands of method contexts so it still works well enough - we can iterate on it over time.

@BruceForstallBruceForstall left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TIHan
TIHan merged commit 02feed0 into dotnet:mainSep 11, 2023
@ghostghost locked as resolved and limited conversation to collaborators Oct 12, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TIHan@jakobbotsch@BruceForstall@MichalStrehovsky