Skip to content

Convert JIT\Regression tests to merged test groups - #83895

Merged
markples merged 33 commits into
dotnet:mainfrom
markples:merge6
Mar 31, 2023
Merged

Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom
markples:merge6

Conversation

@markples

@markplesmarkples commented Mar 24, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.

Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.

@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost assigned markplesMar 24, 2023
@ghost

ghost commented Mar 24, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:markples
Assignees:markples
Labels:

area-System.Reflection.Metadata

Milestone:-

@markplesmarkples added this to the 8.0.0 milestone Mar 24, 2023
@markples

markples commented Mar 29, 2023

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Mar 29, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@markples

markples commented Mar 29, 2023

Copy link
Copy Markdown
ContributorAuthor

fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.

@markples
markples marked this pull request as ready for review March 29, 2023 05:16
@markples

markples commented Mar 29, 2023

Copy link
Copy Markdown
ContributorAuthor

@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.

@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples

markples commented Mar 29, 2023

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Mar 29, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@build-analysisbuild-analysisBot mentioned this pull request Mar 29, 2023
Closed
@JulieLeeMSFTJulieLeeMSFT mentioned this pull request Mar 29, 2023
Closed
17 tasks
@markples

markples commented Mar 30, 2023

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Mar 30, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@markples

markples commented Mar 30, 2023

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Mar 30, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@build-analysisbuild-analysisBot mentioned this pull request Mar 30, 2023
Closed
@markples

markples commented Mar 30, 2023

Copy link
Copy Markdown
ContributorAuthor

@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.

Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result

nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky

MichalStrehovsky commented Mar 31, 2023

Copy link
Copy Markdown
Member

@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?

I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.

@VSadov

VSadov commented Mar 31, 2023

Copy link
Copy Markdown
Member

I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?

@VSadov

VSadov commented Mar 31, 2023•
edited
Loading

Copy link
Copy Markdown
Member

I've obtained the dump and the test binary that crushed. So far not much luck.
The dump is not very useful.

(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.
The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.

I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.

@markples

markples commented Mar 31, 2023

Copy link
Copy Markdown
ContributorAuthor

I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.

@markples

markples commented Mar 31, 2023

Copy link
Copy Markdown
ContributorAuthor

close/reopen to poke the license/cla check

@markplesmarkples reopened this Mar 31, 2023
@markples
markples merged commit 52bbeeb into dotnet:mainMar 31, 2023
@BruceForstall

BruceForstall commented Mar 31, 2023

Copy link
Copy Markdown
Contributor

@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab

(I kicked off a gcstress run immediately after you merged to check this)

@markples

markples commented Mar 31, 2023

Copy link
Copy Markdown
ContributorAuthor

Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).
This will avoid the additional churn that would occur by reverting/remerging.

@ghostghost locked as resolved and limited conversation to collaborators May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects

None yet

Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Convert JIT\Regression tests to merged test groups by markples · Pull Request #83895 · dotnet/runtime · GitHub
Skip to content
Convert JIT\Regression tests to merged test groups  - #83895
Merged
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Mar 31, 2023
Merged
Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Conversation
@markples
@markplesmarkples
commented
Mar 24, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.
Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.
@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost
assigned markplesMar 24, 2023
@ghost
ghost
commented
Mar 24, 2023

Copy link


Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata

See info in area-owners.md if you want to be subscribed.
Issue Details
null
Author:markples
Assignees:markples
Labels:
area-System.Reflection.Metadata
Milestone:-
@markplesmarkples
added this to the 8.0.0 milestone
Mar 24, 2023
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.
@markples
markples
marked this pull request as ready for review
March 29, 2023 05:16
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.
@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 29, 2023

Closed

@JulieLeeMSFTJulieLeeMSFT
mentioned this pull request
Mar 29, 2023

Closed

17 tasks
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 30, 2023

Closed

@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.
Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result
nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky
MichalStrehovsky
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?
I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.
@VSadov
VSadov
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?
@VSadov
VSadov
commented
Mar 31, 2023•

edited

Loading

Copy link


Copy Markdown

Member
I've obtained the dump and the test binary that crushed. So far not much luck.

The dump is not very useful.
(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.

The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.
I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
close/reopen to poke the license/cla check
@markplesmarkples
reopened this
Mar 31, 2023
@markples
markples
merged commit 52bbeeb
into
dotnet:mainMar 31, 2023
@BruceForstall
BruceForstall
commented
Mar 31, 2023

Copy link


Copy Markdown

Contributor
@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab
(I kicked off a gcstress run immediately after you merged to check this)
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).

This will avoid the additional churn that would occur by reverting/remerging.
@ghostghost
locked as resolved and limited conversation to collaborators
May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects


None yet


Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Convert JIT\Regression tests to merged test groups by markples · Pull Request #83895 · dotnet/runtime · GitHub
Skip to content
Convert JIT\Regression tests to merged test groups  - #83895
Merged
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Mar 31, 2023
Merged
Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Conversation
@markples
@markplesmarkples
commented
Mar 24, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.
Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.
@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost
assigned markplesMar 24, 2023
@ghost
ghost
commented
Mar 24, 2023

Copy link


Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata

See info in area-owners.md if you want to be subscribed.
Issue Details
null
Author:markples
Assignees:markples
Labels:
area-System.Reflection.Metadata
Milestone:-
@markplesmarkples
added this to the 8.0.0 milestone
Mar 24, 2023
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.
@markples
markples
marked this pull request as ready for review
March 29, 2023 05:16
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.
@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 29, 2023

Closed

@JulieLeeMSFTJulieLeeMSFT
mentioned this pull request
Mar 29, 2023

Closed

17 tasks
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 30, 2023

Closed

@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.
Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result
nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky
MichalStrehovsky
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?
I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.
@VSadov
VSadov
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?
@VSadov
VSadov
commented
Mar 31, 2023•

edited

Loading

Copy link


Copy Markdown

Member
I've obtained the dump and the test binary that crushed. So far not much luck.

The dump is not very useful.
(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.

The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.
I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
close/reopen to poke the license/cla check
@markplesmarkples
reopened this
Mar 31, 2023
@markples
markples
merged commit 52bbeeb
into
dotnet:mainMar 31, 2023
@BruceForstall
BruceForstall
commented
Mar 31, 2023

Copy link


Copy Markdown

Contributor
@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab
(I kicked off a gcstress run immediately after you merged to check this)
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).

This will avoid the additional churn that would occur by reverting/remerging.
@ghostghost
locked as resolved and limited conversation to collaborators
May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects


None yet


Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Convert JIT\Regression tests to merged test groups by markples · Pull Request #83895 · dotnet/runtime · GitHub
Skip to content
Convert JIT\Regression tests to merged test groups  - #83895
Merged
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Mar 31, 2023
Merged
Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Conversation
@markples
@markplesmarkples
commented
Mar 24, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.
Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.
@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost
assigned markplesMar 24, 2023
@ghost
ghost
commented
Mar 24, 2023

Copy link


Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata

See info in area-owners.md if you want to be subscribed.
Issue Details
null
Author:markples
Assignees:markples
Labels:
area-System.Reflection.Metadata
Milestone:-
@markplesmarkples
added this to the 8.0.0 milestone
Mar 24, 2023
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.
@markples
markples
marked this pull request as ready for review
March 29, 2023 05:16
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.
@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 29, 2023

Closed

@JulieLeeMSFTJulieLeeMSFT
mentioned this pull request
Mar 29, 2023

Closed

17 tasks
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 30, 2023

Closed

@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.
Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result
nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky
MichalStrehovsky
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?
I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.
@VSadov
VSadov
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?
@VSadov
VSadov
commented
Mar 31, 2023•

edited

Loading

Copy link


Copy Markdown

Member
I've obtained the dump and the test binary that crushed. So far not much luck.

The dump is not very useful.
(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.

The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.
I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
close/reopen to poke the license/cla check
@markplesmarkples
reopened this
Mar 31, 2023
@markples
markples
merged commit 52bbeeb
into
dotnet:mainMar 31, 2023
@BruceForstall
BruceForstall
commented
Mar 31, 2023

Copy link


Copy Markdown

Contributor
@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab
(I kicked off a gcstress run immediately after you merged to check this)
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).

This will avoid the additional churn that would occur by reverting/remerging.
@ghostghost
locked as resolved and limited conversation to collaborators
May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects


None yet


Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Convert JIT\Regression tests to merged test groups by markples · Pull Request #83895 · dotnet/runtime · GitHub
Skip to content
Convert JIT\Regression tests to merged test groups  - #83895
Merged
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Mar 31, 2023
Merged
Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Conversation
@markples
@markplesmarkples
commented
Mar 24, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.
Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.
@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost
assigned markplesMar 24, 2023
@ghost
ghost
commented
Mar 24, 2023

Copy link


Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata

See info in area-owners.md if you want to be subscribed.
Issue Details
null
Author:markples
Assignees:markples
Labels:
area-System.Reflection.Metadata
Milestone:-
@markplesmarkples
added this to the 8.0.0 milestone
Mar 24, 2023
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.
@markples
markples
marked this pull request as ready for review
March 29, 2023 05:16
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.
@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 29, 2023

Closed

@JulieLeeMSFTJulieLeeMSFT
mentioned this pull request
Mar 29, 2023

Closed

17 tasks
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 30, 2023

Closed

@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.
Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result
nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky
MichalStrehovsky
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?
I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.
@VSadov
VSadov
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?
@VSadov
VSadov
commented
Mar 31, 2023•

edited

Loading

Copy link


Copy Markdown

Member
I've obtained the dump and the test binary that crushed. So far not much luck.

The dump is not very useful.
(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.

The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.
I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
close/reopen to poke the license/cla check
@markplesmarkples
reopened this
Mar 31, 2023
@markples
markples
merged commit 52bbeeb
into
dotnet:mainMar 31, 2023
@BruceForstall
BruceForstall
commented
Mar 31, 2023

Copy link


Copy Markdown

Contributor
@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab
(I kicked off a gcstress run immediately after you merged to check this)
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).

This will avoid the additional churn that would occur by reverting/remerging.
@ghostghost
locked as resolved and limited conversation to collaborators
May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects


None yet


Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Convert JIT\Regression tests to merged test groups by markples · Pull Request #83895 · dotnet/runtime · GitHub
Skip to content
Convert JIT\Regression tests to merged test groups  - #83895
Merged
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Mar 31, 2023
Merged
Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Conversation
@markples
@markplesmarkples
commented
Mar 24, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.
Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.
@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost
assigned markplesMar 24, 2023
@ghost
ghost
commented
Mar 24, 2023

Copy link


Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata

See info in area-owners.md if you want to be subscribed.
Issue Details
null
Author:markples
Assignees:markples
Labels:
area-System.Reflection.Metadata
Milestone:-
@markplesmarkples
added this to the 8.0.0 milestone
Mar 24, 2023
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.
@markples
markples
marked this pull request as ready for review
March 29, 2023 05:16
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.
@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 29, 2023

Closed

@JulieLeeMSFTJulieLeeMSFT
mentioned this pull request
Mar 29, 2023

Closed

17 tasks
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 30, 2023

Closed

@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.
Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result
nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky
MichalStrehovsky
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?
I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.
@VSadov
VSadov
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?
@VSadov
VSadov
commented
Mar 31, 2023•

edited

Loading

Copy link


Copy Markdown

Member
I've obtained the dump and the test binary that crushed. So far not much luck.

The dump is not very useful.
(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.

The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.
I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
close/reopen to poke the license/cla check
@markplesmarkples
reopened this
Mar 31, 2023
@markples
markples
merged commit 52bbeeb
into
dotnet:mainMar 31, 2023
@BruceForstall
BruceForstall
commented
Mar 31, 2023

Copy link


Copy Markdown

Contributor
@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab
(I kicked off a gcstress run immediately after you merged to check this)
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).

This will avoid the additional churn that would occur by reverting/remerging.
@ghostghost
locked as resolved and limited conversation to collaborators
May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects


None yet


Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
Convert JIT\Regression tests to merged test groups by markples · Pull Request #83895 · dotnet/runtime · GitHub
Skip to content
Convert JIT\Regression tests to merged test groups  - #83895
Merged
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Mar 31, 2023
Merged
Convert JIT\Regression tests to merged test groups #83895
markples merged 33 commits into
dotnet:mainfrom 
markples:merge6
Conversation
@markples
@markplesmarkples
commented
Mar 24, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
The (admittedly long) list of commits are the logical steps involved in converting the tests. Each is relatively homogenous and easier to view than the overall diff. Automated commands are indicated by [command] notation as the start of the commit message.
Usability concerns are listed in #71732. Since JIT\Regression is actively updated, at least some minimum is required before merging (unlike something like il_conformance which is basically not modified). I've attempted to strike a balance there between "perfection" and actually getting this finished by marking a subset as necessary up-front.
@markples
static-none uses args - add comment to RequiresProcessIsolation
ce6b442
@markples
remove unused arg in Runtime_80731.il
638a77f
@markples
Rewrite 4 IL tests to use newarr instead of args
4eb4696
@markples
[cs-main] remove unused args in C# tests
ec49b49
@markples
Update 9 tests with Main(args)
6ac253f
@markples
Workaround: add RequiresProcessIsolation to LdfldaHack
16d7c0a
@markples
Move [Fact] to correct location in b12263
8becab5
@markples
In b06020, wrap $ class so that the C# wrapper can call it
a6dcac8
@markples
[ILTransform -public]
997f40a
@markples
[ILTransform -sr]
b4ebb25
@markples
Manually rename b598031/test and test2.
f7327fb
These were missed by the previous ILTransform -n run.
They no longer clash because run renamed all of their conflicts,
but this renames them for consistency.
@markples
[ILTransform -prociso] Rerun for new tests
59646e6
@markples
Wrappers
f720b0a
@markples
[ILTransform -ilfact]
c086eb2
@markples
Several fixes:
97a58dc
- Manually do missed -collapse-main-sig and fix placement of [Fact]
- Fix b89946 that referred to Main
- Fix Runtime_59444 reflection on Test*
@markples
Fix TestSummary.cs to handle invalid XML characters
660e2e3
@markples
Manual fix for reflection on rename in Runtime_64883
5f5f99b
@markples
[ILTransform -a]
d3462a5
@markples
Revert "Manual fix for reflection on rename in Runtime_64883"
da5c0ed
This reverts commit 5f5f99b.
@markples
Manual fixes
88444fd
@markples
Update test groupings and add extern aliases
3f795e4
@markples
Merge remote-tracking branch 'dotnet/main' into merge6
c08dbb1
@markples
Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.…
2a1d7da
…analyzers errors
@markples
[ILTransform -public]
6989991
@markples
[ILTransform -prociso]
a5f722d
@markples
[ILTransform -ilfact] (undo GitHub_26491)
0631574
@ghostghost
assigned markplesMar 24, 2023
@ghost
ghost
commented
Mar 24, 2023

Copy link


Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata

See info in area-owners.md if you want to be subscribed.
Issue Details
null
Author:markples
Assignees:markples
Labels:
area-System.Reflection.Metadata
Milestone:-
@markplesmarkples
added this to the 8.0.0 milestone
Mar 24, 2023
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
fyi @dotnet/jit-contrib - There is considerable demand to see the effects of merging asap, so I'm currently planning on merging this as soon as I have the last few tests (including new ones as they race against this PR) converted. This will require me monitoring new tests for some of the mistakes that we hope/plan to catch automatically.
@markples
markples
marked this pull request as ready for review
March 29, 2023 05:16
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
@trylek#71732 has the (original) measurements on this, though of course the precise numbers change as more are added, deleted, or split apart. This PR will bring Brian/my efforts to ~2500 with ~1500 probably not far behind. This doesn't count (numbers via dir /s /b *proj | wc) Methodical (~2000), HardwareIntrinsics (161, but there's more going on there), or Loader\classloader\TypeGeneratorTests (~1500). These include priority=1 tests.
@markples
59444 - Changing methods to internal broke reflection calls. Use this
4dc6ecb
opportunity to remove the reflection and simply mark the callees as [Fact]s.
64883/76273 - Suppress the xunit warning so that the code can continue
using the normal (public-only) reflection search.
@markples
markples
commented
Mar 29, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 29, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 29, 2023

Closed

@JulieLeeMSFTJulieLeeMSFT
mentioned this pull request
Mar 29, 2023

Closed

17 tasks
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Mar 30, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 30, 2023

Closed

@markples
markples
commented
Mar 30, 2023

Copy link


Copy Markdown

ContributorAuthor
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere? The closest that I could find was an old issue about OSX (#73299). This is x64 linux. Thanks for any insight.
Console log: 'nativeaot.SmokeTests' from job 54cb44e8-0e3d-48ea-ad42-fbb86e93557d workitem 5d0b02f1-7e34-4da5-9b13-e2a8618b4d50 (ubuntu.1804.amd64.open.rt) executed on machine a00AO7E running Linux-5.4.0-1103-azure-x86_64-with-Ubuntu-18.04-bionic

https://dev.azure.com/dnceng-public/public/_build/results?buildId=222351&view=logs&j=ddb4415b-4613-5bce-e937-0da25336f8b9&t=045c6e99-7b9a-570f-bc7f-14c39fc632af
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-83895-merge-54cb44e80e3d48eaad/nativeaot.SmokeTests/1/console.7cfae149.log?helixlogtype=result
nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh [FAIL]
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

 --------------------------------
Running Test: ThreadLocalStatics.TLSTesting.ThreadLocalStatics_Test
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 3/30/2023 8:17:34 AM
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B60409DF/p
> /datadisks/disk1/work/B60409DF/w/AB650937/e/nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
Expected: True
Actual: False
Stack Trace:
at nativeaot_SmokeTests._DynamicGenerics_DynamicGenerics_DynamicGenerics_._DynamicGenerics_DynamicGenerics_DynamicGenerics_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:
/datadisks/disk1/work/B60409DF/p/nativeaottest.sh: line 15: 8680 Aborted (core dumped) $_DebuggerFullPath $1/native/$exename "${@:3}"

@MichalStrehovsky
MichalStrehovsky
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
@MichalStrehovsky I just hit this failure. This PR adds test merging to JIT\Regression so in theory should have no impact on nativeaot (product or tests). Is this something you've seen elsewhere?
I'm on vacation. Cc @VSadov who just merged a TLS change yesterday. It's either a big coincidence, or related. We should be able to pull down the coredump with runfo.
@VSadov
VSadov
commented
Mar 31, 2023

Copy link


Copy Markdown

Member
I will take a look at the TLS issue. Could be related to the recent change. Does it fail on Windows as well?
@VSadov
VSadov
commented
Mar 31, 2023•

edited

Loading

Copy link


Copy Markdown

Member
I've obtained the dump and the test binary that crushed. So far not much luck.

The dump is not very useful.
(21e8.38cb): Signal SIGABRT code SI_TKILL (Sent by tkill system call) originating from PID 21e8

The stacks are missing any kind of symbolic info though. Hard to tell what happened.

The test binary runs and passes. I would not be surprised if the test will pass if reset. Could indeed be a coincidence.
I will look over the Unix parts of the TLS change. Maybe I can spot something in the code.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
I only saw the Linux x64 failure (in CI), and as you suspected, it passed on retry.
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
close/reopen to poke the license/cla check
@markplesmarkples
reopened this
Mar 31, 2023
@markples
markples
merged commit 52bbeeb
into
dotnet:mainMar 31, 2023
@BruceForstall
BruceForstall
commented
Mar 31, 2023

Copy link


Copy Markdown

Contributor
@markples There are new gcstress timeouts due to this change: https://dev.azure.com/dnceng-public/public/_build/results?buildId=224309&view=ms.vss-test-web.build-test-results-tab
(I kicked off a gcstress run immediately after you merged to check this)
@markples
markples
commented
Mar 31, 2023

Copy link


Copy Markdown

ContributorAuthor
Thanks @BruceForstall. I've attempted to fix these symptoms based on past failures in #84193. They may need additional striping. I think I will prepare a PR that simply sets RequiresProcessIsolation (run one test per process with the simpler wrapper).

This will avoid the additional churn that would occur by reverting/remerging.
@ghostghost
locked as resolved and limited conversation to collaborators
May 1, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@trylektrylektrylek approved these changes

Assignees

@markplesmarkples

Labels

test-enhancementImprovements of test source code

Projects


None yet


Milestone

8.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@markples@BruceForstall@MichalStrehovsky@VSadov@trylek