ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adamsitnik merged 6 commits into from
Jul 22, 2024
Merged

ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch #104972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3092d86
Increasing the tolerance for Mono Runtime for the flaky ProcessThread…
adamsitnik Jul 16, 2024
6d17edc
select the thread based on current thread native id, remove the try c…
adamsitnik Jul 17, 2024
2cc7ecb
Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…
adamsitnik Jul 17, 2024
d9350ff
fix the Linux build: gettid is not exposed in older versions of libc
adamsitnik Jul 17, 2024
0de6947
Make the test conditional
adamsitnik Jul 18, 2024
dabec4f
get SYS_gettid value specific to given architecture (and hopefully fi…
adamsitnik Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
}

private static int GetCurrentThreadId()
{
// The magic values come from https://github.com/torvalds/linux.
int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
{
Architecture.Arm => 224,
Architecture.Arm64 => 178,
Architecture.X86 => 224,
Architecture.X64 => 186,
Architecture.S390x => 236,
Architecture.Ppc64le => 207,
Architecture.RiscV64 => 178,
_ => 178,
};

return syscall(SYS_gettid);
}

[DllImport("libc")]
private static extern int syscall(int nr);
}
}
21 changes: 10 additions & 11 deletions src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
}
}

[Fact]
[PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
[PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
public async Task TestStartTimeProperty()
{
TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
await Task.Factory.StartNew(() =>
{
p.Refresh();
try
{
var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();

@jozkeejozkeeJul 17, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?

Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.

@adamsitnikadamsitnikJul 17, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?

There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.

@adamsitnikadamsitnikJul 17, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.

In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.

@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?

@kouvelkouvelJul 17, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53 in 27776e2

if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}

It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.

Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]

@kouvelkouvelJul 17, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in:

runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621 in 27776e2

ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>

@adamsitnikadamsitnikJul 18, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!

@danmoseleydanmoseleyJul 18, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.

TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.

@kouvelkouvelJul 19, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.

Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
}
catch (InvalidOperationException)
{
// A thread may have gone away between our getting its info and attempting to access its StartTime
}

int newThreadId = GetCurrentThreadId();

ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);

@jozkeejozkeeJul 18, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?

@adamsitnikadamsitnikJul 18, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)

@jozkeejozkeeJul 18, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.

}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
}
}
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch#104972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

adamsitnik
merged 6 commits into

from

Jul 22, 2024
 Merged


ProcessThreadTests.TestStartTimeProperty - Get spawned thread by ID from Process.Threads to avoid mismatch
#104972
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
3092d86

Increasing the tolerance for Mono Runtime for the flaky ProcessThread…


adamsitnik Jul 16, 2024
6d17edc

select the thread based on current thread native id, remove the try c…


adamsitnik Jul 17, 2024
2cc7ecb

Revert "Increasing the tolerance for Mono Runtime for the flaky Proce…


adamsitnik Jul 17, 2024
d9350ff

fix the Linux build: gettid is not exposed in older versions of libc


adamsitnik Jul 17, 2024
0de6947

Make the test conditional


adamsitnik Jul 18, 2024
dabec4f

get SYS_gettid value specific to given architecture (and hopefully fi…


adamsitnik Jul 19, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 22 additions & 0 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.Unix.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,6 +2,7 @@
// The .NET Foundation licenses this file to you under the MIT license.

using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
Expand DownExpand Up@@ -30,5 +31,26 @@ public void TestPriorityLevelProperty_Unix()

 Assert.Throws<PlatformNotSupportedException>(() => thread.PriorityLevel = level);
 }

 private static int GetCurrentThreadId()
 {
 // The magic values come from https://github.com/torvalds/linux.
 int SYS_gettid = RuntimeInformation.ProcessArchitecture switch
 {
 Architecture.Arm => 224,
 Architecture.Arm64 => 178,
 Architecture.X86 => 224,
 Architecture.X64 => 186,
 Architecture.S390x => 236,
 Architecture.Ppc64le => 207,
 Architecture.RiscV64 => 178,
 _ => 178,
 };

 return syscall(SYS_gettid);
 }

 [DllImport("libc")]
 private static extern int syscall(int nr);
 }
}

21 changes: 10 additions & 11 deletions
src/libraries/System.Diagnostics.Process/tests/ProcessThreadTests.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,7 @@
using Microsoft.DotNet.RemoteExecutor;
using Xunit;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace System.Diagnostics.Tests
{
Expand DownExpand Up@@ -107,8 +108,8 @@ public void TestStartTimeProperty_OSX()
 }
 }

 [Fact]
 [PlatformSpecific(TestPlatforms.Linux|TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 [ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
 [PlatformSpecific(TestPlatforms.Linux | TestPlatforms.Windows)] // OSX and FreeBSD throw PNSE from StartTime
 public async Task TestStartTimeProperty()
 {
 TimeSpan allowedWindow = TimeSpan.FromSeconds(2);
Expand DownExpand Up@@ -148,15 +149,13 @@ public async Task TestStartTimeProperty()
 await Task.Factory.StartNew(() =>
 {
 p.Refresh();
 try
 {
 var newest = p.Threads.Cast<ProcessThread>().OrderBy(t => t.StartTime.ToUniversalTime()).Last();
@jozkeejozkeeJul 17, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
Is also possible that the property does have the threads ordered but StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was it wrongly assumed that Process.Threads contained the threads ordered with the most recently created as the Last() one?
There are other tests running in parallel and each of them can create a new thread. Even if we disable that, the ThreadPool can still start a new thread on it's own. So to get 100% deterministic behavior we should IMO perform an ID match.
@adamsitnikadamsitnikJul 17, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StartNew wasn't creating a new thread but reusing an existing one as Dan mentioned.
In theory CLR and Mono are sharing the same thread pool implementation, so I would expect TaskCreationOptions.LongRunning to always create a new thread, but I might be wrong.
@kouvel is it safe to assume that Task.Factory.StartNew(/**/, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); executed from an xUnit test runner will always spawn a new thread?
@kouvelkouvelJul 17, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on this:
runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs

Lines 45 to 53
in
27776e2
if(Thread.IsThreadStartSupported&&(options&TaskCreationOptions.LongRunning)!=0)
{
// Run LongRunning tasks on their own dedicated thread.
newThread(s_longRunningThreadWork)
{
IsBackground=true,
Name=".NET Long Running Task"
}.UnsafeStart(task);
}
It looks like when thread start is supported, a new thread is created for each long-running task. The threads used for long-running tasks are not pooled currently, though they could be. Probably creating a new thread would be more explicit in intent.
Wonder if the failure is happening when thread start is not supported. In the threading tests that create new threads, the following condition is used, which appears to be the same condition as Thread.IsThreadStartSupported:
[ConditionalFact(typeof(PlatformDetection), nameof(PlatformDetection.IsThreadingSupported))]
@kouvelkouvelJul 17, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although using a long-running task would avoid an unhandled exception/crash on assertion failure in the new thread. ThreadTestHelpers.CreateGuardedThread could be used for that instead, as in: 
runtime/src/libraries/System.Threading.Thread/tests/ThreadTests.cs

Line 621
in
27776e2
ThreadTestHelpers.CreateGuardedThread(outwaitForThread,()=>
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kouvel thank you!
@danmoseleydanmoseleyJul 18, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The threads used for long-running tasks are not pooled currently, though they could be.
TIL - I should have checked the code. @kouvel is that worth an issue, as a suggestion? Basically pull a thread from the pool if available, and trigger a replacement to start. I have no idea.
@kouvelkouvelJul 19, 2024

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's done currently is simple and may be sufficient if most long-running work items rarely or never complete. If there's a need where work items would complete frequently enough, and are not short enough and need to be classified as long-running, then it may be worthwhile to pool those threads.
 Assert.InRange(newest.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
 }
 catch (InvalidOperationException)
 {
 // A thread may have gone away between our getting its info and attempting to access its StartTime
 }

 int newThreadId = GetCurrentThreadId();

 ProcessThread[] processThreads = p.Threads.Cast<ProcessThread>().ToArray();
 ProcessThread newThread = Assert.Single(processThreads, thread => thread.Id == newThreadId);

 Assert.InRange(newThread.StartTime.ToUniversalTime(), curTime - allowedWindow, DateTime.Now.ToUniversalTime() + allowedWindow);
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we get the correct thread, could we get rid of the tolerance?
@adamsitnikadamsitnikJul 18, 2024

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reverted the increase of the tolerance. I think it's safer to keep the current one (I am just afraid it would make the test even more flaky)
@jozkeejozkeeJul 18, 2024

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with your fix the test won't be flaky at all 😆 but I might be wrong.
 }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
 }
 }
Expand Down