Skip to content

Fix a deadlock in NonGC + Profiler API - #90847

Merged
EgorBo merged 21 commits into
dotnet:mainfrom
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged

Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom
EgorBo:fix-deadlock-nongc

Conversation

@EgorBo

@EgorBoEgorBo commented Aug 19, 2023

Copy link
Copy Markdown
Member

Fixes#90830

Quick explanation how's the dead-lock happening:
Thread1:
Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.

Thread2 (GC's):
GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.

The fix is #90830 (comment)

@ghostghost added the area-VM-coreclr label Aug 19, 2023
@ghostghost assigned EgorBoAug 19, 2023
@EgorBo

EgorBo commented Aug 19, 2023

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Aug 19, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

jkotas
jkotas reviewed Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
jkotas
jkotas reviewed Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
@EgorBo
EgorBoforce-pushed the fix-deadlock-nongc branch from 64b9fd0 to cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo marked this pull request as ready for review August 20, 2023 13:59
@EgorBo

EgorBo commented Aug 20, 2023

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Aug 20, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

jkotas
jkotas reviewed Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
Comment threadsrc/coreclr/vm/appdomain.hpp Outdated
@EgorBo

EgorBo commented Aug 20, 2023

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Aug 20, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@build-analysisbuild-analysisBot mentioned this pull request Aug 21, 2023
Closed
@davmason
davmason requested a review from a teamAugust 23, 2023 01:02
jkotas
jkotas reviewed Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
@EgorBo

EgorBo commented Aug 23, 2023

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Aug 23, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@build-analysisbuild-analysisBot mentioned this pull request Aug 24, 2023
Closed
EgorBoand others added 3 commits August 24, 2023 20:37
jkotas
jkotas reviewed Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
jkotas
jkotas reviewed Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
_ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
_ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

// FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.

@jkotasjkotasAug 24, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?

We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..

@EgorBoEgorBoAug 25, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes

jkotas
jkotas reviewed Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
jkotas
jkotas approved these changes Aug 25, 2023
View reviewed changes

@jkotasjkotas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!

@build-analysisbuild-analysisBot mentioned this pull request Aug 25, 2023
Closed
jkotas
jkotas reviewed Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
@EgorBo

EgorBo commented Aug 25, 2023

Copy link
Copy Markdown
MemberAuthor

/azp run runtime-coreclr outerloop

@azure-pipelines

azure-pipelinesBot commented Aug 25, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@EgorBo

EgorBo commented Aug 25, 2023

Copy link
Copy Markdown
MemberAuthor

Looks good to me otherwise. Thank you!

Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂

jkotas
jkotas reviewed Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp Outdated
@runfoapprunfoappBot mentioned this pull request Aug 25, 2023
Open
@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo merged commit f9382df into dotnet:mainAug 25, 2023
@EgorBo
EgorBo deleted the fix-deadlock-nongc branch August 25, 2023 18:19
@EgorBo
EgorBo restored the fix-deadlock-nongc branch August 25, 2023 18:19
@EgorBo

EgorBo commented Aug 25, 2023

Copy link
Copy Markdown
MemberAuthor

/backport to release/8.0

@github-actions

github-actionsBot commented Aug 25, 2023

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788

@github-actionsgithub-actionsBot mentioned this pull request Aug 25, 2023
Merged
@EgorBo
EgorBo deleted the fix-deadlock-nongc branch August 25, 2023 18:28
@EgorBoEgorBo mentioned this pull request Aug 25, 2023
Closed
@yowlyowl mentioned this pull request Sep 11, 2023
Merged
@EgorBoEgorBo mentioned this pull request Sep 13, 2023
Merged
@ghostghost locked as resolved and limited conversation to collaborators Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments
+1 more reviewer
@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7

4 participants

@EgorBo@jkotas@cshung@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Fix a deadlock in NonGC + Profiler API by EgorBo · Pull Request #90847 · dotnet/runtime · GitHub
Skip to content
Fix a deadlock in NonGC + Profiler API - #90847
Merged
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged
Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Conversation
@EgorBo
@EgorBoEgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

Member
Fixes#90830
Quick explanation how's the dead-lock happening:
Thread1:

Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.
Thread2 (GC's):

GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.
The fix is #90830 (comment)
@ghostghost
added
the 
area-VM-coreclr

label
Aug 19, 2023
@ghostghost
assigned EgorBoAug 19, 2023
@EgorBo
EgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 19, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBoforce-pushed
the
fix-deadlock-nongc
branch
from
64b9fd0 to
cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo
marked this pull request as ready for review
August 20, 2023 13:59
@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/appdomain.hpp
Outdated

@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 21, 2023

Closed

@davmason
davmason
requested a review
from a teamAugust 23, 2023 01:02
jkotas
jkotas
reviewed
Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 23, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 24, 2023

Closed

EgorBoand others
added 3 commits
August 24, 2023 20:37
jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

 // FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.
@jkotasjkotasAug 24, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?
We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..
@EgorBoEgorBoAug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
approved these changes
Aug 25, 2023
View reviewed changes
@jkotasjkotas
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 25, 2023

Closed

jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 25, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Looks good to me otherwise. Thank you!
Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@runfoapprunfoappBot
mentioned this pull request
Aug 25, 2023

Open

@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo
merged commit f9382df
into
dotnet:mainAug 25, 2023
@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
restored the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/backport to release/8.0
@github-actions
github-actionsBot
commented
Aug 25, 2023

Copy link


Copy Markdown

Contributor
Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 25, 2023
 Merged

@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:28
@EgorBoEgorBo
mentioned this pull request
Aug 25, 2023
 Closed

@yowlyowl
mentioned this pull request
Sep 11, 2023
 Merged

@EgorBoEgorBo
mentioned this pull request
Sep 13, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments

+1 more reviewer

@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7


4 participants

@EgorBo@jkotas@cshung@mangod9

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix a deadlock in NonGC + Profiler API by EgorBo · Pull Request #90847 · dotnet/runtime · GitHub
Skip to content
Fix a deadlock in NonGC + Profiler API - #90847
Merged
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged
Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Conversation
@EgorBo
@EgorBoEgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

Member
Fixes#90830
Quick explanation how's the dead-lock happening:
Thread1:

Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.
Thread2 (GC's):

GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.
The fix is #90830 (comment)
@ghostghost
added
the 
area-VM-coreclr

label
Aug 19, 2023
@ghostghost
assigned EgorBoAug 19, 2023
@EgorBo
EgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 19, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBoforce-pushed
the
fix-deadlock-nongc
branch
from
64b9fd0 to
cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo
marked this pull request as ready for review
August 20, 2023 13:59
@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/appdomain.hpp
Outdated

@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 21, 2023

Closed

@davmason
davmason
requested a review
from a teamAugust 23, 2023 01:02
jkotas
jkotas
reviewed
Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 23, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 24, 2023

Closed

EgorBoand others
added 3 commits
August 24, 2023 20:37
jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

 // FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.
@jkotasjkotasAug 24, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?
We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..
@EgorBoEgorBoAug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
approved these changes
Aug 25, 2023
View reviewed changes
@jkotasjkotas
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 25, 2023

Closed

jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 25, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Looks good to me otherwise. Thank you!
Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@runfoapprunfoappBot
mentioned this pull request
Aug 25, 2023

Open

@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo
merged commit f9382df
into
dotnet:mainAug 25, 2023
@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
restored the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/backport to release/8.0
@github-actions
github-actionsBot
commented
Aug 25, 2023

Copy link


Copy Markdown

Contributor
Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 25, 2023
 Merged

@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:28
@EgorBoEgorBo
mentioned this pull request
Aug 25, 2023
 Closed

@yowlyowl
mentioned this pull request
Sep 11, 2023
 Merged

@EgorBoEgorBo
mentioned this pull request
Sep 13, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments

+1 more reviewer

@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7


4 participants

@EgorBo@jkotas@cshung@mangod9

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix a deadlock in NonGC + Profiler API by EgorBo · Pull Request #90847 · dotnet/runtime · GitHub
Skip to content
Fix a deadlock in NonGC + Profiler API - #90847
Merged
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged
Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Conversation
@EgorBo
@EgorBoEgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

Member
Fixes#90830
Quick explanation how's the dead-lock happening:
Thread1:

Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.
Thread2 (GC's):

GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.
The fix is #90830 (comment)
@ghostghost
added
the 
area-VM-coreclr

label
Aug 19, 2023
@ghostghost
assigned EgorBoAug 19, 2023
@EgorBo
EgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 19, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBoforce-pushed
the
fix-deadlock-nongc
branch
from
64b9fd0 to
cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo
marked this pull request as ready for review
August 20, 2023 13:59
@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/appdomain.hpp
Outdated

@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 21, 2023

Closed

@davmason
davmason
requested a review
from a teamAugust 23, 2023 01:02
jkotas
jkotas
reviewed
Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 23, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 24, 2023

Closed

EgorBoand others
added 3 commits
August 24, 2023 20:37
jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

 // FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.
@jkotasjkotasAug 24, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?
We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..
@EgorBoEgorBoAug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
approved these changes
Aug 25, 2023
View reviewed changes
@jkotasjkotas
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 25, 2023

Closed

jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 25, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Looks good to me otherwise. Thank you!
Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@runfoapprunfoappBot
mentioned this pull request
Aug 25, 2023

Open

@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo
merged commit f9382df
into
dotnet:mainAug 25, 2023
@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
restored the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/backport to release/8.0
@github-actions
github-actionsBot
commented
Aug 25, 2023

Copy link


Copy Markdown

Contributor
Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 25, 2023
 Merged

@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:28
@EgorBoEgorBo
mentioned this pull request
Aug 25, 2023
 Closed

@yowlyowl
mentioned this pull request
Sep 11, 2023
 Merged

@EgorBoEgorBo
mentioned this pull request
Sep 13, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments

+1 more reviewer

@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7


4 participants

@EgorBo@jkotas@cshung@mangod9

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Fix a deadlock in NonGC + Profiler API by EgorBo · Pull Request #90847 · dotnet/runtime · GitHub
Skip to content
Fix a deadlock in NonGC + Profiler API - #90847
Merged
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged
Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Conversation
@EgorBo
@EgorBoEgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

Member
Fixes#90830
Quick explanation how's the dead-lock happening:
Thread1:

Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.
Thread2 (GC's):

GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.
The fix is #90830 (comment)
@ghostghost
added
the 
area-VM-coreclr

label
Aug 19, 2023
@ghostghost
assigned EgorBoAug 19, 2023
@EgorBo
EgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 19, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBoforce-pushed
the
fix-deadlock-nongc
branch
from
64b9fd0 to
cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo
marked this pull request as ready for review
August 20, 2023 13:59
@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/appdomain.hpp
Outdated

@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 21, 2023

Closed

@davmason
davmason
requested a review
from a teamAugust 23, 2023 01:02
jkotas
jkotas
reviewed
Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 23, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 24, 2023

Closed

EgorBoand others
added 3 commits
August 24, 2023 20:37
jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

 // FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.
@jkotasjkotasAug 24, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?
We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..
@EgorBoEgorBoAug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
approved these changes
Aug 25, 2023
View reviewed changes
@jkotasjkotas
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 25, 2023

Closed

jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 25, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Looks good to me otherwise. Thank you!
Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@runfoapprunfoappBot
mentioned this pull request
Aug 25, 2023

Open

@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo
merged commit f9382df
into
dotnet:mainAug 25, 2023
@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
restored the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/backport to release/8.0
@github-actions
github-actionsBot
commented
Aug 25, 2023

Copy link


Copy Markdown

Contributor
Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 25, 2023
 Merged

@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:28
@EgorBoEgorBo
mentioned this pull request
Aug 25, 2023
 Closed

@yowlyowl
mentioned this pull request
Sep 11, 2023
 Merged

@EgorBoEgorBo
mentioned this pull request
Sep 13, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments

+1 more reviewer

@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7


4 participants

@EgorBo@jkotas@cshung@mangod9

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix a deadlock in NonGC + Profiler API by EgorBo · Pull Request #90847 · dotnet/runtime · GitHub
Skip to content
Fix a deadlock in NonGC + Profiler API - #90847
Merged
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged
Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Conversation
@EgorBo
@EgorBoEgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

Member
Fixes#90830
Quick explanation how's the dead-lock happening:
Thread1:

Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.
Thread2 (GC's):

GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.
The fix is #90830 (comment)
@ghostghost
added
the 
area-VM-coreclr

label
Aug 19, 2023
@ghostghost
assigned EgorBoAug 19, 2023
@EgorBo
EgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 19, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBoforce-pushed
the
fix-deadlock-nongc
branch
from
64b9fd0 to
cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo
marked this pull request as ready for review
August 20, 2023 13:59
@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/appdomain.hpp
Outdated

@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 21, 2023

Closed

@davmason
davmason
requested a review
from a teamAugust 23, 2023 01:02
jkotas
jkotas
reviewed
Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 23, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 24, 2023

Closed

EgorBoand others
added 3 commits
August 24, 2023 20:37
jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

 // FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.
@jkotasjkotasAug 24, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?
We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..
@EgorBoEgorBoAug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
approved these changes
Aug 25, 2023
View reviewed changes
@jkotasjkotas
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 25, 2023

Closed

jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 25, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Looks good to me otherwise. Thank you!
Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@runfoapprunfoappBot
mentioned this pull request
Aug 25, 2023

Open

@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo
merged commit f9382df
into
dotnet:mainAug 25, 2023
@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
restored the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/backport to release/8.0
@github-actions
github-actionsBot
commented
Aug 25, 2023

Copy link


Copy Markdown

Contributor
Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 25, 2023
 Merged

@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:28
@EgorBoEgorBo
mentioned this pull request
Aug 25, 2023
 Closed

@yowlyowl
mentioned this pull request
Sep 11, 2023
 Merged

@EgorBoEgorBo
mentioned this pull request
Sep 13, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments

+1 more reviewer

@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7


4 participants

@EgorBo@jkotas@cshung@mangod9

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
Fix a deadlock in NonGC + Profiler API by EgorBo · Pull Request #90847 · dotnet/runtime · GitHub
Skip to content
Fix a deadlock in NonGC + Profiler API - #90847
Merged
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Aug 25, 2023
Merged
Fix a deadlock in NonGC + Profiler API#90847
EgorBo merged 21 commits into
dotnet:mainfrom 
EgorBo:fix-deadlock-nongc
Conversation
@EgorBo
@EgorBoEgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

Member
Fixes#90830
Quick explanation how's the dead-lock happening:
Thread1:

Someone (typically, JIT) tries to allocate an object on NonGC heap. FrozenObjectHeapManager (FOHM) acquires its lock and calls GC's API RegisterNewSegment. That API internally can hit a case when a GC is happening so it has to wait for GC to complete.
Thread2 (GC's):

GC is executing a callback (e.g. GarbageCollectionFinished or *Started) and Profiler uses that callback to enumerate objects on NonGC heap via ICorProfilerInfo14::GetNonGCHeapBounds thus, it also tries to acquire FOHM's lock (to be able to safely enumerate the objects). Thus, GC's thread (Thread2) is wating for FOHM's lock to release (it's taken by Thread1) while Thread1 is waiting for GC to finish.
The fix is #90830 (comment)
@ghostghost
added
the 
area-VM-coreclr

label
Aug 19, 2023
@ghostghost
assigned EgorBoAug 19, 2023
@EgorBo
EgorBo
commented
Aug 19, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 19, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 19, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBoforce-pushed
the
fix-deadlock-nongc
branch
from
64b9fd0 to
cec6257CompareAugust 20, 2023 13:21
@EgorBo
EgorBo
marked this pull request as ready for review
August 20, 2023 13:59
@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
jkotas
jkotas
reviewed
Aug 20, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/profilingenumerators.cpp
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

Comment threadsrc/coreclr/vm/appdomain.hpp
Outdated

@EgorBo
EgorBo
commented
Aug 20, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 20, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 21, 2023

Closed

@davmason
davmason
requested a review
from a teamAugust 23, 2023 01:02
jkotas
jkotas
reviewed
Aug 23, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 23, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 24, 2023

Closed

EgorBoand others
added 3 commits
August 24, 2023 20:37
jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
reviewed
Aug 24, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrent);
 _ASSERT((uint8_t*)obj >= m_pStart + sizeof(ObjHeader) && (uint8_t*)obj < m_pCurrentRegistered);

 // FOH doesn't support objects with non-DATA_ALIGNMENT alignment yet.
@jkotasjkotasAug 24, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to set m_NumComponents for arrays as part of TryAllocateObject?
We are setting it too late and we can end up enumerating arrays without m_NumComponents set that is not going to end wel..
@EgorBoEgorBoAug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! It also allowed to simplify the PublishObject logic a bit. The final API might be simplified a bit with C++ template to allow use of capturing lambdas for simplicity but that needed a bit more changes
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

jkotas
jkotas
approved these changes
Aug 25, 2023
View reviewed changes
@jkotasjkotas
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me otherwise. Thank you!
@build-analysisbuild-analysisBot
mentioned this pull request
Aug 25, 2023

Closed

jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/azp run runtime-coreclr outerloop
@azure-pipelines
azure-pipelinesBot
commented
Aug 25, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 1 pipeline(s).
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
Looks good to me otherwise. Thank you!
Thanks for the help! I wish I could easily spot all possible race conditions/corner cases just like you 🙂
jkotas
jkotas
reviewed
Aug 25, 2023
View reviewed changes
Comment threadsrc/coreclr/vm/frozenobjectheap.cpp
Outdated

@runfoapprunfoappBot
mentioned this pull request
Aug 25, 2023

Open

@EgorBo@jkotas
Update src/coreclr/vm/frozenobjectheap.cpp
2bbfd9c
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
@EgorBo
EgorBo
merged commit f9382df
into
dotnet:mainAug 25, 2023
@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
restored the

fix-deadlock-nongc

branch
August 25, 2023 18:19
@EgorBo
EgorBo
commented
Aug 25, 2023

Copy link


Copy Markdown

MemberAuthor
/backport to release/8.0
@github-actions
github-actionsBot
commented
Aug 25, 2023

Copy link


Copy Markdown

Contributor
Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/5979164788
@github-actionsgithub-actionsBot
mentioned this pull request
Aug 25, 2023
 Merged

@EgorBo
EgorBo
deleted the

fix-deadlock-nongc

branch
August 25, 2023 18:28
@EgorBoEgorBo
mentioned this pull request
Aug 25, 2023
 Closed

@yowlyowl
mentioned this pull request
Sep 11, 2023
 Merged

@EgorBoEgorBo
mentioned this pull request
Sep 13, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Sep 24, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@jkotasjkotasjkotas approved these changes
@mangod9mangod9mangod9 left review comments

+1 more reviewer

@cshungcshungcshung left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@EgorBoEgorBo

Labels

area-VM-coreclr

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

ProfilingAPI: ICorProfilerInfo14::GetNonGCHeapBounds deadlock on .NET 8.0-preview7


4 participants

@EgorBo@jkotas@cshung@mangod9