[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

[MONO][Marshal] Fix race condition in marshal callback installation - #77383

Closed
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix
Closed

[MONO][Marshal] Fix race condition in marshal callback installation#77383
naricc wants to merge 2 commits into
dotnet:mainfrom
naricc:naricc/marshal-racefix

Conversation

@naricc

@nariccnaricc commented Oct 24, 2022

Copy link
Copy Markdown
Contributor

This fixes a race condition on ilgen_cb_inited; this flag was used to determine if callbacks were already installed, but had no synchronization around it. Replaced with a pointer to a dynamically allocated buffer and a cas.

Fixes: #74603
Fixes: #77090

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-extra-platforms

@naricc

Copy link
Copy Markdown
ContributorAuthor

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

void
mono_install_marshal_callbacks_ilgen (MonoMarshalIlgenCallbacks *cb)
{
g_assert (!ilgen_cb_inited);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the race occur ? This should be called during startup.

@nariccnariccOct 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkotas Was reporting what looks like a race condition here: #77090. I'm not sure exactly what interleaving can cause this though, and have not been able to trigger it locally.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vargaz@naricc since e107bcb, mono_marshal_ilgen_initmay be called by embedders early, but it doesn't have to be called. In the case where the driver doesn't call it early (for example the normal desktop mono driver doesn't call it), get_marshal_cb will initialize lazily. The race is in the lazy case where one thread can see ilgen_cb_inited == TRUE, but the callbacks are all still null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't it get called during runtime startup ?

I was surprised, but apparently it doesn't. Going back to the very first PR it's always been lazy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now on wasm at least, all such initialization needs to happen before the runtime is initialized, i.e.
https://github.com/dotnet/runtime/blob/main/src/mono/wasm/runtime/driver.c#L593
When mono_jit_init_version () is called, it can check whenever the embedder has made some custom changes, and if not, initialize things the default way.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I don't have any objection to just using mono_marshal_ilgen_init to set a flag and doing all the callback initialization late during startup. @naricc I think that's what you wanted to do for the component in the first place right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lambdageek That is what I am doing for the componentization change.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in the case that we are using NOILGEN and no one has called mono_marshal_ilgen_init we just want to go ahead and install the noilgen callbacks?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea that sounds right.

So is that what we should do here too, instead of the thing with atomics? mono_marshal_ilgen_init sets a flag; we check that flag during, say, mini_init(). At which point we do callback installation if it is set?

Yea, basically at the point where in the componentized versino we'd call the component init function, here we'd just set up the callbacks.

I think writing down #77383 (comment) helped me to think about what the real problem is. It's not concurrency. It's that we have a confusing API for choosing between ilgen/noilgen. We should init the marshaling stack at startup - there's no need for it to be lazy. You were right in your proposal about mono_marshal_ilgen_init (just use it to set a flag), but I didn't understand the problem properly.

lambdageek
lambdageek previously approved these changes Oct 24, 2022

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. needs barriers. also some style nits

memcpy (&ilgen_marshal_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use g_malloc/g_free

MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL ))
if (mono_atomic_cas_ptr((void**)&ilgen_marshal_cb, local_cb, NULL )!=NULL)

nit: usually the pattern is if (mono_atomic_cas_ptr (dest, newValue, expectedValue) != expectedValue)

ilgen_cb_inited = TRUE;
}
MonoMarshalIlgenCallbacks* local_cb = (MonoMarshalIlgenCallbacks*)malloc(sizeof(MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a memory barrier between the memcpy and the CAS

Suggested change
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalIlgenCallbacks));
mono_memory_barrier ();

Comment threadsrc/mono/mono/metadata/marshal.c Outdated
MonoMarshalLightweightCallbacks* local_cb = (MonoMarshalLightweightCallbacks*)malloc(sizeof(MonoMarshalLightweightCallbacks));
memcpy (local_cb, cb, sizeof (MonoMarshalLightweightCallbacks));

if (mono_atomic_cas_ptr((void**)&marshal_lightweight_cb, local_cb, NULL))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comments as the other method:

  1. use g_malloc/g_free
  2. barrier after memcpy
  3. add != NULL to the if condition

@lambdageek

Copy link
Copy Markdown
Member

Added fix for race condition.

Please use a better summary. something like [mono][marshal] Fix race condition in callback initialization

@lambdageek
lambdageek dismissed their stale reviewOctober 24, 2022 19:24

I guess we're going with a flag instead of atomics

@nariccnaricc changed the title Added fix for race condition.[MONO] Fix race condition in marshal callback installationOct 24, 2022
@nariccnaricc changed the title [MONO] Fix race condition in marshal callback installation[MONO][Marshal] Fix race condition in marshal callback installationOct 24, 2022
@naricc

Copy link
Copy Markdown
ContributorAuthor

I am abandoing this PR in favor of doing it the flag way ( see discussion above). That is a different change, so I will do it in a different PR.

@nariccnaricc closed this Oct 25, 2022
@ghostghost locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Mono] System.Text.RegularExpressions.Tests source generator tests crash in DeflateInit2_ Race condition on marshal-ilgen init

3 participants

@naricc@lambdageek@vargaz