Fixing memory ordering issue in ConcurrentQueue

[VSadov]

Fixes: #76501

[ghost]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes: #76501

Author:	VSadov
Assignees:	VSadov
Labels:	area-System.Collections
Milestone:	-

[VSadov]

I am running the repro in multiple processes for about 20 min now with no failures. It typically fails in under one minute.

[VSadov]

No failures for an hour. So it looks like the scenario has no other issues.

[VSadov]

thanks @filipnavara for a repro scenario!

[filipnavara]

Thanks for the fix, LGTM. It should probably be backported to 7.0.

[stephentoub]

Thanks. Nice find.

[stephentoub]

It should probably be backported to 7.0.

I'm fine with that; CQ is at the heart of the system due to its usage in ThreadPool. Just note this isn't new; been this way since I missed adding the volatile in 2016 :)
https://github.com/dotnet/corefx/pull/14254/files#diff-c12f070bb2033f631c1e51c214b49d678ecb1c871b24652213654486f8cb187bR567

[VSadov]

Thanks!!

[VSadov]

The issue is on ToArray/CopyTo/Enumerate path. I think it would be unusual to do such operations while the queue is mutated. Any kind of synchronization that guarantees quiescence will likely make this issue disappear.

Also it requires weak architecture and hardware that utilizes that weakness aggressively. - Missing write fences often cause troubles as write-buffering is common. Missing read fences require that CPU speculates far ahead. These are probably still not very common. Evidently, we have only seen this issue on M1.

I think chances of seeing this causing problems in actual programs are low. On the other hand it is possible, and the fix is very low risk and 7.0 may see a bigger share of weak architectures than prior releases.

I think it may be worth porting.

[VSadov]

I also think this issue may become a test/stress nuisance in 7.0

It was quite annoying for NativeAOT on OSX lately as it failed fairly often

[VSadov]

/backport to release/7.0

[github-actions]

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3439145097