Enable parallel compilation features by default

[majocha]

This fixes remaining CI failures and enables by default --parallelcompilation. This includes:

• Parallel reference resolution
• Parallel optimizations Parallel optimization - behind a test flag --test:ParallelOptimization #14390
• Parallel ILxGen IlxGen parallelization #14372
• Graph based type checking Parallel type-checking in compilation (behind a feature flag) #14494

NOTE: For deterministic builds only parallel reference resolution will be enabled.

--parallelcompilation- switch can be used to revert all of the above to previous behavior.

See tracking issue #18989

TODO:

• restore switches
• double check that with everything off things still work

Some il baselines changed because of reordered methods.

This work was encouraged and kindly sponsored by Amplifying F# initiative 🚀

[github-actions]

❗ Release notes required

---

✅ Found changes and release notes in following paths:

Change path	Release notes path	Description
src/Compiler	docs/release-notes/.FSharp.Compiler.Service/11.0.0.md

[T-Gro]

Can I ignore the other three?
This one is fully green now.

[majocha]

Can I ignore the other three? This one is fully green now.

Yes, we can continue with just this one.

[majocha]

System.Exception : EVIL_PROVIDER_NamespaceName_Exception.err EVIL_PROVIDER_NamespaceName_Exception.bsl differ; "diff between [D:\a_work\1\s\artifacts\Temp\FSharp.Test.Utilities\72306301\6278ff13\typeProviders\negTests\EVIL_PROVIDER_NamespaceName_Exception.bsl] and [D:\a_work\1\s\artifacts\Temp\FSharp.Test.Utilities\72306301\6278ff13\typeProviders\negTests\EVIL_PROVIDER_NamespaceName_Exception.err]
diff at line 4
+
Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
at FSharp.Compiler.CompilerImports.clo@1108-481.Invoke(TcImportsLockToken tcitok)
at Internal.Utilities.Library.Lock1.AcquireLock[a](FSharpFunc2 f)
at FSharp.Compiler.CompilerImports.TcImportsSafeDisposal.Finalize()
"

Type providers don't like parallel imports, another random fail in evil provider.

[T-Gro]

Type providers don't like parallel imports, another random fail in evil provider.

I wonder if it because of the evil provider, or if the compiler should not synchronizing access to type provider code for all of them. (I assume that at time of TP creation, nothing in the compiler was parallel and access to TP libraries was always sequential).

[majocha]

Type providers don't like parallel imports, another random fail in evil provider.

I wonder if it because of the evil provider, or if the compiler should not synchronizing access to type provider code for all of them. (I assume that at time of TP creation, nothing in the compiler was parallel and access to TP libraries was always sequential).

I suspect it was because of one missing lock 6b96083
There was a comment that this requires a lock, but the lock was not there.

[majocha]

Type providers don't like parallel imports, another random fail in evil provider.

I wonder if it because of the evil provider, or if the compiler should not synchronizing access to type provider code for all of them. (I assume that at time of TP creation, nothing in the compiler was parallel and access to TP libraries was always sequential).

Yes, the oldest thing I remember was the Reactor, that made everything sequential. But then there's this #10310

[T-Gro]

Ok this makes sense, all access to TP synchronized.
We cannot tell from the outside if TP internals are thread safe or not.

[majocha]

I suspect there is still a lot that can be optimized w.r.t. graph creation and how Transparent Compiler caches it. But I feel it would be best to do it separately.

[Copilot]

Pull Request Overview

This PR enables parallel compilation features by default after stabilization, including parallel reference resolution, parallel optimizations, parallel ILxGen, and graph-based type checking. The --parallelcompilation- switch can be used to revert to previous behavior. Setting --deterministic also makes compilation sequential except for parallel imports. IL baselines changed due to reordered methods from parallel processing.

Key Changes:

• Enabled parallel compilation by default in configuration and build settings
• Fixed deterministic mode to exclude graph-based type checking
• Improved script handling and signature-to-implementation file mapping in graph checking
• Updated thread safety for type provider disposal

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/FSharp.Build/Microsoft.FSharp.NetSdk.props	Changed ParallelCompilation default to true
src/FSharp.Build/Fsc.fs	Changed ParallelCompilation default from false to true
src/Compiler/Optimize/Optimizer.fs	Changed default processing mode from Sequential to Parallel
src/Compiler/Driver/ParseAndCheckInputs.fs	Added deterministic mode check to exclude graph-based type checking
src/Compiler/Driver/GraphChecking/Types.fsi	Added IsScript property and TryGetWrongOrderSignatureToImplementationIndex method
src/Compiler/Driver/GraphChecking/Types.fs	Improved file pair mapping logic and added script detection
src/Compiler/Driver/GraphChecking/DependencyResolution.fs	Added script compilation handling and wrong-order signature detection
src/Compiler/Driver/CompilerOptions.fs	Updated parallel compilation switch to control reference resolution
src/Compiler/Driver/CompilerImports.fs	Refactored assembly import to be fully async and fixed thread safety
src/Compiler/Driver/CompilerConfig.fs	Changed defaults to enable parallel features
docs/release-notes/.FSharp.Compiler.Service/11.0.0.md	Added release notes entry
tests/fsharp/typecheck/sigs/neg56_a.bsl	Updated baseline with new error
tests/fsharp/typecheck/sigs/neg14.bsl	Updated baseline with reordered errors
tests/fsharp/tests.fs	Removed skipped test
tests/FSharp.Compiler.ComponentTests/Miscellaneous/MigratedTypeCheckTests.fs	Re-enabled previously skipped test
Multiple IL baseline files	Updated baselines reflecting method reordering from parallel compilation

[nojaf]

Great to see another step in the right direction. That said, don’t present this as "everything is now parallel." It needs more transparency and nuance.

[nojaf]

This is quite vague about what is actually changed here. Anyone reading this, will need to scour through the PR to know what is actual up.

[majocha]

@nojaf thanks for the careful review👍 , I'll do a follow up addressing it.

[nbevans]

I had to add --parallelcompilation- to my FSharpChecker.Compile args to prevent a huge delay (1 minute or so) on my Async.Parallel compiled .fsx scripts

Anyone know why that is? My expectation was that parallel compilation should work perfectly with Async.Parallel invoked compilation of a dozen or so .fsx scripts?

If I leave parallel compilation args alone, and use Async.Sequential, it's also fine.

Thanks

[majocha]

I had to add --parallelcompilation- to my FSharpChecker.Compile args to prevent a huge delay (1 minute or so) on my Async.Parallel compiled .fsx scripts

I understand this is not parallel compilation of multiple .fsx files in one FSharpChecker.Compile call but many parallel Compile calls to the same checker, each with a single .fsx?

Anyone know why that is? My expectation was that parallel compilation should work perfectly with Async.Parallel invoked compilation of a dozen or so .fsx scripts?

Checker uses a single DependencyProvider instance. I guess if there are some #r nuget references in the scripts it chokes on some shared state, maybe even filesystem. I don't think we have a test for such usecase.

[nbevans]

There's a dozen of so "main entry point" .fsx which could then have a large tree of #load'd .fsx. The hierarchy depth could reach 20 to 30 in some cases.

There's a single FSharpChecker instance and then Async.Parallel is used to compile those dozen or so main entry point .fsx in parallel. This was working great on the F#9 era FSCS but after updating to F#10 FSCS I encountered this very large delay before any of the Async.Parallel compilations start completing. Which instinctively makes me think of like a mutex lock or something holding things up.

After getting Codex to summarise the release notes and PRs between the relevant FSCS versions for potentially relevant changes to this issue, it highlighted the --parallelcompilation option changes. So I disabled the --parallelcompilation option and the F#9 FSCS performant behaviour returned.

I mean I could just be satisfied I fixed my issue per se, but now I'm wondering whether a few small tweaks could unlock better performance by keeping the new parallel compilation features enabled.

[majocha]

Now I remember. Presence of script files in the sources should disable graph checking and just type check sequentially with no extra penalty comparing to --parallelcompilation-.

fsharp/src/Compiler/Driver/GraphChecking/DependencyResolution.fs

Lines 257 to 273 in a5df95f

	// If there is a script in the project, we just process sequentially all the files that may have been added as part of the script closure.
	// That means all files up to the last script file.
	letscriptCompilationLength=
	files
	|> Array.tryFindIndexBack (fun f -> f.IsScript)
	|> Option.map (fun idx -> idx +1)
	|> Option.defaultValue 0
	letsequentialPartForScriptCompilation=
	files
	|> Array.take scriptCompilationLength
	|> Array.map (fun file ->
	file.Idx,
	[|
	if file.Idx >0then
	file.Idx -1
	|])

Clearly something is not working correctly.

But the fact this does not happen with Async.Sequential makes me think there is contention somewhere at file system / dependecy manager level.

[nbevans]

The largest .fsx entry point script actually just hangs. It would normally compile in about 20-25seconds. But with the parallel compilation, I was waiting for about 5 minutes or so before I realised it had essentially hanged. I attached the VS debugger and it looked like a lot of WaitOne's in several worker threads used by FSCS were all essentially in deadlock with each other.

Out of interest, what makes .fsx so different that it, essentially, isn't fully supported by parallel compilation?

[majocha]

I remember there were a few test cases failing, with mixed use of .fsx and .fs in a compilation. So, this was basically done to unblock the feature in such cases. I can't find the test cases out of my head, but it would be good to revisit this. I opened a draft to identify the failing tests #19649

[T-Gro]

@nbevans :

We would love to have a representative of your setup as one of regression tests in our matrix, do you this is possible?
to e.g. anonymize parts of it, but maintain and similarily-sized setup of the .fsx files and their dependencies (i.e. even similar number of shared and loaded dependencies etc.)

[nbevans]

I'll have a think how we could somehow isolate the relevant parts

It's basically a Host process which when it boots up uses FSharpChecker.Compile to compile several .fsx scripts. And on developer machines, it uses Async.Parallel to speed up the boot of the Host process. On prod environments we just use Async.Sequential, so no issues there.

These .fsx scripts serve as core configuration for the Host process. They are quite deeply nested in a hierarchy. Some of the script hierarchies, for sure, reference the same absolute path'd .fsx (via different relative paths via #load directives) multiple times.

Because these .fsx scripts can in some cases take anything from 1 to 30 sec to compile, we actually compute a hash of all the .fsx's (paths + content) in the hierarchy first, and the .dll assembly is stamped with that hash, so re-compilation of the assembly is not necessary if nothing has changed in the whole hierarchy of that .fsx script entry point.