Use thread-local APIs when possible

[itamarst]

Given the existence in the wild (e.g. in scikit-learn) of nested Python thread pools, being able to set limits per-thread is that much more important for reliable oversaturation prevention.

Some libraries provide both process-wide and thread-local settings:

• MKL: https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-0/mkl-set-num-threads-local.html
• BLIS: https://github.com/flame/blis/blob/master/docs/Multithreading.md#locally-at-runtime

In practice BLIS is probably not viable.

Long-term, I think there should be a ecosystem-wide push to provide thread-local APIs.

In the short term, exposing MKL at least would be helpful for #215.

Update: After some discussion of the downstream use case of Python thread pools (#215), conclusion was that rather than it being useful to have thread-local scoped limiting APIs, they should in fact be the default.

[itamarst]

My first thought was just adding it for MKL, but given #211 one could actually do this dynamically for additional APIs, if sufficiently motivated, so might wait for that.

UPDATE: Maybe not actually useful to do this dynamically? You're still stuck with whatever the API gives you, so nested Python thread pools are still a problem if all you have is process-level setting scope. So maybe I'll revert to initial plan.

[itamarst]

Did most of the work in a branch, but will wait until #214 is merged so I can write a test.

[ogrisel]

In practice BLIS is probably not viable.

I agree: I took at the BLIS API, and it seems that using it would require callers to be recompiled to use non-standard BLAS function equivalent to pass the locally configured runtime object as extra argument:

https://github.com/flame/blis/blob/master/docs/Multithreading.md#locally-at-runtime-using-the-expert-interfaces

[ogrisel]

For the record, OpenBLAS also has a openblas_set_num_threads_local function, but it is not documented, and after looking at the code I doubt that this is equivalent to what mkl_set_num_threads_local. This function does not seem to be tested either.

If needed, we could try to contribute to OpenBLAS to actually make this function use a thread-local configuration, with similar semantics as the MKL API but that sounds complex to get right.

[itamarst]

One of my longer term todos is writing an extensive blog post about why it's important to have thread-local limiting APIs and therefore thread-local thread pool sizes, and then file issues with every relevant library using the blog post as evidence.

[itamarst]

So for OpenBLAS with OpenMP, using the OpenMP limiting API instead of OpenBLAS limiting API will mean it mostly (maybe even always?) has a thread-scoped limiting API. The downside is that if someone sets different limits for OpenMP and BLAS, this will clash; any idea what the use case is for that, and if anyone actually does it?

OpenBLAS with pthreads is such an outlier that I am going to open a separate issue to deal with it.

[ogrisel]

The downside is that if someone sets different limits for OpenMP and BLAS, this will clash; any idea what the use case is for that, and if anyone actually does it?

I think we don't need to handle this case: we can leave it as undefined behavior.

[itamarst]

Annoyingly OpenMP on Windows (vcomp) has a process-wide limiting API. So on Windows OpenBLAS won't benefit from OpenMP for limiting API purposes (but it will benefit if it means it's not using a process-wide thread pool, so might be worth encouraging anyway, once it's clear whether vcomp has a process-wide shared thread pool or not.)

[itamarst]

Extra fun: due to OpenMathLib/OpenBLAS#5806 it's not really possible to set the number of threads when OpenBLAS is using OpenMP, except with the OMP_NUM_THREADS environment variable before OpenBLAS is first loaded 😭