Skip to content

FIX OpenBLAS pthreads can only be controlled from one thread at a time - #275

Draft
itamarst wants to merge 6 commits into
joblib:masterfrom
itamarst:openblas-pthreads-one-thread-only
Draft

itamarst wants to merge 6 commits into
joblib:masterfrom
itamarst:openblas-pthreads-one-thread-only

Conversation

@itamarst

@itamarst itamarst commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The expected joblib usage pattern is that it has a thread-local counter, which gets incremented on ThreadedBackend.configure(), and decremented on ThreadedBackend.finalize(), and if it hits zero then OpenBLASController.clear_controlling_thread() is called.

The result is that so long as you have Parallel() running in a particular thread, typically the main thread, other threads can't set OpenBLAS-pthreads thread pool size.

A separate companion PR will expose an API to allow joblib to override thread limits so they're either 1 or all cores, since we want that too, I think.

Haven't written all the relevant tests since seems worth some design feedback first @ogrisel

TODO

  • CI passes
  • Tests
  • Changelog
  • actually write the corresponding joblib logic and make sure it works

@itamarst itamarst changed the title FIX OpenBLAS pthreads can only be contrlled from one thread at a time FIX OpenBLAS pthreads can only be controlled from one thread at a time Oct 22, 2026
@ogrisel

ogrisel commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

I like this approach. If we ever have a second case of process wide shared threadpool, we could even introduce a mixin class for those controller methods. Or we could use ducktyping on the controllers from joblib.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants