Hang at shutdown: take_gil() clears the GIL drop request after releasing gil->mutex

[ashm-dev]

Bug report

Bug description:

A daemon thread that is waiting for the GIL during interpreter shutdown can leave the main thread blocked forever in drop_gil().

In take_gil(), the _PyThreadState_MustExit() branch releases gil->mutex first (Python/ceval_gil.c:346) and only then clears the drop request it set earlier on the GIL holder (:354). Between these two lines the holder can go through drop_gil(): it still sees _PY_GIL_DROP_REQUEST_BIT set (:260, read without the mutex) and waits on gil->switch_cond with no timeout (:271). The only thread that could take the GIL and signal switch_cond is the daemon thread, and it goes straight to _PyThreadState_HangThread(). The process never exits.

(Line numbers are from main at a5a4659.)

Sequence:

1. The main thread holds the GIL in C code for longer than the switch interval. The daemon thread times out in take_gil() and sets the drop request bit on the main thread.
2. Finalization marks the daemon thread as shutting down (_PyThreadState_SetShuttingDown()).
3. The main thread keeps holding the GIL (shutdown GC). The daemon thread times out again and enters the _PyThreadState_MustExit() branch: MUTEX_UNLOCK(gil->mutex) at :346.
4. Before the daemon thread reaches :354, the main thread runs a __del__ from the shutdown GC, handles the eval breaker and calls drop_gil(). The bit is still set, so it waits on switch_cond.
5. The daemon thread clears the bit (too late) and hangs. Nobody signals switch_cond.

This needs exactly one thread waiting for the GIL. If another thread were waiting, it would take the GIL and signal switch_cond.

Reproducer

gc_bug.py:

import atexit
import threading
import time


class Finalizer:
    def __del__(self):
        pass


def spin():
    while True:
        time.sleep(0)


threading.Thread(target=spin, daemon=True).start()
time.sleep(0.1)

# Many tracked objects make the shutdown gc.collect() hold the GIL for a while.
junk = [[] for _ in range(10**6)]

# A cycle with __del__: shutdown GC runs Python code, which releases the GIL.
f = Finalizer()
f.cycle = f
del f

# Hold the GIL in C, with no eval-breaker checks, right before finalization.
atexit.register(sum, range(10**7))

The race window is a few instructions wide, so a plain ./python gc_bug.py exits normally almost every time. To make it deterministic, stop the daemon thread inside the window with a debugger and let only the main thread run:

$ PYTHON_JIT=0 lldb ./python
(lldb) breakpoint set -f ceval_gil.c -l 353 -c drop_requested
(lldb) process launch -- gc_bug.py
(lldb) thread continue 1

The process never exits. Main thread backtrace after interrupting (PYTHON_JIT=0 only keeps the debugger from stopping on JIT code registration):

pthread_cond_wait
drop_gil                Python/ceval_gil.c:271
_Py_HandlePending       Python/ceval_gil.c:1418
check_periodics
_PyEval_EvalFrameDefault
...
call_unbound_noarg      (Finalizer.__del__)
finalize_garbage
PyGC_Collect
_Py_Finalize
Py_RunMain

If the breakpoint is set at :357 instead (after the bit is cleared), the process exits normally every time. So the hang comes from clearing the bit after releasing the mutex.

Build

CC=clang-21 CXX=clang++-21 LDFLAGS='-fuse-ld=lld-21' ./configure --enable-experimental-jit=yes
make

./python -VV: Python 3.16.0a0 (heads/main:a5a4659548e) [Clang 21.1.8]

CPython versions tested on:

CPython main branch, 3.16

Operating systems tested on:

Linux

Linked PRs

• gh-158090: Fix hang at shutdown when a daemon thread resets the GIL drop request #158101

[picnixz]

This really looks like very niche and I am not entirely sure the reproducer is faithful. Maybe the interaction with the debugger and the OS are at fault? And even with that, how frequently do we hit this case? is there a realistic reproducer?

cc @ZeroIntensity

[lazerg]

I checked it without a debugger. I added usleep(200000) between MUTEX_UNLOCK(gil->mutex) and the drop request reset in take_gil(), then ran the reproducer from the issue on a --with-pydebug build. It hung in 3 of 3 runs. With the same sleep in the order from gh-158101 (reset first, unlock second), it exited normally in 3 of 3 runs. The sleep only widens a window that is already in the code. Without it the window is a few instructions wide, so a real hang is rare.

[ashm-dev]

@picnixz Reproduces under gdb and under lldb; also checked a plain run and a single CPU (taskset)

[ZeroIntensity]

I haven't had a chance to look at this too closely, but this does look like a real race (though extremely rare -- you probably have a better chance of a cosmic ray crashing the interpreter than encountering this deadlock in production).