Double release of `PyBytesWriter` in `assemble_init()` error path corrupts the bytes writer freelist

[lpyu001]

Bug report

Bug description:

Summary

When a memory allocation fails inside assemble_init() in Python/assemble.c, the writers that were already created are released twice: once by the error: label in assemble_init() and again by assemble_free(), which _PyAssemble_MakeCodeObject() always calls. The second release corrupts the bytes_writers freelist. On a debug build, a later compile() aborts with Assertion 'fl->size > 0' failed or segfaults. On a release build, no assertion fires. Instead, later compile() calls silently return code objects with corrupted co_code and co_linetable, and the process can segfault when freelists are cleared. Any compile() call reaches this path; the source text does not matter.

This reintroduces the double-release bug fixed by gh-151112/#151142.#155747 reintroduced it for the linetable writer, and #157349 extended it to all three writers.

Reproduction Code

Requires _testcapi (for set_nomemory). The assertion requires a --with-pydebug build.

import _testcapi
compile("x", "<s>", "exec")  # warm up (initialise AST state) before injecting
for n in range(1, 300):
    _testcapi.set_nomemory(n, n + 1)
    try:
        compile("x", "<s>", "exec")
    except MemoryError:
        pass
    finally:
        _testcapi.remove_mem_hooks()
print("no crash")

On a release build, the corruption shows up as wrong bytecode:

import sys, _testcapi
ref = compile("x", "<s>", "exec")
for n in range(500):
    _testcapi.set_nomemory(n, n + 1)
    try:
        compile("x", "<s>", "exec")
    except MemoryError:
        pass
    finally:
        _testcapi.remove_mem_hooks()
co = compile("x", "<s>", "exec")
print(co.co_code == ref.co_code, co.co_linetable == ref.co_linetable)
print(ref.co_code.hex(), co.co_code.hex())
print(ref.co_linetable.hex(), co.co_linetable.hex())

Actual Behavior

Debug build, first script:

python: ./Include/internal/pycore_freelist.h:79: _PyFreeList_PopNoStats: Assertion `fl->size > 0' failed.
Aborted (core dumped)

Running each injection point in a separate process (one failing allocation, then three more compile("x") calls) on a debug build gives this: failing the linetable writer allocation (assemble.c:73) triggers the assertion above. Failing the exception table writer allocation (assemble.c:77) crashes with SIGSEGV. Every other injection point raises MemoryError normally.

Release build, second script. The last two lines vary between runs because the overwritten bytes are pointer values:

False False
8000000059001d004d072100 63bc00005f7000004d072100
f103010101db0001 f0bc92785f700000

On a release build, running the first script with the original source "try:\n pass\nexcept* (A, B) as e:\n raise\nfinally:\n del x\n" (after import ast, n from 1 to 200) prints no crash and then segfaults during interpreter shutdown (exit code 139):

#0  insert_to_freepool (pool=0x7ffff7b28000, state=...) at Objects/obmalloc.c:2599
#1  pymalloc_free (_unused_ctx=<optimized out>, p=0x7ffff7b2bcf0, state=...) at Objects/obmalloc.c:2817
#2  _PyObject_Free (ctx=<optimized out>, p=0x7ffff7b2bcf0) at Objects/obmalloc.c:2831
#4  clear_freelist (dofree=<optimized out>, is_finalization=<optimized out>, freelist=<optimized out>) at Objects/object.c:908
#5  _PyObject_ClearFreeLists (freelists=..., is_finalization=is_finalization@entry=0) at Objects/object.c:954
#6  _PyGC_ClearAllFreeLists (interp=<optimized out>) at Python/gc_gil.c:14
#7  gc_collect_main (tstate=..., generation=generation@entry=2, reason=reason@entry=_Py_GC_REASON_MANUAL) at Python/gc.c:1611

The release build's symptoms depend on heap layout. A variant of the second script without import sys returned correct code objects.

Expected Behavior

compile() should raise MemoryError when an allocation fails, and leave interpreter state intact. Later compile() calls should succeed and produce the same code objects as before the failure. This was the behavior after the gh-151112 fix (#151142).

Detail

Faulty control flow

static int
assemble_init(struct assembler *a, int firstlineno)
{
    memset(a, 0, sizeof(struct assembler));
    ...
    a->a_bytecode_writer = PyBytesWriter_Create(DEFAULT_CODE_SIZE);      // :69
    ...
    a->a_linetable_writer = PyBytesWriter_Create(DEFAULT_CNOTAB_SIZE);   // :73
    ...
    a->a_except_table_writer = PyBytesWriter_Create(DEFAULT_LNOTAB_SIZE); // :77
    ...
    return SUCCESS;
error:
    PyBytesWriter_Discard(a->a_bytecode_writer);      // fields are not set to NULL
    PyBytesWriter_Discard(a->a_linetable_writer);
    PyBytesWriter_Discard(a->a_except_table_writer);
    return ERROR;
}

assemble_emit() returns ERROR, and _PyAssemble_MakeCodeObject() then calls assemble_free() unconditionally (assemble.c:820). assemble_free() calls PyBytesWriter_Discard() on the same three pointers again.

Freelist corruption mechanism (debug build)

Py_bytes_writers_MAXFREELIST is 1, so the second release does not form a self-loop in the freelist. gdb breakpoints on PyBytesWriter_Create and PyBytesWriter_Discard, for the round where the linetable allocation fails, show this sequence:

=== INJECTED FAIL in byteswriter_create, FmData.count=195 freelist.size=0 head=(nil)
#2  assemble_init (a=a@entry=0x7fffffffd1e0, firstlineno=<optimized out>) at Python/assemble.c:73
--- Discard(0x7ffff77a4170) caller:
#1  0x00005555557c72f9 in assemble_init (a=a@entry=0x7fffffffd1e0, firstlineno=<optimized out>) at Python/assemble.c:83
--- Discard(0x7ffff77a4170) caller:
#1  0x00005555557c80ef in assemble_free (a=a@entry=0x7fffffffd1e0) at Python/assemble.c:92
--- Create head=0x7ffff77a4170 size=1
--- Create head=0xdddddddddddddddd size=0
python: ./Include/internal/pycore_freelist.h:79: _PyFreeList_PopNoStats: Assertion `fl->size > 0' failed.

1. The bytecode writer B is allocated with PyMem_Malloc, because the freelist is empty.
2. The linetable allocation fails. error: calls Discard(B), and B is pushed onto the freelist (size 1). a->a_bytecode_writer still points to B.
3. assemble_free() calls Discard(B) again. The freelist is full, so _PyFreeList_Push() refuses the push and B is passed to PyMem_Free() while it is still the freelist head.
4. The next PyBytesWriter_Create() pops the freed B, which is a use-after-free. The freelist head becomes 0xdddddddddddddddd and size becomes 0.
5. The next PyBytesWriter_Create() finds a non-NULL head with size 0 and hits the assertion.

Full stack at the second release:

#0  PyBytesWriter_Discard (writer=0x7ffff77a4170) at Objects/bytesobject.c:3810
#1  assemble_free (a=a@entry=0x7fffffffd1e0) at Python/assemble.c:92
#2  _PyAssemble_MakeCodeObject (...) at Python/assemble.c:820
#3  optimize_and_assemble_code_unit (...) at Python/compile.c:1496
#4  _PyCompile_OptimizeAndAssemble (c=c@entry=0x7ffff77c3740, addNone=addNone@entry=1) at Python/compile.c:1524
#5  compiler_mod (...) at Python/compile.c:902
#6  _PyAST_Compile (...) at Python/compile.c:1537
#7  _Py_CompileString (...) at Python/pythonrun.c:1595

If the exception table allocation (:77) fails instead, error: releases B (pushed to the freelist) and the linetable writer L (freelist full, so passed to PyMem_Free()). assemble_free() then reads the freed L inside PyBytesWriter_Discard():

#0  PyObject_TypeCheck (ob=ob@entry=0xdddddddddddddddd, type=0x555555b95b00 <PyByteArray_Type>) at ./Include/object.h:375
#1  PyByteArray_AS_STRING (op=0xdddddddddddddddd) at ./Include/cpython/bytearrayobject.h:26
#2  _PyBytesWriter_GetData (writer=writer@entry=0x7ffff7b59470) at ./Include/internal/pycore_bytesobject.h:110
#3  byteswriter_data (writer=0x7ffff7b59470) at Objects/bytesobject.c:3627
#4  byteswriter_check_canary_byte (writer=writer@entry=0x7ffff7b59470) at Objects/bytesobject.c:3651
#5  PyBytesWriter_Discard (writer=0x7ffff7b59470) at Objects/bytesobject.c:3816
#6  assemble_free (a=a@entry=0x7fffffffd0f0) at Python/assemble.c:93

Without the debug canary check, this path would pass L to PyMem_Free() a second time.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-158241: Prevent double release of assembler byte writers #158242