gh-141749: Align pure Python unpickler exceptions with the C one

[v0ropaev]

Closes #141749. Picks up @djoume's #141754, which was closed unmerged yesterday after going stale.

The pure Python unpickler leaks IndexError and KeyError out of load() where the C one raises UnpicklingError. Measured on main, same malformed input through both implementations:

input	pickle._Unpickler	_pickle.Unpickler
b"\xff."	KeyError: 255	UnpicklingError: invalid load key, '\xff'.
b"\x85." (TUPLE1)	IndexError: list index out of range	UnpicklingError: unpickling stack underflow
b"R." (REDUCE)	IndexError: pop from empty list	UnpicklingError: unpickling stack underflow
b"(I1\nd." (odd DICT)	IndexError: list index out of range	UnpicklingError: odd number of items for DICT
b"1." (POP_MARK, no MARK)	IndexError: pop from empty list	UnpicklingError: could not find MARK
b"(\x85." (past a MARK)	IndexError: list index out of range	UnpicklingError: unexpected MARK found

Across sixteen such inputs: 0 of 16 agreed on main, 16 of 16 agree on this branch, message text included.

That matters beyond tidiness — pickle.load documents UnpicklingError for malformed data, so code that catches it correctly is still crashed by a hostile pickle whenever the pure Python implementation is in use.

Both of @picnixz's review points from #141754

Do not catch the KeyError for the call. Store the dispatcher separately.

Done — the dispatch lookup and the call are separate, so a KeyError raised inside a handler is no longer swallowed as an invalid opcode:

try:
    handler = dispatch[key[0]]
except KeyError:
    self._invalid_opcode(key[0])
handler(self)

Do we still need to catch IndexError?

@djoume answered on that thread, and the answer explains why this PR is wider than theirs:

Yes, IndexError is still needed. This PR only fixes specific cases (invalid opcodes, missing MARK). Many other operations like self.stack[-1], self.stack.pop() can still raise IndexError on an empty stack during normal unpickling operations.

Exactly so — which is why every handler that indexes or pops the stack is guarded here, not just the two cases. Once they all are, nothing reaches the caller as IndexError, and the answer to the question becomes no.

The two distinct C messages are preserved: a pending MARK hides the rest of the stack, so an opcode reaching past it reports unexpected MARK found while an exhausted stack reports unpickling stack underflow.

The test expectations got stricter, not looser

Worth flagging, since the diff touches existing test expectations and the policy rightly forbids loosening them. In Lib/test/test_pickle.py the tuples shrink:

-    bad_stack_errors = (IndexError,)
+    bad_stack_errors = (pickle.UnpicklingError,)
-    bad_stack_errors = (pickle.UnpicklingError, IndexError)
+    bad_stack_errors = (pickle.UnpicklingError,)

IndexError was previously accepted; it no longer is. No assertion was relaxed and none was deleted.

New cases in pickletester.py pin the exact message for each invalid-opcode escape form and for each of the five distinct stack messages.

Compatibility

This is a behaviour change. UnpicklingError is not a subclass of IndexError or KeyError, so code with except IndexError around pickle.load stops catching — but only on the pure Python path, only for malformed input, and only where the C implementation already raised UnpicklingError. Anyone handling both implementations already had to catch UnpicklingError.

main only; no backport.

Verification

On a build of main (3.16.0a0):

./python.exe -m test test_pickle test_pickletools   →  run=1,294, skipped=61, SUCCESS

Reverting Lib/pickle.py to main while keeping the tests: FAILURE.

Two of the guards, BINPERSID and READONLY_BUFFER, were initially reachable by no test — reverting just those two left the suite green. They now have cases, and reverting just those two fails.

Misc/NEWS.d/next/Library/ entry added, using @picnixz's suggested wording from that thread.

• Issue: Pure Python pickle.py Raises Wrong Exceptions for Invalid Data #141749