gh-67486: Say what is wrong when detect_encoding gets a text readline

[v0ropaev]

Closes #67486.

Passing a text readline to tokenize.detect_encoding fails inside the BOM check, so you get an error about startswith rather than about what you did wrong:

>>> tokenize.detect_encoding(io.StringIO("x=1\n").readline)
TypeError: startswith first arg must be str or a tuple of str, not bytes

@berkerpeksag pointed out in 2019 that the case still reachable is tokenize.open() together with tokenize.tokenize(), since open() returns a text stream. That is still how it behaves on main today, with his own example file:

>>> with tokenize.open("hello.py") as f:
...     list(tokenize.tokenize(f.readline))
TypeError: startswith first arg must be str or a tuple of str, not bytes

With this change both say:

TypeError: detect_encoding() requires a readline callable returning bytes,
but it returned str; read the source as bytes, or use
tokenize.generate_tokens() to tokenize text

The check lives in read_or_stop() inside detect_encoding, after the StopIteration handler, so a readline that stops immediately still returns ('utf-8', []). It rejects str rather than demanding bytes, so bytearray keeps working — that path isn't documented, but it works today and it seemed wrong to break it in passing.

The exception type doesn't change: it was TypeError before and still is. The message names detect_encoding rather than tokenize, because detect_encoding is public and called directly from trace.py, idlelib/iomenu.py and importlib/_bootstrap_external.py — pointing at tokenize() would name a function that isn't in the caller's traceback.

Three new cases in TestDetectEncoding: str on the first line, str on the second, and bytearray as a control that the check didn't tighten into isinstance(line, bytes). Changing the check to not isinstance(line, bytes) fails the suite, so the bytearray case is doing work.

./python.exe -m test test_tokenize test_trace test_inspect test_linecache is 600 tests, all passing. No existing test asserted the old message, so nothing had to be adjusted.

One line added to the detect_encoding docs, since the issue has carried a docs label since 2019. main only; this improves an error message rather than fixing a crash, so I'm not proposing a backport.

• Issue: Clarify error when ‘tokenize.detect_encoding’ receives text #67486

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34747803 | 📁 Comparing c515d37 against main (a5b03fa)

  🔍 Preview build  

2 files changed

± library/tokenize.html
± whatsnew/changelog.html

[v0ropaev]

The macOS failure is test_ssl, with ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca and an ENV CHANGED result — nothing to do with tokenize. My four other PRs off the same base passed macOS-26 in the same window, so it looks like a bad moment rather than anything here. Could someone re-run it when convenient?