[FridrichMethod]

Why

The scheduled Check wiki sources run of 2026-10-19 (run 39224451765) failed in discover_bwiki(). BWIKI answered a pages() batch with HTTP 567 (EdgeOne rate limiting) three times in a row:

requests.exceptions.HTTPError: 567 Server Error: Unknown Status for url: https://wiki.biligame.com/deemo/api.php?...

567 was already in RETRY_STATUSES, but it was retried on the same schedule as a server error: 3 attempts, 1 s and 2 s apart. The run stopped about 3 s after the first 567, well inside any rate-limit window.

What changes

scripts/fetch_wikis.py

• New RATE_LIMITED = {429, 567}. These statuses get RATE_LIMIT_ATTEMPTS (6) attempts. When the server sends no Retry-After, the waits are 5, 10, 20, 30 and 30 s, about 95 s in all.
• Connection errors, timeouts and 500/502/503/504 keep 3 attempts 1 s and 2 s apart, so a host that is down still fails fast.
• The two kinds are counted apart: each has its own attempt limit and its own waits, whatever came before it. A request is made at most 8 times, and every failure advances one bounded counter, so the loop always ends.
• A Retry-After still replaces any wait, capped at 30 s (unchanged).
• download() no longer tries a Fandom candidate's other URLs after rate limiting. They're all on the host that is limiting, and without this one candidate could cost about 3 × 95 s. A server error still moves on to the next URL.
• http_status() is the one place retried(), retry_delay() and download() read the status from.

docs/sources-wikis.md, docs/sources-wikis.zh-CN.md: the retry sentence describes both schedules and the separate counting. The Fandom paragraph says rate limiting fails the file at once.

tests/test_fetch_wikis.py

• Pins both schedules (including connection errors and a 404 that fails at once) and five mixed sequences.
• Pins the download rule for 429 and 567, with 503 for contrast.
• Pins both doc mirrors against retry_delay() and the constants. The doc checks now share one assertDocumented() helper.
• RetryTests caps the fake web at 20 requests, so a broken bound fails instead of hanging.

Review

A review of the first commit found that one shared counter let a 503 after two 567s go unretried, and that rate-limit waits didn't follow the documented schedule after other errors. Both are fixed in the second commit. Findings left as they are:

• Ctrl-C delay during a rate-limited download: after the URL fix, the longest wait matches what a Retry-After: 30 could already cause.
• Discover job's 30-minute timeout: a request that keeps failing still gives up after about 95 s with the named error.
• Jitter and per-host pacing: larger design changes. They're the follow-up if the backoff alone proves insufficient.
• Doc wording on other RequestExceptions: that was already true before this PR.

Validation

• The new tests fail on the original code with the same HTTPError: 567 as the CI run. Undoing either review fix also fails them.
• Every tests/test_*.py passes under coverage; total coverage is 89.6% (fail_under = 89).
• pre-commit run on the changed files passes with the pinned hooks (ruff 0.17.0 check and format, mypy).

Caveat

BWIKI isn't reachable from the environment this was written in, so the new backoff hasn't been tried against the live site.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AZypKz9iupFY3Dhsd8Q1rK

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.