Repository navigation
fix(scan): time out stalled HF requests and stop page URLs passing silently - #138
Conversation
…lently - Hugging Face listing pages and range reads now carry a (10s connect, 60s read) timeout. A stall fails the scan with exit 1 and a "Timed out" message instead of hanging CI. - A network failure mid-file is reported as a fetch failure, not as an unparseable file. - HF web-page URLs are rewritten when they map exactly (repo page, tree/main -> hf://; blob/<rev>/<path> -> /resolve/ URL) and refused with exit 1 and a suggested target otherwise (UnsupportedHFPageURL). - A URL that no scanner claims exits 1 (UnsupportedFileType); a query string no longer hides the extension.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61a354a6d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if parts and parts[0] in ("datasets", "spaces"): | ||
| kind = parts[0][:-1] | ||
| return None, ( | ||
| f"{target} is a Hugging Face {kind} page; aisbom scans model repos " | ||
| "only. Pass a model repo as hf://<owner>/<repo>." | ||
| ) |
There was a problem hiding this comment.
Preserve direct file URLs in datasets and Spaces
When a supported model file is hosted in a dataset or Space, this check rejects its direct download URL—for example, https://huggingface.co/spaces/org/app/resolve/main/model.safetensors—as an unsupported page without making a request. Previously, direct HTTPS file targets reached RemoteStream regardless of repository type. Refusing dataset and Space web pages should not also disable scanning individual model files hosted there; recognize their /resolve/ URLs before applying this page-only rejection.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in c121d84. A dataset or Space /resolve/ URL is left alone again and reaches RemoteStream as before. A dataset or Space blob/<rev>/<path> page now maps to its /resolve/ URL, like a model repo's. Only the repo-level pages are refused. Tests cover both, plus datasets/<o>/<r>/tree/main still being refused.
Refusing dataset and Space repo pages also refused their /resolve/ file URLs, which always reached RemoteStream. Leave those alone, and map a dataset or Space blob page onto its /resolve/ URL like a model repo's.
What
Two ways a Hugging Face target could end in a false result now fail explicitly.
Timeouts. The
hf://listing pages and everyRemoteStreamrange read now carryHF_REQUEST_TIMEOUT = (10, 60): 10 seconds to connect, 60 seconds of silence per read. A slow transfer that keeps sending is never cut off. A stall fails the scan with exit1and its own message ("Timed out waiting for huggingface.co…") instead of hanging CI. A failure part-way through a file is now reported as a fetch failure rather than as an unparseable file.RemoteStreamrecords the error, and the scanner drops the inspector's wrong verdict.Page URLs. A Hugging Face page URL used to find nothing and exit
0. Now:huggingface.co/<owner>/<repo>,…/tree/mainhf://<owner>/<repo>…/blob/<rev>/<path>…/resolve/<rev>/<path>tree/<branch>,tree/main/<folder>, a dataset or Space page, a non-repo page1with the target to use (UnsupportedHFPageURL)Each rewrite prints a dim note on stderr. The rewrite happens in
DeepScanner.__init__, soscan,scoreandcomplyall get it.Unclaimed URLs. A URL that no scanner claims now exits
1asUnsupportedFileTypewithout making a request. A query string no longer hides the extension.Behaviour change
A pipeline that scans a URL nothing claims (e.g. a
.zip) will now fail. This is called out at the top of the CHANGELOGUnreleasedsection.Telemetry
The closed set gains one reason,
UnsupportedHFPageURL. The README privacy section now lists the full set (it was also missingListingTruncated). No URL or repo id is sent, and a test checks this.Verification
poetry run pytest --cov=aisbom --cov-fail-under=85: 1823 passed, 94.72% coverage.aisbom bypass-scorecard --checkpasses.hf://google-bert/bert-base-uncasedanddemo_data --offlineare identical once serial number and timestamp are removed. Both exit 0.tree/mainpage gives the same SBOM ashf://.blob/main/model.safetensorspage scans that file.tree/refs%2Fpr%2F1exits 1 with thehf://suggestion.