Skip to content

fix(scan): time out stalled HF requests and stop page URLs passing silently - #138

Merged
lab700xdev merged 2 commits into
mainfrom
hf-target-robustness
Nov 15, 2026
Merged

lab700xdev merged 2 commits into
mainfrom
hf-target-robustness

Conversation

@lab700xdev

Copy link
Copy Markdown
Contributor

What

Two ways a Hugging Face target could end in a false result now fail explicitly.

Timeouts. The hf:// listing pages and every RemoteStream range read now carry HF_REQUEST_TIMEOUT = (10, 60): 10 seconds to connect, 60 seconds of silence per read. A slow transfer that keeps sending is never cut off. A stall fails the scan with exit 1 and its own message ("Timed out waiting for huggingface.co…") instead of hanging CI. A failure part-way through a file is now reported as a fetch failure rather than as an unparseable file. RemoteStream records the error, and the scanner drops the inspector's wrong verdict.

Page URLs. A Hugging Face page URL used to find nothing and exit 0. Now:

Input Result
huggingface.co/<owner>/<repo>, …/tree/main scanned as hf://<owner>/<repo>
…/blob/<rev>/<path> scanned as …/resolve/<rev>/<path>
tree/<branch>, tree/main/<folder>, a dataset or Space page, a non-repo page exit 1 with the target to use (UnsupportedHFPageURL)

Each rewrite prints a dim note on stderr. The rewrite happens in DeepScanner.__init__, so scan, score and comply all get it.

Unclaimed URLs. A URL that no scanner claims now exits 1 as UnsupportedFileType without making a request. A query string no longer hides the extension.

Behaviour change

A pipeline that scans a URL nothing claims (e.g. a .zip) will now fail. This is called out at the top of the CHANGELOG Unreleased section.

Telemetry

The closed set gains one reason, UnsupportedHFPageURL. The README privacy section now lists the full set (it was also missing ListingTruncated). No URL or repo id is sent, and a test checks this.

Verification

  • poetry run pytest --cov=aisbom --cov-fail-under=85: 1823 passed, 94.72% coverage. aisbom bypass-scorecard --check passes.
  • Old vs new SBOMs for hf://google-bert/bert-base-uncased and demo_data --offline are identical once serial number and timestamp are removed. Both exit 0.
  • Against live Hugging Face:
    • The bert tree/main page gives the same SBOM as hf://.
    • The blob/main/model.safetensors page scans that file.
    • tree/refs%2Fpr%2F1 exits 1 with the hf:// suggestion.

…lently

- Hugging Face listing pages and range reads now carry a (10s connect,
  60s read) timeout. A stall fails the scan with exit 1 and a "Timed out"
  message instead of hanging CI.
- A network failure mid-file is reported as a fetch failure, not as an
  unparseable file.
- HF web-page URLs are rewritten when they map exactly (repo page,
  tree/main -> hf://; blob/<rev>/<path> -> /resolve/ URL) and refused
  with exit 1 and a suggested target otherwise (UnsupportedHFPageURL).
- A URL that no scanner claims exits 1 (UnsupportedFileType); a query
  string no longer hides the extension.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T15:48:10.796575Z 61a354a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61a354a6d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread aisbom/remote.py
Comment on lines +313 to +318
if parts and parts[0] in ("datasets", "spaces"):
kind = parts[0][:-1]
return None, (
f"{target} is a Hugging Face {kind} page; aisbom scans model repos "
"only. Pass a model repo as hf://<owner>/<repo>."
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve direct file URLs in datasets and Spaces

When a supported model file is hosted in a dataset or Space, this check rejects its direct download URL—for example, https://huggingface.co/spaces/org/app/resolve/main/model.safetensors—as an unsupported page without making a request. Previously, direct HTTPS file targets reached RemoteStream regardless of repository type. Refusing dataset and Space web pages should not also disable scanning individual model files hosted there; recognize their /resolve/ URLs before applying this page-only rejection.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c121d84. A dataset or Space /resolve/ URL is left alone again and reaches RemoteStream as before. A dataset or Space blob/<rev>/<path> page now maps to its /resolve/ URL, like a model repo's. Only the repo-level pages are refused. Tests cover both, plus datasets/<o>/<r>/tree/main still being refused.

Refusing dataset and Space repo pages also refused their /resolve/ file
URLs, which always reached RemoteStream. Leave those alone, and map a
dataset or Space blob page onto its /resolve/ URL like a model repo's.
@lab700xdev
lab700xdev merged commit f34a80c into main Nov 15, 2026
2 checks passed
@lab700xdev
lab700xdev deleted the hf-target-robustness branch November 15, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant