Repository navigation
feat(comply): pre-fill more fields from scan data - #137
Conversation
Fill more of the comply pack's TODO fields from data the scan already reaches, each tagged with its source and none guessed. SBOM (additive; old-vs-new diff on demo_data and bert shows no change to existing fields): - modelCard properties: aisbom:hf:author, base_model, language, created_at (skipping HF's 2022 migration stamp), parameters_total and co2_eq_emissions; model-index results as typed quantitativeAnalysis.performanceMetrics (capped at 50) - aisbom:safetensors:parameters per file, summed from header shapes - aisbom:pypi:author on dependencies, from the existing PyPI request (licence cache schema bumped so old entries are refetched) comply: - EU AI Act: model dependencies from base_model; parameter count from HF's total and/or a complete safetensors set; modality from the task (AIsbom's mapping); "declared" rows for languages, first published on Hugging Face, training hardware, CO2 and evaluation results, leaving the regulatory field itself TODO - FDA: supplier and end-of-support columns appended to component-support.csv. End-of-support dates come from endoflife.date for packages it tracks (package names only; --no-eol-lookup, AISBOM_NO_EOL_LOOKUP=1 and --offline skip it; disclosed in README) docs/comply.md mapping updated; TEMPLATE_VERSION 2026-11-15.2.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d14ef00cbf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (str(c["name"]), str(c["version"])) | ||
| for c in _components(doc) | ||
| if c.get("type") != _ML_COMPONENT_TYPE and c.get("name") and c.get("version") |
There was a problem hiding this comment.
Preserve exact-pin information before selecting EOL lookups
For an unpinned requirement such as numpy>=2.2.0, DeepScanner._parse_requirements records version="2.2.0" and pinned=False, but CycloneDX serialization drops the pinned flag. This predicate consequently includes that lower bound as though it were an installed version, and the FDA pack fills end_of_support_date using the 2.2 cycle even though the requirement permits later cycles with different support dates. Preserve pin provenance through serialization and restrict EOL enrichment to exact pins; otherwise unpinned dependencies should remain TODO.
Useful? React with 👍 / 👎.
| shard = _SHARD.match(name) | ||
| if shard: | ||
| key = (shard["stem"], int(shard["total"])) |
There was a problem hiding this comment.
Group local shards by their originating directory
When scanning a local directory tree, artifact names contain only basenames (_inspect_safetensors uses local_path.name), so this key merges shards from unrelated model directories. For example, a tree containing only a/model-00001-of-00002.safetensors and b/model-00002-of-00002.safetensors produces a supposedly complete parameter count in Annex XI even though neither model is complete. Carry the originating directory or another model-set identity into the grouping, and leave the count TODO when that identity cannot be established.
Useful? React with 👍 / 👎.
…er directory endoflife.date dates are now looked up only for exact == pins, taken from the scan's dependency list (the SBOM drops the pin flag and records numpy>=2.2.0 as 2.2.0), and joined to components by bom-ref. Shard-set completeness is now decided by the scanner, which sees directories, and written as aisbom:safetensors:set_parameters / set_shards on each shard. comply no longer groups by component name, which is a basename for local scans and would treat halves of two models in different directories as one set.
Summary
aisbom complynow pre-fills more of the fields it used to leave asTODO (provider), from data the scan already reaches. Every value names its source, and nothing is guessed:SBOM (additive only)
modelCardproperties:aisbom:hf:author,base_model,language,created_at,parameters_total,co2_eq_emissions(+ source, training type, hardware, location).created_atis skipped when it is the2022-03-02T23:29:04Zstamp that every repo older than HF's migration carries.model-indexresults go into the typedquantitativeAnalysis.performanceMetricsslot, capped at 50.aisbom:safetensors:parametersis added per file: the sum of tensor-shape products from the header, recorded only when every shape is well formed.aisbom:pypi:authoris added on dependencies, read from the PyPI request already made for licences. The licence cache schema is bumped so old entries are fetched again.comply packs
base_model.-0000k-of-0000Nshard).component-support.csvgets four new columns at the end:supplier,supplier_source,end_of_support_lookup,end_of_support_lookup_source.pkg:pypi/purl, matched to the pin's release cycle. A provider answer always keeps the date cell.Privacy
endoflife.datelookup forcomply --framework fdaonly. It sends package names; versions stay local.--no-eol-lookup,AISBOM_NO_EOL_LOOKUP=1or--offline.scannever makes this request.docs/comply.mdmapping is updated;TEMPLATE_VERSIONis now2026-11-15.2.Verification
poetry run pytest --cov=aisbom --cov-fail-under=85: 1777 passed, 94.63%. New suites: modelcard declared fields, safetensors parameters, PyPI author, endoflife.date, comply scan fields (present/absent per field, partial and duplicated shard sets, answers coexistence, CLI opt-outs).aisbom bypass-scorecard --checkpasses.main) vs new SBOM, compared field by field fordemo_dataandhf://google-bert/bert-base-uncased: no existing field changed; only the properties above were added.