Skip to content

Redact secrets written inside indexed files - #262

Merged
devmaxxx merged 2 commits into
mainfrom
feat/redact-content-secrets
Oct 19, 2026
Merged

devmaxxx merged 2 commits into
mainfrom
feat/redact-content-secrets

Conversation

@devmaxxx

Copy link
Copy Markdown
Owner

Summary

  • The walk already refused credential files by name (.env, *.pem, …), but a secret written inside an indexed file — a token in a source file, a key exported in a shell script, a private key pasted into a README — reached graph.json, the BM25 and dense indexes, and --rerank prompts. Every text read for extraction now goes through secrets_scanner's bundled (gitleaks-compatible) rules first. Each secret becomes [REDACTED_SECRET], newlines inside it are kept so line numbers hold, and inline gitleaks:allow markers are not honoured. update logs per file how many were redacted.
  • Compiling all ~1,000 rules costs 1.8 s per process, which every reader's inline update would pay after each edit. The scanner already skips any rule whose keywords a text lacks, so each run builds one scanner from only the rules its texts' keywords name. A one-file update takes 0.14 s in all (release build), and a test pins that this subset finds exactly what the full ruleset finds on every .rs/.md file in this repository. Texts go through in 64 MiB batches, and a no-op update builds no scanner.
  • GRAMMAR 6 → 7, so stores written before this change are re-read once instead of keeping the secrets of files that never change. The regex crates build optimised in the dev profile. Debug builds were 7× slower at compiling the rules, which pushed tests/reader_budget.rs from 3.5 s to 52.7 s.

Known gap: zerograph families prints the first line where each ID prefix appears without redacting it. That output goes to the terminal, not the store; left for a follow-up.

Test plan

  • cargo test: 1,546 lib tests and every integration binary pass
  • cargo clippy --all-targets -- -D warnings is clean
  • End-to-end: a GitHub token in a document reaches no file under .zerograph/. The test fails with redaction disabled.
  • Subset scanner and full ruleset give identical findings on every .rs/.md file in the repo
  • Multi-line PEM block: line numbers below it are unchanged
  • Timing on a copy of this repo (release): full --no-dense build 5.08 s, one-file update 0.14 s
  • CI is green on all three platforms

The walk already refuses credential files by name, but a token pasted
into a source file, a README or a shell script reached graph.json, the
lexical and dense indexes and rerank prompts. Every text read for
extraction now passes through secrets_scanner's bundled rules first;
each secret becomes [REDACTED_SECRET], keeping its newlines so line
numbers hold, and inline gitleaks:allow markers are not honoured.

Compiling all ~1,000 rules costs 1.8 s per process, which a reader's
inline update would pay on every edit. The scanner already skips a rule
whose keywords a text lacks, so a run builds one scanner from only the
rules its texts' keywords name: a one-file update takes 0.14 s in all.
A test pins that the subset finds what the full ruleset finds on every
.rs and .md file in this repository.

The regex crates build optimised in the dev profile, since every test
that builds a store compiles the subset.
Texts are scanned and extracted in 64 MiB batches instead of all at
once, and an empty batch builds no scanner, so a no-op update stays
free of the rules. GRAMMAR moves to 7 so a store written before
redaction is re-read once rather than keeping the secrets of files
that never change. The redaction count is the markers written, not
the findings, and the docs name the paths the rules' allowlist skips.
@devmaxxx
devmaxxx merged commit a0a4bdd into main Oct 19, 2026
4 checks passed
@devmaxxx
devmaxxx deleted the feat/redact-content-secrets branch October 19, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant