Skip to content

fix(report): the multi-language reachability figures are per-language — each language's filter record carried through the merge, the scan-wide figures summed across languages (#707) - #870

Merged
gadievron merged 2 commits into
masterfrom
fix/707-per-language-reachability
Oct 26, 2026

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

What

Closes #707 — PR #251's "Scope honesty" residual, disclosed but never filed: on a two-language scan, pipeline_stats.reachable_units was language A's kept count while total_units spanned A+B — the derived reduction percentage described A's slice while labelled scan-wide, and reachability_filter_applied read true from A's record even when B ran unfiltered. The merge spread only the first language's metadata and built per_language without the reachability record.

The fix

The merge carries each language's reachability_filter record into metadata.per_language[lang] (present-only — an unfiltered language keeps no key, never a fabricated record).

The reader builds the scan-wide aggregate when any record exists:

Single-language datasets fall back to the top-level record (the pre-#707 shape, still written) — unchanged.

Disclosed limits

The #602 denominator marker (reachability_baseline_languages) is not emitted on the parse-time merge path — no producer writes the per-record key (only the scanner's aggregate derives it), so the merge path can emit units_newly_reachable without the marker: absence, not fabrication; the mirror-the-scanner remediation is recorded. The first-wins regression is caught by the reader-level sibling pin (the artifact-level advisory pin's fixture carries B's advisory only — its docstring cites the sibling). The pruned_by_file cap matches the scanner's top-20.

Evidence (runs/707)

  • RED gate at 6f4b1f60..ccd93281: 19 nodes / 4 witnesses, all assertion-red at base (the reviewer re-ran the suite at base: the witnesses red by assertion, the legacy pins green).
  • C4 at the pushed head: 3 behavioral reverts (the merge carry, the reader aggregate, the telemetry threading) + 1 prose claim (the docstring); the test scope includes the neighboring merge suite.
  • fable adversarial review: 4 rounds — the fabricated zeros (violating the in-file LLM-reach report gaps: the unknown-unit_id skip line loses kind/confidence and has no counter; entry_points_promoted is not decomposed #602 contract), the first-wins advisories, the mixed-level claim, the scanner-shape per_language breaking the artifact contract on --llm-reachability runs, the advisory double-append — every finding executed, remediated, and mutation-probed; the terminal round plus the post-audit pin round both TERMINAL DRY.
  • astra bundle audit: two rounds — the artifact-level end-to-end gaps (the scanner-path projected-ints pin + the language-B advisory through the final artifact) demanded and landed; the re-audit: BUNDLE-ACCEPTABLE.
  • T8 standing-promises table, C2 anchor queries (consumed==total), A4 census (master unmoved; the open PR disjoint), reconciliation.
  • Parity — the legs' own terminal lines (the full logs in the bundle):
    # base (6f4b1f60)
    3 failed, 13244 passed, 38 skipped, 2 xfailed, 1 warning in 1602.34s
    # head (ccd93281)
    3 failed, 13262 passed, 40 skipped, 2 xfailed, 1 warning in 1588.09s
    
    the standing trio identity-matched in both legs; the +18 passing = this suite's 19 tests + the 1 parametrized item, minus 2 Go-binary tests flipped to skip by the fresh worktree's go_parser build race (the same first-run environment class as the npm race — both pass on the warmed re-run); the collect-diff: +20 items, 0 removed; the Go module: EXIT 0.
  • The battery — the runs' own terminal line: 96 passed in 0.21s (this suite's 19 + the merge + report suites).

sys.path.insert(0, str(_CORE_ROOT))

import core.dataset_merge as _prod_merge # noqa: E402
import core.reporter as _prod_reporter # noqa: E402
gadievron and others added 2 commits October 26, 2026 20:19
… — each language's filter record carried through the merge, the scan-wide figures summed across languages (#707)

PR #251's disclosed-but-never-filed residual: on a two-language scan,
pipeline_stats.reachable_units was language A's kept count while
total_units spanned A+B — the derived reduction percentage described
A's slice while labelled scan-wide, and reachability_filter_applied
read true from A's record even when B ran unfiltered. The merge spread
only the first language's metadata and built per_language without the
reachability record.

The fix (the issue's own direction): the merge carries each language's
reachability_filter record into metadata.per_language[lang] (present-
only — an unfiltered language keeps no key, never a fabricated record);
the reporter's reader builds the scan-wide aggregate when any record is
present — reachable_units and original_units are the sums (an
UNFILTERED language contributing its whole unit count to BOTH sides, its
units being all reachable, so the numerator's language scope always
matches the denominator's), the reduction percentage re-derived
scan-wide. The numeric telemetry keys are PER-KEY present-only (a key
emitted only when some record carried it — never a fabricated zero, the
#602 contract); the advisories are JOINED and ATTRIBUTED per language
(never first-wins — dropping language B's advisory re-opened the
#323/#520 vector); a MIXED-LEVEL scan claims no single
effective_processing_level (a warning instead, the scanner's own
refilter-aggregate precedent); a MALFORMED record (no integer
reachable_units) is named in a warning and treated as unfiltered (its
units both sides, never denominator-only fabricated pruning); a
top-level record already carrying its own per_language block (the
scanner's newer scan-wide aggregate) outranks the parse-time records.
Single-language datasets fall back to the top-level record.

The falsifier holds BOTH disjuncts: the scan-wide figures' denominator
matches the numerator's language scope, and the artifact now carries
the additive reachability_per_language decomposition (each language's
own kept count) beside the sums.
…y the numerator

The malformed-record gate admitted a record on ONE count:

    if not isinstance(rec.get("reachable_units"), int) or \
            isinstance(rec.get("reachable_units"), bool):

while the summing loop below adds each side INDEPENDENTLY — `kept` to
reachable_total, `orig` to original_total, each behind its own isinstance
check. So a record carrying a VALID integer reachable_units beside a
missing or non-integer original_units passed the gate, landed in
`records[lang]`, and then contributed to the NUMERATOR and not to the
DENOMINATOR. The language's own `units` count is never consulted on that
path, because a dict record with a valid kept count is routed to
`records[lang]` and the `units` fallback belongs to the UNFILTERED branch.

Two things could therefore ship that must not: more reachable units than
total units, and a NEGATIVE reduction percentage. That is the exact
invariant #707 exists to establish — the PR's own stated falsifier is "the
scan-wide figures' denominator matches the numerator's language scope" —
failing on a narrower input than the falsifier was written against.

EXECUTED through the real merge and the real _load_reachability_metadata
(fixture: python {units 20, rf 10/20} + go {units 500, kept 50, NO
original_units}):
  head ccd9328  : reachable=60  original=20   reduction=-200.0  <-- inverted
  this commit    : reachable=510 original=520  reduction=1.9     <-- sane
  canonical 2-lang fixture: 50.0% on BOTH sides, unchanged
and identically for original_units given as the string "500", as a bool,
as a float, as None and as a list.

THE REPAIR: the gate validates every field the reduction READS. The fields
are named ONCE, in `_REDUCTION_COUNT_FIELDS`, and checked by
`_unusable_reduction_counts` — the defect was that the gate knew one member
of a set the arithmetic reads two members of, so a list the gate iterates
cannot fall out of step the way two hand-written isinstance pairs did. A
rejected record falls to the UNFILTERED branch, which is this PR's own
documented policy for a malformed record ("its units both sides, never
denominator-only fabricated pruning"); the mirrored direction, numerator-
only, fabricates NEGATIVE pruning and is now named as such.

The malformed-record sentence moves into `_malformed_record_warning` so the
three emitting sites cannot drift, and the reason clause names the field
that actually failed. With reason == "no integer reachable_units" it
reproduces the pre-repair text BYTE-FOR-BYTE: the numerator case's wording
is unchanged, and a regression test pins that byte-equality.

WHY NO AUTHORED TEST CAUGHT IT, and the hole this closes: the suite's
record helper is `_rf(kept: int, original: int, ...)` — BOTH counts are
required positional arguments, so no fixture in the file could express the
defective record. 19/19 passed at the head and 19/19 on the control. The
suite gains `_partial_rf(**fields)`, the deliberate counterpart that CAN
omit a count, and the PR's falsifier is now asserted as a predicate
(reachable <= original AND reduction >= 0) over six malformed shapes
rather than stated as a sentence. This is the second consecutive defect in
this series hidden by a helper that cannot express the bad shape; the
inexpressiveness is what the new helper removes.

Reachability, stated against this change rather than for it: every one of
the ELEVEN production record producers writes original_units beside
reachable_units — the canonical reachability_filter_record
(entry_point_detector.py:648), parsers/swift/test_pipeline.py:248 and
core/parser_adapter.py:593,659, with rust/zig/go/c/ruby/php/javascript all
routing through the canonical one. So this gate can only ever reject a
record NO production path emits (a legacy, hand-assembled, third-party or
truncated dataset.json), and the canonical corpus is untouched. The defect
was latent, not an active regression; the gate is correspondingly in the
over-seed direction.

Rule-40 receipt (instrument checked FIRST, hash-bound
eb5c07f4f713f5a9 -> 555733f047c049a7), three dimensions — the #707 suite,
the inversion predicate, the canonical fixture: 6 mutants, 6 KILLED,
0 survivors.
  I0 a deliberate no-op comment  -> NOT scored a kill (instrument valid)
  M1 _REDUCTION_COUNT_FIELDS narrowed to ("reachable_units",) -- THE
     DEFECT ITSELF                                      -> KILLED (3 failed,
     the predicate speaking the exact divergence: assert 60 <= 20)
  M2 the bool tooth removed                             -> KILLED (2 failed)
  M3 narrowed to ("original_units",)                    -> KILLED (3 failed)
  M4 the gate disabled (predicate returns [])           -> KILLED (6 failed)
  M5 the warning wording drifted                        -> KILLED (1 failed)
  M6 the reason clause hardcoded to the numerator field -> KILLED (1 failed)
Control restored BYTE-EXACT to 555733f047c049a7 after every mutant. M5 and
M6 kill on the suite dimension only and pass the inversion predicate, which
is correct: they touch wording, not arithmetic.

Review-arc commits: this commit (the hunks below had NO independent
reviewer — the reviewer became the author under the ADDENDUM 42 grant).
authored-by: arc(ses_e8704fffdffehwcool1yeMiODl)

Agent: PR-MERGE ses_e8704fffdffehwcool1yeMiODl
@gadievron
gadievron force-pushed the fix/707-per-language-reachability branch from 99ba15d to f8b403d Compare October 26, 2026 18:21
@gadievron

Copy link
Copy Markdown
Collaborator Author

R3/R4 — PR 870, at head f8b403d9 (the repaired + master-rebased head)

@sounil — eligible non-author CODEOWNER, re-derived from .github/CODEOWNERS at live master (a959dd3b) this arc and not cached: a single * rule naming seven owners, of whom this PR's author is one, so R5 applies. A notice for the record, not a gate on any merge.

DISCLOSURE, re-derived live at this head: repo-visibility public · enforce_admins=false · required_reviews unmet · escape-hatch not allowance (the allowance exists and names ar7casper, not the acting identity). Read separately: ruleset ci-required (id 23956162, active) carries 0 bypass_actors across its 14 required contexts, so nothing waives CI; branch protection carries require_code_owner_reviews=true. Approving review count 0, none manufactured.

WHAT CHANGED SINCE ccd93281, AND WHY

The previous arc halted at the whole-diff R2 review on a real finding, handed up rather than self-accepted. The maintainer granted the repair (ADDENDUM 42, the 14a(iii) grant made explicit). Two commits now carry the branch:

commit what
84636e33 the authored #707 work, rebased onto a959dd3b (unchanged content)
f8b403d9 the denominator-gate repair + 4 tests (+166/−15)

THE FINDING. The malformed-record gate admitted a record on one count (reachable_units), while the summing loop adds each side independently. So a record with a valid integer reachable_units beside a missing or non-integer original_units passed the gate, landed in records[lang], and contributed to the numerator and not the denominator. Two things could ship that must not: more reachable units than total units, and a negative reduction percentage. That is the exact invariant #707 exists to establish — its own stated falsifier is "the scan-wide figures' denominator matches the numerator's language scope".

EXECUTED through the real merge and the real _load_reachability_metadata (fixture python {units 20, rf 10/20} + go {units 500, kept 50, NO original_units}):

tree reachable original reduction
head ccd93281 60 20 −200.0 ← inverted
this head f8b403d9 510 520 1.9 ← sane
canonical 2-language fixture 15 30 50.0 (unchanged)

Identical inversion for original_units as the string "500", as a bool, as a float, as None and as a list.

THE REPAIR. The gate validates every field the reduction reads; the fields are named once in _REDUCTION_COUNT_FIELDS and checked by _unusable_reduction_counts, so a list the gate iterates cannot fall out of step with the arithmetic the way two hand-written isinstance pairs did. A rejected record falls to the unfiltered branch — this PR's own documented policy for a malformed record. The malformed sentence moves into _malformed_record_warning so the three emitting sites cannot drift, and with reason no integer reachable_units it reproduces the pre-repair text byte-for-byte (a regression test pins that byte-equality).

WHY NO AUTHORED TEST CAUGHT IT. The suite's helper is _rf(kept: int, original: int) — both counts are required positional arguments, so no fixture could express the defective record. 19/19 passed at the head and 19/19 on the control. The suite gains _partial_rf(**fields), the deliberate counterpart that can omit a count, and the falsifier is now asserted as a predicate over six malformed shapes. Three of the four new tests are RED on ccd93281 (assert 60 <= 20) and GREEN here; the fourth is the wording control that must pass on both sides.

REACHABILITY, stated against the change rather than for it. All eleven production record producers write original_units beside reachable_units — the canonical reachability_filter_record (entry_point_detector.py:648), parsers/swift/test_pipeline.py:248, core/parser_adapter.py:593,659, with rust/zig/go/c/ruby/php/javascript routing through the canonical one. So the gate can only reject a record no production path emits; the canonical corpus is untouched and the defect was latent, not an active regression.

RULE-40 (instrument checked FIRST, hash-bound eb5c07f4f713f5a9 → 555733f047c049a7), three dimensions — the #707 suite, the inversion predicate, the canonical fixture: 6 mutants, 6 KILLED, 0 survivors. I0 a deliberate no-op was not scored a kill. M1 the gate narrowed back to numerator-only — the defect itself — KILLED, the predicate speaking the exact divergence. M2 the bool tooth removed, M3 denominator-only, M4 the gate disabled, M5 wording drift, M6 the reason hardcoded — all KILLED. Control restored byte-exact after each. One mutant (M5, first attempt) produced a file hash equal to the control, i.e. the edit never applied; it was re-run rather than scored as a survivor.

SUITE: 13,259 passed / 41 skipped / 2 xfailed on the repaired tree. The 9 failures (tests/patch/test_run_cve_batch.py ×5, tests/patch/test_run_metadata.py::TestCollectGitInfo ×2, tests/test_llm_sdk_contract_floor.py ×2) reproduce identically on the unrepaired head — the control was run before the attribution, not after — and are not attributable to this change. ruff check . clean. CI 19/19 at this head, run_attempt > 1 census = 0, so no rerun family was invoked and none is claimed.

THE REBASE, DISCLOSED. The repair's first push (99ba15dc) was a fast-forward, but it then read mergeable: dirty: my helper block and 869's landed _finding_cwe block insert at the same anchor, so the two independent additions conflicted. The pre-repair head ccd93281 merged master cleanly — the conflict was mine, not the author's. Resolved by 14b's named rebase onto a959dd3b, keeping both blocks; the author's commit rebased without conflict and its content is unchanged. behind_by is now 0, so the rebase served as F3 and no separate update-branch was issued. Every verification leg was re-run on the rebased bytes.

THE PUSH. [PUSH-GRANT] form=prose scope=870 branch=fix/707-per-language-reachability parent=ccd93281, riding ADDENDUM 42's committed "the 14a(iii) grant made explicit". The author lane was probed immediately before each act and read ACTIVE (mid-turn) — but its live work is OpenAnt-709 (issue #709), it holds no claim on this branch, and its last push here was 9.73 h earlier; the head was unmoved across three endpoints. Lease pinned, readback byte-equal, master never touched. Letter-deviation disclosed: the literal [TAKEOVER-GRANT] PR=… head=… token is still absent — the grant names clause 14a(iii) and declares it explicit, which §1273 admits as "the maintainer's explicit waiver of the literal form", but it is a clause-naming, not the token. Recorded rather than papered over.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant