Skip to content

feat(p014): add opt-in project-aware binding coverage - #423

Merged
PhysShell merged 9 commits into
mainfrom
feat/p014-prod0-coverage
Oct 21, 2026
Merged

PhysShell merged 9 commits into
mainfrom
feat/p014-prod0-coverage

Conversation

@PhysShell

@PhysShell PhysShell commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

What this is

P-014 PROD0. It adds one opt-in, observation-only extractor flag, --binding-coverage FILE. For one restored
.csproj it writes a per-project binding-coverage record:

  • binding rates for call, type and event sites;
  • compiler errors;
  • the reference manifest, and how each assembly conflict was decided;
  • a status with named reasons.

Without the flag nothing changes. Extractor behaviour, OwnIR, SARIF, exit codes and diagnostics stay the same.
PROD1 remains forbidden. New symbols are not fed to diagnostics. #407 stays deferred.

Final head: 4b0321e240beda354f193b54502ccd4f18592478. Base: main 22b29d5b.

Verdict history (both decisions kept)

Head Verdict Where
f84943d PASS-P014-PROD0 issued on G0–G3, then refuted by the owner's adversarial review → KILL-P014-PROD0 (final) REPORT.md (unrewritten), 01-review-falsifiers.md, 02-review-result.md
4b0321e PASS-P014-FIX0: a separate, bounded experiment, pre-registered before the code change; CI green on this head 03-fix0-preregistration.md, 04-fix0-result.md

The review found three defects on f84943d, all in what the sidecar said about its own completeness. Facts and
exit codes were never affected.

Falsifier f84943d 4b0321e
F-PROD0-1 #if NET8_0, no DefineConstants COMPLETE (it measured the #else branch) SOURCE_SET_INCOMPLETE [preprocessor-conditional]
F-PROD0-2 "/*probe:" in a source no sidecar (exception in the research probe parser) sidecar written
F-PROD0-3 unreadable .cs unreadable = 1 together with COMPLETE SOURCE_SET_INCOMPLETE [unreadable-source]
F-PROD0-4 Root → Child, Child unreadable both COMPLETE Child [unreadable-source]; Root [p2p-unit-source_set_incomplete:Child]

Production changes

File Change
frontend/roslyn/OwnSharp.Extractor/BindingCoverage.cs (new) resolver and coverage code in namespace OwnSharp.BindingCoverage, plus the sidecar Runner; FIX0 changes listed below
frontend/roslyn/OwnSharp.Extractor/Program.cs (+17) --binding-coverage FILE, --binding-coverage-tfm TFM, usage text, and one call after the facts are written. A failure goes to stderr; the exit code is unchanged
.github/workflows/ci.yml (one step in wpf-extractor) facts are cmp-identical with and without the flag; directory and unrestored inputs are UNSUPPORTED; the restored sample is measured, deterministic, and COMPLETE exactly when no reason is present
frontend/roslyn/README.md, docs/proposals/P-014-semantic-resolution.md the flag is documented, and PROD1 is marked as not authorised

FIX0, in BindingCoverage.cs:

  • any #if/#elif (Roslyn directive trivia) gives preprocessor-conditional;
  • an unreadable source gives unreadable-source;
  • a project-reference member's status reaches its parent in Compile, after the member's sources are read;
  • /*probe:*/ parsing runs only under Measure(recordProbes: true), with a bounds check, and the Runner never
    enables it.

Fail closed

COMPLETE requires all of the following:

  • a complete reference set: packs and packages present, identity checked by content hash, and every same-file-name
    conflict decided by the dotnet/sdk order (overrides → AssemblyVersion → FileVersion);
  • and no source-set reason: generators, XAML code generation, implicit usings, defines, #if/#elif,
    Directory.Build.*, package MSBuild assets, an unreadable source, or an incomplete project-reference member;
  • and nothing unsupported.

Defines and implicit usings are not applied; they are only reported.

Evidence on 4b0321e (docs/evidence/p014-prod0-2026-10/04-fix0-result.md)

  • Facts. Byte-identical across main 22b29d5b, this branch without the flag and this branch with it:
    P4 AD528B8A…, P5 5F0EA76D…, samples F82A32D8….
  • Controls. K1, K2, K4 (CS0101 in one shared compilation, CS0433 at the site when projects are compiled
    separately), K5, K6, K7, F2.3–F2.5, K11, K12 and the guard, run against the production code: all PASS, and
    identical to the f84943d run.
  • SDK oracle. Unchanged: K8a 163 = 163, K8b 163 = 163, K8c 206 = 206, G0 Wpf.Ui.dll 209 = 209, K9 163 = 163
    and 167 = 167.
  • Comparison with the f84943d run. In 12 sidecars every field is identical; the only change is the added
    preprocessor-conditional reason on P4's root project and on P5. P4/P5 are never COMPLETE.
  • Determinism. Sidecars are byte-identical across 3 runs.
  • Cost on P5. The flag adds 1578 ms, against a limit of 3 × legacy-merged warm in the same session = 2721 ms;
    peak about 297 MB.
  • Build. No new warning.

Tests

  • ruff check . passes.
  • CI is green on 4b0321e.
  • Local python tests/run_tests.py exits 1 on this Windows host.
    • Setup: clean committed tree, cargo on PATH.
    • The failures: the same 87 test_cli_ownir_fixtures cases as the main baseline in the same environment
      (frozen LF bytes against Windows CRLF). There are no other failures.
    • That CRLF issue is a separate cross-platform golden-test problem and is not addressed here.

Limits

  1. A complete reference set does not mean a complete compilation. The coverage numbers are binding rates, not a
    share of verified code.
  2. preprocessor-conditional is conservative: even #if DEBUG makes a unit incomplete.
  3. foreign_binding = 0 guards against leaks between projects. It is not independent proof that the reference
    manifest is right; that rests on the SDK-oracle checks.
  4. Not qualified: equal-version package-vs-package conflicts (left CONFLICT_UNRESOLVED), the rank and platform
    steps, the full SDK contract.
  5. Host environment: SDK 11.0.100-rc.2, packs 8.0.32 and 10.0.13.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG

PhysShell and others added 6 commits October 21, 2026 12:36
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
…vation only)

One CSharpCompilation per project from the existing project.assets.json, local
targeting packs and the packages folder (project references as compilation
references; dotnet/sdk conflict order; fail-closed statuses), written to a
separate file after the facts. Facts, findings, SARIF and exit codes are
unchanged; directory/.sln/unrestored inputs are recorded UNSUPPORTED. CI step
in wpf-extractor asserts byte-identical facts and the fail-closed status.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
G0 Wpf.Ui.dll package-vs-package conflict equals the SDK oracle (209=209);
G1 P4/P5 all SOURCE_SET_INCOMPLETE; G2 facts byte-identical main/off/on,
controls K1-K12 against production code, fixtures vs oracle, foreign 0,
deterministic sidecars; G3 P5 +1390 ms vs limit 3063 ms, ~302 MB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
…only CRLF module, same on main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
…D0-1..3)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
…w; FIX0 gate proposed

F-PROD0-1 (#if NET8_0 without DefineConstants -> COMPLETE), F-PROD0-2 (probe
marker text -> no sidecar) and F-PROD0-3 (unreadable source with COMPLETE) all
confirmed on the f84943d build. Facts and exit codes unaffected. Fix gate is
proposed only, not started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
@PhysShell

Copy link
Copy Markdown
Owner Author

CODE-REVIEW-HOLD / KILL-P014-PROD0 on f84943d. Three adversarial falsifiers confirmed: false COMPLETE under conditional compilation, sidecar failure on an unterminated probe marker, and COMPLETE with an unreadable source. FIX0 is authorized as a separate bounded experiment. Merge remains blocked.

Details:

  • The runs used the extractor built from f84943d's own sources; the fixtures are synthetic net8.0 projects restored offline.
    • F-PROD0-1: #if NET8_0 with no <DefineConstants> gives unit_status = COMPLETE. The sidecar measured the #else branch.
    • F-PROD0-2: the source line Console.WriteLine("/*probe:"); gives no sidecar (ArgumentOutOfRangeException in the research probe parser). The exit code stays 0.
    • F-PROD0-3: an unreadable .cs gives files.unreadable = 1 together with unit_status = COMPLETE.
  • Facts and exit codes are not affected. The defects are in what the sidecar claims about its own completeness.
  • The review evidence (pre-registration of the falsifiers, then the results) is kept on the branch locally for now. It will be pushed to this PR together with FIX0's result.

PhysShell and others added 3 commits October 21, 2026 15:16
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
…sources; probes off in production

- any #if/#elif (Roslyn directive trivia) in a unit's sources -> SOURCE_SET_INCOMPLETE(preprocessor-conditional):
  sources are parsed without the SDK's TFM defines, so the measured branch may not be the compiled one;
- an unreadable source -> SOURCE_SET_INCOMPLETE(unreadable-source);
- project-reference status propagation moved to Compile (post-order), after each member's sources are read,
  so an incompleteness found while reading reaches every parent;
- /*probe:*/ parsing only behind Measure(recordProbes: true), bounds-checked; the sidecar Runner never enables it.
Facts, findings and exit codes unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
…ed; G0-G3 re-run)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
@PhysShell

Copy link
Copy Markdown
Owner Author

FIX0 pushed: new head 4b0321e. Local verdict: PASS-P014-FIX0. The required CI cycle on this head is pending, and merge stays blocked until it passes and the owner reviews it.

Verdict history:

  • PASS-P014-PROD0 (REPORT.md) was refuted by review.
  • KILL-P014-PROD0 for f84943d is final (02-review-result.md).
  • FIX0 is a separate, bounded experiment (03-fix0-preregistration.md, committed before the code change), and this PASS applies only to the new head.

Commits, in order:

  1. b446be4: review falsifiers, pre-registered.
  2. b693a62: review result, KILL.
  3. f60f35f: FIX0 pre-registration and the F-PROD0-4 fixture.
  4. af18f0b: the fix.
  5. 4b0321e: the results (04-fix0-result.md).

The fix (BindingCoverage.cs only; facts, findings and exit codes are unchanged):

  • Conditional compilation. Any #if/#elif, detected through Roslyn directive trivia, now gives SOURCE_SET_INCOMPLETE(preprocessor-conditional).
  • Unreadable source. An unreadable source now gives SOURCE_SET_INCOMPLETE(unreadable-source).
  • Status propagation. A project-reference member's status now reaches its parent after the member's sources are read (post-order in Compile).
  • Probe markers. /*probe:*/ parsing runs only under Measure(recordProbes: true), with a bounds check; the production Runner never enables it.

Falsifiers:

f84943d 4b0321e
F-PROD0-1 #if NET8_0 COMPLETE SOURCE_SET_INCOMPLETE [preprocessor-conditional]
F-PROD0-2 "/*probe:" no sidecar sidecar written
F-PROD0-3 unreadable .cs COMPLETE SOURCE_SET_INCOMPLETE [unreadable-source]
F-PROD0-4 Root → Child, Child unreadable both COMPLETE Child [unreadable-source]; Root [p2p-unit-source_set_incomplete:Child]

G0–G3 re-run:

  • Facts are byte-identical across main, the branch without the flag and the branch with it.
  • The controls are identical to the f84943d run.
  • The oracle fixtures are unchanged.
  • In 12 sidecars every field is identical; the only change is the added preprocessor-conditional reason on P4's root and on P5.
  • Sidecars are deterministic.
  • Cost on P5: the flag adds 1578 ms, against a limit of 2721 ms, with a peak of about 297 MB.

Tests. ruff passes. python tests/run_tests.py, run on the clean committed tree with cargo on PATH, exits 1 with the same 87 test_cli_ownir_fixtures CRLF cases as the main baseline, and nothing else.

Not claimed. foreign_binding = 0 guards only against leaks between projects. The correctness of the reference manifest rests on the SDK-oracle checks.

@PhysShell
PhysShell merged commit 1750769 into main Oct 21, 2026
84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant