Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405

Classification

• Type: research / test infrastructure spike
• Priority: P2 / backlog, not a request for production implementation or a new diagnostic
• Area: Roslyn extractor → OwnIR → ownership analysis → diagnostics
• Inspiration: nkhitrov/typemut, which mutates Python type annotations and observes whether type checkers reject the mutant
• Related: Canonical first-party call facts: preserve interprocedural calls instead of collapsing them into frontend release/use/escape guesses #382 (canonical call facts), ConsumesParam flattens guarded release into unconditional handoff, causing false OWN009 and false must summaries #380 (guarded disposal), Prove value: oracle sweep over new OSS repos + draft triage table #201 (real-OSS differentiation oracle), Research: borrowed-view invalidation on collection mutation (CollectionsMarshal pilot) #400 (another bounded research issue)

Problem

We have regression fixtures and real-project mining, but no systematic way to answer: for a given ownership/lifetime bug family, what small changes can we make to correct C# programs that should turn into a diagnostic, and which of these changes does Own.NET actually detect?

Hand-written fixtures can encode the same assumptions as the analyzer. A semantic mutation harness could expose blind spots across aliases, branches and method boundaries. It would complement, not replace, the real-OSS miner and independent bug validation.

Unlike Stryker.NET, which tests the strength of runtime tests, this spike measures whether Own.NET's static analyzer reacts to known, validity-checked changes in C# ownership semantics.

Scope: first pilot only (3 mutation families)

Start with a tiny checked-in corpus of compiling .NET examples that are known to be clean in baseline runs, and create one minimal edit per mutant:

1. Premature release → use-after-release: insert Dispose() before a subsequent use; variant with an alias.
2. Lost release obligation: remove or move the sole release off a live path (e.g. guarded / early-return path); verify that this genuinely changes the lifetime obligation rather than only removing a redundant call.
3. Ownership handoff / interprocedural release: change a first-party callee from non-consuming to consuming (or introduce a guarded consume), while the caller still uses the resource. Treat method-summary precision / Canonical first-party call facts: preserve interprocedural calls instead of collapsing them into frontend release/use/escape guesses #382 and guarded disposal / ConsumesParam flattens guarded release into unconditional handoff, causing false OWN009 and false must summaries #380 as known areas to probe, not assumed bug reports.

Keep the matrix intentionally small: one straight-line and, where valid, one alias/control-flow or call-boundary variant per family. Do not start with arbitrary AST mutations, new OwnIR semantics, or full-repository fuzzing.

Required oracle and result classification

A detected diagnostic is meaningful only after ruling out invalid mutants and pre-existing errors.

For every case, record:

• baseline source and its expected clean diagnostic set;
• patch/diff, operator ID, stable mutant ID, relevant method/line;
• compilation / Roslyn-extraction result and whether the intended defect is actually present;
• expected diagnostic category and location/range, based on independently reasoned source semantics;
• Own.NET diagnostics in the baseline and mutant, plus difference;
• optional competitor results (CodeQL / Infer#) only when runnable, otherwise explicitly not tested.

Separate statuses instead of treating every nonzero exit as a success:

• Detected: valid defect mutant, expected new finding attributable to the injected defect.
• Missed: valid defect mutant, no expected finding.
• Invalid: does not compile / extraction fails / edit changes unrelated semantics.
• Equivalent or oracle-unresolved: no provable new defect (exclude from scored denominator).
• Infrastructure error / timeout: report separately, never count as detected.

A compiler error is not an analyzer detection. A warning at an unrelated site is not a kill. If a supposed clean baseline already has the target diagnostic, reject or fix the baseline before measuring coverage. Also include a couple of semantics-preserving negative controls (e.g. rename or safe refactor) that should not create diagnostics, to catch regressions in precision.

Implementation direction

• Prefer a small standalone harness that invokes existing project build / Roslyn extraction / own-check entry points, not a new analyzer subsystem.
• Start from immutable checked-in fixtures plus deterministic diff patches; use Roslyn transformations only if text patches prove insufficient.
• Keep outcomes and reproducer artifacts readable in JSON/Markdown, with a by-operator matrix and links/paths to cases. Do not collapse quality into a single mutation score.
• Reuse existing tests or mining/oracle infrastructure where it is actually compatible; no mandatory Stryker.NET dependency or third-party engine in the first slice.
• Keep cost bounded. No parallel worktree engine, global corpus crawl, continuous CI gate or automatically filed bug reports in phase 1.

Deliverables / acceptance criteria

• 3 operators with at least 2 reviewed case variants each where semantically applicable, plus 2 clean negative controls.
• All baselines compile and have no target findings; all scored mutants compile, extract correctly, and contain the claimed defect.
• Deterministic local command that generates/runs cases and emits machine-readable results with Detected / Missed / Invalid / Equivalent-or-unresolved / Infra-error distinctions.
• For each detected case, the reported finding matches the expected category and source location, not just a nonzero process exit.
• Results are independently reproducible with the exact revision/toolchain recorded; no source tree left modified after a run.
• A short report separates confirmed analyzer gaps from invalid tests or known limitations, and recommends either a narrow follow-up or STOP.

Stop / promotion gate

Timebox implementation to a small proof of concept. Promote only if it reveals at least one independently validated, previously untracked blind spot or shows substantial incremental regression coverage over existing fixtures at acceptable maintenance cost.

If the findings duplicate current tests/issues, mutants are predominantly invalid/equivalent, or the oracle is too subjective, document that and stop. Synthetic coverage is not evidence of real-world defect frequency or superiority over competing analyzers.

Out of scope: generic .NET mutation-testing product, new OWN rules, full differential competition suite, CI mutation-score threshold, claims that every surviving mutant is a bug.