Support `--learn` for inline expectation tests by d10c · Pull Request #22228 · github/codeql

d10c · 2026-07-23T14:08:33Z

Teaches the shared inline-expectations test library to drive codeql test run --learn for inline // $ Alert expectations: when a postprocess test query is run under --learn, the library now emits a learnEdits relation describing how to rewrite the annotated source so the comments match the actual query results.

Requires the matching CLI support for consuming learnEdits; the library change is incompatible CLIs older than that. The CI on this PR should pass only once a supported CLI gets released to nightly.

What it does

For each result/expectation mismatch on the standard // $ tag1 tag2=value tag3[query] // rest of comment…-style tags, learnEdits records a structured edit (append a new comment, remove or rewrite an existing one). The CLI applies these edits to the source file, so --learn updates the inline expectations directly rather than writing a failure grid into .expected. Without --learn the relation is invisible and behaviour is unchanged.

API

TestPostProcessing::InputSig gains getStartCommentMarker(...) (and a defaulted getEndCommentMarker), keyed on the source file of a location so a single database can mix languages with different comment syntax. Each per-language utils/test/InlineExpectationsTestQuery.ql supplies its marker (//, #, ...); block-comment languages are left for a follow-up but the end-marker hook is already in place.

Follow-ups not implemented in this PR

Block-comment languages (XML/YAML/HTML/ERB/Razor …)
Multi-query reconciliation (// $ Alert[query1] Alert[query2])
Splitting the tag1,tag2=value shorthand where only one tag needs to be updated.
Support for non-qlref/non-postprocessor inline expectations tests.

owen-mc

getStartCommentMarker would actually be (mildly) useful for ~~models-as-data~~ as well. (Correction, I meant inline expectations test, specifically for this regex.) Would you consider putting it somewhere else, e.g. in the main library for each language, so that it could be used for that?

Kamil-0425 · 2026-09-24T15:55:53Z

+       * that report a line comment as ending at column 1 of the next line.
+       */
+      private int getReplacementEndColumn(TestImpl2::ExpectationComment comment) {
+        exists(string relativePath, int sl, int el, int ec |


Copilot

Copilot review overview

🟡 Changes recommended

Mandatory signature implementations are missing, and block comments can be rewritten into source-breaking line comments.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5 · 1 · 1

Open (7)

Restrict Go learning rewrites to line comments · New
Avoid rewriting block comments with line-comment markers · New
Preserve Rust block-comment delimiters during rewrites · New
Implement comment markers in Kotlin test utilities · New
Exclude Swift block comments from line-comment rewrites · New
Suppress appends when any expectation comment occupies the line · New
Add automated coverage for the new learn-mode edit protocol · New

Resolved since last review (1)

This documentation overstates the block-comment hook: the note-only replacement path at lines…

+  string getStartCommentMarker(string relativePath) {
+    // Go databases can also contain XML, whose block-comment syntax is not yet supported, so
+    // we only render for Go sources.
+    relativePath.matches("%.go") and
+    result = "//"


+  string getStartCommentMarker(string relativePath) {
+    // JavaScript databases can also contain HTML, whose block-comment syntax is not yet
+    // supported, so we only render for line-comment source files.
+    relativePath.regexpMatch(".*\\.(js|cjs|mjs|jsx|ts|cts|mts|tsx)") and
+    result = "//"


+  string getStartCommentMarker(string relativePath) {
+    // Rust databases can also contain YAML, whose `#` comment syntax differs, so we only
+    // render for Rust sources.
+    relativePath.matches("%.rs") and
+    result = "//"


+  bindingset[relativePath]
+  string getStartCommentMarker(string relativePath);


+  string getStartCommentMarker(string relativePath) {
+    // The unified extractor currently ingests only Swift sources, which use `//`. Gating on
+    // the extension keeps this correct if it gains a language with a different comment syntax.
+    relativePath.regexpMatch(".*\\.(swift|swiftinterface)") and
+    result = "//"


+          not exists(TestImpl2::ExpectationComment existing |
+            isRewritableComment(existing) and
+            parseLocationString(existing.getLocation().getRelativeUrl(), relativePath, el, _, _, _)
+          ) and


+      query predicate learnEdits(
+        string file, int line, string operation, int startColumn, int endColumn, string text
+      ) {


`getAnExpectation` -- the low-level parse of a `// $ ...` expectation comment -- lived as a private predicate inside `Make<Impl>`, reachable only by that module's expectation classes. The upcoming `--learn` postprocessing needs the same parse (to see every expectation on a comment, including ones the running test ignores, so it can preserve them), but it lives in `TestPostProcessing`, outside `Make`. Move the predicate into a new top-level `private module ExpectationParser<Impl>` that both consumers `private import`, so neither has to expose the parse as part of its API nor reach into the other's internals. This is a pure refactor: the predicate body and its callers' behavior are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083

The `--learn` support that follows needs to render inline-expectation comments in the syntax of the file being edited, and a single CodeQL database can mix source languages (e.g. Java plus XML), so the marker must be chosen per file rather than per database. Extend `TestPostProcessing::InputSig` with `getStartCommentMarker(relativePath)` (no result -> that file is left untouched by `--learn`) and a defaulted `getEndCommentMarker(relativePath)` (`none()` by default: line-comment languages render no closing marker; block-comment languages can override it later). Each language's `InlineExpectationsTestQuery.ql` implements `getStartCommentMarker`, gated to that language's own source extensions so that other file types (XML/YAML/HTML/ERB/Razor) extracted into the same database are not rewritten with the wrong syntax. This commit only wires the marker API through the signature and the per-language inputs; nothing consumes it yet, so behavior is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083

`codeql test run --learn` could previously only rewrite `.expected` files; inline expectations (the `// $ Alert` comments checked by `InlineExpectationsTest`) had to be fixed by hand. This teaches the shared test library to compute those source edits so the test runner can apply them. The `test-postprocess` query exposes a `learnEdits` relation (`file, line, operation, startColumn, endColumn, text`) describing the minimal source rewrite that makes the inline expectations match the actual query results. Replace edits use 1-based half-open `[startColumn, endColumn)` ranges. Equal boundaries represent insertion, and an end column of zero denotes replacement through the end of the line. Same-line inclusive CodeQL location ends convert to exclusive boundaries; locations ending on the following line use the to-EOL form. The relation covers: - appending a fresh comment carrying every tag learned for a line that has an unexpected result and no existing comment to merge into; - rewriting an existing expectation comment as a whole so it matches the current results: dropping fixed-spurious tags, promoting a `MISSING:` expectation that now fires, clearing a stale `SPURIOUS:` annotation, and merging in freshly learned tags, re-rendering the remaining expectations (or deleting the comment when none remain); - preserving expectations this test does not own -- e.g. a tag annotated with a different query's id that shares the source file -- and any trailing regular note (`// $ Alert // note`); - recording any unexpected result, not just `Alert`. The comment syntax comes from the `getStartCommentMarker` / `getEndCommentMarker` markers added to `InputSig` in the previous commit, so `LearnEditsImpl` renders each edit in the target file's own syntax and stays language-agnostic. The Actions postprocessor supplies `#` for `.yml` and `.yaml` sources when implementing the same signature. Edits are emitted as a query predicate rather than applied here: the engine consumes `learnEdits` only under `--learn` and ignores it otherwise, so ordinary `test run` output is unchanged. The supporting predicates are wrapped in a `private module LearnEditsImpl` so only the `query predicate learnEdits` is re-exposed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083

`getAnExpectation` is a relation that holds between an expectation comment and the parsed parts it carries, not a function returning a single value, so the `get` prefix is misleading and trips the CodeQL predicate-naming style check. Rename it to `hasExpectation`, matching the `has`-prefixed convention for such predicates (and the sibling `hasExpectationWithValue`). Pure rename with no behavior change; reformatted with `codeql query format`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083

codeql test run --learn recognises a postprocess query as an inline-expectation test, and so may rewrite the inline `// $ ...` comments in its source, by the `@tags inline-expectation-test` on the query. Add that tag to every language's InlineExpectationsTestQuery.ql so their tests are learnable; without it --learn declines to edit their source and fails, since it cannot group the query with its siblings into a shared-source cohort. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083

Emit append edits only when an unexpected result starts and ends on the same source line. Multi-line locations need language-aware source anchors; leaving them unedited lets the tagged test fail safely instead of inserting a line comment inside a construct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1adfa307-c48e-43f5-94e3-8286e4a8c083

d10c · 2026-09-24T19:11:27Z

getStartCommentMarker would actually be (mildly) useful for ~~models-as-data~~ as well. (Correction, I meant inline expectations test, specifically for this regex.) Would you consider putting it somewhere else, e.g. in the main library for each language, so that it could be used for that?

@owen-mc Once the API has stabilized, I can. I already suspect that I'll need to support both // and /* syntaxes in the same language, for instance.

github-actions Bot added C# JS C++ Java Python Go Ruby QL-for-QL Rust Pull requests that update Rust code Swift labels Jul 23, 2026

github-advanced-security Bot found potential problems Jul 23, 2026

View reviewed changes

Comment thread shared/util/codeql/util/test/InlineExpectationsTest.qll Fixed

d10c force-pushed the d10c/learn-inline-expectations-lib branch 3 times, most recently from 63edc17 to 52f2ac6 Compare July 28, 2026 16:03

d10c requested a review from cklin July 28, 2026 16:17

d10c marked this pull request as ready for review July 28, 2026 16:17

Copilot AI review requested due to automatic review settings July 28, 2026 16:17

d10c requested review from a team as code owners July 28, 2026 16:17

Copilot started reviewing on behalf of d10c July 28, 2026 16:18 View session

d10c added the no-change-note-required This PR does not need a change note label Jul 28, 2026

aschackmull reviewed Jul 29, 2026

View reviewed changes

Comment thread shared/util/codeql/util/test/InlineExpectationsTest.qll Outdated

aschackmull reviewed Jul 29, 2026

View reviewed changes

Comment thread shared/util/codeql/util/test/InlineExpectationsTest.qll Outdated

d10c force-pushed the d10c/learn-inline-expectations-lib branch from 52f2ac6 to 62a0475 Compare July 30, 2026 17:05

d10c marked this pull request as draft August 13, 2026 16:08

d10c force-pushed the d10c/learn-inline-expectations-lib branch 2 times, most recently from 4b9a47d to f8bdb41 Compare August 13, 2026 16:46

d10c marked this pull request as ready for review August 13, 2026 17:26

owen-mc reviewed Aug 14, 2026

View reviewed changes

Comment thread java/ql/lib/utils/test/InlineExpectationsTestQuery.ql Outdated

d10c force-pushed the d10c/learn-inline-expectations-lib branch from f8bdb41 to 8d9192d Compare August 20, 2026 09:56

d10c force-pushed the d10c/learn-inline-expectations-lib branch from 9b870e6 to fc542a0 Compare September 1, 2026 17:33

d10c force-pushed the d10c/learn-inline-expectations-lib branch from fc542a0 to 4234e9b Compare September 8, 2026 18:39

d10c force-pushed the d10c/learn-inline-expectations-lib branch from 4234e9b to 7774288 Compare September 24, 2026 15:50

d10c requested a balanced review from Copilot September 24, 2026 15:52

github-advanced-security Bot found potential problems Sep 24, 2026

View reviewed changes

Copilot started reviewing on behalf of d10c September 24, 2026 15:53 View session

Copilot AI reviewed Sep 24, 2026

View reviewed changes

d10c force-pushed the d10c/learn-inline-expectations-lib branch from 7774288 to ae24f4e Compare September 24, 2026 17:57

d10c and others added 2 commits September 24, 2026 14:08

github-actions Bot added the Actions Analysis of GitHub Actions label Sep 24, 2026

d10c and others added 4 commits September 24, 2026 14:42

d10c force-pushed the d10c/learn-inline-expectations-lib branch from ae24f4e to 6b67283 Compare September 24, 2026 19:04

d10c requested a review from a team as a code owner September 24, 2026 19:04

github-actions Bot added the Kotlin label Sep 24, 2026

Navigation Menu

Support `--learn` for inline expectation tests - #22228

Support `--learn` for inline expectation tests#22228
d10c wants to merge 6 commits into
github:mainfrom
d10c:d10c/learn-inline-expectations-lib

d10c commented Jul 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

owen-mc left a comment •

edited

Loading

Uh oh!

Uh oh!

Kamil-0425 Sep 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

d10c commented Sep 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

		bindingset[relativePath]
		string getStartCommentMarker(string relativePath);

Conversation

d10c commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it does

API

Follow-ups not implemented in this PR

Uh oh!

Uh oh!

Uh oh!

Uh oh!

owen-mc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kamil-0425 Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

d10c commented Sep 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

d10c commented Jul 23, 2026 •

edited

Loading

owen-mc left a comment •

edited

Loading