Skip to content

fix(skills/copilot-review-loop): report the effort level a review states - #96

Merged
sourcehawk merged 1 commit into
mainfrom
fix/copilot-review-effort-line
Oct 12, 2026
Merged

sourcehawk merged 1 commit into
mainfrom
fix/copilot-review-effort-line

Conversation

@sourcehawk

Copy link
Copy Markdown
Owner

Description

Fixes #95

The skill said that nothing in a Copilot review records its effort level, and told the agent not to report that a review was balanced. That statement is false: a review in the current format states its level in a labelled line of its body. This PR tells the agent to report the level from that line, and to report no level when the line is absent.

Changes

  • ## Review effort level is a precondition: the agent reads the line **Review effort:** <level> in the review body, reports that level, and names the line as its source.
  • Same section: if the line is absent, the agent says that the review does not state its level. A missing line does not mean lite, and a mention of effort in the prose of the body is not the line.
  • Step 4 names the review body as the place where a review states its level.

Challenges

Not every review carries the line. Copilot reviews on six pull requests of this repository show two formats. Each review in the current format (#78, #90, #91, #94) carries the line, with or without an emoji before it, and not always as the last line. Reviews in the older format (#28, #32) carry no such line, and one body on #32 mentions the effort level in its prose summary. The rule for the absent line and the rule for a prose mention come from those reviews.

Testing

Each run gave a new agent the skill file and a review body, and asked for the triage report to the user.

On the unedited skill, six runs reported the level from the body: three on a lite review and three on a balanced review. So the failure in the issue did not occur in these runs. But two of the three runs on a balanced review told the user that the skill was wrong, and did not obey it.

With the new text, five runs passed. Two runs on a balanced review reported Balanced and quoted the line, with no argument against the skill. Two runs on a body with no line, and with a summary that reads as thorough, said that the review does not state its level and guessed none. One run on a lite review reported Lite with the line as its source.

No script changed, so the script tests did not run. The JSON files in .claude-plugin/ parse.

🤖 Generated with Claude Code

…w body

The section `Review effort level is a precondition` said that nothing in the review payload records the effort level, and told the agent not to report that a review was balanced. That statement is false. A Copilot review in the current format states its level in a labelled line of its body, `**Review effort:** <level>`. An agent that obeyed the skill could not tell the user a fact that the review states.

The section now tells the agent to report the level from that line and to name the line as the source. If the line is absent, the agent says that the review does not state its level, and reports none. The rule against a guess from how thorough the review looks stays. Step 4 now names the body as the place where a review states its level.

Copilot reviews on six pull requests of this repository show when the line is present. Each review in the current format carries the line, with or without an emoji before it, and not always as the last line. Reviews in the older format carry no such line, and one of them mentions the effort level in its prose summary.

In a baseline on the unedited skill, six runs reported the level from the body. Two of the three runs on a balanced review said that the skill was wrong and did not obey it. With the new text, five runs passed: two on a balanced review, two on a body with no line, and one on a lite review. No run argued against the skill, and no run guessed a level for the body with no line.

Fixes #95

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 12, 2026 21:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused guidance matches the documented review formats and handles missing labels safely.

0 open findings

What changed in this PR

Fixes #95 by teaching the Copilot review loop to report only explicitly stated review effort levels.

Changes:

  • Parses the labelled effort line from review bodies.
  • Avoids inferring a level when the line is absent.
  • Clarifies that Step 4 reads effort metadata from the body.
File Description
skills/​copilot-review-loop/​SKILL.md Documents reliable effort-level reporting during review triage.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@sourcehawk
sourcehawk merged commit 94002b6 into main Oct 12, 2026
6 checks passed
@sourcehawk
sourcehawk deleted the fix/copilot-review-effort-line branch October 12, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

copilot-review-loop: the skill says the review payload does not record the effort level, but the review body states it

2 participants