Skip to content

ci: label unlabelled issues with a model and check PR descriptions - #4554

Open
justinmclean wants to merge 3 commits into
apache:masterfrom
justinmclean:ci/model-issue-labels
Open

justinmclean wants to merge 3 commits into
apache:masterfrom
justinmclean:ci/model-issue-labels

Conversation

@justinmclean

@justinmclean justinmclean commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR address?

N/A

Rationale

Issues opened blank, or with "Other / not sure" as the area, get no area label from the issue form, so a maintainer has to label them by hand. Contributor PRs often arrive without a linked issue or test notes, and reviewers have to ask for them.

What changed?

The issue form labelling moves into scripts/ci/issue-labeler.mjs, which the issue workflow now runs. When the form gives no area, the script asks a self-hosted model on ASF infrastructure (LLMAO, default gemma4-26b) for up to three area labels, plus bug or enhancement when the issue has no type label. Only exact matches to the existing allowlist are applied, so a prompt injection in the issue body can at worst pick a wrong label from that list. Without LLMAO_API_KEY and LLMAO_BASE_URL the model part does nothing, and model errors are warnings, not failures.

A new PR Description check, scripts/ci/pr-description-check.mjs, fails a contributor PR whose description has no linked issue (or explicit N/A), no description, or no test notes, and names each gap in the job summary. It is plain pattern matching, with no model, and the workflow skips bots, drafts and authors with write access.

Both workflows check out only their script, from the default branch or the PR's base commit, without persisted credentials. Maintainers can run either script by hand on any issue or PR:

scripts/ci/issue-labeler.mjs --issue 1234 --dry-run
scripts/ci/pr-description-check.mjs --pr 1234

--dry-run prints the labels the labeler would add and changes nothing on GitHub. Each script's header lists the configuration it reads.

Local Execution

  • Passed: the labeler dry run gives the form labels on issues 4573 and 4544. On blank issue 4531 it skips the model without a key, and against a mock endpoint it handles a normal reply, a prompt injection, a think block, a prose reply and HTTP 503. The PR check passes every contributor PR in the last 12 and fails only member and bot PRs, which the workflow skips.
  • Pre-commit hooks ran

Not yet run against the live LLMAO endpoint, which needs the secret and variable set first.

Setup (PMC member)

  1. Create an LLMAO personal key with Project set to Apache Iggy, then add it as a secret. The command prompts for the value, so the key never appears in shell history:

    gh secret set LLMAO_API_KEY -R apache/iggy
  2. Set the base URL from the LLMAO page, ending in /v1:

    gh variable set LLMAO_BASE_URL -R apache/iggy --body "https://<llmao-host>/v1"

After merge I'll check the results on the next few issues and contributor PRs.

AI Usage

Claude Code wrote the scripts, the workflow steps and the test harnesses. I reviewed the changes and the test results.

Issues the form leaves without an area get labels from a self-hosted model on ASF infrastructure, filtered to the allowlist. Contributor PRs fail a check when the description has no linked issue, description or test notes.
@hubcio

hubcio commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

@justinmclean please use just title ci: ... :)

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer
  • /pin - exempt the PR from the stale bot, /unpin to undo

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Oct 15, 2026
@hubcio

hubcio commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

@justinmclean could you extract the inline logic into standalone scripts, with the workflows calling the same code? Maintainers should be able to run them manually for a specific issue or PR. Please add --dry-run to the labeler so it prints the labels it would add without changing anything on GitHub, and include usage examples and the required configuration. This would make the logic easier to test, debug and maintain.

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above comment

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Oct 15, 2026
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.02%. Comparing base (4c53f2a) to head (dd65b9b).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4554      +/-   ##
============================================
- Coverage     89.48%   89.02%   -0.47%     
  Complexity     1785     1785              
============================================
  Files          1330     1330              
  Lines        284416   284416              
  Branches     242865   242866       +1     
============================================
- Hits         254502   253192    -1310     
- Misses        24273    25606    +1333     
+ Partials       5641     5618      -23     
Components Coverage Δ
Rust Core 90.71% <ø> (-0.02%) ⬇️
Java SDK 70.78% <ø> (ø)
C# SDK 66.42% <ø> (-12.92%) ⬇️
Python SDK 90.43% <ø> (ø)
PHP SDK 86.02% <ø> (ø)
Node SDK 95.12% <ø> (-0.01%) ⬇️
Go SDK 70.88% <ø> (-0.03%) ⬇️
C++ SDK 84.44% <ø> (ø)
see 67 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The workflows now run scripts/ci/issue-labeler.mjs and scripts/ci/pr-description-check.mjs, so maintainers can run them by hand on any issue or PR. The labeler has --dry-run.
@justinmclean justinmclean changed the title ci(ci): label unlabelled issues with a model and check PR descriptions ci: label unlabelled issues with a model and check PR descriptions Oct 16, 2026
@justinmclean

Copy link
Copy Markdown
Member Author

Done. The labeler and the PR check are now scripts/ci/issue-labeler.mjs and scripts/ci/pr-description-check.mjs, and the workflows run them. The labeler has --dry-run, and usage and configuration are in each script header and the PR description.

@justinmclean
justinmclean requested a review from hubcio October 18, 2026 21:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author PR is waiting on author response

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants