test(evals): pin the Nikita source reference workflow

[harlan-zw]

🔗 Linked issue

Follow-up to #232.

❓ Type of change

• 📖 Documentation
• 🐞 Bug fix
• 👌 Enhancement
• ✨ New feature
• 🧹 Chore
• ⚠️ Breaking change

📚 Description

Upstream edits and truncated tool output can fail the Nikita reference eval despite a correct lookup. Pin the Skill and reference graders to one immutable commit, and grade commands directly instead of truncated output. Regression cases reject mismatched commits and accept long tool output.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

[harlan-zw]

Agent verification: a live GLM 5.3 Flash/opencode run scored 1.00 with the Skill and 0.58 without it. The successful commands loaded the pinned root Skill and read onboarding.md at the same f8b8a1828cd53e4aedaa39c3e2abb01e71660bc0 commit. Both returned wroteSkillFiles: false. The run also recorded an initial direct-delivery failure in released CLI 3.6.0 before the hosted read succeeded.

[harlan-github-agent]

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-08 13:21 UTC.

GitHub merged this pull request.

• Logged (20/100): The new happy-path test cannot detect a grader that silently stops being scored, so a future regex typo in a grader file would weaken the eval while tests pass. View code
  • Ask an agent to verify and fix this finding

f3feb490-61e6-4fce-bf29-287acb17aaed

Selected findings run after merge and open separate pull requests.

[harlan-github-agent]

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: READY.

Read the Review results