test(evals): cover pinned root Skill reference reads

[harlan-zw]

🔗 Linked issue

❓ Type of change

• 📖 Documentation
• 🐞 Bug fix
• 👌 Enhancement
• ✨ New feature
• 🧹 Chore
• ⚠️ Breaking change

📚 Description

The existing CLI evals do not require an Agent to read a root Skill's supporting file.
Add a Nikita source lookup that checks the reference answer, the loaded commit, and absence of installation.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

[harlan-zw]

Agent verification of head 6731684.

The real production case ran once per arm on opencode 1.18.32 and GLM 5.3 Flash. With the skilld Skill: 1.00. Without: 0.50. The with arm read onboarding.md at the loaded commit and installed nothing. No grader was skipped.

The same-commit grader accepted the recorded trace and rejected a changed reference commit.

Released skilld 3.6.0 installed the Nikita Artifact into an isolated opencode copy target. Lockfile restore recovered every source and target byte exactly. An older hosted Artifact reported pinned. A scratch lockfile fixture changed only its source selector to track the Repository, simulating a pre-push install. Update then advanced from 7e7f68d to f8b8a18 and refreshed the reference in both copies. Source verification passed after install, restore, and update.

The production refresh audit found a progressing hourly batch with no failures. Three refreshed nested references returned matching commit provenance and sizes. An unpinned reference returned the expected temporary 503. The audit did not claim all legacy snapshots had refreshed.

[harlan-github-agent]

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-08 11:50 UTC.

GitHub merged this pull request.

• Logged (30/100): A scored grader (weight 2 of 14) can false-fail correct runs because its evidence only survives an incidental 2000-character output truncation in the trace, making a healthy case look like a Skill regression. View code
  • Ask an agent to verify and fix this finding
• Logged (25/100): The case's reference answer is pinned to a live third-party file that can change at any time, so the eval can rot silently and report a skilld Skill regression that is really upstream content drift. View code
  • Ask an agent to verify and fix this finding
• Logged (10/100): The README score table was not updated with the new case's recorded results, leaving the eval documentation inconsistent with the run the pull request reports. View code
  • Ask an agent to verify and fix this finding

98680614-6f85-4137-824f-61bb51447e26

Selected findings run after merge and open separate pull requests.

[harlan-github-agent]

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: READY.

Read the Review results