Repository navigation
fix(lh-132058): align harness grading and skill guidance with runs - #41
Merged
Merged
Conversation
Bedrock reliability runs showed the missing-profile scenario failing on correct answers that said the configuration command "isn't exposed" instead of one of three exact phrasings. The fixture now accepts those equivalent wordings, including curly apostrophes, while still failing an answer that never says the command is missing. The CLI skill told agents that API tokens come from developer.cisco.com, which only hosts the documentation, so nearly every missing-profile answer sent users to the wrong place. It now states that tokens are generated in the SCC Firewall Manager UI by an API-only user. The skill also no longer names the profile file location, which agents repeated despite the rule against stating a path not seen in tool output. The Bedrock lane inlines the skill text without its location, so the setup skill could not resolve its helper scripts and agents spent many commands searching for them. The system prompt now states the path of the staged skill file the agent can reach inside its sandbox.
huides00
requested review from
Scoombe,
afercal and
siddhuwarrier
as code owners
October 2, 2026 12:22
Two urllib3 advisories, PYSEC-2026-4175 and PYSEC-2026-4177, were published after the last green main build and now fail the runtime dependency audit on every branch. Both are fixed in urllib3 2.8.0, but the SCC Firewall Manager SDK requires urllib3 below 2.1.0 in both the locked release and the newest one on PyPI, so the patched version cannot be resolved. The two IDs join the existing exception list kept for the same SDK pin, and the guard test that counts the exceptions is raised to match so further additions still need an explicit change. The exceptions should be removed once the SDK permits a patched urllib3 release.
Scoombe
approved these changes
Oct 2, 2026
The fix commit on this branch makes the next release a patch, and release preparation requires the target version to be recorded in source. Add the 0.43.1 entry covering the harness grading, skill guidance, Bedrock skill path, and urllib3 audit exception changes.
Consistency CheckNo consistency issues found. Checker output |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://cisco-sbg.atlassian.net/browse/LH-132058
Description
Recent Bedrock reliability runs passed 449 of 450 samples, but the review showed that the one
recurring failure and several weaker signals came from the test harness and the CLI skill rather
than from agent behavior. The missing-profile scenario rejected correct answers that described the
configuration command as not exposed, because it only accepted three exact phrasings. It now
accepts those equivalent wordings and still fails an answer that never says the command is missing.
The CLI skill now tells users that API tokens are generated in the SCC Firewall Manager UI by an
API-only user, replacing guidance that sent them to the developer documentation site, which does
not issue tokens. The skill also stops naming the profile file location, which agents had been
repeating to users despite being told not to state paths they had not observed.
The Bedrock lane now tells the agent where its loaded skill file lives, so the setup skill can find
its helper scripts without the repeated searching seen in the runs. A follow-up Bedrock run is
still needed to confirm the improvement in practice.