diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index fdb4a931a..c749413a5 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -100,9 +100,30 @@ but the whole command is searched: the first 2000 characters, then the rest in w start and end on logical-line boundaries. Agents write long heredoc scripts that end in the command a task checks for (`cat > x <` and `run --resume` re-grade with the current rules, so a re-graded older run can score differently from its recorded result. The `framework_version` of a run shows which release graded it. + +**A match is text, not proof.** The pattern is searched in the command text, so `echo 'cmd'`, `# cmd` and `true || cmd` all match a `cmd` pattern, although `cmd` does not run. When the effect of the command is what you grade, also check that effect (a file, a state, an output). + +**Shell normalization.** For a shell command, both `command_pattern` and `exclude_pattern` are matched against the raw command text **and** its shell-normalized form — the `bash`/`sh`/`zsh -lc "..."` wrapper stripped and shell quoting resolved with `shlex` — and a hit on *either* form counts. So a pattern like `curated_channels` matches whether the agent wrote the argument bare, `'single'`-quoted, `"double"`-quoted, or `\"escaped\"`; you do **not** hand-encode shell quoting. Because the same haystacks also feed `exclude_pattern` and the `max_count` gate, normalization is **not** purely additive: a quote-obfuscated call can now be caught by an exclusion or a `max_count: 0` gate that the raw text alone would have missed — and, conversely, an unedited `exclude_pattern` may now exclude a call it previously let through. Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline. + +**Long commands.** A shell command is searched in full: the first 2,000 characters, then the rest in windows of at most 2,000 characters that start and end on line boundaries. So a command that follows a long heredoc in the same Bash call still counts. JSON parameters are searched only in their first 2,000 characters, so a long Write or Edit body is not read as a command. The windows are not purely additive either: an `exclude_pattern` or a `max_count: 0` gate can now catch a call late in a long script that it did not see before, so suites should re-baseline. A match that crosses a window boundary is not guaranteed, and `^` and `$` can also match at a window edge. **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry. +With a pattern, `tool_name: null` searches the command of a shell call, not the parameters of every tool (see **Which text is searched**). + ### `cli_called` Checks whether a CLI invocation matching a **structured** pattern was recorded, by reading a JSON Lines invocation log the sandbox produced. **Binary scoring.** diff --git a/docs/agents/OPENCODE.md b/docs/agents/OPENCODE.md index 09327d30f..3f8896902 100644 --- a/docs/agents/OPENCODE.md +++ b/docs/agents/OPENCODE.md @@ -228,7 +228,7 @@ call's `command` parameter) score identically whether the run used Claude, Codex or OpenCode. Tool **argument keys** are normalized the same way, because `command_executed` -serializes `parameters` to JSON for every tool but `Bash` — so a criterion like +serializes the `parameters` of a named tool with no `command` to JSON — so a criterion like `{type: command_executed, tool_name: Read, command_pattern: 'file_path.*app\.py'}` would otherwise match on Claude and score 0 on OpenCode for identical behaviour: diff --git a/plugins/coder-eval/reference/criteria.md b/plugins/coder-eval/reference/criteria.md index 526e1fbdd..037e3cbd3 100644 --- a/plugins/coder-eval/reference/criteria.md +++ b/plugins/coder-eval/reference/criteria.md @@ -105,12 +105,12 @@ Optional: | Field | What it is | | --- | --- | -| `tool_name` | Tool name filter (e.g., 'Bash'). None = any tool. | -| `command_pattern` | Regex to match command parameters. None = any command. For a Bash command the pattern is matched against the raw command text OR its shell-normalized form — `shlex`-resolved quoting with a `bash`/`zsh -lc` wrapper stripped — whichever hits, so you need not hand-encode shell quoting/escaping (`'single'`, `"double"`, bare). | +| `tool_name` | Tool name filter (e.g., 'Bash'). None = any tool. A pattern searches only the `command` param of a shell call: a Bash call, or any call with a `command` param, such as a shell tool under another name. A named tool without a `command` param is searched on its JSON params. With None, a call without a `command` param is not searched: a mention in a Write, Edit or TodoWrite body does not count, and a `max_count: 0` guard (also the legacy `command_not_executed` without a tool_name) does not see it. Name the tool to search its params. | +| `command_pattern` | Regex to match command parameters. None = any command. For a shell command (see tool_name) the pattern is matched against the raw command text OR its shell-normalized form — `shlex`-resolved quoting with a `bash`/`zsh -lc` wrapper stripped — whichever hits, so you need not hand-encode shell quoting/escaping (`'single'`, `"double"`, bare). | | `min_count` | Minimum matching commands required. ``0`` allows the criterion to pass when no commands match (combine with ``max_count: 0`` to express ``must NOT match``). | | `max_count` | Optional maximum matching commands allowed (inclusive). ``None`` means no upper bound — the current default. When set, the criterion passes iff ``min_count <= match_count <= max_count``. | -| `require_success` | If True, only count successful commands. | -| `exclude_pattern` | Regex that must NOT match. Commands matching both command_pattern and exclude_pattern are skipped. Like command_pattern, this is matched against the raw Bash command OR its shell-normalized form, so it also excludes quote-obfuscated variants of the same call. | +| `require_success` | If True, only count successful commands. This is the reported success of the whole tool call, so `cmd \| tail` and `cmd; true` can pass when cmd fails. | +| `exclude_pattern` | Regex that must NOT match. Commands matching both command_pattern and exclude_pattern are skipped. Like command_pattern, this is matched against the raw shell command OR its shell-normalized form, so it also excludes quote-obfuscated variants of the same call. | ### `commands_efficiency` diff --git a/plugins/coder-eval/reference/task-rubric.md b/plugins/coder-eval/reference/task-rubric.md index ca5961c9b..d21470ee1 100644 --- a/plugins/coder-eval/reference/task-rubric.md +++ b/plugins/coder-eval/reference/task-rubric.md @@ -52,7 +52,7 @@ test, the criteria are wrong — not the agent. Then walk the mechanical checks: | # | Check | Fix | |---|---|---| -| 1 | Does a `command_executed` criterion credit an invocation that **failed**? | Set `require_success: true` whenever the command's success is what is being graded. The default is `false`, which counts a crashed invocation as evidence the work was done. | +| 1 | Does a `command_executed` criterion credit an invocation that **failed**? | Set `require_success: true` whenever the command's success is what is being graded. The default is `false`, which counts a crashed invocation as evidence the work was done. It reads the reported success of the whole tool call. So `cmd \| tail`, `cmd; true` and `echo cmd` can also pass. When the effect is what you grade, also check the effect. | | 2 | Does the pattern also match a help probe, or a longer word that contains it? | Add `exclude_pattern` and word boundaries. Cover **both** `--help` and `-h`; an agent that runs `tool --help` and gives up otherwise satisfies a bare `tool` pattern. | | 3 | Would an agent that does **nothing** pass? | List the criteria satisfiable by inaction — `min_count: 0`, a `max_count: 0` negative, a `file_exists` on a file the setup already created. A task passes only when **every** scoring criterion (`weight` above 0) meets its own `pass_threshold`, so if inaction satisfies all of them the task is a no-op detector. If it satisfies only some, total their `weight` against the task's: that share is how much of the score is free. | | 4 | Does a regex alternation launder the assertion? | An alternation branch that makes the payload opaque — matching an indirection flag such as `--from-file` instead of the content that flag points at — proves nothing. Keep the shape criterion on the shape-visible form and add a companion check on the payload itself. | diff --git a/plugins/coder-eval/skills/task/SKILL.md b/plugins/coder-eval/skills/task/SKILL.md index 440de214f..b4800aad6 100644 --- a/plugins/coder-eval/skills/task/SKILL.md +++ b/plugins/coder-eval/skills/task/SKILL.md @@ -115,7 +115,8 @@ Rules that matter: produced. Set `require_success: true` whenever the command's success is what you are grading; the permissive default (`false`) counts a crashed invocation as evidence the work was done, and survives only for a genuine exception — a probe whose failure is an - acceptable outcome. + acceptable outcome. It reads the reported success of the whole tool call (`cmd | tail` can + pass when `cmd` fails), so also check the effect of the command. - When the prompt genuinely must name a literal — a flag like `--json`, an output filename — a criterion matching that literal is a **smoke check**, not evidence: it only proves the agent typed back what it was told. Keep it if you like, at a low diff --git a/src/coder_eval/criteria/command_executed.py b/src/coder_eval/criteria/command_executed.py index d7d51d51e..1637274ab 100644 --- a/src/coder_eval/criteria/command_executed.py +++ b/src/coder_eval/criteria/command_executed.py @@ -133,20 +133,17 @@ def _search_windows(cmd_text: str) -> list[str]: def _match_haystacks(cmd_text: str, *, is_shell: bool) -> list[str]: """Strings a pattern may match against for one command. - When ``is_shell``, every search window of ``cmd_text`` (see - :func:`_search_windows`) plus the quote-resolved, wrapper-stripped form of each - window (see :func:`_normalize_shell`). Otherwise only the leading - ``_MAX_PATTERN_SEARCH_LEN`` characters: a non-shell tool's params are JSON, and a - later window of a Write/Edit body is file content, not something the agent ran. - Normalizing per window keeps every haystack within the ReDoS bound and caps - ``shlex`` input for free. Matching is "either" -- a pattern hits the command if it - matches ANY haystack. - - ``is_shell`` is decided once by the caller (a Bash tool whose ``command`` is - a non-empty ``str``) and passed in, rather than re-derived here from - ``tool_name`` alone: a Bash record with a missing/empty ``command`` serializes - its params to JSON, where shell tokenization is meaningless, and must NOT be - normalized (else stripped JSON quotes could newly satisfy an exclusion). + When ``is_shell`` (``cmd_text`` is a shell command string), every search window of + ``cmd_text`` (see :func:`_search_windows`) plus the quote-resolved, + wrapper-stripped form of each window (see :func:`_normalize_shell`). Otherwise + ``cmd_text`` is JSON and only its leading ``_MAX_PATTERN_SEARCH_LEN`` characters + are searched: a later window of a Write/Edit body is file content, not something + the agent ran. JSON is not normalized, because stripped JSON quotes could newly + satisfy an exclusion. Normalizing per window keeps every haystack within the ReDoS + bound and caps ``shlex`` input for free. Matching is "either" -- a pattern hits + the command if it matches ANY haystack. + + The caller decides ``is_shell`` in :func:`_search_text`. """ if not is_shell: return [cmd_text[:_MAX_PATTERN_SEARCH_LEN]] @@ -159,6 +156,58 @@ def _match_haystacks(cmd_text: str, *, is_shell: bool) -> list[str]: return haystacks +def _is_shell_call(cmd: "CommandTelemetry") -> bool: + """True for a ``Bash`` call or any call with a ``command`` param (a shell tool under another name).""" + return cmd.tool_name == "Bash" or "command" in cmd.parameters + + +def _search_text(cmd: "CommandTelemetry", *, tool_named: bool) -> tuple[str, bool] | None: + """The text a pattern searches in one call, and whether it is a shell command string. + + The shape of the call decides, not the criterion. A shell call (:func:`_is_shell_call`) + is searched on its command: + + - a non-empty ``str`` command is searched with shell normalization and windows; + - another non-empty command (an argv list) is searched as its JSON; + - a missing or empty command gives no text. + + Another call is searched on its JSON params only when the criterion names its tool + (``tool_named``); otherwise it gives no text, because a Write body or a TodoWrite + item can name a command that never ran. + + ``None`` means no text: a ``command_pattern`` does not match the call and an + ``exclude_pattern`` does not exclude it. + + Rationale: .claude/notes/contracts.md § Which text a pattern searches + """ + raw_command = cmd.parameters.get("command") + if _is_shell_call(cmd): + if not raw_command: + return None + if isinstance(raw_command, str): + return raw_command, True + # An argv LIST is searched as JSON; it never reaches shlex. + return json.dumps(raw_command), False + if tool_named: + return json.dumps(cmd.parameters), False + return None + + +def _unsearched_note( + criterion: CommandExecutedCriterion, unsearched: list["CommandTelemetry"], match_count: int +) -> str: + """A details line that says why calls that only name the command did not count, or ``""``. + + The hint to set ``tool_name`` is given only for a call that naming its tool makes searchable. + """ + if not unsearched or match_count >= criterion.min_count: + return "" + hint = ( + " Set tool_name to search the params of another tool." if any(not _is_shell_call(c) for c in unsearched) else "" + ) + return f"\n{len(unsearched)} call(s) without a command were not searched.{hint}" + + @register_criterion class CommandExecutedChecker(BaseCriterion[CommandExecutedCriterion]): """Checker for CommandExecutedCriterion. @@ -175,56 +224,42 @@ def _matching_commands( all_commands: list["CommandTelemetry"], pattern: re.Pattern[str] | None, exclude_re: re.Pattern[str] | None, - ) -> list[str]: + ) -> tuple[list[str], list["CommandTelemetry"]]: """Filter + label the commands matching this criterion. Shared by ``_check_impl`` and ``live_verdict`` so a command can never count for one and not the other (the live trigger and the authoritative score always agree on WHICH commands match). Callers compile the patterns and own error handling. + + Returns the labels of the matching commands, and the calls a + ``command_pattern`` did not search because they gave no text (see + :func:`_search_text`). """ matching: list[str] = [] + unsearched: list[CommandTelemetry] = [] for cmd in all_commands: - # Filter by tool name if criterion.tool_name is not None and cmd.tool_name != criterion.tool_name: continue - - # Filter by success status if criterion.require_success and cmd.result_status != "success": continue - # A ``command`` value is not guaranteed to be a ``str`` -- Codex rollout - # recovery can carry it as an argv LIST. Narrow with ``isinstance`` so a - # non-str never reaches ``shlex.split`` and zeroes the whole criterion. - raw_command = cmd.parameters.get("command") - if cmd.tool_name == "Bash" and isinstance(raw_command, str) and raw_command: - cmd_text = raw_command - is_shell = True - else: - cmd_text = json.dumps(cmd.parameters) - is_shell = False - - # Raw command AND its quote-resolved form. ``is_shell`` is decided once - # above; the helper must not re-derive it from tool_name alone. - # Rationale: .claude/notes/contracts.md § Normalizing a shell command before matching it - haystacks = _match_haystacks(cmd_text, is_shell=is_shell) - - # Filter by command pattern + searched = _search_text(cmd, tool_named=criterion.tool_name is not None) + # Raw text AND, for a shell command, its quote-resolved form, so neither + # the pattern nor the exclusion can be dodged by quoting. + haystacks = _match_haystacks(searched[0], is_shell=searched[1]) if searched else [] if pattern is not None and not any(pattern.search(h) for h in haystacks): + if searched is None: + unsearched.append(cmd) continue - - # Apply exclusion pattern (skip commands matching the exclusion). - # Same both-haystacks logic so exclusion can't be dodged by quoting. if exclude_re is not None and any(exclude_re.search(h) for h in haystacks): continue - # Build a display label for the matched command (same narrowing as above) - if cmd.tool_name == "Bash" and isinstance(raw_command, str) and raw_command: - label = raw_command + if searched is not None and searched[1]: + matching.append(searched[0]) else: - label = f"{cmd.tool_name}({json.dumps(cmd.parameters)[:80]})" - matching.append(label) - return matching + matching.append(f"{cmd.tool_name}({json.dumps(cmd.parameters)[:80]})") + return matching, unsearched def live_verdict( self, @@ -256,7 +291,7 @@ def live_verdict( except re.error: return "undecided" - match_count = len(self._matching_commands(criterion, all_commands, pattern, exclude_re)) + match_count = len(self._matching_commands(criterion, all_commands, pattern, exclude_re)[0]) if criterion.max_count is not None and match_count > criterion.max_count: return "fail" if criterion.max_count is None and criterion.min_count > 0 and match_count >= criterion.min_count: @@ -328,7 +363,7 @@ def _check_impl( # Filter and count matching commands (shared with live_verdict so the # live trigger and the authoritative score never disagree on matches). - matching_commands = self._matching_commands(criterion, all_commands, pattern, exclude_re) + matching_commands, unsearched = self._matching_commands(criterion, all_commands, pattern, exclude_re) match_count = len(matching_commands) @@ -357,6 +392,7 @@ def _check_impl( else: range_text = f"{match_count} matches (allowed range {criterion.min_count}..{criterion.max_count})" details = f"Matched {range_text} commands (filters: {filter_text})" + details += _unsearched_note(criterion, unsearched, match_count) if matching_commands: # Show up to 3 example matches examples = matching_commands[:3] diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index 1e661aa73..2717d53a9 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -884,12 +884,23 @@ class CommandExecutedCriterion(LiveSuccessCriterion): requires_agent: ClassVar[bool] = True type: Literal["command_executed"] = "command_executed" - tool_name: str | None = Field(default=None, description="Tool name filter (e.g., 'Bash'). None = any tool.") + tool_name: str | None = Field( + default=None, + description=( + "Tool name filter (e.g., 'Bash'). None = any tool. A pattern searches only the `command` " + "param of a shell call: a Bash call, or any call with a `command` param, such as a shell " + "tool under another name. A named tool without a `command` param is searched on its JSON " + "params. With None, a call without a `command` param is not searched: a mention in a " + "Write, Edit or TodoWrite body does not count, and a `max_count: 0` guard (also the " + "legacy `command_not_executed` without a tool_name) does not see it. Name the tool to " + "search its params." + ), + ) command_pattern: str | None = Field( default=None, description=( - "Regex to match command parameters. None = any command. For a Bash " - "command the pattern is matched against the raw command text OR its " + "Regex to match command parameters. None = any command. For a shell " + "command (see tool_name) the pattern is matched against the raw command text OR its " "shell-normalized form — `shlex`-resolved quoting with a " "`bash`/`zsh -lc` wrapper stripped — whichever hits, so you need not " "hand-encode shell quoting/escaping (`'single'`, `\"double\"`, bare)." @@ -913,12 +924,18 @@ class CommandExecutedCriterion(LiveSuccessCriterion): "passes iff ``min_count <= match_count <= max_count``." ), ) - require_success: bool = Field(default=False, description="If True, only count successful commands.") + require_success: bool = Field( + default=False, + description=( + "If True, only count successful commands. This is the reported success of the whole tool " + "call, so `cmd | tail` and `cmd; true` can pass when cmd fails." + ), + ) exclude_pattern: str | None = Field( default=None, description=( "Regex that must NOT match. Commands matching both command_pattern and exclude_pattern are skipped. " - "Like command_pattern, this is matched against the raw Bash command OR its shell-normalized form, so " + "Like command_pattern, this is matched against the raw shell command OR its shell-normalized form, so " "it also excludes quote-obfuscated variants of the same call." ), ) diff --git a/tests/lint/live_verdict_contract.py b/tests/lint/live_verdict_contract.py index c56e4340e..6aa4b6550 100644 --- a/tests/lint/live_verdict_contract.py +++ b/tests/lint/live_verdict_contract.py @@ -74,11 +74,12 @@ def _cmd_crit( min_count: int = 1, max_count: int | None = None, require_success: bool = False, + tool_name: str | None = "Bash", ) -> CommandExecutedCriterion: return CommandExecutedCriterion( type="command_executed", description=f"command_executed[{pattern}]", - tool_name="Bash", + tool_name=tool_name, command_pattern=pattern, min_count=min_count, max_count=max_count, @@ -274,6 +275,26 @@ def _skill(name: str, *, sequence_number: int) -> CommandTelemetry: ), reaches="pass", ), + ContractCase( + label="null tool_name: a Write body naming the command never counts; the shell call does", + criterion=_cmd_crit(min_count=1, max_count=None, tool_name=None), + commands=( + cmd("Write", {"file_path": "fetch.sh", "content": "curl https://example.com"}, sequence_number=0), + cmd("TodoWrite", {"todos": [{"content": "curl https://example.com"}]}, sequence_number=1), + cmd("shell", {"command": "curl https://example.com"}, sequence_number=2), + ), + reaches="pass", + ), + ContractCase( + label="null tool_name must-NOT-run guard: a Write body does not trip it; running it fails", + criterion=_cmd_crit(pattern="rm -rf", min_count=0, max_count=0, tool_name=None), + commands=( + cmd("Write", {"file_path": "clean.sh", "content": "rm -rf /tmp/scratch"}, sequence_number=0), + _bash("echo done", sequence_number=1), + cmd("shell", {"command": "rm -rf /tmp/scratch"}, sequence_number=2), + ), + reaches="fail", + ), ), } diff --git a/tests/test_command_executed.py b/tests/test_command_executed.py index edc1ce2ff..ba9108e2b 100644 --- a/tests/test_command_executed.py +++ b/tests/test_command_executed.py @@ -2,15 +2,18 @@ from datetime import datetime +import pytest + from coder_eval.criteria.command_executed import ( _MAX_PATTERN_SEARCH_LEN, _WINDOW_OVERLAP, + CommandExecutedChecker, _match_haystacks, _normalize_shell, _search_windows, ) from coder_eval.evaluation.checker import SuccessChecker -from coder_eval.models import CommandExecutedCriterion +from coder_eval.models import CommandExecutedCriterion, CriterionResult from coder_eval.models.results import TurnRecord from coder_eval.models.telemetry import CommandTelemetry @@ -1051,15 +1054,13 @@ def test_argv_joined_not_collapsed_into_false_anchored_match(self): assert result.score == 1.0 def test_bash_record_without_command_is_not_shell_normalized(self): - """A Bash record with no ``command`` key must not have its JSON params tokenized. - - ``_match_haystacks`` is now told whether cmd_text is a shell command - (``is_shell``, decided once at the extraction site) instead of - re-deriving it from ``tool_name == "Bash"`` alone. A Bash record whose - ``command`` is missing (reachable on the Codex path) falls back to the - JSON blob and must NOT be normalized — otherwise shlex strips the JSON - quotes, adding a haystack that can newly satisfy an ``exclude_pattern`` - and wrongly drop the command below ``min_count``. + """A Bash record with no ``command`` key must not be dropped by an exclusion. + + A Bash record whose ``command`` is missing, which some harnesses record, + has no text to search (``_search_text``), so no exclusion can drop it. + An earlier version searched its JSON params, and shlex-stripped JSON + quotes could then newly satisfy an ``exclude_pattern`` and wrongly drop + the command below ``min_count``. """ sandbox = MockSandbox() # Bash tool, but params carry a `description`, not a `command`. @@ -1069,9 +1070,8 @@ def test_bash_record_without_command_is_not_shell_normalized(self): criterion = CommandExecutedCriterion( description="counts the bash record", tool_name="Bash", - # Matches the shlex-stripped JSON (`{description: run ...}`) but NOT the - # raw JSON (`{"description": "run ...}`); with the is_shell fix the raw - # is the only haystack, so the command is not excluded. + # Matches the shlex-stripped JSON (`{description: run ...}`), which is + # never a haystack: the record is not searched, so it is not excluded. exclude_pattern=r"description:\s+run", min_count=1, ) @@ -1182,8 +1182,198 @@ def test_non_shell_tool_body_past_the_bound_is_not_searched(self): turn_records = [_make_turn([write])] criterion = CommandExecutedCriterion( description="must NOT use retired `uip or users list`", + # Named: with tool_name null, a pattern never searches a Write at all. + tool_name="Write", command_pattern=r"uip\s+or\s+users\s+list", min_count=0, max_count=0, ) assert SuccessChecker(MockSandbox()).check(criterion, turn_records=turn_records).score == 1.0 + + +# Tool calls that name `uip solution deploy` without running it. +_MENTIONS_WITHOUT_RUNNING = [ + pytest.param("Write", {"file_path": "NOTES.md", "content": "Next: uip solution deploy"}, id="write"), + pytest.param("Grep", {"pattern": "uip solution deploy", "path": "."}, id="grep"), + pytest.param("TodoWrite", {"todos": [{"content": "Run uip solution deploy", "status": "pending"}]}, id="todo"), + # A call with a command: only the command is searched, not its other params. + pytest.param( + "ExecutePowershellCommand", + {"command": "echo ok", "description": "Run uip solution deploy later"}, + id="other-param", + ), + pytest.param("Bash", {"description": "Run uip solution deploy"}, id="bash-without-command"), +] + +_DEPLOY = r"uip\s+solution\s+deploy" + + +def _deployed( + *, + tool_name: str | None = None, + command_pattern: str | None = _DEPLOY, + exclude_pattern: str | None = None, + min_count: int = 1, + max_count: int | None = None, + require_success: bool = True, + description: str = "Agent deployed the solution", +) -> CommandExecutedCriterion: + return CommandExecutedCriterion( + description=description, + tool_name=tool_name, + command_pattern=command_pattern, + exclude_pattern=exclude_pattern, + min_count=min_count, + max_count=max_count, + require_success=require_success, + ) + + +def _check(criterion: CommandExecutedCriterion, *calls: tuple[str, dict]) -> CriterionResult: + commands = [ + _make_command(tool_name=tool, parameters=params, tool_id=f"t{i}") for i, (tool, params) in enumerate(calls) + ] + return SuccessChecker(MockSandbox()).check(criterion, turn_records=[_make_turn(commands)]) + + +def _score(criterion: CommandExecutedCriterion, tool: str, params: dict) -> float: + return _check(criterion, (tool, params)).score + + +class TestNullToolNameSearchesOnlyTheCommand: + """With ``tool_name`` null, a pattern searches only a call's ``command``.""" + + @pytest.mark.parametrize(("tool", "params"), _MENTIONS_WITHOUT_RUNNING) + def test_mention_in_another_tool_does_not_count(self, tool, params): + assert _score(_deployed(), tool, params) == 0.0 + + @pytest.mark.parametrize(("tool", "params"), _MENTIONS_WITHOUT_RUNNING) + def test_mention_in_another_tool_does_not_live_pass(self, tool, params): + # The early-stop trigger shares the matcher: no pass, so no on_pass stop. + turn_records = [_make_turn([_make_command(tool_name=tool, parameters=params)])] + assert CommandExecutedChecker().live_verdict(_deployed(), turn_records) == "undecided" + + @pytest.mark.parametrize(("tool", "params"), _MENTIONS_WITHOUT_RUNNING) + def test_mention_in_another_tool_does_not_trip_a_negative_guard(self, tool, params): + guard = _deployed(description="must NOT deploy", min_count=0, max_count=0, require_success=False) + assert _score(guard, tool, params) == 1.0 + + @pytest.mark.parametrize( + ("tool", "command"), + [ + pytest.param("Bash", "uip solution deploy --name billing", id="bash"), + pytest.param("Bash", ["bash", "-lc", "uip solution deploy --name billing"], id="argv-list"), + # A shell tool under another name: a Delegate host tool without an alias, + # or a hydrated ATIF call. Its `command` param still marks it as a shell call. + pytest.param("ExecutePowershellCommand", "uip solution deploy --name billing", id="unaliased"), + pytest.param("shell", "uip solution deploy --name billing", id="atif-shell"), + pytest.param("shell", ["bash", "-lc", "uip solution deploy --name billing"], id="other-name-argv-list"), + ], + ) + def test_shell_call_still_counts(self, tool, command): + assert _score(_deployed(), tool, {"command": command}) == 1.0 + + def test_named_tool_params_are_still_searched(self): + params = {"file_path": "NOTES.md", "content": "Next: uip solution deploy"} + assert _score(_deployed(tool_name="Write"), "Write", params) == 1.0 + + def test_without_a_pattern_every_tool_still_counts(self): + turn_records = [_make_turn([_make_command(tool_name=t, tool_id=t) for t in ("Read", "Write", "Bash")])] + criterion = CommandExecutedCriterion(description="any 3 tool calls", min_count=3) + assert SuccessChecker(MockSandbox()).check(criterion, turn_records=turn_records).score == 1.0 + + +class TestTheCallShapeChoosesTheText: + """The shape of a call, not ``tool_name``, decides which text a pattern searches.""" + + @pytest.mark.parametrize("command_pattern", [None, r"uip"], ids=["exclude-only", "with-pattern"]) + @pytest.mark.parametrize( + ("exclude", "expected"), + [ + pytest.param("dry-run", 1.0, id="exclude-text-in-another-param"), + pytest.param(r"solution\s+deploy", 0.0, id="exclude-text-in-the-command"), + ], + ) + def test_exclusion_searches_the_command_with_or_without_a_pattern(self, command_pattern, exclude, expected): + params = {"command": "uip solution deploy", "description": "dry-run first"} + criterion = _deployed(command_pattern=command_pattern, exclude_pattern=exclude, require_success=False) + assert _score(criterion, "shell", params) == expected + + @pytest.mark.parametrize("tool", ["exec_tool", "shell", "ExecutePowershellCommand"]) + @pytest.mark.parametrize("named", [False, True], ids=["null", "named"]) + def test_naming_a_shell_tool_keeps_an_anchored_match(self, tool, named): + criterion = _deployed(tool_name=tool if named else None, command_pattern=r"^npm test") + assert _score(criterion, tool, {"description": "run tests", "command": "npm test"}) == 1.0 + + @pytest.mark.parametrize("tool_name", [None, "Bash"]) + @pytest.mark.parametrize( + "params", + [ + pytest.param({"description": "Run uip solution deploy"}, id="missing"), + pytest.param({"command": "", "description": "Run uip solution deploy"}, id="empty"), + ], + ) + def test_bash_call_without_a_command_is_not_searched(self, tool_name, params): + assert _score(_deployed(tool_name=tool_name), "Bash", params) == 0.0 + + @pytest.mark.parametrize("tool_name", [None, "shell"]) + def test_shell_call_with_an_empty_command_is_not_searched(self, tool_name): + params = {"command": "", "description": "Run uip solution deploy"} + assert _score(_deployed(tool_name=tool_name), "shell", params) == 0.0 + + @pytest.mark.parametrize("tool", ["Bash", "shell", "ExecutePowershellCommand"]) + @pytest.mark.parametrize( + "command", + [ + # Matches only in the shell-normalized form. + pytest.param("bash -lc \"uip 'solution' deploy\"", id="quoted"), + # Matches only in a window after the first 2,000 characters. + pytest.param("echo build\n" * 300 + "uip solution deploy", id="after-2000-chars"), + ], + ) + def test_every_shell_tool_gets_normalization_and_windows(self, tool, command): + assert _score(_deployed(), tool, {"command": command}) == 1.0 + + def test_naming_the_tool_never_drops_a_pattern_match(self): + calls = [ + ("Bash", {"command": "uip solution deploy"}), + ("shell", {"command": "uip solution deploy", "description": "x"}), + ("shell", {"command": ["bash", "-lc", "uip solution deploy"]}), + ("Write", {"file_path": "NOTES.md", "content": "uip solution deploy"}), + ("Bash", {"description": "uip solution deploy"}), + ] + for tool, params in calls: + assert _score(_deployed(tool_name=tool), tool, params) >= _score(_deployed(), tool, params), tool + + def test_label_of_a_shell_call_under_another_name_is_its_command(self): + params = {"description": "x" * 100, "command": "uip solution deploy --name billing"} + assert "uip solution deploy --name billing" in _check(_deployed(), ("shell", params)).details + + +class TestUnsearchedCallsInDetails: + """A failing criterion says how many calls it did not search.""" + + def test_null_tool_name_counts_them_and_says_how_to_search_them(self): + result = _check(_deployed(), ("Write", {"file_path": "NOTES.md", "content": "Next: uip solution deploy"})) + assert result.score == 0.0 + assert "1 call(s) without a command were not searched. Set tool_name" in result.details + + def test_named_bash_counts_them_without_the_hint(self): + result = _check(_deployed(tool_name="Bash"), ("Bash", {"description": "uip solution deploy"})) + assert "1 call(s) without a command were not searched." in result.details + assert "Set tool_name" not in result.details + + def test_no_hint_when_naming_the_tool_cannot_help(self): + # A Bash call without a command has no text under any tool_name. + result = _check(_deployed(), ("Bash", {"description": "uip solution deploy"})) + assert "1 call(s) without a command were not searched." in result.details + assert "Set tool_name" not in result.details + + def test_a_passing_criterion_has_no_note(self): + result = _check( + _deployed(), + ("Write", {"content": "uip solution deploy"}), + ("Bash", {"command": "uip solution deploy"}), + ) + assert result.score == 1.0 + assert "not searched" not in result.details