FEAT: program execution scorer (scorer phase 10b)

doc/code/framework.md

@@ -289,6 +289,13 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 - Surface sources read what a location holds for a `SurfaceScorable`, such as files under one
   root directory. `FileWriteScorer` evaluates its `ContentWritten` condition against that
   evidence. The caller owns workspace isolation; a snapshot does not identify its writer.
+- Program-execution sources run a program for a `ProgramExecutionScorable` through an injected
+  client from the experimental `pyrit._code_execution` module, and turn its results into scoring evidence.
+  `ProgramExecutionScorer` splits its `ProgramBehaves` condition so the source receives only the
+  runtime profile and invocations, never the expected results. A program that fails is observed
+  evidence; a sandbox that cannot run it is an acquisition error, so the verdict is undetermined
+  rather than false.
+  Each scoring call runs afresh; an observation identifies the run and supports explicit replay.
 - Observation lifecycle is optional. A scorer that implements `AttemptObserver`, whether it is a
   root scorer or wrapped inside one, is asked by the attack to start observing after setup, once
   its attempt ID is allocated and before anything is sent, and to stop when the attempt ends. A
@@ -362,6 +369,20 @@ The below talks about responsibilities of most modules in the PyRIT library
 
 - Components that need credentials should go through these helpers rather than handling tokens themselves.
 
+## Code Execution (Experimental)
+
+**Responsibility**: Connect to caller-configured execution backends and return typed invocation results.
+
+- The client API is private and experimental in `pyrit._code_execution`; it may change without a deprecation cycle.
+- Targets, converters, dataset generators, and scorers can inject the same `SandboxClient` contract
+  from `pyrit._code_execution`. Execution inputs and results are private types in `pyrit.models.code_execution`.
+- `runtime_profile_name` selects a client-owned configuration by name, not by supplying an image or command.
+  Clients advertise the available names through `get_runtime_profile_names()`.
+- PyRIT does not own environments. The one-image Docker example is the only exception:
+  `DockerSandboxClient` manages disposable containers; callers supply the daemon and pinned image.
+- **Does not own**: interpreting results, scoring criteria, message or seed creation, memory
+  persistence, or infrastructure setup. Those remain with the consumer or caller.
+
 ## [Exceptions](../contributing/9_exception)
 
 **Responsibility**: Define PyRIT's exception hierarchy and the retry behavior built around it.

doc/code/scoring/0_scoring.ipynb

@@ -275,7 +275,8 @@
     "the target need not be a language model. Its kind is `scorer_target_response`.\n",
     "Media observation capture remains deferred until its evidence can be snapshotted.\n",
     "Trace-backed tool observations are covered in [Tool-call scoring](5_tool_call_scorer.ipynb), and file-system\n",
-    "observations in [File-write scoring](6_file_write_scorer.ipynb).\n",
+    "observations in [File-write scoring](6_file_write_scorer.ipynb), and sandboxed program runs in\n",
+    "[Program-execution scoring](7_program_execution_scorer.ipynb).\n",
     "\n",
     "Replaying a judgment is different from evaluating a stored run against a new expectation.\n",
     "A retained target judgment answers the original expectation; changing that expectation\n",

doc/code/scoring/0_scoring.py

@@ -161,7 +161,8 @@
 # the target need not be a language model. Its kind is `scorer_target_response`.
 # Media observation capture remains deferred until its evidence can be snapshotted.
 # Trace-backed tool observations are covered in [Tool-call scoring](5_tool_call_scorer.ipynb), and file-system
-# observations in [File-write scoring](6_file_write_scorer.ipynb).
+# observations in [File-write scoring](6_file_write_scorer.ipynb), and sandboxed program runs in
+# [Program-execution scoring](7_program_execution_scorer.ipynb).
 #
 # Replaying a judgment is different from evaluating a stored run against a new expectation.
 # A retained target judgment answers the original expectation; changing that expectation

doc/code/scoring/7_program_execution_scorer.ipynb

@@ -0,0 +1,175 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# Program execution and scoring\n",
+    "\n",
+    "`pyrit._code_execution` is experimental; its API may change without a deprecation cycle.\n",
+    "Its shared data types in `pyrit.models.code_execution` are also private and experimental.\n",
+    "The caller provides a Docker daemon and this exact pre-pulled image:\n",
+    "`docker pull python@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1`.\n",
+    "PyRIT does not start Docker or pull images. This example needs no model credentials.\n",
+    "Docker must support cgroup memory, swap, CPU, and process limits. Runtime images must not\n",
+    "declare volumes. Unsupported backends and images are rejected before execution.\n",
+    "\n",
+    "## Execute without scoring or memory\n",
+    "\n",
+    "A runtime profile name selects a trusted client configuration, not an arbitrary image.\n",
+    "The client runs code verbatim and returns bounded output and exit information.\n",
+    "Targets, converters, and generators can use this same interface."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Direct execution: 9\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pyrit._code_execution import DockerRuntimeProfile, DockerSandboxClient\n",
+    "from pyrit.models.code_execution import _ProgramInvocation\n",
+    "\n",
+    "client = DockerSandboxClient(\n",
+    "    profiles={\n",
+    "        \"python\": DockerRuntimeProfile(\n",
+    "            image=\"python@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1\",\n",
+    "            command=(\"python3\", \"-I\"),\n",
+    "        )\n",
+    "    }\n",
+    ")\n",
+    "program = \"import sys\\nprint(int(sys.argv[1]) ** 2)\\n\"\n",
+    "invocation = _ProgramInvocation(arguments=(\"3\",))\n",
+    "assert \"python\" in client.get_runtime_profile_names()\n",
+    "run = await client.execute_async(  # type: ignore\n",
+    "    program=program, runtime_profile_name=\"python\", invocations=(invocation,)\n",
+    ")\n",
+    "assert run.events[0].exit_code == 0\n",
+    "assert run.events[0].stdout == \"9\\n\"\n",
+    "print(f\"Direct execution: {run.events[0].stdout.strip()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
+    "## Score and save the evidence\n",
+    "\n",
+    "The source receives only code and invocation inputs; expected results stay with the scorer.\n",
+    "Every `score_async` call starts a fresh execution, identified by its saved observation ID.\n",
+    "Wrong output, unexpected exits, and enforced resource/time limits fail a case. Infrastructure\n",
+    "failures or insufficient evidence leave the verdict undetermined."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Score: True - All 1 case(s) produced the expected result under python.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pyrit.memory import CentralMemory\n",
+    "from pyrit.models import ContentScorable, Equals, ProgramBehaves, ProgramCase, ProgramResult, ScoringExpectation\n",
+    "from pyrit.score import ProgramExecutionScorer, ProgramExecutionSource\n",
+    "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n",
+    "\n",
+    "await initialize_pyrit_async(  # type: ignore\n",
+    "    memory_db_type=IN_MEMORY, load_defaults=False, env_files=[], silent=True\n",
+    ")\n",
+    "case = ProgramCase(invocation=invocation, expected=ProgramResult(stdout=Equals(value=\"9\")))\n",
+    "expectation = ScoringExpectation(conditions=(ProgramBehaves(runtime_profile_name=\"python\", cases=(case,)),))\n",
+    "scorer = ProgramExecutionScorer(source=ProgramExecutionSource(sandbox=client))\n",
+    "score = (await scorer.score_async(scorable=ContentScorable(value=program), expectation=expectation))[0]  # type: ignore\n",
+    "assert score.get_value() is True\n",
+    "print(f\"Score: {score.get_value()} - {score.score_rationale}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4",
+   "metadata": {},
+   "source": [
+    "## Replay without another execution\n",
+    "\n",
+    "Replay uses the saved outputs, not the Docker client. Expected results may change, but the\n",
+    "runtime profile name and invocation inputs must still match the saved run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Replay: False - The program did not produce the expected result under python: case 0 output did not match.\n"
+     ]
+    }
+   ],
+   "source": [
+    "memory = CentralMemory.get_memory_instance()\n",
+    "observation = (await memory.get_observations_async(observation_ids=score.observation_ids))[0]\n",
+    "different_case = ProgramCase(invocation=invocation, expected=ProgramResult(stdout=Equals(value=\"8\")))\n",
+    "different_expectation = ScoringExpectation(\n",
+    "    conditions=(ProgramBehaves(runtime_profile_name=\"python\", cases=(different_case,)),)\n",
+    ")\n",
+    "replayed = (\n",
+    "    await scorer.score_observation_async(observation=observation, expectation=different_expectation)  # type: ignore\n",
+    ")[0]\n",
+    "assert replayed.get_value() is False\n",
+    "assert replayed.observation_ids == score.observation_ids\n",
+    "print(f\"Replay: {replayed.get_value()} - {replayed.score_rationale}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6",
+   "metadata": {},
+   "source": [
+    "Real execution and fault coverage is in `tests/integration/score/test_program_execution_scorer_integration.py`.\n",
+    "Opt in with `RUN_ALL_TESTS=true`; the tests require the daemon and pinned image above."
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "cell_metadata_filter": "-all"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.14.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

doc/code/scoring/7_program_execution_scorer.py

@@ -0,0 +1,96 @@
+# ---
+# jupyter:
+#   jupytext:
+#     cell_metadata_filter: -all
+#     text_representation:
+#       extension: .py
+#       format_name: percent
+#       format_version: '1.3'
+# ---
+
+# %% [markdown]
+# # Program execution and scoring
+#
+# `pyrit._code_execution` is experimental; its API may change without a deprecation cycle.
+# Its shared data types in `pyrit.models.code_execution` are also private and experimental.
+# The caller provides a Docker daemon and this exact pre-pulled image:
+# `docker pull python@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1`.
+# PyRIT does not start Docker or pull images. This example needs no model credentials.
+# Docker must support cgroup memory, swap, CPU, and process limits. Runtime images must not
+# declare volumes. Unsupported backends and images are rejected before execution.
+#
+# ## Execute without scoring or memory
+#
+# A runtime profile name selects a trusted client configuration, not an arbitrary image.
+# The client runs code verbatim and returns bounded output and exit information.
+# Targets, converters, and generators can use this same interface.
+
+# %%
+from pyrit._code_execution import DockerRuntimeProfile, DockerSandboxClient
+from pyrit.models.code_execution import _ProgramInvocation
+
+client = DockerSandboxClient(
+    profiles={
+        "python": DockerRuntimeProfile(
+            image="python@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1",
+            command=("python3", "-I"),
+        )
+    }
+)
+program = "import sys\nprint(int(sys.argv[1]) ** 2)\n"
+invocation = _ProgramInvocation(arguments=("3",))
+assert "python" in client.get_runtime_profile_names()
+run = await client.execute_async(  # type: ignore
+    program=program, runtime_profile_name="python", invocations=(invocation,)
+)
+assert run.events[0].exit_code == 0
+assert run.events[0].stdout == "9\n"
+print(f"Direct execution: {run.events[0].stdout.strip()}")
+
+# %% [markdown]
+# ## Score and save the evidence
+#
+# The source receives only code and invocation inputs; expected results stay with the scorer.
+# Every `score_async` call starts a fresh execution, identified by its saved observation ID.
+# Wrong output, unexpected exits, and enforced resource/time limits fail a case. Infrastructure
+# failures or insufficient evidence leave the verdict undetermined.
+
+# %%
+from pyrit.memory import CentralMemory
+from pyrit.models import ContentScorable, Equals, ProgramBehaves, ProgramCase, ProgramResult, ScoringExpectation
+from pyrit.score import ProgramExecutionScorer, ProgramExecutionSource
+from pyrit.setup import IN_MEMORY, initialize_pyrit_async
+
+await initialize_pyrit_async(  # type: ignore
+    memory_db_type=IN_MEMORY, load_defaults=False, env_files=[], silent=True
+)
+case = ProgramCase(invocation=invocation, expected=ProgramResult(stdout=Equals(value="9")))
+expectation = ScoringExpectation(conditions=(ProgramBehaves(runtime_profile_name="python", cases=(case,)),))
+scorer = ProgramExecutionScorer(source=ProgramExecutionSource(sandbox=client))
+score = (await scorer.score_async(scorable=ContentScorable(value=program), expectation=expectation))[0]  # type: ignore
+assert score.get_value() is True
+print(f"Score: {score.get_value()} - {score.score_rationale}")
+
+# %% [markdown]
+# ## Replay without another execution
+#
+# Replay uses the saved outputs, not the Docker client. Expected results may change, but the
+# runtime profile name and invocation inputs must still match the saved run.
+
+# %%
+memory = CentralMemory.get_memory_instance()
+observation = (await memory.get_observations_async(observation_ids=score.observation_ids))[0]
+different_case = ProgramCase(invocation=invocation, expected=ProgramResult(stdout=Equals(value="8")))
+different_expectation = ScoringExpectation(
+    conditions=(ProgramBehaves(runtime_profile_name="python", cases=(different_case,)),)
+)
+replayed = (
+    await scorer.score_observation_async(observation=observation, expectation=different_expectation)  # type: ignore
+)[0]
+assert replayed.get_value() is False
+assert replayed.observation_ids == score.observation_ids
+print(f"Replay: {replayed.get_value()} - {replayed.score_rationale}")
+
+# %% [markdown]
+# Real execution and fault coverage is in `tests/integration/score/test_program_execution_scorer_integration.py`.
+# Opt in with `RUN_ALL_TESTS=true`; the tests require the daemon and pinned image above.

doc/myst.yml

@@ -162,6 +162,7 @@ project:
             - file: code/scoring/4_scorer_metrics.ipynb
             - file: code/scoring/5_tool_call_scorer.ipynb
             - file: code/scoring/6_file_write_scorer.ipynb
+            - file: code/scoring/7_program_execution_scorer.ipynb
         - file: code/memory/0_memory.md
           children:
             - file: code/memory/1_sqlite_memory.ipynb

pyrit/_code_execution/__init__.py

@@ -0,0 +1,39 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+# ruff: noqa: F401
+
+"""
+Experimental clients for caller-configured code-execution backends.
+
+This private package is an explicit opt-in. Its API may change without a deprecation cycle.
+"""
+
+from typing import TYPE_CHECKING
+
+from pyrit.common.lazy_imports import get_lazy_dir, resolve_lazy_export
+
+if TYPE_CHECKING:
+    from pyrit._code_execution.client import SandboxClient, SandboxExecutionError
+    from pyrit._code_execution.docker_client import DockerRuntimeProfile, DockerSandboxClient
+
+_LAZY_EXPORTS: dict[str, str] = {
+    "SandboxClient": "pyrit._code_execution.client",
+    "SandboxExecutionError": "pyrit._code_execution.client",
+    "DockerRuntimeProfile": "pyrit._code_execution.docker_client",
+    "DockerSandboxClient": "pyrit._code_execution.docker_client",
+}
+
+__all__ = list(_LAZY_EXPORTS)
+
+
+def __getattr__(name: str) -> object:
+    return resolve_lazy_export(
+        name=name,
+        module_name=__name__,
+        module_globals=globals(),
+        exports=_LAZY_EXPORTS,
+    )
+
+
+def __dir__() -> list[str]:
+    return get_lazy_dir(module_globals=globals(), exports=_LAZY_EXPORTS)