[WatchTree-19]

Description

Adds experimental program execution and behavior scoring, implementing phase 10b of the scorer design and roadmap. It checks what submitted code actually does for specified inputs, rather than judging a description of its behavior.

Reusable execution, separate from scoring

• pyrit._code_execution provides a small client contract for caller-configured execution backends. Targets, converters, dataset generators, and scorers can use it without importing scoring or memory. Its shared input and result types are private and experimental.
• The client accepts verbatim code, a trusted runtime profile name, arguments, and stdin. It returns bounded output, exit information, timeout/resource-limit outcomes, and available runtime provenance. It does not interpret results or receive expected answers.
• The caller supplies the backend and trusted runtimes. PyRIT does not start the daemon, build or pull images, or maintain reusable environments.

Optional Docker adapter

Each invocation runs in a fresh container using a pre-pulled, digest-pinned image. The adapter disables networking and daemon logging, runs as a non-root user with capabilities dropped and privilege escalation disabled, mounts the program read-only, and applies memory, CPU, process, time, and output limits. It rejects backends that do not report the required resource support and images that declare volumes. Submitted source stays inside a private host workspace; cleanup handles cancellation and removes owned containers and anonymous volumes.

Stronger isolation can use the same client contract through a different caller-provided backend.

Scoring and replay

ProgramExecutionSource resolves program references and captures evidence. ProgramExecutionScorer keeps the expected results from ProgramBehaves separate and compares them with that evidence. Every scoring call executes afresh; the saved observation identifies the run. Explicit replay evaluates saved output without another execution.

All cases passing yields true; any proven case failure yields false. Infrastructure failures and insufficient evidence leave the verdict undetermined. Truncated output retains its uncertainty. Code is not extracted from surrounding prose, and existing scorer behavior is unchanged.

[WatchTree-19]

Thanks Roman, all six addressed in the latest commit, each with a regression test:

• Concurrent callers for the same attempt now share one in-flight run, cancelled only when every caller has gone.
• The client inspects the container status and only reports an exit code for an exited container; an attach failure while it is still running is a SandboxExecutionError.
• Management commands are killed and reaped in a finally, so cancellation no longer leaves one running.
• A failed rm is reported with the container name, kept in the observation's sandbox_warnings metadata, and stops further cases.
• The program file is made writable before the directory is removed, covered by a mocked-invocation test that runs on every platform.
• ContentEntryScorable is accepted as a program reference, with a round-trip test that scores a saved score's anchor.

[WatchTree-19]

Thanks Roman, all three are addressed in the latest commit.

1. The timeout now covers the whole invocation, including the client exiting after its streams close. If the client stalls, it gets killed, and the container's own state decides the result, so a container that is still running comes back as a sandbox error rather than a pass. There are two new tests where a fake client closes stdout and stderr and then hangs, one over a finished container and one over a running one.
2. When the last caller is cancelled, it now waits for the sandbox cleanup to finish before its own cancellation completes. The test holds cleanup open and checks that cancelling waits for it.
3. I took out the scorer's pre-check. Memory now links an identical stored observation inside the same transaction, under its row lock, and still rejects a different observation that reuses a stored ID. I added a test with two concurrent score_async calls on real SQLite, held together at the write. It reproduces your error on the previous code and now stores one observation that both scores link to.

[WatchTree-19]

Thanks Rich, all five are in the latest commit.

1. Containers now run with --log-driver none, so the daemon keeps nothing beyond what the client retains. The test asserts the flag on the generated arguments, and on a real daemon attached output still arrives and is capped (a 1 MB write kept 128 bytes).
2. An attempt's reusable evidence now has an ID derived from the source and the attempt, and on a cache miss the source loads that observation from memory before running anything. Your sequence (score, score, 256 other attempts, score again) now runs the program once and both scores link to one observation. Errors keep fresh IDs and are retried. Evidence that was never stored can still run again after eviction, but memory refuses a different run of an attempt that already has stored evidence, so contradicting results are never stored. There is a test with two sources racing one attempt.
3. --entrypoint now comes from the profile, so the recorded command is the one that runs. The fake Docker client applies an image entrypoint the way Docker does, and on a real daemon an image with ENTRYPOINT ["python3"] exited 2 before this change and now passes.
4. Creating and removing the program directory runs in a worker thread, and a cancelled call waits for that work before its cancellation completes, so cleanup order is kept. The tests block the file work until the event loop releases it, and check that cancelling during directory creation still leaves nothing behind.
5. _run_arguments builds the command line and _collect_output_async handles the streams against the one deadline. _run_invocation_async keeps the process and container cleanup.