Repository navigation
FEAT: Add ScenarioSuite registry and one-click launch - #3490
hannahwestra25 wants to merge 7 commits into
Conversation
Merge code-registered read-only suites with stored editable definitions and surface suite origin across the API and UI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Keep Fluent table-cell layout so descriptions and advisory issues determine row height instead of overflowing into an internal scrollbar. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Add transient registry-backed suite execution with ordered scenario submission, optimistic definition snapshots, durable child metadata, progress polling, and cancellation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Keep full exception details in server logs while exposing only stable user-safe failure messages through suite execution responses and the UI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Keep patched ScenarioRunService helpers compatible with the internal initial_metadata keyword added for suite child attribution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Preserve admitted child runs across suite cancellation, keep transient status-read failures nonterminal, and reject capacity when only active executions remain. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
The latest lifecycle changes address three of the earlier findings. These comments cover the remaining lifecycle and polling issues, the shared-model boundary, the metadata key, and the proposed UI flow.
| ValueError: If a suite is already registered under this name. | ||
| """ | ||
| try: | ||
| self.instances.register(suite, name=suite.name, tags=tags) |
There was a problem hiding this comment.
Can we reset the code-registered suites during live reinitialization? reset_setup_registries() does not include ScenarioSuiteRegistry, so an initializer that registers a suite succeeds at startup but fails with "already registered" when the same configuration is applied again. Removing the initializer also leaves its suites in the registry. The registered instances should follow the setup lifecycle; the stored suite documents should remain unchanged.
| return | ||
| } | ||
| const previous = latestExecutionRef.current | ||
| setState({ |
There was a problem hiding this comment.
Can we keep polling with a delay after a refresh failure, or add a Retry button to the stale-data view? After two consecutive failures, this branch preserves the last execution but does not schedule another poll. ScenarioSuiteExecutionPage only exposes Retry when there is no execution, so a temporary outage leaves an active suite's progress frozen until the user reloads the page.
|
|
||
| from pydantic import BaseModel, ConfigDict, Field | ||
|
|
||
| SCENARIO_SUITE_EXECUTION_METADATA_KEY = "scenario_suite_execution_v1" |
There was a problem hiding this comment.
Let's use a stable metadata key, scenario_suite_execution, without _v1 in the name. The payload already has a version field, which is where the schema version belongs. Encoding it in both places makes readers track multiple keys for the same concept when the schema changes. Please keep the version inside the payload and update the readers and writers together.
| ) | ||
|
|
||
|
|
||
| class LaunchScenarioSuiteRequest(BaseModel): |
There was a problem hiding this comment.
Can we keep the shared suite launch/execution contracts in pyrit.models, following RunScenarioRequest and ScenarioRunSummary, and keep only the HTTP-specific envelopes here? I would also separate reusable suite/preset resolution from the backend task lifecycle. The backend should own admission, scheduling, and cancellation, but resolving a suite into child requests should not require importing a backend service. Reusing the existing run states and launch-field validation would also avoid a second contract drifting from the ordinary scenario path. This matches the model and backend responsibilities in framework.md.
| appearance="primary" | ||
| icon={<PlayRegular />} | ||
| disabled={!runnable || !defaultsReady} | ||
| onClick={() => setLaunching(item)} |
There was a problem hiding this comment.
I'd like to keep the registry for defining configurations and use the Scanner for launching them. That would better match the existing UI. Can we organize the flow like this?
- Combine presets and suites under one Scenario presets registry page. Put Scenario Preset Groups (or Scenario Suites) at the top, with individual presets below.
- Add an Add to scanner yes/no switch to each preset. The tip should explain that this controls whether the preset appears as an individual scanner entry; presets used only within a suite can stay hidden from that list.
- Remove Launch actions from the registry pages and move launching to the Scanner.
- In the Scanner, show suites/groups first in alphabetical order, with an expandable list of their member scenarios, then the normal individual scanners. Separate Single and Groups tabs could also work if that layout is clearer.
- Keep the existing launch flow: select an item, review its configuration, and click Launch. We can allow configuration edits before launch for individual scenarios; suite-level edits do not need to be part of this change.
Small, non-blocking note: Refresh is in the opposite position from other pages. I would rather remove manual refresh across the UI eventually, so I would not spend time on that here.
Reset code-registered ScenarioSuites during live setup replacement while leaving stored suite documents intact. Continue polling active suite executions after consecutive transient refresh failures so stale progress can recover without a page reload. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Description
ScenarioSuiteRegistry, modeled onScenarioPresetRegistry, so code-registered read-only suites coexist with editable stored suites while preserving stored precedence and actionable registered-name conflicts.None, while stored definitions retain opaque optimistic-concurrency versions.ScenarioSuiteStorageas persistence.ScenarioSuiteExecutionServicethat preflights the complete reviewed suite/preset snapshot before accepting work, then submits ordinary scenario runs in suite order throughScenarioRunService.scenario_suite_execution_v1metadata atomically on each successful childScenarioResult, without adding parent run records or database migrations.Tests and Documentation
Added backend coverage for registry resolution, optimistic snapshots, ordered submission, admission-safe cancellation, transient status recovery, terminal-aware retention, lifecycle behavior, and safe failure responses. Added frontend coverage for launch gating, polling, cancellation, routing, backlinks, and error redaction. Updated suite configuration documentation.
Focused validation passed: 115 backend and 5 frontend tests, Ruff, formatting, and whitespace. Earlier branch validation passed 590 backend/import and 200 frontend tests plus TypeScript, ESLint, and production build. Cached
ty==0.0.86reported no code errors; pinnedty==0.0.89could not download due PyPI TLS, so CI reruns it. JupyText was not run because no notebooks changed.