Skip to content

FEAT: Add ScenarioSuite registry and one-click launch - #3490

Open
hannahwestra25 wants to merge 7 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-scenario-suite-registry
Open

hannahwestra25 wants to merge 7 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-scenario-suite-registry

Conversation

@hannahwestra25

@hannahwestra25 hannahwestra25 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • Adds ScenarioSuiteRegistry, modeled on ScenarioPresetRegistry, so code-registered read-only suites coexist with editable stored suites while preserving stored precedence and actionable registered-name conflicts.
  • Exposes suite and preset origin plus nullable versions; registered definitions use None, while stored definitions retain opaque optimistic-concurrency versions.
  • Routes suite CRUD, resolution, source configuration, and startup scanning through the registry while retaining ScenarioSuiteStorage as persistence.
  • Adds a process-local ScenarioSuiteExecutionService that preflights the complete reviewed suite/preset snapshot before accepting work, then submits ordinary scenario runs in suite order through ScenarioRunService.
  • Persists versioned scenario_suite_execution_v1 metadata atomically on each successful child ScenarioResult, without adding parent run records or database migrations.
  • Adds launch, grouped progress, and best-effort cancellation APIs plus runtime admission/shutdown integration.
  • Makes cancellation admission-safe, keeps temporary child-status read errors nonterminal, and retains active executions at the 256-entry boundary; capacity exhaustion returns 429 instead of evicting active grouped control state.
  • Adds a launch dialog, registered/stored launch actions, ordered progress page, child-run links/backlinks, transient polling recovery, and cancellation UI.
  • Keeps suite execution failures user-safe: full exception details stay in server logs while API responses and the progress page expose stable actionable messages.
  • Preserves stored-suite editing, optimistic concurrency, advisory status projection, authorization, existing scenario-run behavior, and current suite routes.
image image image

Tests and Documentation

Added backend coverage for registry resolution, optimistic snapshots, ordered submission, admission-safe cancellation, transient status recovery, terminal-aware retention, lifecycle behavior, and safe failure responses. Added frontend coverage for launch gating, polling, cancellation, routing, backlinks, and error redaction. Updated suite configuration documentation.

Focused validation passed: 115 backend and 5 frontend tests, Ruff, formatting, and whitespace. Earlier branch validation passed 590 backend/import and 200 frontend tests plus TypeScript, ESLint, and production build. Cached ty==0.0.86 reported no code errors; pinned ty==0.0.89 could not download due PyPI TLS, so CI reruns it. JupyText was not run because no notebooks changed.

Copilot AI added 3 commits October 23, 2026 09:01
Merge code-registered read-only suites with stored editable definitions and surface suite origin across the API and UI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Keep Fluent table-cell layout so descriptions and advisory issues determine row height instead of overflowing into an internal scrollbar.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Add transient registry-backed suite execution with ordered scenario submission, optimistic definition snapshots, durable child metadata, progress polling, and cancellation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
@hannahwestra25 hannahwestra25 changed the title FEAT: Add ScenarioSuite registry support FEAT: Add ScenarioSuite registry and one-click launch Oct 23, 2026
Copilot AI added 3 commits October 23, 2026 12:04
Keep full exception details in server logs while exposing only stable user-safe failure messages through suite execution responses and the UI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Keep patched ScenarioRunService helpers compatible with the internal initial_metadata keyword added for suite child attribution.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd
Preserve admitted child runs across suite cancellation, keep transient status-read failures nonterminal, and reject capacity when only active executions remain.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest lifecycle changes address three of the earlier findings. These comments cover the remaining lifecycle and polling issues, the shared-model boundary, the metadata key, and the proposed UI flow.

ValueError: If a suite is already registered under this name.
"""
try:
self.instances.register(suite, name=suite.name, tags=tags)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reset the code-registered suites during live reinitialization? reset_setup_registries() does not include ScenarioSuiteRegistry, so an initializer that registers a suite succeeds at startup but fails with "already registered" when the same configuration is applied again. Removing the initializer also leaves its suites in the registry. The registered instances should follow the setup lifecycle; the stored suite documents should remain unchanged.

return
}
const previous = latestExecutionRef.current
setState({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep polling with a delay after a refresh failure, or add a Retry button to the stale-data view? After two consecutive failures, this branch preserves the last execution but does not schedule another poll. ScenarioSuiteExecutionPage only exposes Retry when there is no execution, so a temporary outage leaves an active suite's progress frozen until the user reloads the page.


from pydantic import BaseModel, ConfigDict, Field

SCENARIO_SUITE_EXECUTION_METADATA_KEY = "scenario_suite_execution_v1"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use a stable metadata key, scenario_suite_execution, without _v1 in the name. The payload already has a version field, which is where the schema version belongs. Encoding it in both places makes readers track multiple keys for the same concept when the schema changes. Please keep the version inside the payload and update the readers and writers together.

)


class LaunchScenarioSuiteRequest(BaseModel):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep the shared suite launch/execution contracts in pyrit.models, following RunScenarioRequest and ScenarioRunSummary, and keep only the HTTP-specific envelopes here? I would also separate reusable suite/preset resolution from the backend task lifecycle. The backend should own admission, scheduling, and cancellation, but resolving a suite into child requests should not require importing a backend service. Reusing the existing run states and launch-field validation would also avoid a second contract drifting from the ordinary scenario path. This matches the model and backend responsibilities in framework.md.

appearance="primary"
icon={<PlayRegular />}
disabled={!runnable || !defaultsReady}
onClick={() => setLaunching(item)}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to keep the registry for defining configurations and use the Scanner for launching them. That would better match the existing UI. Can we organize the flow like this?

  1. Combine presets and suites under one Scenario presets registry page. Put Scenario Preset Groups (or Scenario Suites) at the top, with individual presets below.
  2. Add an Add to scanner yes/no switch to each preset. The tip should explain that this controls whether the preset appears as an individual scanner entry; presets used only within a suite can stay hidden from that list.
  3. Remove Launch actions from the registry pages and move launching to the Scanner.
  4. In the Scanner, show suites/groups first in alphabetical order, with an expandable list of their member scenarios, then the normal individual scanners. Separate Single and Groups tabs could also work if that layout is clearer.
  5. Keep the existing launch flow: select an item, review its configuration, and click Launch. We can allow configuration edits before launch for individual scenarios; suite-level edits do not need to be part of this change.

Small, non-blocking note: Refresh is in the opposite position from other pages. I would rather remove manual refresh across the UI eventually, so I would not spend time on that here.

Reset code-registered ScenarioSuites during live setup replacement while leaving stored suite documents intact. Continue polling active suite executions after consecutive transient refresh failures so stale progress can recover without a page reload.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2403a4a9-6af6-46b2-b371-46de73a018fd

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants