feat(workflows): compose workflows with a unified execution tree

[markuswondrak]

Summary

Reimplements workflow composition from main (c00dc055) with one persisted execution tree and one executor. Replaces #4724 and implements the scoped single-run contract approved in #4680 (comment).

Closes #4680.

Included workflows receive private, strictly bound inputs and return only declared outputs alongside workflow/status/error metadata. Targets must exactly match safe, installed, enabled IDs in the current project. Bound overlay-resolved definitions survive resume, cycles are path-based, and included depth is limited to 16.

Why replace #4724

The previous implementation accumulated separate scope identities, result keys, cursor state, snapshot files, and execution paths. Its latest review identified a real fan-out isolation bug: nested calls had distinct scope keys but shared the caller-result alias.

Each execution occurrence now owns its result, children, and workflow binding. Authored step IDs are local expression aliases; concurrent items have independent contexts. Persistence and resume follow the same tree.

Diff	#4724	Replacement
Added lines	5,990	1,655
Removed lines	154	434
Production additions	2,590	869

The replacement includes 702 added test lines. Existing tests on main are retained.

Architecture and compatibility

• _execution.py owns tree traversal, resume, control flow, workflow scope entry, validation, and reporting projections.
• composition.py owns target resolution and typed boundaries, reusing existing input coercion.
• engine.py keeps public lifecycle and RunState persistence, replacing the previous execution loops with thin adapters.
• YAML strings inside JSON preserve definition/expansion scalar types without external snapshot files. Larger checkpoints are the tradeoff.
• Resume retains selected expansions and completed occurrences across branches, loops, fan-out, and workflow calls. Explicit input updates rebind incomplete calls; ordinary resume keeps persisted inputs.
• Output-evaluation failures retry finalization without repeating completed child commands.
• Main-format legacy checkpoints adapt from their top-level index. Unreleased checkpoint formats from feat(workflows): compose installed workflows via a scoped workflow step #4724 are not migrated.
• Tree-backed checkpoints left running after a crash can resume. One executing process per run remains the assumption; side effects before their checkpoint may repeat.
• Nested gate and scope reporting follow the tree. Root result keys and generated loop/fan-out aliases remain compatibility views.

Evidence

test_concurrent_nested_calls_keep_downstream_aliases_local was run against #4724 commit 7ece7a16 using an isolated import path. It failed with consumed outputs {1: 1, 2: 1} instead of {1: 1, 2: 2}. The same case passes here.

45 focused cases cover concurrent downstream consumption, scoped outputs, frozen branch/custom expansion resume, loops and fan-out, terminal sibling reuse, stable definitions, strict target/input/output rejection, cycles/depth, output-finalization retry, malformed checkpoints, checkpoint/log failures, and legacy adaptation. A CLI integration case exercises run → nested gate status in JSON and human formats → resume.

Test selection reasoning

Changed area	Affects	Verification
Engine, execution tree, composition, registration	Execution, validation, resume	Existing workflow suites and occurrence-level regressions
Run/resume/status helpers	specify workflow run/resume/status	Command suites and composed-gate CLI integration
Workflow documentation	Authoring and publishing	Reference/architecture/publishing updates and diff review

Results

• Clean-main workflow baseline: 1,259 passed, 1 skipped.
• New focused suite: 45 passed.
• Full .venv/bin/python -m pytest tests -q: 8,435 passed, 212 skipped, 4 failed in 328 seconds. All four failures also reproduce against clean main c00dc055: preset-update missing-argument wording and three checksum tests expecting English FAILED under a German locale. All workflow tests pass in the full run.
• Ruff 0.15.0 passes for the workflow package and changed tests; git diff --check passes.

AI disclosure

Implemented and submitted on behalf of @markuswondrak by OpenCode, model gpt-6-astra (github-copilot/gpt-6-astra), in autonomous mode with user-directed scope. AI assistance covered decision-history research, planning, source code, tests, documentation, regression reproduction, automated validation, commit/push, and this fully AI-drafted PR description. Human line-by-line review or manual testing is not attested. This rewrite incorporates requirements and findings from the AI-assisted implementation and review history in #4724.

[Copilot]

Copilot review overview

🟡 Changes recommended

Child step implementation exceptions are incorrectly converted into ordinary workflow failures and may be swallowed by continue_on_error.

Review effort: Balanced
Findings: 1

Open (1)

• Propagate child execution exceptions instead of swallowing them · New

What changed in this PR

Introduces composable workflows backed by a unified, persisted execution tree.

Changes:

• Adds scoped workflow calls with typed inputs and declared outputs.
• Unifies execution, resume, fan-out, and nested-state persistence.
• Adds CLI reporting, documentation, and regression coverage.

File	Description
src/​specify_cli/​workflows/​_commands.py	Reports nested scopes and gates.
src/​specify_cli/​workflows/​_execution.py	Implements tree-based execution.
src/​specify_cli/​workflows/​__init__.py	Registers workflow steps.
src/​specify_cli/​workflows/​command_resume.py	Documents crash recovery.
src/​specify_cli/​workflows/​command_status.py	Displays composed scopes.
src/​specify_cli/​workflows/​composition.py	Handles workflow boundaries.
src/​specify_cli/​workflows/​engine.py	Integrates persistence and execution.
src/​specify_cli/​workflows/​step/​gate/​__init__.py	Normalizes gate messages.
src/​specify_cli/​workflows/​step/​workflow/​__init__.py	Defines the workflow step.
tests/​specify_cli/​workflows/​test_command_status.py	Tests composed CLI lifecycle.
tests/​workflows/​test_composition_execution.py	Covers composition and resume.
design/​workflow-step.md	Updates execution guidance.
docs/​reference/​workflows.md	Documents composition semantics.
workflows/​ARCHITECTURE.md	Describes the execution tree.
workflows/​PUBLISHING.md	Adds workflow-step validation guidance.
workflows/​README.md	Lists the new step type.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.