Skip to content

TEST-201 docs: bring Simulations pages up to date with Composer test generation and the run API - #1342

Open
scott-lowe-vapi wants to merge 2 commits into
mainfrom
scottlowe/test-201-simulations-docs-catch-up
Open

scott-lowe-vapi wants to merge 2 commits into
mainfrom
scottlowe/test-201-simulations-docs-catch-up

Conversation

@scott-lowe-vapi

Copy link
Copy Markdown
Contributor

Description

Linear: TEST-201. Companion in-repo docs: VapiAI/vapi#23428.

  • Composer
    • New section Generate tests for an assistant (/composer#generate-tests-for-an-assistant): requirements draft → Generate coverage → suite card with mocked tools and not-tested gaps → Run suite, plus edits and a tip to run five or more iterations.
    • Limitations: "Cannot test your agent" is replaced by "Cannot test every behavior".
    • Safety › Deletion no longer claims Composer can delete resources. It has no delete tools; only discard-draft, for your own unpublished draft.
  • Advanced
    • "Mock tool responses" (anchor unchanged) says generated suites come pre-mocked and explains how to review those mocks.
    • New sections: "Handoffs and transfers in simulations", covering the static-handoff refusal and dynamic handoffs, with a gaps table; and "Hooks and call endings", covering the removed customer.speech.timeout and the tester hang-up ending as customer-ended-call.
    • Four troubleshooting rows.
  • Manage (API tabs)
    • The Idempotency-Key header on POST /eval/simulation/run.
    • Run, item and suite filters (suiteId, targetType, targetId, status, simulationId, assistantId, squadId, name).
    • A note that any query parameter switches these lists to { metadata, results }.
  • Quickstart: the unmocked-tools warning now applies to hand-built suites only.
  • Best practices / Run and maintain: one iteration while setting up, then five or more before trusting a pass rate.
  • Overview, Manage, Voice testing: the handoff mentions link to the new limits section.

Evidence

Every behavior was checked against VapiAI/vapi main:

  • Dashboard labels: features/simulationsOnboarding/constants.ts.
  • Idempotency: simulationRunIdempotencyKeyParse.ts and simulationRunIdempotentReplayGet.ts.
  • Filters: simulationRun.types.ts and GetSimulationSuitesDTO.ts.
  • Handoffs and hooks: handoffSimulationRefusalGet.ts and simulationTargetReengagementHooks/.

Testing Steps

  • fern check (5.151.1) passes: 0 errors. The 14 warnings were already on main.
  • Check the preview deployment for the changed pages

…generation and the run API

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lightsage-app

lightsage-app Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Lightsage docs evals

Lightsage could not queue docs evals for this PR.

Docs URL: https://vapi-preview-01a1527c-c7e7-742f-bb13-ed300ead57a9.docs.buildwithfern.com
Commit: d16141f
Reason: No evals are selected for GitHub PR evals. Open Lightsage and enable the GitHub checkbox for at least one eval.

@github-actions

Copy link
Copy Markdown
Contributor

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant