Skip to content

TEST-203 docs: add the Simulations onboarding API guide - #1344

Draft
scott-lowe-vapi wants to merge 2 commits into
scottlowe/test-201-simulations-docs-catch-upfrom
scottlowe/test-203-onboarding-api-guide
Draft

scott-lowe-vapi wants to merge 2 commits into
scottlowe/test-201-simulations-docs-catch-upfrom
scottlowe/test-203-onboarding-api-guide

Conversation

@scott-lowe-vapi

Copy link
Copy Markdown
Contributor

Description

Draft: merge when VapiAI/vapi#23422 (TEST-200) deploys. That PR publishes the onboarding endpoints and lists this guide as a merge gate. Linear: TEST-203. Stacked on #1342.

  • New page Generate a test suite with the API (/observability/simulations-generate-api), added to the nav after Advanced. It covers:
    • beta key access, and the 404 for orgs without it
    • the five-step flow with cURL: draft, revise, generate with idempotencyKey, edit with patchKey, run with Idempotency-Key
    • request and response field tables
    • coverage and tool gap reasons
    • idempotency and retries: the in-progress and key-reused 409s
    • daily limits: 60 drafts, 20 generations and 60 edits per UTC day, at most 2 generating at once, with what doesn't count
    • the error table
  • Advanced: troubleshooting rows for the TEST-192 run-create 400 (VapiAI/vapi#23407) and the Failed to create target call failure reason (VapiAI/vapi#23400).
  • Composer: a Limitations bullet on the shared daily allowance.

Sources are the open stack's diffs (#23400, #23407, #23418–#23422). The 524 in the retry list comes from the endpoint's own operation description.

Before merge:

  • Re-check every value against the merged code.
  • Resolve the MDX TODO in Limits: do the limits apply to Enterprise orgs, and can they be raised?
  • Confirm "contact support" is the right path for beta key access.
  • Link the API reference pages once fern api update pulls the routes in.

What's New blurb:

Generate a simulation suite with the API (beta). Three new endpoints under /eval/simulation/onboarding run the same test-generation flow as Composer: draft numbered requirements from an assistant's prompt, tools, and handoffs, revise them with your feedback, then generate a saved suite whose simulations mock every tool that would reach an external system. The response reports which requirements are covered, which aren't and why, and any handoffs or transfers the simulations don't exercise. Edit the suite in place, then run it with POST /eval/simulation/run; an idempotencyKey makes every retry safe. Private API keys work for organizations enabled during the beta, so contact support to turn it on, and generation shares a free daily allowance per organization with Composer that resets at 00:00 UTC. Read Generate a test suite with the API.

Testing Steps

  • fern check (5.151.1) passes: 0 errors. The 14 warnings were already on main.
  • Check the preview deployment once the endpoints are live on staging

@github-actions

Copy link
Copy Markdown
Contributor

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

scott-lowe-vapi and others added 2 commits October 18, 2026 21:45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scott-lowe-vapi
scott-lowe-vapi force-pushed the scottlowe/test-203-onboarding-api-guide branch from ef56047 to b6e5fd4 Compare October 19, 2026 04:46
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant