From 5309b88ed82495754a8d148e682c7e8d7b2bfe07 Mon Sep 17 00:00:00 2001 From: Scott Lowe Date: Sun, 18 Oct 2026 21:26:22 -0700 Subject: [PATCH 1/2] TEST-203 docs: add the Simulations onboarding API guide Co-Authored-By: Claude Opus 5.5 --- fern/composer.mdx | 1 + fern/docs.yml | 3 + fern/observability/simulations-advanced.mdx | 2 + .../simulations-generate-api.mdx | 371 ++++++++++++++++++ 4 files changed, 377 insertions(+) create mode 100644 fern/observability/simulations-generate-api.mdx diff --git a/fern/composer.mdx b/fern/composer.mdx index 0d1b638d9..728ae632c 100644 --- a/fern/composer.mdx +++ b/fern/composer.mdx @@ -65,6 +65,7 @@ Composer understands voice agent architecture and Vapi's capabilities. It can cr - **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration. - **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these. - **Cannot deploy to production automatically**: You control when and how changes go live. +- **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC. ## Common use cases diff --git a/fern/docs.yml b/fern/docs.yml index a63911bab..1dcd3a945 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -692,6 +692,9 @@ navigation: - page: Advanced path: observability/simulations-advanced.mdx icon: fa-light fa-flask-vial + - page: Generate with the API + path: observability/simulations-generate-api.mdx + icon: fa-light fa-wand-magic-sparkles - page: Configure an AI tester path: observability/simulations-configure-ai-tester.mdx icon: fa-light fa-robot diff --git a/fern/observability/simulations-advanced.mdx b/fern/observability/simulations-advanced.mdx index fed724f55..db8d65aff 100644 --- a/fern/observability/simulations-advanced.mdx +++ b/fern/observability/simulations-advanced.mdx @@ -379,6 +379,8 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario | --- | --- | | Every iteration fails | Review the transcript, then verify the evaluation's description, data type, comparator, and expected value. | | A run stays queued or running | Check the assistant or squad configuration and its provider credentials. Fetch the run and its items to find a failure reason. | +| Creating a voice run returns `400 Target assistant would fail to start a call: ` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. | +| A run item fails with `Failed to create target call: ` | The assistant or squad under test failed call validation when the run item started its call. Fix what the message names, then rerun. No AI tester call is started for that item. | | Results vary between runs | Make the intent and evaluations more specific, then run multiple iterations to measure consistency. | | There is no audio or recording | Check whether the run used chat mode. Use voice mode to test or record audio. | | Start or end webhooks do not trigger | Confirm the corresponding hook and its URL are configured on the scenario. | diff --git a/fern/observability/simulations-generate-api.mdx b/fern/observability/simulations-generate-api.mdx new file mode 100644 index 000000000..5b69beae8 --- /dev/null +++ b/fern/observability/simulations-generate-api.mdx @@ -0,0 +1,371 @@ +--- +title: Generate a test suite with the API +subtitle: Draft requirements for an assistant, generate a simulation suite from them, edit it, and run it +description: "Use the Simulations onboarding API to draft test requirements from an assistant's prompt and tools, generate a simulation suite with mocked tools, edit it, and run it." +slug: observability/simulations-generate-api +--- + +The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It's the same flow [**Composer**](/composer#generate-tests-for-an-assistant) runs when you ask it to generate tests, exposed as three endpoints you can call from a script, a CI job, or a coding agent: + +| Step | Endpoint | Takes up to | +| -- | -- | -- | +| Draft requirements | `POST /eval/simulation/onboarding/requirements` | 2 minutes | +| Generate a suite | `POST /eval/simulation/onboarding/suite` | 3 minutes | +| Edit the suite | `PATCH /eval/simulation/onboarding/suite/{suiteId}` | 2 minutes | + +You then run the suite with the existing [`POST /eval/simulation/run`](/observability/simulations-quickstart) endpoint. + + + **Beta.** Private API keys can call these endpoints only for organizations enabled during the beta. For any other organization they return `404`. [Contact support](/support) to enable your organization. Composer and the Dashboard work without this. + + +## Prerequisites + +- An [**assistant**](/assistants) to test. Copy its ID from its page in the Dashboard. +- A private API key in `VAPI_API_KEY`. The key needs permission to create and update simulation suites. +- An HTTP client with a long timeout. Suite generation can take up to 3 minutes, so set a client timeout of at least 200 seconds. + + + Generated simulations mock every tool that would reach an external system, and the response lists them in `mockedTools`. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). + + +## Generate and run a suite + + + + Send the assistant ID. Vapi reads the assistant's prompt, tools, and handoffs, and drafts the behaviors it must get right as numbered requirements, plus a few open questions. + + ```bash + curl -sS -X POST "https://api.vapi.ai/eval/simulation/onboarding/requirements" \ + --max-time 200 \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ "assistantId": "" }' \ + > draft.json + ``` + + The response is a draft: + + ```json + { + "version": 1, + "sources": ["System prompt", "Tools: bookAppointment, checkAvailability"], + "requirements": [ + { "id": "R1", "text": "Confirms the date, time, and time zone before booking." }, + { "id": "R2", "text": "Offers another slot when the requested time is unavailable." }, + { + "id": "R3", + "text": "Does not book outside office hours.", + "tag": "assumption", + "note": "Office hours are not in the prompt; assumed Monday to Friday, 9am to 5pm." + } + ], + "openQuestions": [ + { + "id": "Q1", + "text": "What are the office hours?", + "why": "R3 depends on them." + } + ] + } + ``` + + Review the requirements and answer the open questions. A requirement tagged `assumption` rests on a guess; one tagged `not-testable` describes something a chat simulation can't check, such as pronunciation. + + + + To change the draft, call the same endpoint again. Send the draft exactly as you received it in `previousDraft`, and what you want changed in `feedback`. The two fields must be sent together. + + ```bash + jq -n --slurpfile draft draft.json '{ + assistantId: "", + previousDraft: $draft[0], + feedback: "Office hours are Monday to Friday, 8am to 6pm. Drop R2: we never offer another slot." + }' > revise.json + + curl -sS -X POST "https://api.vapi.ai/eval/simulation/onboarding/requirements" \ + --max-time 200 \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d @revise.json \ + > draft.json + ``` + + The new draft has a higher `version` and a `changeNote` describing what changed. Repeat until you agree with the requirements. Each revision counts toward the [daily limit](#limits). + + + + Send the agreed requirements and an `idempotencyKey`. Use one key per agreed draft, and reuse it for every retry of that draft. The key makes retries safe: a retry with the same key never generates a second suite. See [**Idempotency and retries**](#idempotency-and-retries). + + ```bash + jq -n --slurpfile draft draft.json '{ + assistantId: "", + idempotencyKey: "booking-agent-draft-2", + requirements: $draft[0].requirements + }' > suite.json + + curl -sS -X POST "https://api.vapi.ai/eval/simulation/onboarding/suite" \ + --max-time 200 \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d @suite.json + ``` + + Vapi plans scenarios that cover the requirements, writes each scenario and its success criteria, mocks the tools, then saves the suite and assigns it to the assistant. The response describes the saved suite: + + ```json + { + "suite": { "id": "", "name": "Riley: core behaviours" }, + "assistantId": "", + "simulations": [ + { + "id": "", + "scenarioId": "", + "name": "Books a slot inside office hours", + "personalityName": "Skeptical Sam", + "successCriteria": ["Did the assistant confirm the date, time, and time zone before booking?"], + "requirementIds": ["R1", "R3"], + "mockedTools": ["bookAppointment", "checkAvailability"] + } + ], + "coverage": { + "covered": 2, + "total": 3, + "gaps": [ + { "requirementId": "R2", "reason": "budget", "detail": "It did not fit in this generation." } + ], + "retryable": 1 + }, + "requirements": [{ "id": "R1", "text": "Confirms the date, time, and time zone before booking." }], + "docs": { "bestPractices": "https://docs.vapi.ai/...", "toolMocks": "https://docs.vapi.ai/..." }, + "reused": false + } + ``` + + Check `coverage.gaps` for requirements no simulation covers, and `toolGaps` for tool paths the simulations don't exercise. See [**Coverage gaps**](#coverage-gaps) and [**Tool gaps**](#tool-gaps). + + + + Edit the suite with a new `patchKey` for each edit. In one request, you can: + + - Remove simulations with `removeSimulationIds`. They are unlinked from the suite, not deleted. + - Rewrite up to 10 simulations with `edits`, each an instruction in your own words. Put every change for a simulation in one instruction. + - Regenerate the scenarios that failed or did not fit with `retryFailed: true`. `coverage.retryable` is the number it would regenerate. + + ```bash + curl -sS -X PATCH "https://api.vapi.ai/eval/simulation/onboarding/suite/" \ + --max-time 200 \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "patchKey": "booking-agent-edit-1", + "edits": [ + { + "simulationId": "", + "instruction": "Make the caller ask for 7pm first, then accept 5pm." + } + ], + "retryFailed": true + }' + ``` + + The response is the updated suite. If an edit could not be applied, for example because the rewrite did not generate in time, its simulation ID is listed in `failedEditSimulationIds`. Retry those with a new `patchKey`. + + + + Run the suite against the assistant with [`POST /eval/simulation/run`](/observability/simulations-quickstart). Send an `Idempotency-Key` header so a retried request returns the run the first request created instead of starting another. + + ```bash + curl -sS -X POST "https://api.vapi.ai/eval/simulation/run" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -H "Idempotency-Key: booking-agent-run-1" \ + -d '{ + "simulations": [{ "type": "simulationSuite", "simulationSuiteId": "" }], + "target": { "type": "assistant", "assistantId": "" }, + "transport": { "provider": "vapi.webchat" }, + "iterations": 5 + }' + ``` + + Poll `GET /eval/simulation/run/` until `status` is `ended`, as in the [**Simulations quickstart**](/observability/simulations-quickstart). Run five or more iterations before you trust a pass rate, and read the failed transcripts. See [**Test outcomes with Simulations**](/test/simulations-best-practices). + + + +## Request fields + +### Draft requirements + +| Field | Type | Description | +| -- | -- | -- | +| `assistantId` | string (UUID) | **Required.** The assistant to draft requirements for. | +| `previousDraft` | object | The draft being revised, exactly as this endpoint returned it. Send it with `feedback`. | +| `feedback` | string | What you want changed in `previousDraft`, up to 8,000 characters. Send it with `previousDraft`. | + +### Generate a suite + +| Field | Type | Description | +| -- | -- | -- | +| `assistantId` | string (UUID) | **Required.** The assistant the suite tests and is assigned to. | +| `idempotencyKey` | string | **Required.** 1 to 200 characters. One key per agreed draft. | +| `requirements` | array | **Required.** 1 to 50 requirements from the draft, each with a unique `id`. Pass them as the draft returned them. | + +### Edit a suite + +`suiteId` in the path must be a suite generated by `POST /eval/simulation/onboarding/suite` (or by Composer) in your organization. + +| Field | Type | Description | +| -- | -- | -- | +| `patchKey` | string | **Required.** 1 to 200 characters. A new key for each edit. | +| `removeSimulationIds` | array of UUIDs | Simulations to unlink from the suite, up to 100. | +| `edits` | array | Up to 10 `{ simulationId, instruction }` objects, at most one per simulation. `instruction` is up to 8,000 characters. | +| `retryFailed` | boolean | Regenerate the scenarios that failed or did not fit when the suite was generated. | + +## Response fields + +### Requirements draft + +| Field | Description | +| -- | -- | +| `version` | Draft version, `1` for a first draft. | +| `changeNote` | What changed since the previous draft. | +| `sources` | What the draft was based on, such as the prompt and tools. | +| `requirements` | Up to 50 requirements. Each has an `id` (such as `R3`), `text` (one observable behavior), and optionally a `tag` (`assumption` or `not-testable`) and a `note` explaining what was assumed or why it can't be tested. | +| `openQuestions` | Up to 3 questions to answer before you agree. Each has an `id` (such as `Q1`), `text`, and `why` (which requirement the answer changes, and how). | + +### Suite + +Generating and editing a suite both return this object. + +| Field | Description | +| -- | -- | +| `suite` | The suite's `id` and `name`. Pass the `id` to edit or run the suite. | +| `assistantId` | The assistant the suite is assigned to. | +| `simulations` | The suite's simulations, in order. Each has an `id`, `scenarioId`, `name`, optional `personalityName`, `successCriteria` (each as the yes/no question it is judged by), `requirementIds` (the requirements it covers), and `mockedTools`. | +| `coverage` | `covered` and `total` requirements, `gaps` (requirements no simulation covers, each with a `requirementId`, `reason`, and optional `detail`), and `retryable` (planned scenarios an edit with `retryFailed` would regenerate). | +| `requirements` | The agreed requirements the suite was generated from. | +| `docs` | `bestPractices` and `toolMocks` documentation links. | +| `reused` | `true` when the request matched one already applied (the same `idempotencyKey` or `patchKey`) and nothing changed. | +| `toolGaps` | Tool paths the simulations don't exercise as configured. Each has a `toolName`, an `assistantName` when the tool belongs to a handoff destination, a `reason`, and optionally the `destinationNames` the simulations don't reach. Omitted when there are none. | +| `failedEditSimulationIds` | Edit responses only. Simulations an edit asked to rewrite that were not rewritten. | + +## Coverage gaps + +A requirement with no simulation covering it appears in `coverage.gaps` with one of these reasons: + +| Reason | What it means | What to do | +| -- | -- | -- | +| `not-testable` | The requirement is tagged as something a chat simulation can't check, such as pronunciation or latency. | Test it another way, for example with a [voice run](/test/voice-testing). | +| `unmapped` | No planned scenario exercises the requirement, or its simulation was removed from the suite. | Reword the requirement as an observable behavior and generate a new suite, or add a simulation by hand. | +| `failed` | Its scenario could not be generated or saved in time. | Edit the suite with `retryFailed: true`. | +| `budget` | It didn't fit in one generation's time budget. | Edit the suite with `retryFailed: true`. | + +## Tool gaps + +`toolGaps` lists tool paths the simulations don't exercise as configured. Most are handoffs or transfers that Vapi mocks because running them could reach your server, a phone number, or a destination it couldn't check safely. A mocked handoff or transfer answers as if it went through, so the conversation continues past it, but the destination isn't tested. New reasons may be added. + +| Reason | What it means | +| -- | -- | +| `dynamic-handoff` | Your server chooses the handoff destination at call time, so the handoff is mocked. | +| `destination-not-plain` | The destination assistant uses MCP tools, a pinned assistant version, or other configuration that generation doesn't resolve, so the handoff into it is mocked. | +| `destination-unresolved` | The destination assistant couldn't be loaded, so the handoff into it is mocked. | +| `destination-truncated` | The destination wasn't checked because the assistant has too many handoff destinations to check in time, or it didn't load in time. The handoff into it is mocked. | +| `transfer-external` | A `transferCall` can leave Vapi, to a phone number, a SIP address, or a destination your server chooses, so it is mocked. | +| `destination-name-unscoped` | A handoff names a destination assistant outside a squad, which can't be resolved at runtime, so that destination is never reached. The handoff isn't mocked. | +| `native-tool-shadowed` | A tool that normally runs as is (such as `endCall` or `transferCall`) shares its name with a mocked tool, so the mock answers it too. | + +To test a mocked path, run a separate simulation against the destination assistant, or replace the mock with a sandbox integration. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). + +## Idempotency and retries + +**Suite generation.** Send one `idempotencyKey` per agreed draft, and retry with the same key: + +- **On a timeout or a `502`, `504`, or `524`**, retry with the same key. You get the saved suite, a `409` while it is still generating, or a fresh generation if the first one never saved. +- **On a `409` with `SIMULATION_ONBOARDING_GENERATION_IN_PROGRESS`**, the first request is still generating. Wait `Retry-After` seconds (15) and retry with the same key until you get the suite. +- **On a `409` with `SIMULATION_ONBOARDING_IDEMPOTENCY_KEY_REUSED`**, the key was already used for a different assistant or different requirements. Don't retry with it; send a new key. + +A retry that returns a saved suite sets `reused` to `true` and doesn't count toward your limits. To generate a new suite from a revised draft, use a new key. + +**Edits.** Send a new `patchKey` for each edit. Replaying a key that was already applied changes nothing and returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. So you can retry a timeout or a `5xx` with the same key. A suite remembers its last 50 patch keys, so never reuse one. + +**Drafts.** Drafting has no key. Retrying a failed draft drafts again and counts toward the limit. + +## Limits + +Each organization has a free daily allowance per UTC day, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC. + +| Operation | Daily limit per organization | +| -- | -- | +| Requirement drafts and revisions | 60 | +| Suite generations | 20 | +| Suite edits | 60 | + +An organization can also have at most **2 suite generations** running at once. + +These don't count toward the daily limits: + +- A retry that returns a saved suite, or a replayed `patchKey`. +- An edit that only removes simulations, or `retryFailed` when nothing is left to retry. +- A request refused before generation starts, such as a `404` or a `422`. + +A request counts once generation starts, even if it later fails with a `502` or times out. + +{/* TODO verify: whether these limits apply to Enterprise orgs or can be raised on request. The source only says "free per-org quota". */} + +When you reach a limit, the API returns `429`: + +| Code | Meaning | What to do | +| -- | -- | -- | +| `SIMULATION_ONBOARDING_QUOTA_EXCEEDED` | The organization used its daily allowance for this operation. | Don't retry. The limit resets at 00:00 UTC; `Retry-After` is the number of seconds until then. | +| `SIMULATION_ONBOARDING_CONCURRENCY_LIMITED` | The organization already has 2 suites generating. | Wait `Retry-After` seconds (30), then retry. | + +## Errors + +The `409` and `429` responses carry a JSON body and a `Retry-After` header when waiting helps: + +```json +{ + "code": "SIMULATION_ONBOARDING_QUOTA_EXCEEDED", + "message": "This org has used its 20 free suite generations for today. The limit resets at 00:00 UTC.", + "retryAfterSeconds": 41230, + "limit": 20 +} +``` + +| Field | Description | +| -- | -- | +| `code` | Which refusal this is. Branch on it. | +| `message` | What happened, written for the user. | +| `retryAfterSeconds` | Seconds to wait before retrying, the same as `Retry-After`. For a daily limit, the seconds until 00:00 UTC. | +| `limit` | The daily limit that was reached, on a daily-limit refusal only. | + +| Status | When | What to do | +| -- | -- | -- | +| `400` | The body is invalid; `previousDraft` and `feedback` were not sent together; or the generated or edited suite uses something your organization's data-region policy doesn't allow, so it isn't saved. | Fix the request. | +| `401` | The API key is missing or invalid. | Use a valid private API key. | +| `403` | The key can't create or edit simulation suites. | Use a key with that permission. | +| `404` | The assistant wasn't found; the suite isn't a generated suite in your organization; or your organization doesn't have API access yet. | Check the ID, or [contact support](/support) to enable access. | +| `409` | `SIMULATION_ONBOARDING_GENERATION_IN_PROGRESS` or `SIMULATION_ONBOARDING_IDEMPOTENCY_KEY_REUSED`. Suite generation only. | See [**Idempotency and retries**](#idempotency-and-retries). | +| `422` | None of the requirements can be checked in a chat simulation (generation only), or an MCP server's tools couldn't be listed, so simulations would call them for real instead of a mock (generation and edits). | Add a testable requirement, or check that the MCP server is reachable, then retry. | +| `429` | A daily limit or the concurrent-generation limit was reached. | See [**Limits**](#limits). | +| `500` | Vapi couldn't read the assistant, its tools, or the suite. | Retry. For generation and edits, retry with the same key. | +| `502` | The model failed to generate a usable result. | Retry. For generation and edits, retry with the same key. | +| `503` | Simulation generation is paused, or no caller personalities are configured. | Retry later. | +| `504` | The request took too long. | Retry. For generation and edits, retry with the same key: a generation may still save. | + +## Next steps + + + + Generate the same suite through a conversation in the Dashboard. + + + Review and change the mocks a generated suite uses. + + + Write observable criteria and repeat critical tests. + + + Review and rerun results, and maintain the suite over time. + + From b6e5fd43787c2647ad8cad62a2b4cbd91313837b Mon Sep 17 00:00:00 2001 From: Scott Lowe Date: Sun, 18 Oct 2026 21:45:44 -0700 Subject: [PATCH 2/2] TEST-203 docs: note unmocked hook tools in the API guide Co-Authored-By: Claude Opus 5.5 --- fern/observability/simulations-generate-api.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/observability/simulations-generate-api.mdx b/fern/observability/simulations-generate-api.mdx index 5b69beae8..4a117b06e 100644 --- a/fern/observability/simulations-generate-api.mdx +++ b/fern/observability/simulations-generate-api.mdx @@ -26,7 +26,7 @@ You then run the suite with the existing [`POST /eval/simulation/run`](/observab - An HTTP client with a long timeout. Suite generation can take up to 3 minutes, so set a client timeout of at least 200 seconds. - Generated simulations mock every tool that would reach an external system, and the response lists them in `mockedTools`. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). + Generated simulations mock every tool in the assistant's tool list that would reach an external system, and the response lists them in `mockedTools`. Tools that only an assistant hook calls aren't mocked, and the assistant's server messages, such as the end-of-call report, still go to its server URL. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). ## Generate and run a suite