diff --git a/fern/composer.mdx b/fern/composer.mdx index 80f62d5a0..0d1b638d9 100644 --- a/fern/composer.mdx +++ b/fern/composer.mdx @@ -54,15 +54,16 @@ Composer understands voice agent architecture and Vapi's capabilities. It can cr - Configure integrations and webhooks - Update agent prompts and settings - Troubleshoot technical issues +- Create and run [simulations](/observability/simulations-overview), and [generate a test suite](#generate-tests-for-an-assistant) for an assistant - Answer questions about Vapi features - Recommend best practices for your use case ### Limitations -- **Cannot delete resources**: Composer cannot delete assistants, tools, phone numbers, or any other resources. This is an intentional safety measure. Use the dashboard sidebar to delete resources manually. +- **Cannot delete resources**: Composer cannot delete assistants, tools, phone numbers, or any other resources. This is an intentional safety measure. Use the dashboard sidebar to delete resources manually. Composer can discard your own unpublished assistant or tool draft, which never changes the live resource. - **Cannot access external systems**: Composer operates within Vapi and cannot reach your CRM, database, or other third-party systems directly. - **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration. -- **Cannot test your agent**: You need to test the agent yourself and provide feedback. +- **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these. - **Cannot deploy to production automatically**: You control when and how changes go live. ## Common use cases @@ -171,6 +172,37 @@ Composer understands voice agent architecture and Vapi's capabilities. It can cr +## Generate tests for an assistant + +Ask Composer to generate tests for an assistant, for example "Generate a test suite for my dental office agent." Composer reads the assistant's prompt and tools, agrees a list of testable requirements with you, then builds a [simulation suite](/observability/simulations-overview) that covers them. + + + + Composer drafts a numbered list of **Test requirements**. Each one is a behavior a simulated caller could trigger. Composer tags a requirement **Assumption** when the prompt implies it without stating it, and **Can't be simulated** when a chat simulation can't check it, such as pronunciation or the exact wording of the first message. It also asks up to three open questions when an answer would change a requirement. + + + Select **Generate coverage** to accept the draft, or type the changes you want in the chat. Composer revises the draft and asks you to review it again. It never generates a suite from requirements you haven't agreed to. + + + Composer generates the suite and shows it on a card. Each simulation covers one caller goal, and each requirement it covers becomes one of its success criteria. The card shows how many requirements are covered and why any aren't, and lists the handoffs and transfers the suite doesn't test as **Not tested**, with the reason. Every tool in the assistant's tool list that would reach an external system is [mocked by default](/observability/simulations-advanced#mock-tool-responses), and each simulation lists the tools it mocks. Tools that only a hook calls aren't mocked. + + + Select **Run suite** on the card, or ask Composer to run it. The suite runs once, in chat, against the assistant it was generated for. Composer doesn't wait for the run: progress and results appear at the run's dashboard URL. When the run ends, ask Composer about the results. + + + +After generating a suite, ask Composer to change it in the same conversation: + +- **Remove simulations**: Removed simulations are unlinked from the suite, not deleted. +- **Rewrite a simulation**: Describe the change, for example "make the caller refuse to give a date of birth." +- **Retry failed simulations**: When some simulations couldn't be generated, Composer offers to try them again. + +The requirements you agreed to are saved with the suite, and edits always work from that saved set. Revising the requirements later doesn't change an existing suite: Composer generates a new suite from the new requirements. + + +One passing run shows the assistant **can** succeed, not that it does so reliably. Run five or more iterations before you trust a pass rate: ask Composer, or [rerun the suite](/observability/simulations-manage#rerun-a-suite) with more **Iterations**. See [testing best practices](/test/simulations-best-practices). + + ## Safety features Composer includes safeguards to prevent accidental or irreversible changes to your account. You control how much Composer can do on its own with per-conversation approval modes. @@ -188,7 +220,7 @@ Approval enforcement always happens on the server, and read operations never req ### Deletion -Composer can delete resources such as assistants, tools, phone numbers, squads, and files. Deletion follows your approval mode: unless you have selected Bypass, Composer asks for your explicit approval before it deletes anything. Because deletion is permanent, keep an approval mode on if you want a confirmation step before a resource is removed. +Composer does not delete resources. Delete assistants, tools, phone numbers, squads, and files from the dashboard yourself. The approval settings include a delete category, so a policy you set now also applies if Composer gains deletion later. When Composer removes a simulation from a suite it generated, the simulation is unlinked from the suite, not deleted. ### How the approval flow works diff --git a/fern/observability/simulations-advanced.mdx b/fern/observability/simulations-advanced.mdx index 591986a63..fed724f55 100644 --- a/fern/observability/simulations-advanced.mdx +++ b/fern/observability/simulations-advanced.mdx @@ -41,6 +41,12 @@ Open a suite and select **Edit**, then **Next**. The review step contains the ** During a simulation, the [**assistant**](/assistants) or [**squad**](/squads) under test runs its real [**tools**](/tools). Mock a tool to return fixed text instead, so that tool never calls its live service or API. A mock intercepts only the tool it names; every other tool the assistant or squad calls still runs for real. + +Suites that [Composer generates](/composer#generate-tests-for-an-assistant) come with a mock already set for every tool that would reach an external system, including handoffs and transfers that can't run safely. Tools that only end or route the conversation, such as end call, voicemail, DTMF, and static handoffs, run as they would on a real call. To review or change a generated mock, edit the suite and open **Tool mocks & webhooks**: change the **Mock result**, or turn off **Enabled** to run that tool live. Suites you build yourself start with no mocks, so every tool runs for real until you add one. + +Generated mocks cover the tools in the assistant's tool list and its handoff destinations. A tool that only an assistant [**hook**](/assistants/assistant-hooks) calls isn't mocked, and the assistant's [server messages](/server-url), such as the end-of-call report, still go to its server URL. Check both before you run a generated suite against production systems. + + @@ -126,6 +132,32 @@ Return plain text to test how the assistant handles a service that is unavailabl These strings replace the selected tool's output. They do not produce an HTTP error, add network latency, or cause a network timeout. +## Handoffs and transfers in simulations + +During any simulation, a [**handoff**](/squads/handoff) that lists its destinations, none of them dynamic, runs only when the model names one of those destinations. If the model gives no destination, asks for `dynamic`, or names an assistant the handoff doesn't list, the handoff is refused: the assistant doesn't switch, and the model is asked to call the handoff again with one of its destinations. A refused handoff never asks your server for a destination. + +A [**dynamic handoff**](/squads/handoff/destinations#dynamic-handoffs) runs as it does on a live call, so it asks your server for a destination unless you [mock it](#mock-tool-responses). + +### Gaps in generated suites + +When [Composer generates a suite](/composer#generate-tests-for-an-assistant), it lists each handoff or transfer the suite doesn't exercise as **Not tested**, with the reason: + +| Reason | What the suite does | +| --- | --- | +| Your server chooses the destination (a dynamic handoff) | Mocks the handoff. | +| It transfers to a phone number or your server | Mocks the transfer. | +| The destination assistant couldn't be checked safely: it uses MCP tools or pinned versions, couldn't be loaded, or was past the number of assistants Composer reads | Mocks the handoff into it. | +| It hands off by assistant name outside a [**squad**](/squads) | Leaves the handoff live, but the path isn't reached. | +| It shares its name with a mocked tool | Mocks it too. | + +Other handoffs and transfers run as they would on a real call. Cover the paths above with test calls against sandbox systems. + +## Hooks and call endings + +Simulations keep the [**hooks**](/assistants/assistant-hooks) on the assistant or squad under test, so in a voice simulation a `call.ending` hook runs as it would on a real call. The exception is `customer.speech.timeout`: Vapi removes those hooks from the assistant and every squad member, so the assistant doesn't prompt the AI tester when it pauses. If a scenario sets `targetOverrides.hooks`, Vapi uses those hooks exactly as set, which lets you test re-engagement behavior. + +When the AI tester hangs up first in a voice simulation, for example after saying goodbye, the call to the assistant under test ends as `customer-ended-call`, the same as when a real caller hangs up. Hooks and logic that depend on a caller hang-up run as they would on a real call. + ## Send webhooks on run start and end Turn on webhooks to notify your server when a simulation iteration starts or ends. Each webhook sends a `POST` request to the URL you configure. @@ -351,6 +383,10 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario | There is no audio or recording | Check whether the run used chat mode. Use voice mode to test or record audio. | | Start or end webhooks do not trigger | Confirm the corresponding hook and its URL are configured on the scenario. | | A tool mock does not apply | Confirm that `toolName` matches the configured tool name exactly and that `enabled` is `true`. | +| A handoff is refused in the transcript | The model gave no destination, `dynamic`, or an assistant the handoff doesn't list. Check the handoff's destinations and the prompt's routing instructions. See [**Handoffs and transfers in simulations**](#handoffs-and-transfers-in-simulations). | +| A generated suite lists a handoff or transfer as **Not tested** | Composer couldn't run that path safely, so it mocked it or couldn't reach it. Cover it with a test call. See [**Gaps in generated suites**](#gaps-in-generated-suites). | +| The assistant never asks whether the AI tester is still there | Simulations remove `customer.speech.timeout` hooks. To test them, set `targetOverrides.hooks` on the scenario. See [**Hooks and call endings**](#hooks-and-call-endings). | +| A voice simulation's assistant call ends as `customer-ended-call` | The AI tester hung up first, which matches a real caller hanging up. | | A tool looks up the wrong caller | Set `customer.number` in the scenario's variable values. See [**Simulate a call from a specific phone number**](#simulate-a-call-from-a-specific-phone-number). | | A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. | | Latency limits show as skipped | Check whether the run used chat mode or the assistant under test uses GPT-Live. Both skip latency limits. | diff --git a/fern/observability/simulations-manage.mdx b/fern/observability/simulations-manage.mdx index 9bfb4c14f..720c3641e 100644 --- a/fern/observability/simulations-manage.mdx +++ b/fern/observability/simulations-manage.mdx @@ -92,6 +92,25 @@ curl -X GET "https://api.vapi.ai/eval/simulation/run/" \ The run includes `status` and `itemCounts` (`total`, `passed`, `failed`, and so on). Fetch the result for each simulation iteration with `GET /eval/simulation/run//item`. Each item's `results` holds its `evaluations` and, when the scenario sets latency limits, its `latencyEvaluations`. See [**Review latency results**](/observability/simulations-advanced#review-latency-results). +Filter the lists with query parameters: + +| Endpoint | Filters | +| --- | --- | +| `GET /eval/simulation/run` | `suiteId` (repeat for up to 100 suites), `targetType` (`assistant` or `squad`), `targetId`, and `status` (`queued`, `running`, or `ended`). `targetType` and `targetId` each work alone. | +| `GET /eval/simulation/run//item` | `status` (`queued`, `running`, `evaluating`, `passed`, `failed`, or `canceled`) and `simulationId`. | +| `GET /eval/simulation/suite` | `assistantId` and `squadId` (repeat each for up to 100 IDs; sending both requires both to match) and `name`, which matches part of the suite name. | + +```bash +curl -X GET "https://api.vapi.ai/eval/simulation/run?suiteId=&status=ended" \ + -H "Authorization: Bearer $VAPI_API_KEY" +``` + +Each list also accepts `limit` (default 100, maximum 1,000) and `page`. + + + The response shape depends on the query. With no query parameters, these endpoints return a plain array. With any query parameter, the run and run item lists return a paginated object, `{ "metadata": { ... }, "results": [ ... ] }`, with at most `limit` records. The suite list does the same when you send `limit`, `page`, `name`, `assistantId`, or `squadId`. If your integration parses the response as an array, read `results` whenever you send a query parameter. + + @@ -121,6 +140,27 @@ curl -X POST "https://api.vapi.ai/eval/simulation/run" \ Use `"provider": "vapi.websocket"` for voice. +To retry a run request without starting a second run, send an `Idempotency-Key` header with a value you generate, such as a UUID: + +```bash +curl -X POST "https://api.vapi.ai/eval/simulation/run" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -H "Idempotency-Key: " \ + -d '{ + "simulations": [{ "type": "simulationSuite", "simulationSuiteId": "" }], + "target": { "type": "assistant", "assistantId": "" }, + "transport": { "provider": "vapi.webchat" }, + "iterations": 1 + }' +``` + +- A retry with the same key and the same body returns the run the first request created, with `reused: true`. It doesn't start or bill the run again. +- Reusing a key with a different body returns `409`. Use a new key for a different run. +- If a request returns `503` with `code: SIMULATION_WORK_DISABLED`, simulations are paused across Vapi. Retry later. Any other `503` can be retried with the same key. +- A key is up to 255 printable ASCII characters and is scoped to your organization. An empty or invalid key returns `400`. +- A key stays bound to its run for as long as the run exists. It does not expire after a set time. + @@ -138,7 +178,7 @@ Create a regression test when you fix a defect or find an unexpected response. R ### Edge-case tests -Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. +Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Some handoffs and transfers [behave differently in simulations](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. ### Design evaluations diff --git a/fern/observability/simulations-overview.mdx b/fern/observability/simulations-overview.mdx index 264be58d4..987264758 100644 --- a/fern/observability/simulations-overview.mdx +++ b/fern/observability/simulations-overview.mdx @@ -89,7 +89,7 @@ Simulations test live interactions. An AI tester with a defined personality and Choose Evals when you want to lock down a specific response or verify a tool call's arguments with fast, rerunnable checks. -Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice. +Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice. Some handoffs and transfers [behave differently in simulations](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). See [testing voice agents](/test/voice-testing) for guidance on combining both tools into one coverage plan. diff --git a/fern/observability/simulations-quickstart.mdx b/fern/observability/simulations-quickstart.mdx index 511f73ffe..052f970b2 100644 --- a/fern/observability/simulations-quickstart.mdx +++ b/fern/observability/simulations-quickstart.mdx @@ -18,7 +18,7 @@ A reusable **simulation suite** that asks an AI tester to book an appointment wi You create Riley and the structured output during this quickstart. - Simulations run unmocked tools for real. Before running, inspect Riley's tools and use sandbox integrations or [mock every tool](/observability/simulations-advanced#mock-tool-responses) that could create a booking, send a message, or affect a real customer. + In a suite you build yourself, every tool without a [mock](/observability/simulations-advanced#mock-tool-responses) runs for real. Before running, inspect Riley's tools and use sandbox integrations or mock every tool that could create a booking, send a message, or affect a real customer. Suites that [Composer generates](/composer#generate-tests-for-an-assistant) mock tools that reach external systems by default. ## Build and run a simulation diff --git a/fern/test/run-and-maintain-tests.mdx b/fern/test/run-and-maintain-tests.mdx index f887e4d87..d7efcaf60 100644 --- a/fern/test/run-and-maintain-tests.mdx +++ b/fern/test/run-and-maintain-tests.mdx @@ -38,6 +38,7 @@ Rerun affected tests whenever you change prompts, tools, models, voices, transcr AI conversations can take different paths from the same starting point. One passing Simulation shows that one path worked, not that the journey is reliable. - Use one iteration while you build or debug a Simulation. +- Run five or more iterations before you trust a pass rate. - Repeat release checks based on the impact of failure and how much the result has varied before. - Use more iterations for critical journeys and intermittent failures when the extra evidence justifies the time and cost. diff --git a/fern/test/simulations-best-practices.mdx b/fern/test/simulations-best-practices.mdx index 5fc6aa2b8..38d1e514a 100644 --- a/fern/test/simulations-best-practices.mdx +++ b/fern/test/simulations-best-practices.mdx @@ -117,7 +117,7 @@ Build and debug a scenario in chat mode, then run important paths in voice mode ## Repeat critical tests and review the conversation -Agent behavior can vary between runs. One pass shows that the agent **can** succeed; repeated passes provide stronger evidence that it succeeds consistently. Use more iterations for outcomes where a failure would harm a customer or the business. +Agent behavior can vary between runs. One pass shows that the agent **can** succeed; repeated passes provide stronger evidence that it succeeds consistently. Use one iteration while you set up a test, then run five or more iterations before you trust a pass rate. Use more iterations for outcomes where a failure would harm a customer or the business. Read every failed transcript and sample passed transcripts. Ask: diff --git a/fern/test/voice-testing.mdx b/fern/test/voice-testing.mdx index 2d978bcac..30256727d 100644 --- a/fern/test/voice-testing.mdx +++ b/fern/test/voice-testing.mdx @@ -47,7 +47,7 @@ voice agents need both. | **Core question** | At this point in the conversation, did the agent make the right next decision? | By the end of the conversation, did the agent reach the right outcome? | | **Scope** | Best kept focused on one or a few decision checkpoints, with a known conversation history | A complete conversation with an AI tester | | **Conversation path** | Fixed context; checks what happens next | Flexible; the tester adapts as the conversation unfolds | -| **Best for** | Tool selection and arguments, routing, refusals, required questions, and specific responses | Task completion, recovery from problems, handoffs, caller behavior, and the overall experience | +| **Best for** | Tool selection and arguments, routing, refusals, required questions, and specific responses | Task completion, recovery from problems, handoffs ([with limits](/observability/simulations-advanced#handoffs-and-transfers-in-simulations)), caller behavior, and the overall experience | | **Mode** | Text-based mock conversation | Chat or voice | For example, use an **Eval** to check that an agent calls the rescheduling tool only after it verifies the caller. Use a **Simulation** to check that a caller can reschedule, hear the correct new time, and finish the conversation successfully. The Simulation should pass even if the agent asks the questions in a different valid order.