diff --git a/fern/docs.yml b/fern/docs.yml index a63911bab..38c0f3e9e 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -689,6 +689,9 @@ navigation: - page: Quickstart path: observability/simulations-quickstart.mdx icon: fa-light fa-rocket + - page: Test an assistant + path: observability/simulations-test-tab.mdx + icon: fa-light fa-vial - page: Advanced path: observability/simulations-advanced.mdx icon: fa-light fa-flask-vial diff --git a/fern/observability/simulations-quickstart.mdx b/fern/observability/simulations-quickstart.mdx index 052f970b2..9aea02da5 100644 --- a/fern/observability/simulations-quickstart.mdx +++ b/fern/observability/simulations-quickstart.mdx @@ -7,6 +7,8 @@ slug: observability/simulations-quickstart In about 10 minutes, you'll run an AI tester through an appointment-booking conversation with Riley and get a pass or fail result. You'll also have a reusable simulation suite that you can run again after any change. +Testing a single assistant? Start from its [**Test tab**](/observability/simulations-test-tab), where Composer can generate a suite for you. + ## What you'll build A reusable **simulation suite** that asks an AI tester to book an appointment with Riley, Vapi's appointment-scheduler template assistant. The first criterion checks whether Riley reports the requested date and time with a confirmation number. This is a conversation-only demonstration, not proof that an appointment exists in a calendar. diff --git a/fern/observability/simulations-test-tab.mdx b/fern/observability/simulations-test-tab.mdx new file mode 100644 index 000000000..bc88b880c --- /dev/null +++ b/fern/observability/simulations-test-tab.mdx @@ -0,0 +1,89 @@ +--- +title: Test an assistant +subtitle: Generate, run, and track simulation suites from an assistant's Test tab +description: "Use an assistant's Test tab to generate a simulation suite with Composer, run it, and follow its latest results and run history without leaving the assistant." +slug: observability/simulations-test-tab +--- + +The **Test** tab on an assistant lists the [**simulation suites**](/observability/simulations-overview) assigned to it. From the tab, you can generate a suite with Composer, run or rerun a suite, and open its results and run history without leaving the assistant. + +## Prerequisites + +- An [**assistant**](/assistants) in your Vapi account. +- Permission to read test suites and test runs. Without both, the **Test** tab does not appear. +- Permission to create test suites to generate a suite, and permission to create test runs to run one. + + + Runs use the published version of the assistant. If you're editing a draft, the tab shows **Runs use the published version**. Publish your changes to test them. If [**traffic splitting**](/assistants/versioning/traffic-splitting) is enabled, publish the new version with 0% traffic so it handles no live calls. + + +## Generate a suite with Composer + +Open the assistant and select the **Test** tab. If no suites are assigned to the assistant, the tab shows **No test suites yet** with **Generate with Composer** and an **About simulations** link. Once the assistant has suites, select **Generate suite** above the table to create another. + +{/* Screenshot: Test tab empty state with "No test suites yet" and the Generate with Composer button */} + +Both buttons open [**Composer**](/composer#generate-tests-for-an-assistant) in a new thread for this assistant: + + + + Composer reads the assistant's prompt and tools and drafts **Test requirements**. Select **Generate coverage** to accept them, or type the changes you want in the chat. + + + + Composer generates simulations from the requirements and saves them as a suite assigned to the assistant. The suite card shows **Saved to the Test tab**. Tools that reach your systems are mocked by default, and each simulation lists any tools it mocks. + + + + Select **Run suite** on the card to start a chat run with one iteration. The suite also appears in the **Test** tab. + + + +## Read the suite table + +The table shows up to 100 suites assigned to the assistant, one row per suite, with the latest run of that suite against this assistant. + +{/* Screenshot: Test tab suite table with Status, Progress, and Last run columns */} + +| Column | What it shows | +| -- | -- | +| **Test suites assigned** | The suite name. Once the suite has run, select the row to open its latest run. | +| **Simulations** | How many simulations the latest run executed, with its chat or voice mode. Before the first run, the number of simulations in the suite. | +| **Status** | The latest run's status: **Queued**, **Running**, **Passed**, **Failed**, **Errored**, or **Canceled**. A suite that has never run for this assistant shows **Not run yet**. | +| **Progress** | A bar of the latest run's outcomes. During a run, it shows how many simulations are complete, such as `2 of 5 complete`. After the run, it summarizes the outcomes, such as `4 passed ยท 1 failed or errored`. | +| **Last run** | When the latest run ended, such as `5 minutes ago`. It updates every minute while the tab is open. | + +Each row ends with a **Run** or **Rerun** button. Once a suite has run, the row also shows a run history button that opens **Simulations** > **Runs** filtered to that suite and this assistant. To see every run for the assistant, select **View run history** above the table. + +## Run a suite + +Select **Run** on a suite that has never run for this assistant. If the suite can run in chat, it starts immediately as a chat run with one iteration, and the row updates as it progresses. + +The **Run suite** dialog opens instead, so you can choose the mode and iterations, when: + +- The assistant needs voice, such as one that uses a GPT Live model. **Voice** is the only mode offered. +- One of the suite's simulations is missing. +- A scenario has hooks configured. + +To run a suite again, select **Rerun**. The dialog always opens so you can choose **Chat** or **Voice** and the number of **Iterations**. It starts with the mode of the previous run. Each rerun creates a new run and picks up any changes made to the suite. + + + A suite with no simulations can't run. Add at least one simulation before running it. Run and Rerun stay disabled while a run of the suite is queued or running. + + +## Next steps + + + + Build and run a simulation suite step by step. + + + Edit suites, review results, and maintain coverage. + + + Control what tools return during a simulation. + + + Generate tests for an assistant in conversation. + +