From b83d68f7aa4429d7798fb993feb82f77feaf837c Mon Sep 17 00:00:00 2001 From: Herve Labas Date: Thu, 1 Oct 2026 18:30:40 +0200 Subject: [PATCH] docs: add a Hobby path to the Agentic Check migration guide [AI-557] Hobby accounts don't include the AI Assistant, so they replace Agentic Checks with their own AI agent connected to the Checkly MCP Server. The guide now covers both paths, uses the in-product Copy prompt and Open AI assistant buttons, and tells CLI-managed projects to replace the construct in code. AI Assistant conversations now use one AI invocation when they first run, so the usage notes on the migration, automatic repair and Rocky AI pages say so. Co-Authored-By: Claude Opus 5.5 --- ai/rocky-ai.mdx | 2 +- .../agentic-checks/migrate.mdx | 44 +++++++++++++++---- resolve/automatic-check-repair.mdx | 2 +- 3 files changed, 37 insertions(+), 11 deletions(-) diff --git a/ai/rocky-ai.mdx b/ai/rocky-ai.mdx index 6bff2b7a..8ac38f57 100644 --- a/ai/rocky-ai.mdx +++ b/ai/rocky-ai.mdx @@ -27,7 +27,7 @@ Set an intent and enable [automatic repair](../resolve/automatic-check-repair) f ## Consumption and Invocations -Every root cause analysis — whether triggered automatically or manually — counts as one invocation against your plan's Resolve quota. If the same analysis result is pushed to multiple alert channels, it is still counted as a single invocation. An automatic check repair counts as one additional invocation. AI Assistant conversations and **Initialize from saved check** do not currently count against this allowance. +Every root cause analysis — whether triggered automatically or manually — counts as one invocation against your plan's Resolve quota. If the same analysis result is pushed to multiple alert channels, it is still counted as a single invocation. An automatic check repair counts as one additional invocation. Each AI Assistant conversation counts as one invocation when it first runs. **Initialize from saved check** does not currently count against this allowance. When your invocation quota is reached, no overages are charged. Invocations refresh at the start of your next billing cycle. You can monitor your current usage and configure automatic analysis in the [AI settings](https://app.checklyhq.com/accounts/rocky-ai). diff --git a/detect/synthetic-monitoring/agentic-checks/migrate.mdx b/detect/synthetic-monitoring/agentic-checks/migrate.mdx index 1dc0f1bc..22e7359b 100644 --- a/detect/synthetic-monitoring/agentic-checks/migrate.mdx +++ b/detect/synthetic-monitoring/agentic-checks/migrate.mdx @@ -10,24 +10,50 @@ hidden: true Agentic Checks are deprecated and will stop running on **December 1, 2026**. Replace each active Agentic Check before that date to keep monitoring its journey. -Create a Browser or Multistep Check for the same journey, verify the replacement, then deactivate the Agentic Check. The AI Assistant can create other check and monitor types too; this guide uses Browser and Multistep Checks because automatic repair can edit their scripts in Checkly. The replacement has its own results and history. +Create a Browser or Multistep Check for the same journey, verify the replacement, then deactivate the Agentic Check. This guide uses Browser and Multistep Checks because automatic repair can edit their scripts in Checkly. The replacement has its own results and history. + +How you create the replacement depends on your plan: + +- **Trial, Starter, Team, and Enterprise:** use the [AI Assistant](../../../ai/ai-assistant) in Checkly. +- **Hobby:** use your own AI agent connected to the [Checkly MCP Server](../../../ai/mcp-server). -- A Trial, Starter, Team, or Enterprise account with access to the [AI Assistant](../../../ai/ai-assistant) - Permission to create and edit checks - Access to each Agentic Check's prompt, schedule, locations, alert settings, and any variables or secrets it uses +- For the AI Assistant: Rocky AI enabled for the account. An account owner can enable it in the [AI settings](https://app.checklyhq.com/accounts/rocky-ai). +- For your own agent: an MCP client connected to the Checkly MCP Server. See [Set up the MCP server](../../../ai/mcp-server/setup). -## Replace an Agentic Check + +If a Checkly CLI project manages the Agentic Check, create the replacement in that project instead: replace the `AgenticCheck` construct with a `BrowserCheck` or `MultiStepCheck` construct and deploy. [Checkly Skills](../../../ai/skills) help your agent write and test the check. Changes made in the Checkly app or through the MCP Server to a check that a CLI project manages are overwritten by its next deploy. + + +## Create the replacement with the AI Assistant -1. Open the Agentic Check and copy its prompt. Note its frequency, locations, alert channels, referenced variables or secrets, and any custom skills. -2. Open **AI Assistant**, select **New**, and ask it to create a check for the same journey. Give it the original prompt and the pass and fail criteria. Choose a **Browser Check** for a web journey or a **Multistep Check** for an API workflow. +1. Open the Agentic Check and select **Copy prompt**. Note its frequency, locations, alert channels, referenced variables or secrets, and any custom skills. +2. Select **Open AI assistant**, paste the prompt, and ask for a check for the same journey. Include the pass and fail criteria. Choose a **Browser Check** for a web journey or a **Multistep Check** for an API workflow. 3. Review the proposed plan and select **Create checks**. Inspect the generated script, assertions, settings, and check intent. Confirm that important outcomes and guardrails from the original prompt are present. If the Agentic Check used custom skills, verify that the replacement script covers the behavior you still need. -4. Run the replacement check and confirm it reports the expected result. Correct the script or settings if needed. Match the original check's frequency, locations, and alerting before relying on the replacement. -5. In the replacement check's **Automatic Repair** settings, review its intent, select **On** under **Automatic repair behavior**, and save. [Learn how automatic repair works](../../../resolve/automatic-check-repair). -6. Once the replacement is running as expected, deactivate the Agentic Check. Keep it for its existing configuration and history, or remove it later according to your retention needs. + +## Create the replacement with your own AI agent + +1. Open the Agentic Check and select **Copy prompt**. Note its frequency, locations, alert channels, referenced variables or secrets, and any custom skills. +2. In your MCP client, ask your agent to create the replacement through the Checkly MCP Server. Give it the copied prompt, the check type, and the original schedule and locations: + +```text title="Prompt" wrap +Use Checkly to create a Browser Check named "Checkout flow" that monitors the journey below. Run it every 10 minutes from eu-west-1 and us-east-1. The prompt references variables as {{VARIABLE_NAME}}; read them in the script with process.env.VARIABLE_NAME. Show me the script before you create the check. + + +``` + +3. Open the new check in Checkly and inspect its script, assertions, and settings. Confirm that important outcomes and guardrails from the original prompt are present. If the Agentic Check used custom skills, verify that the replacement script covers the behavior you still need. + +## Verify and switch over + +1. Run the replacement check and confirm it reports the expected result. Correct the script or settings if needed. Match the original check's frequency, locations, and alerting before relying on the replacement. +2. In the replacement check's **Automatic Repair** settings, review its intent, select **On** under **Automatic repair behavior**, and save. If the check has no intent yet, select **Initialize from saved check** first. [Learn how automatic repair works](../../../resolve/automatic-check-repair). +3. Once the replacement is running as expected, deactivate the Agentic Check. Keep it for its existing configuration and history, or remove it later according to your retention needs. Migrate one journey at a time. If the original prompt covered several unrelated flows, create a separate replacement check for each flow and verify each one before deactivating the Agentic Check. @@ -39,4 +65,4 @@ You can enable automatic repair on Browser and Multistep Checks you already use. ## AI invocation usage -AI Assistant conversations and **Initialize from saved check** currently do not consume your Resolve AI invocation allowance. Root cause analysis and a subsequent automatic repair each consume an invocation. See [AI invocation usage](../../../resolve/automatic-check-repair#ai-invocation-usage). +Each AI Assistant conversation uses one AI invocation from your plan's allowance when it first runs. Root cause analysis and a subsequent automatic repair each use an invocation. **Initialize from saved check** does not currently use one, and neither does creating checks through the MCP Server. See [AI invocation usage](../../../resolve/automatic-check-repair#ai-invocation-usage). diff --git a/resolve/automatic-check-repair.mdx b/resolve/automatic-check-repair.mdx index 10236a0b..8329d089 100644 --- a/resolve/automatic-check-repair.mdx +++ b/resolve/automatic-check-repair.mdx @@ -34,4 +34,4 @@ You can also write the intent yourself. Use this option when the script does not ## AI invocation usage -Each new root cause analysis uses one AI invocation. If Checkly also starts a repair, that repair uses one additional invocation from the same Resolve allowance. Each click on **Initialize from saved check** requests a new AI suggestion, but it does not currently draw from that allowance. AI Assistant conversations do not draw from it either. See [Rocky AI consumption and invocations](../ai/rocky-ai#consumption-and-invocations). +Each new root cause analysis uses one AI invocation. If Checkly also starts a repair, that repair uses one additional invocation from the same Resolve allowance. Each click on **Initialize from saved check** requests a new AI suggestion, but it does not currently draw from that allowance. Each AI Assistant conversation uses one invocation from the same allowance when it first runs. See [Rocky AI consumption and invocations](../ai/rocky-ai#consumption-and-invocations).