diff --git a/docs/admin/config/site-config.mdx b/docs/admin/config/site-config.mdx index a716ce546..7f0979088 100644 --- a/docs/admin/config/site-config.mdx +++ b/docs/admin/config/site-config.mdx @@ -140,6 +140,20 @@ All advanced configuration options and their default values are shown below. // Maximum number of batch spec templates to display in the template library UI. Default is 20. "batchChanges.templateLibrary.displayLimit": 20, + // Enables/disables the AI agent that configures auto-indexing for a repository on behalf of administrators. Disabled by default. The agent is available only if auto-indexing is enabled and the license includes AI Terms 2025. + "codeIntelAutoIndexing.agent.enabled": false, + + // The maximum number of auto-indexing configuration agent runs that may execute concurrently. Default is 2. + "codeIntelAutoIndexing.agent.maxConcurrentRuns": 2, + + // The maximum number of index jobs a single auto-indexing configuration agent run may execute before giving up. Default is 4. + "codeIntelAutoIndexing.agent.maxIndexAttempts": 4, + + // The model the auto-indexing configuration agent runs with, as a model reference such as "anthropic::2024-10-22::claude-sonnet-5-5". Only the agent's default model and the alternatives declared for it in internal/inference/models are accepted; saving any other value is rejected, and a stored value that is no longer supported after an upgrade falls back to the default model. Leave unset to use the default model. + // Other example values: + // - "anthropic::2024-10-22::claude-sonnet-5-5" + "codeIntelAutoIndexing.agent.model": null, + // Whether auto-indexing policies may apply to all repositories on the Sourcegraph instance. Default is false. The policyRepositoryMatchLimit setting still applies to such auto-indexing policies. "codeIntelAutoIndexing.allowGlobalPolicies": false, diff --git a/docs/self-hosted/observability/alerts.mdx b/docs/self-hosted/observability/alerts.mdx index 8a7566925..a3f4baa0f 100644 --- a/docs/self-hosted/observability/alerts.mdx +++ b/docs/self-hosted/observability/alerts.mdx @@ -4512,7 +4512,7 @@ Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~" ] ``` -*Managed by the Sourcegraph Code Plane team.* +*Managed by the Sourcegraph Platform team.*
Technical details @@ -6700,6 +6700,82 @@ Custom query for warning alert: `max(((sum(increase(src_workerutil_dbworker_stor
+## codeintel-autoindexing: codeintel_indexing_agent_error_rate + +

agent run handler error rate over 1h

+ +**Descriptions** + +- warning codeintel-autoindexing: 20%+ agent run handler error rate over 1h for 30m0s +- critical codeintel-autoindexing: 50%+ agent run handler error rate over 1h for 30m0s + +**Next steps** + +- Check worker logs for "indexing agent failed" to find the failing run IDs and underlying errors. The run`s steps are also visible in the repository`s code graph dashboard. +- An error is a handler failure (LLM provider error or timeout, gitserver or database error, quota check failure), not an agent verdict: runs that end in needs_intervention or gave_up count as successes here. +- Check for upstream LLM provider issues or timeouts. +- A spike that correlates with resets in the "Stalled job resetters" group points at worker crashes rather than per-run failures. +- More help interpreting this metric is available in the [dashboards reference](/self-hosted/observability/dashboards#codeintel-autoindexing-codeintel_indexing_agent_error_rate). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_codeintel-autoindexing_codeintel_indexing_agent_error_rate", + "critical_codeintel-autoindexing_codeintel_indexing_agent_error_rate" +] +``` + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Custom query for warning alert: `max(((sum(increase(src_codeintel_indexing_agent_errors_total{op="Handle"\}[3h])) / (sum(increase(src_codeintel_indexing_agent_total\{op="Handle"\}[3h])) + sum(increase(src_codeintel_indexing_agent_errors_total\{op="Handle"\}[3h]))) * 100) and ((sum(increase(src_codeintel_indexing_agent_total\{op="Handle"\}[3h])) + sum(increase(src_codeintel_indexing_agent_errors_total\{op="Handle"}[3h]))) >= 5)) > 20)` + +Custom query for critical alert: `max(((sum(increase(src_codeintel_indexing_agent_errors_total{op="Handle"\}[3h])) / (sum(increase(src_codeintel_indexing_agent_total\{op="Handle"\}[3h])) + sum(increase(src_codeintel_indexing_agent_errors_total\{op="Handle"\}[3h]))) * 100) and ((sum(increase(src_codeintel_indexing_agent_total\{op="Handle"\}[3h])) + sum(increase(src_codeintel_indexing_agent_errors_total\{op="Handle"}[3h]))) >= 5)) > 50)` + +
+ +
+ +## codeintel-autoindexing: codeintel_indexing_agent_subagent_context_limit_rate + +

subagent runs that hit the context limit by agent

+ +**Descriptions** + +- warning codeintel-autoindexing: more than 10% of a subagent's runs hit the context limit over 1h +- critical codeintel-autoindexing: more than 25% of a subagent's runs hit the context limit over 1h + +**Next steps** + +- Subagents have no compaction. When the task plus accumulated tool results outgrow the input token budget of the subagent model, the subagent stops early. Usually it returns a summary of partial findings with a note that they may be incomplete (max_input_tokens_summarized); if even that summary does not fit or fails, the parent gets "sub-agent exceeded maximum input tokens; retry with a narrower task" instead (max_input_tokens). Either way the parent answers with incomplete findings or spends extra turns retrying. +- Check the "Agent tool calls" group for the same agent: a tool whose output grew recently, or a jump in its call rate, is the usual cause. +- A rise that starts at a deploy points at a changed subagent prompt, tool, or model input token budget. +- If the results of one tool dominate, tighten its result limits or the subagent instructions so it reads less per turn. +- More help interpreting this metric is available in the [dashboards reference](/self-hosted/observability/dashboards#codeintel-autoindexing-codeintel_indexing_agent_subagent_context_limit_rate). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_codeintel-autoindexing_codeintel_indexing_agent_subagent_context_limit_rate", + "critical_codeintel-autoindexing_codeintel_indexing_agent_subagent_context_limit_rate" +] +``` + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Custom query for warning alert: `max(((sum by (workload, agent) (increase(src_agent_runs_total{agent!="main",stop_reason=~"max_input_tokens\\|max_input_tokens_summarized",workload=~"code_intel_indexing_agent"\}[1h])) / sum by (workload, agent) (increase(src_agent_runs_total\{agent!="main",workload=~"code_intel_indexing_agent"\}[1h])) * 100) and (sum by (workload, agent) (increase(src_agent_runs_total\{agent!="main",workload=~"code_intel_indexing_agent"}[1h])) >= 10)) > 10)` + +Custom query for critical alert: `max(((sum by (workload, agent) (increase(src_agent_runs_total{agent!="main",stop_reason=~"max_input_tokens\\|max_input_tokens_summarized",workload=~"code_intel_indexing_agent"\}[1h])) / sum by (workload, agent) (increase(src_agent_runs_total\{agent!="main",workload=~"code_intel_indexing_agent"\}[1h])) * 100) and (sum by (workload, agent) (increase(src_agent_runs_total\{agent!="main",workload=~"code_intel_indexing_agent"}[1h])) >= 10)) > 25)` + +
+ +
+ ## codeintel-uploads: codeintel_uploads_janitor_scip_documents_oldest_dereference_log_age

approximate age of the oldest SCIP document dereference log record, based on the lowest-ID record

diff --git a/docs/self-hosted/observability/dashboards.mdx b/docs/self-hosted/observability/dashboards.mdx index 03dc96f05..cb2e7b8cc 100644 --- a/docs/self-hosted/observability/dashboards.mdx +++ b/docs/self-hosted/observability/dashboards.mdx @@ -19244,7 +19244,7 @@ Refer to the [alerts reference](/self-hosted/observability/alerts#worker-pods_av To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103000` on your Sourcegraph instance. -*Managed by the Sourcegraph Code Plane team.* +*Managed by the Sourcegraph Platform team.*
Technical details @@ -31638,6 +31638,446 @@ sum by (op)(increase(src_codeintel_autoindexing_janitor_expired_errors_total{job
+### Code Intelligence > Autoindexing: Auto-indexing agent: worker + +#### codeintel-autoindexing: codeintel_indexing_agent_active_handlers + +

Agent runs in progress

+ +Number of agent runs executing concurrently. The ceiling is the codeIntelAutoIndexing.agent.maxConcurrentRuns site setting (default 2), read when the worker starts; sustained values at the ceiling mean queued runs are waiting for a slot. Each run holds its slot while its index jobs execute on executors. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101000` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum(src_codeintel_indexing_agent_handlers) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_processing_rate + +

Agent runs completed per hour

+ +Rate of agent runs whose handler finished without an error, whatever verdict the agent reached (see the outcomes panel). Handler errors appear in the error rate panel instead. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101001` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum(rate(src_codeintel_indexing_agent_total{op="Handle"}[30m])) * 3600 +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_p95_duration + +

95th percentile agent run duration

+ +95th percentile wall-clock time of a run: the agent loop including every LLM request and every index job it waited for on executors. A run executes at most codeIntelAutoIndexing.agent.maxIndexAttempts index jobs (default 4) of up to 20 minutes each, so runs over an hour are expected for slow builds. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101010` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum(rate(src_codeintel_indexing_agent_duration_seconds_bucket{op="Handle"}[1h])) by (le)) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_p50_duration + +

50th percentile agent run duration

+ +Median wall-clock time of a run. A shift here without a change in outcomes usually means executor queue wait or build time changed, not the agent. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101011` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.50, sum(rate(src_codeintel_indexing_agent_duration_seconds_bucket{op="Handle"}[1h])) by (le)) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_error_rate + +

Agent run handler error rate over 1h

+ +Percentage of handler invocations that ended in an error. The alerts evaluate the rate over 3h and are gated on at least 5 attempts in that window, so they stay quiet during idle periods. + +Refer to the [alerts reference](/self-hosted/observability/alerts#codeintel-autoindexing-codeintel_indexing_agent_error_rate) for 2 alerts related to this panel. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101020` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum(increase(src_codeintel_indexing_agent_errors_total{op="Handle"}[1h])) / (sum(increase(src_codeintel_indexing_agent_total{op="Handle"}[1h])) + sum(increase(src_codeintel_indexing_agent_errors_total{op="Handle"}[1h]))) * 100 +``` +
+ +
+ +### Code Intelligence > Autoindexing: Auto-indexing agent: queue and outcomes + +#### codeintel-autoindexing: codeintel_indexing_agent_queue_depth + +

Agent runs waiting in the queue

+ +Number of runs enqueued (by an administrator from the repository`s code graph dashboard) but not yet picked up by a handler. Each repository has at most one live run, so this is bounded by the number of repositories administrators have triggered. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101100` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_workerutil_queue_depth{domain="codeintel_indexing_agent_worker"}) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_queue_wait + +

Oldest queued agent run wait time

+ +How long the oldest queued run has been waiting for a handler; this is the delay an administrator sees between clicking the button and the agent starting. Zero when the queue is empty. Long waits with active handlers at the ceiling mean codeIntelAutoIndexing.agent.maxConcurrentRuns is too low for the number of repositories being onboarded. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101101` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_workerutil_queue_duration_seconds{domain="codeintel_indexing_agent_worker"}) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_outcomes + +

Finished agent runs per hour by outcome

+ +What finished runs concluded. configured means a proven configuration was saved to the repository; needs_intervention means an administrator must act, broken down by kind (missing_secrets, unsupported_project, indexer_failure, other); gave_up means the attempt budget ran out without a clear next step. A rising share of gave_up or indexer_failure across many repositories points at the prompt, the index attempt budget, or an indexer image rather than at the repositories. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101110` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (outcome, intervention_kind) (rate(src_codeintel_indexing_agent_run_outcomes_total[1h])) * 3600 +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_index_jobs + +

Index jobs executed by the agent per hour

+ +Index jobs the agent enqueued on the codeintel executor queue to test candidate configurations. This is the agent`s load on executors; compare with the Executor dashboard`s codeintel queue size. Divide by the runs completed per hour to get the average attempts a run needs; values near codeIntelAutoIndexing.agent.maxIndexAttempts mean most runs exhaust their budget. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101111` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum(rate(src_codeintel_indexing_agent_index_jobs_total[1h])) * 3600 +``` +
+ +
+ +### Code Intelligence > Autoindexing: Agent runs + +#### codeintel-autoindexing: codeintel_indexing_agent_runs_by_stop_reason + +

Agent runs per minute by agent and stop reason

+ +Why agent loop runs ended. no_tool_calls is a normal answer, awaiting_tool_result is a pause on an external tool result such as ask_user, max_input_tokens means the prompt did not fit even after compaction, max_input_tokens_summarized means a subagent ran out of context and stopped with a summary of partial findings, and error or context_done are runs that failed or were canceled. Agent is main for the top-level loop or the name of the subagent tool whose loop ran. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101200` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, agent, stop_reason) (rate(src_agent_runs_total{workload=~"code_intel_indexing_agent"}[5m])) * 60 +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_subagent_context_limit_rate + +

Subagent runs that hit the context limit by agent

+ +Percentage of subagent loop runs that ran out of context, per subagent: stop reason max_input_tokens_summarized (stopped with partial findings) or max_input_tokens (stopped with an error). The panel uses a 5m window; the alerts use a 1h window per subagent and require at least 10 runs in it, so an isolated oversized task does not fire them. + +Refer to the [alerts reference](/self-hosted/observability/alerts#codeintel-autoindexing-codeintel_indexing_agent_subagent_context_limit_rate) for 2 alerts related to this panel. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101210` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +(sum by (workload, agent) (rate(src_agent_runs_total{workload=~"code_intel_indexing_agent",agent!="main",stop_reason=~"max_input_tokens|max_input_tokens_summarized"}[5m])) or (0 * sum by (workload, agent) (rate(src_agent_runs_total{workload=~"code_intel_indexing_agent",agent!="main"}[5m])))) / (sum by (workload, agent) (rate(src_agent_runs_total{workload=~"code_intel_indexing_agent",agent!="main"}[5m])) > 0) * 100 +``` +
+ +
+ +### Code Intelligence > Autoindexing: Agent tool calls + +#### codeintel-autoindexing: codeintel_indexing_agent_tool_call_rate + +

Tool calls per minute by agent, tool, and status

+ +Executions of registered agent tools. Agent is main for the top-level loop or the subagent tool whose loop made the call. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101300` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, agent, tool, status) (rate(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent"}[5m])) * 60 +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_tool_error_rate + +

Tool call error rate over 5m by agent and tool

+ +Percentage of tool calls ending in an execution error, timeout, or unavailable backend. Invalid requests and caller cancellations are shown separately as workload-quality signals and do not indicate server reliability. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101301` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +(sum by (workload, agent, tool) (rate(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent",status=~"error|timeout|unavailable"}[5m])) or (0 * sum by (workload, agent, tool) (rate(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent",status=~"success|error|timeout|unavailable"}[5m])))) / (sum by (workload, agent, tool) (rate(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent",status=~"success|error|timeout|unavailable"}[5m])) > 0) * 100 +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_tool_p50_duration + +

Median tool call duration over 5m by agent, tool, and status

+ +Typical tool execution time. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101310` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.50, sum by (le, workload, agent, tool, status) (rate(src_agent_tool_call_duration_seconds_bucket{workload=~"code_intel_indexing_agent"}[5m]))) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_tool_p95_duration + +

95th percentile tool call duration over 5m by agent, tool, and status

+ +Tail tool execution time. A sharp step near a round number usually marks a tool or upstream timeout. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101311` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum by (le, workload, agent, tool, status) (rate(src_agent_tool_call_duration_seconds_bucket{workload=~"code_intel_indexing_agent"}[5m]))) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_tool_timeout_count + +

Tool calls timing out every 5m

+ +Timeout means the tool observed a deadline exceeded error and is included in the server-reliability error rate. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101320` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, agent, tool) (increase(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent",status="timeout"}[5m])) +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_tool_invalid_request_rate + +

Invalid tool requests per minute by agent and tool

+ +Requests rejected before useful tool execution because their arguments were invalid. A rise indicates the model or caller is producing arguments that do not match the tool schema or validation rules, not a server-reliability failure. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101330` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, agent, tool) (rate(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent",status="invalid_request"}[5m])) * 60 +``` +
+ +
+ +#### codeintel-autoindexing: codeintel_indexing_agent_tool_canceled_rate + +

Canceled tool calls per minute by agent and tool

+ +Calls whose caller context ended before completion. A rise can indicate abandoned or superseded agent work and wasted workload, but is not a server-reliability failure. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-autoindexing/codeintel-autoindexing?viewPanel=101331` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, agent, tool) (rate(src_agent_tool_calls_total{workload=~"code_intel_indexing_agent",status="canceled"}[5m])) * 60 +``` +
+ +
+ ## Code Intelligence > Code Nav

The service at internal/codeintel/codenav`.