Skip to content

Add privacy-safe DA script error reporting for WSR - #486

Merged
amilandi merged 6 commits into
mainfrom
amilandin/adk-script-errors-8169047
Nov 11, 2026
Merged

amilandi merged 6 commits into
mainfrom
amilandin/adk-script-errors-8169047

Conversation

@amilandi

@amilandi amilandi commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/8169047

  • Add adk.script.error using the existing consent, identity, sessions, asynchronous 1DS transport, and bounded retry buffer.
  • Observe unhandled exceptions at 32 explicit DA CLI boundaries, including FlightCheck. Preserve the original exception, exit code, and cancellation behavior; nested boundaries emit once.
  • Emit an allow-listed script/module and numeric line, fixed diagnostic code/category/message, random event ID, and existing common dimensions. Never stringify the exception or collect its arbitrary message, traceback/source text, arguments, locals, paths, or response bodies. Unknown/custom exception types receive a fixed fallback.
  • Add deduplicated daily script trends and safe WSR code-location/action-item queries. Retry envelopes retain the same event ID.
  • Code targets DA main only; no main-ca code changes. Separately, obsolete chart families were removed from both DA and CA dashboard views at the user's explicit request.

Coverage and limits

The explicit registry covers DA setup/ALM/reconciliation, ServiceNow/Workday connection CLIs, push/publish, checkpoint, scans, topic/workflow debug utilities, evaluation CLIs, analytics pointer, MCP configuration, branding validation, and the FlightCheck CLI.

Not a complete count of all script failures: handled errors, normal/nonzero SystemExit, cancellation, pre-boundary import/syntax failures, background tasks, process termination/native crashes, unsupported/legacy entry points, and the inline evaluation-review CLI are excluded. Handled FlightCheck checkpoint Error findings retain their existing reporting and are not escaping CLI crashes. There is no complete invocation denominator, so this is not a skill success/failure rate. Delivery remains best-effort with a shared one-second crash flush window. Do not combine API/build/FlightCheck errors with this event as independent failures.

Privacy follows CONTRIBUTING and the Maker Kit disclosure: regex scrubbing is not permission to collect arbitrary error text. Source locations are interpreted against the emitted commit, and can represent the deepest ADK call site when an external library fails. Local uncommitted edits are not represented by the SHA.

Aria

Published ADK: Script Errors by Script Trend (DA) on ADK Platform (External DA), plus matching DA-filtered trends on the combined ADK Platform and FlightCheck tabs:

https://portal.aria.ms/dashboard/gJ3tJywEt63-MRJUJXu8QX6GDHZ4HUhjyAUUbnK5v0MHk2vIoQqvUyo?projectId=311254257bbc417e860c76781d4863c8&isEU=false

  • Cube: gEPXwR--MRJUJXu8QX6GDHZ4HUhjyHZ2Qb5gJk9HlIGroWE4MkM.
  • Chart: gNGdJXtmd7tNdEF-MRJUJXu8QX6GDHZ4HUhjyGPOZvUmDEYnkzxMEAEiuK4.
  • Daily event count split only by script, with existing External DA/customer and all existing known-test-tenant exclusions.
  • Only script/agent type/tenant class/tenant ID cube dimensions; no error-message, source-line, or SHA chart dimensions.
  • Read back persisted cube/chart/tab. The initial publication preserved existing charts; see the subsequently authorized cleanup below.
  • RTA counts can include delivery duplicates; raw WSR queries deduplicate on event_id.
  • Combined ADK chart: gNGdJXtmd7tNdEF-MRJUJXu8QX6GDHZ4HUhjyArdqdt1jkOEuh7Ur7IXZjU, DA agent filter and script split, includes internal traffic.
  • Combined FlightCheck chart: gNGdJXtmd7tNdEF-MRJUJXu8QX6GDHZ4HUhjyLBibXkg8UMZoH3OLmA1bFk, DA agent filter and script_id == "flightcheck.cli", includes internal traffic.
  • Ingestion verified with Edge Playwright after 45.66 minutes on 2026-11-10: combined ADK chart renders setup_existing_da = 1 and flightcheck.cli = 1; combined FlightCheck chart renders flightcheck.cli = 1. Live chart queries and internal-tenant-scoped queries agree. Screenshots and query responses were saved; this verifies actual data/rendering, not just chart definitions or collector acceptance.
  • At the user's explicit request, exactly two synthetic error envelopes (one setup script, one FlightCheck CLI) were sent in a single request; collector accepted HTTP 204 at 2026-11-10T17:01:04Z. Both have EmployeeHub tenant 935884d7-bdee-469b-a461-fcc530a3ac83 and tenant_class == "internal". All 53 remaining live external chart layers were re-audited after ingestion and exclude this tenant/class. Customer-class queries scoped to this tenant return no positive counts. No customer-class synthetic event, session-start, run, or check event was sent, and no event was re-emitted during verification. The one-time verification schedule was stopped.

Authorized dashboard cleanup

The combined ADK tab hit Aria's 50-tile limit. The user explicitly requested removing agents-created/deployed/API-outcomes charts for DA and CA and investigating/removing meaningless build/deploy charts, totals and trends.

Investigation: setup.py emits “agent create” after writing a local workspace, not creating a new agent. Both adk.build.complete and adk.agent.deploy are emitted by the same legacy Dataverse push.py operation, not separate builds or publish/deployment operations. The current native DA path returns before this block. CA still has legacy push emitters; absence of chart data does not prove that no CA operation can emit. Live 30-day build/deploy queries returned no data, including unsegmented queries. “Created” did have counts, but measures the wrong thing for the chart title.

Removed 48 dashboard tiles: agents-created, agents-deployed, API outcomes, build outcomes, deploy-by-target, build duration, and deploy duration, including totals and trends. Scope: combined ADK (24), External DA (12), External CA (12). Remaining charts were compacted; raw events, cubes, and saved visualization documents were preserved. No FlightCheck existing charts were removed. All retained visualization definitions were read back unchanged.

Direct API access returned 401 and Azure CLI could not obtain an Aria token. Used the existing dedicated authenticated Edge profile; did not take over the shared Playwright MCP profile.

Validation

  • python -m pytest -q tests\test_adk_telemetry.py: 295 passed.
  • After FlightCheck coverage: the complete telemetry suite has 297 passed; four FlightCheck CLI suites have 210 passed. Ruff, syntax compilation, and whitespace checks passed.
  • Final asynchronous-entry-point/privacy/consent/retry/exit-focused run: python -m pytest -q tests\test_adk_telemetry.py -k script: 45 passed.
  • Broader python -m pytest -q tests\scripts tests\test_adk_telemetry.py: 2631 passed, 15 failed. All 15 failures are pre-existing assertions in test_installer_launch.py / test_installer_maker_profile.py; rerunning those files on clean main at the same base (7e9ffa1f) reproduces exactly 15 failed, 21 passed. Installer files/settings are unchanged by this PR.
  • Ruff across production scripts, MCP sources, and the changed test file: passed.
  • compileall for production scripts and MCP sources; git diff --check: passed.
  • All telemetry tests isolate local state and mock transport. Tests exercise each actual CLI guard with a failing main and asynchronous dispatch, privacy sentinels (including an exception that cannot be stringified), deepest approved frame selection, nested deduplication, independent invocation IDs, opt-out, CA rejection, delivery retries, and reporting failures preserving the original exception.

Example event (mocked transport; not sent)

Fake example identifiers in the dev project; this is the complete envelope captured from the emitter. toolkit_git_sha is the sample base revision, not a promise that local uncommitted source matches that revision. The original exception message was PRIVATE MESSAGE NOT EMITTED and is intentionally absent.

{
  "ver": "4.0",
  "name": "adk.script.error",
  "time": "2026-11-09T21:23:52.233Z",
  "iKey": "o:08e397b2c6c243eeaeb341e111c36167",
  "data": {
    "schema_version": "1.8.0",
    "instance_id": "11111111-1111-1111-1111-111111111111",
    "tenant_id": "22222222-2222-2222-2222-222222222222",
    "tenant_class": "customer",
    "tenant_name": "",
    "session_id": "87c6ac2e-9bba-4c98-81e1-8f2432370f7b",
    "surface": "cli",
    "adk_version": "0.4.37",
    "toolkit_git_sha": "7e9ffa1",
    "toolkit_git_branch": "other",
    "agent_type": "declarative_agent",
    "timestamp": "2026-11-09T21:23:52.211Z",
    "script_error_schema_version": "1",
    "event_id": "69e1640c-f14c-4f59-8c78-4c943a1251b9",
    "script_id": "setup_existing_da",
    "adk_capability": "setup",
    "failure_module": "setup_existing_da",
    "failure_line": 2994,
    "error_category": "runtime",
    "error_code": "UNHANDLED_RUNTIME_ERROR",
    "error_message": "An unexpected runtime error caused the script to fail.",
    "env": "dev"
  }
}

After commit 9eb8b253 was pushed, a deliberate failure was added locally and uncommitted only at setup_existing_da.py:2994, before parsing/authentication/remote writes. It is not part of this PR. Verified the actual CLI exits 1 with the expected message; a mocked asynchronous run captures exactly one safe event without the injected message. This initial local check sent no events; the two separately authorized internal-only ingestion probes are recorded above.

Run /setup from C:\Users\amilandin\adk-script-errors-8169047\solutions\ess-maker-skills; the deterministic dev-only direct trigger is:

Set-Location C:\Users\amilandin\adk-script-errors-8169047\solutions\ess-maker-skills
$env:ESS_ADK_ARIA_ENV = "dev"
python scripts\setup_existing_da.py cached-accounts

The local message is LOCAL MANUAL TEST: ADK script-error reporting. Telemetry intentionally replaces it with the fixed runtime diagnostic. The production dashboard will not show this dev-project test, and its External DA filters also exclude internal/unknown tenants.

WSR Contoso tenant-classification follow-up

ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/8183293

The November 2-8 external DA ADK report exposed a second Contoso demo tenant: 811c4ca4-538a-4a66-8c4b-d29ab413b37f (Graph domain M365CPI87546542.onmicrosoft.com), confirmed internal by the report owner. It was absent from upstream main's shared classifier and the report's saved exclusions. Commit 773f929a adds the exact GUID to the shared internal/test registry and historical customer-query exclusions, with a fixed-ID regression covering ADK common dimensions. No display-name/domain heuristic is introduced.

Validation: python -m pytest -q tests\test_adk_telemetry.py tests\flightcheck\test_telemetry.py -k 'tenant or customer': 52 passed; Ruff and diff checks pass. The intentional local setup failure remains uncommitted and is not included.

Avery Milandin and others added 4 commits November 9, 2026 13:25
Capture unhandled exceptions at 31 explicit DA CLI boundaries with fixed diagnostic metadata and source locations. Add focused privacy, async, deduplication, consent, retry, and exit-preservation tests, plus WSR queries and reporting documentation.

ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/8169047

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Reuse the privacy-safe script-error boundary for FlightCheck CLI exceptions and allow-listed nested frames. Document the combined ADK and FlightCheck chart deployments. Preserve handled check findings and normal exit/cancellation behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Exclude the owner-confirmed internal M365CPI87546542 tenant from future customer classification and historical customer queries. Add a fixed-ID regression covering the shared ADK dimensions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Avery Milandin and others added 2 commits November 10, 2026 14:31
Keep the investigation guide focused on shared telemetry investigation behavior. Preserve the README privacy disclosure and link to the raw query pack, retaining query-specific interpretation notes without dashboard-count claims.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Approving. Nothing here blocks merge given the timeline. The privacy design holds up: the payload is an allowlist, the exception is never turned into text, and mutation tests confirmed the tests catch redaction breaks. Please pick these up in a follow-up PR next week, in this order:

  1. (First priority) Respect FlightCheck's --no-telemetry flag. Right now adk.script.error and adk.session.start are still sent for FlightCheck runs started with --no-telemetry, including the runs launched by connect_servicenow_da.py and maker_diagnostics.py. This is an explicit opt-out, so it should be fixed first.
  2. Wrap the script_error_boundary import in a try/except that falls back to a no-op, as the three scripts that already imported telemetry do. Otherwise an import-time bug in the telemetry code would stop all 32 scripts from starting.
  3. Match error types through __mro__ instead of the exact type. Today FileNotFoundError, PermissionError, JSONDecodeError, CalledProcessError and requests.HTTPError all fall back to UNHANDLED_EXCEPTION.
  4. Register scripts/flightcheck_setup.py (added by #493). Also add a test that requires every __main__ script to be either in the registry or on an explicit exclusion list.
  5. Coordinate with #436, which also bumps the SCHEMA_VERSION you changed (to 1.9.0), and with #495, which deletes 4 scripts that this PR registers.

Verified at e732fea: 936 tests passed. The 8 failures in test_installer_launch.py also fail on main.

@amilandi
amilandi merged commit 9ecca95 into main Nov 11, 2026
9 checks passed
Harsheet jain (jainharsheet77) pushed a commit that referenced this pull request Nov 12, 2026
The fresh-session bootstrap test copies mcp_config.py into a temporary kit
root, so the script resolves that root from its own path. Since #486,
mcp_config.py runs its CLI inside adk_telemetry.script_error_boundary, and
the lone copy cannot import adk_telemetry once this branch merges with main.

Put the real scripts directory on PYTHONPATH for the subprocess. The copy
still resolves the temporary root, and the suite's telemetry-off guard is
inherited.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 66275402-dce2-4892-a314-f02931047e7e
@daeunJe0ng
Dawn Jeong (daeunJe0ng) deleted the amilandin/adk-script-errors-8169047 branch November 12, 2026 18:21
@amilandi

Copy link
Copy Markdown
Contributor Author

Addressed the five follow-up items from the approval review in PR #494, commit 8021abe. FlightCheck now applies parsed opt-out to SDK session/API/crash emissions; all 33 registered CLI boundaries fail open on telemetry import errors; classification uses allow-listed bases through mro; flightcheck_setup.py and an exhaustive CLI coverage-decision test are added. Schema remains 1.8.0 here; coordination details are also posted on #436 and #495. Privacy remains fixed-field/fixed-text only. Focused offline regressions and the pinned pre-commit gate passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants