Repository navigation
Add privacy-safe DA script error reporting for WSR - #486
Conversation
Capture unhandled exceptions at 31 explicit DA CLI boundaries with fixed diagnostic metadata and source locations. Add focused privacy, async, deduplication, consent, retry, and exit-preservation tests, plus WSR queries and reporting documentation. ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/8169047 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Reuse the privacy-safe script-error boundary for FlightCheck CLI exceptions and allow-listed nested frames. Document the combined ADK and FlightCheck chart deployments. Preserve handled check findings and normal exit/cancellation behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Exclude the owner-confirmed internal M365CPI87546542 tenant from future customer classification and historical customer queries. Add a fixed-ID regression covering the shared ADK dimensions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Keep the investigation guide focused on shared telemetry investigation behavior. Preserve the README privacy disclosure and link to the raw query pack, retaining query-specific interpretation notes without dashboard-count claims. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Dawn Jeong (daeunJe0ng)
left a comment
There was a problem hiding this comment.
[copilot-review] Approving. Nothing here blocks merge given the timeline. The privacy design holds up: the payload is an allowlist, the exception is never turned into text, and mutation tests confirmed the tests catch redaction breaks. Please pick these up in a follow-up PR next week, in this order:
- (First priority) Respect FlightCheck's
--no-telemetryflag. Right nowadk.script.errorandadk.session.startare still sent for FlightCheck runs started with--no-telemetry, including the runs launched byconnect_servicenow_da.pyandmaker_diagnostics.py. This is an explicit opt-out, so it should be fixed first. - Wrap the
script_error_boundaryimport in a try/except that falls back to a no-op, as the three scripts that already imported telemetry do. Otherwise an import-time bug in the telemetry code would stop all 32 scripts from starting. - Match error types through
__mro__instead of the exact type. TodayFileNotFoundError,PermissionError,JSONDecodeError,CalledProcessErrorandrequests.HTTPErrorall fall back toUNHANDLED_EXCEPTION. - Register
scripts/flightcheck_setup.py(added by #493). Also add a test that requires every__main__script to be either in the registry or on an explicit exclusion list. - Coordinate with #436, which also bumps the
SCHEMA_VERSIONyou changed (to 1.9.0), and with #495, which deletes 4 scripts that this PR registers.
Verified at e732fea: 936 tests passed. The 8 failures in test_installer_launch.py also fail on main.
The fresh-session bootstrap test copies mcp_config.py into a temporary kit root, so the script resolves that root from its own path. Since #486, mcp_config.py runs its CLI inside adk_telemetry.script_error_boundary, and the lone copy cannot import adk_telemetry once this branch merges with main. Put the real scripts directory on PYTHONPATH for the subprocess. The copy still resolves the temporary root, and the suite's telemetry-off guard is inherited. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 66275402-dce2-4892-a314-f02931047e7e
|
Addressed the five follow-up items from the approval review in PR #494, commit 8021abe. FlightCheck now applies parsed opt-out to SDK session/API/crash emissions; all 33 registered CLI boundaries fail open on telemetry import errors; classification uses allow-listed bases through mro; flightcheck_setup.py and an exhaustive CLI coverage-decision test are added. Schema remains 1.8.0 here; coordination details are also posted on #436 and #495. Privacy remains fixed-field/fixed-text only. Focused offline regressions and the pinned pre-commit gate passed. |
Summary
ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/8169047
adk.script.errorusing the existing consent, identity, sessions, asynchronous 1DS transport, and bounded retry buffer.mainonly; nomain-cacode changes. Separately, obsolete chart families were removed from both DA and CA dashboard views at the user's explicit request.Coverage and limits
The explicit registry covers DA setup/ALM/reconciliation, ServiceNow/Workday connection CLIs, push/publish, checkpoint, scans, topic/workflow debug utilities, evaluation CLIs, analytics pointer, MCP configuration, branding validation, and the FlightCheck CLI.
Not a complete count of all script failures: handled errors, normal/nonzero
SystemExit, cancellation, pre-boundary import/syntax failures, background tasks, process termination/native crashes, unsupported/legacy entry points, and the inline evaluation-review CLI are excluded. Handled FlightCheck checkpoint Error findings retain their existing reporting and are not escaping CLI crashes. There is no complete invocation denominator, so this is not a skill success/failure rate. Delivery remains best-effort with a shared one-second crash flush window. Do not combine API/build/FlightCheck errors with this event as independent failures.Privacy follows CONTRIBUTING and the Maker Kit disclosure: regex scrubbing is not permission to collect arbitrary error text. Source locations are interpreted against the emitted commit, and can represent the deepest ADK call site when an external library fails. Local uncommitted edits are not represented by the SHA.
Aria
Published ADK: Script Errors by Script Trend (DA) on ADK Platform (External DA), plus matching DA-filtered trends on the combined ADK Platform and FlightCheck tabs:
https://portal.aria.ms/dashboard/gJ3tJywEt63-MRJUJXu8QX6GDHZ4HUhjyAUUbnK5v0MHk2vIoQqvUyo?projectId=311254257bbc417e860c76781d4863c8&isEU=false
gEPXwR--MRJUJXu8QX6GDHZ4HUhjyHZ2Qb5gJk9HlIGroWE4MkM.gNGdJXtmd7tNdEF-MRJUJXu8QX6GDHZ4HUhjyGPOZvUmDEYnkzxMEAEiuK4.event_id.gNGdJXtmd7tNdEF-MRJUJXu8QX6GDHZ4HUhjyArdqdt1jkOEuh7Ur7IXZjU, DA agent filter and script split, includes internal traffic.gNGdJXtmd7tNdEF-MRJUJXu8QX6GDHZ4HUhjyLBibXkg8UMZoH3OLmA1bFk, DA agent filter andscript_id == "flightcheck.cli", includes internal traffic.2026-11-10: combined ADK chart renderssetup_existing_da = 1andflightcheck.cli = 1; combined FlightCheck chart rendersflightcheck.cli = 1. Live chart queries and internal-tenant-scoped queries agree. Screenshots and query responses were saved; this verifies actual data/rendering, not just chart definitions or collector acceptance.2026-11-10T17:01:04Z. Both have EmployeeHub tenant935884d7-bdee-469b-a461-fcc530a3ac83andtenant_class == "internal". All 53 remaining live external chart layers were re-audited after ingestion and exclude this tenant/class. Customer-class queries scoped to this tenant return no positive counts. No customer-class synthetic event, session-start, run, or check event was sent, and no event was re-emitted during verification. The one-time verification schedule was stopped.Authorized dashboard cleanup
The combined ADK tab hit Aria's 50-tile limit. The user explicitly requested removing agents-created/deployed/API-outcomes charts for DA and CA and investigating/removing meaningless build/deploy charts, totals and trends.
Investigation:
setup.pyemits “agent create” after writing a local workspace, not creating a new agent. Bothadk.build.completeandadk.agent.deployare emitted by the same legacy Dataversepush.pyoperation, not separate builds or publish/deployment operations. The current native DA path returns before this block. CA still has legacy push emitters; absence of chart data does not prove that no CA operation can emit. Live 30-day build/deploy queries returned no data, including unsegmented queries. “Created” did have counts, but measures the wrong thing for the chart title.Removed 48 dashboard tiles: agents-created, agents-deployed, API outcomes, build outcomes, deploy-by-target, build duration, and deploy duration, including totals and trends. Scope: combined ADK (24), External DA (12), External CA (12). Remaining charts were compacted; raw events, cubes, and saved visualization documents were preserved. No FlightCheck existing charts were removed. All retained visualization definitions were read back unchanged.
Direct API access returned 401 and Azure CLI could not obtain an Aria token. Used the existing dedicated authenticated Edge profile; did not take over the shared Playwright MCP profile.
Validation
python -m pytest -q tests\test_adk_telemetry.py: 295 passed.python -m pytest -q tests\test_adk_telemetry.py -k script: 45 passed.python -m pytest -q tests\scripts tests\test_adk_telemetry.py: 2631 passed, 15 failed. All 15 failures are pre-existing assertions intest_installer_launch.py/test_installer_maker_profile.py; rerunning those files on clean main at the same base (7e9ffa1f) reproduces exactly 15 failed, 21 passed. Installer files/settings are unchanged by this PR.compileallfor production scripts and MCP sources;git diff --check: passed.Example event (mocked transport; not sent)
Fake example identifiers in the dev project; this is the complete envelope captured from the emitter.
toolkit_git_shais the sample base revision, not a promise that local uncommitted source matches that revision. The original exception message wasPRIVATE MESSAGE NOT EMITTEDand is intentionally absent.{ "ver": "4.0", "name": "adk.script.error", "time": "2026-11-09T21:23:52.233Z", "iKey": "o:08e397b2c6c243eeaeb341e111c36167", "data": { "schema_version": "1.8.0", "instance_id": "11111111-1111-1111-1111-111111111111", "tenant_id": "22222222-2222-2222-2222-222222222222", "tenant_class": "customer", "tenant_name": "", "session_id": "87c6ac2e-9bba-4c98-81e1-8f2432370f7b", "surface": "cli", "adk_version": "0.4.37", "toolkit_git_sha": "7e9ffa1", "toolkit_git_branch": "other", "agent_type": "declarative_agent", "timestamp": "2026-11-09T21:23:52.211Z", "script_error_schema_version": "1", "event_id": "69e1640c-f14c-4f59-8c78-4c943a1251b9", "script_id": "setup_existing_da", "adk_capability": "setup", "failure_module": "setup_existing_da", "failure_line": 2994, "error_category": "runtime", "error_code": "UNHANDLED_RUNTIME_ERROR", "error_message": "An unexpected runtime error caused the script to fail.", "env": "dev" } }After commit
9eb8b253was pushed, a deliberate failure was added locally and uncommitted only atsetup_existing_da.py:2994, before parsing/authentication/remote writes. It is not part of this PR. Verified the actual CLI exits 1 with the expected message; a mocked asynchronous run captures exactly one safe event without the injected message. This initial local check sent no events; the two separately authorized internal-only ingestion probes are recorded above.Run
/setupfromC:\Users\amilandin\adk-script-errors-8169047\solutions\ess-maker-skills; the deterministic dev-only direct trigger is:The local message is
LOCAL MANUAL TEST: ADK script-error reporting. Telemetry intentionally replaces it with the fixed runtime diagnostic. The production dashboard will not show this dev-project test, and its External DA filters also exclude internal/unknown tenants.WSR Contoso tenant-classification follow-up
ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/8183293
The November 2-8 external DA ADK report exposed a second Contoso demo tenant:
811c4ca4-538a-4a66-8c4b-d29ab413b37f(Graph domainM365CPI87546542.onmicrosoft.com), confirmed internal by the report owner. It was absent from upstream main's shared classifier and the report's saved exclusions. Commit773f929aadds the exact GUID to the shared internal/test registry and historical customer-query exclusions, with a fixed-ID regression covering ADK common dimensions. No display-name/domain heuristic is introduced.Validation:
python -m pytest -q tests\test_adk_telemetry.py tests\flightcheck\test_telemetry.py -k 'tenant or customer': 52 passed; Ruff and diff checks pass. The intentional local setup failure remains uncommitted and is not included.