Repository navigation
Conversation
Add scoped native captures, activation guards, New Chat evidence and owned temporary diagnostics. Retire the replaced legacy drivers with their tests and telemetry registrations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Preserve the native testing layer while incorporating the resolved Planner and authoring guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Dawn Jeong (daeunJe0ng)
left a comment
There was a problem hiding this comment.
Review: native topic testing and debugging
Verdict: request changes. Target binding is strict and fails closed: account, tenant, agent, environment, exact Studio URL, loopback-only CDP (Chrome DevTools Protocol) endpoint, and a single matching page are all checked, and evidence files are never overwritten or replayed. Two items need fixing before merge; the rest are minor. Design notes refer to ESS-ADK-Topic-Authoring-Developer-Design (draft dated 2026-11-08).
Must fix
1. Diagnostic plant and cleanup crash on an empty mapping (scripts/debug_plant.py:55)
_content_endcallsmax()over a mapping's children withoutdefault. The list branch on line 56 has one. A valid action containingextra: {}raisesValueError: max() arg is an empty sequence. Reproduced for both plant and strip.- Why it matters: the maker cannot add the diagnostic or remove it again, and gets a raw Python error instead of
DebugPlantError. - Suggested fix:
default=node.end_mark.line, plus tests with an empty mapping in the anchor action and in the owned node.
2. Topic testing loses all script-error telemetry (scripts/adk_telemetry.py:111, scripts/native_topic_test.py:760)
- The PR removes
drive_topic,plant_debug, andstrip_debug, which were registered inSCRIPT_ERROR_ENTRY_POINTSand wrapped inscript_error_boundary.native_topic_testis not registered andmainis not wrapped. - Why it matters: script-error coverage for
topic_testgoes from three entry points to zero. Design sections 2.3 and 4.2 require telemetry parity. - Suggested fix: register
native_topic_testundertopic_test, wrapmaininscript_error_boundary("native_topic_test"), and add it to the registry test.
Minor
- Unquoted activity text (
scripts/debug_plant.py:142):activity: {activity}is written into YAML without quoting or re-parsing. The docs' own exampleReady for review: total={Topic.Total}produces a YAML ScannerError. Quote inside the function, re-parse, and raiseDebugPlantErroron mismatch. - Malformed evidence files raise raw
KeyError(scripts/native_topic_test.py:331, 437): for exampleinspection["component"]["id"],execution["prompt"].maindoes not catchKeyError, so the user gets a traceback instead of the documented blocked result. Validate shape up front or addKeyError/TypeErrorto the handled set. - Playwright: core or optional?
scripts/requirements.txt:14adds it as a core dependency, whilenative_topic_test.py:419says basic installs do not support browser testing. The docs also don't mention the separateplaywright installbrowser step. Pick one model and align both. - Docs ask for unsupported CDP calls (
src/reference/ess-docs/operations/native-topic-testing.md:97): the session is told to sendEmulation.clearDeviceMetricsOverrideand refresh the window, which no supported operation does. This conflicts with "capture does not launch, navigate, reset or reload" (design 3.9). Add a narrow operation or have the agent block and report. - New chat is not confirmed (
scripts/native_topic_test.py:746, 756): onlynewChatOutcome: clickedis recorded. Design 3.9 asks for the action and its confirmation as separate facts. Record an empty-conversation check after the click. - Activation readback compares server-generated fields (
scripts/native_topic_test.py:715): other code strips them with_without_server_generated_fields. If activation changes any of them, a successful write is reported asincomplete. Strip the same fields, or confirm against a live tenant. failedmixes two states (scripts/native_topic_test.py:364): an exact reply plus topic drift returnsfailed. Report drift asincompletesofailedmeans a wrong reply only.- Breaking change has no migration pointer:
cdp_driver.py,drive_topic.py,plant_debug.py,strip_debug.py,topic_checker_capture.py, and the live debug_plant API are removed. A short "replaced by" table innative-topic-testing.mdwould help anyone who scripted the old CLIs. - Nit:
tests/scripts/test_analytics_pointer.py:207still says "mirrors drive_topic.py's fallback".
Tests run
python -m pytest -q tests/scripts/test_native_topic_test.py tests/scripts/test_daga_chat_driver.py tests/test_debug_plant.py tests/test_drive_surface.py tests/scripts/test_authoring_guidance.py tests/scripts/test_setup_existing_da.py tests/test_adk_telemetry.py tests/scripts/test_emit_capability.py tests/setup/test_da_setup_router.py: 739 passed, 2 failed. Both failures also fail on the base branch (#504): test_every_canonical_capability_is_actually_emitted (already noted in the PR description) and the Windows line-ending hash test.
ruff check on the changed Python files: passed.
Not verified: live Copilot Studio and browser behavior (Edge launch, New chat and Test pane selectors, streaming replies).
Preserve both blank-account validation and the testing-layer inventory-reuse coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Use topic_test for the testing CLI and update related guidance, evidence names and current consumers consistently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Reject malformed saved evidence before browser actions, preserve crash reporting and normalize only generated activation-dialog fields. Handle empty YAML mappings, clarify serialized activity input and retain manual debugging fallback with session-directed recovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Description
Add bounded native DA-GA topic testing and debugging on top of the authoring and Push layer.
This is layer 2 of the three-PR split of the integration PR. It targets
users/nkemd/topic-authoring-native-split, notmain. Shared authoring/testing handoff contracts are introduced by the parent; this layer supplies their runtime and operational references. Topic deletion remains unavailable until layer 3.Related issue
Source integration: #495
Type of change
Testing
test_every_canonical_capability_is_actually_emitted, with the same four stale capability names as the source integration.PLAYWRIGHT_NODEJS_PATH; no shared installation or browser profile was changed.Checklist