Skip to content

feat(testing)!: native topic testing and debugging - #505

Open
nkemms wants to merge 5 commits into
users/nkemd/topic-authoring-native-splitfrom
users/nkemd/topic-testing-native-split
Open

nkemms wants to merge 5 commits into
users/nkemd/topic-authoring-native-splitfrom
users/nkemd/topic-testing-native-split

Conversation

@nkemms

@nkemms nkemms commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Description

Add bounded native DA-GA topic testing and debugging on top of the authoring and Push layer.

  • Resolve the selected saved topic, prepare an owned test browser, and require explicit activation and test scope.
  • Capture one approved turn with correlated reply, card, completion and saved-state evidence. Observe uncertain sends without replaying them.
  • Support New Chat through the available direct-button or navigation path while preserving successful actions and cleanup failures separately.
  • Add owned temporary diagnostics with guarded cleanup, keeping interpretation, fixes, recovery and retesting in session guidance.
  • Retire the replaced legacy browser and debug adapters together with their telemetry registrations and tests.

This is layer 2 of the three-PR split of the integration PR. It targets users/nkemd/topic-authoring-native-split, not main. Shared authoring/testing handoff contracts are introduced by the parent; this layer supplies their runtime and operational references. Topic deletion remains unavailable until layer 3.

Related issue

Source integration: #495

Type of change

  • New feature
  • Breaking change: retire replaced legacy topic-testing and live-debug CLIs.
  • Documentation update
  • Refactor / cleanup

Testing

  • Focused Python suite: 739 passed, 1 known inherited failure in test_every_canonical_capability_is_actually_emitted, with the same four stale capability names as the source integration.
  • Includes isolated browser DOM tests, native operation/evidence tests, activation concurrency guards, diagnostic cleanup, authoring compatibility and shared connection tests.
  • The local Playwright package lacks its bundled Node executable. Tests passed using the installed Node runtime through process-scoped PLAYWRIGHT_NODEJS_PATH; no shared installation or browser profile was changed.
  • Ruff, Prettier, four-file Markdownlint and whitespace checks passed.
  • No new live-agent trial or remote mutation was performed for this split. Existing integration runtime evidence is not a new independent runtime test of this intermediate layer.

Checklist

  • Runtime, guidance, retirement and tests are kept together.
  • Existing capture host and profiles remain untouched.
  • No merge, publication or force-push is included.

Add scoped native captures, activation guards, New Chat evidence and owned temporary diagnostics. Retire the replaced legacy drivers with their tests and telemetry registrations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Preserve the native testing layer while incorporating the resolved Planner and authoring guidance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: native topic testing and debugging

Verdict: request changes. Target binding is strict and fails closed: account, tenant, agent, environment, exact Studio URL, loopback-only CDP (Chrome DevTools Protocol) endpoint, and a single matching page are all checked, and evidence files are never overwritten or replayed. Two items need fixing before merge; the rest are minor. Design notes refer to ESS-ADK-Topic-Authoring-Developer-Design (draft dated 2026-11-08).

Must fix

1. Diagnostic plant and cleanup crash on an empty mapping (scripts/debug_plant.py:55)

  • _content_end calls max() over a mapping's children without default. The list branch on line 56 has one. A valid action containing extra: {} raises ValueError: max() arg is an empty sequence. Reproduced for both plant and strip.
  • Why it matters: the maker cannot add the diagnostic or remove it again, and gets a raw Python error instead of DebugPlantError.
  • Suggested fix: default=node.end_mark.line, plus tests with an empty mapping in the anchor action and in the owned node.

2. Topic testing loses all script-error telemetry (scripts/adk_telemetry.py:111, scripts/native_topic_test.py:760)

  • The PR removes drive_topic, plant_debug, and strip_debug, which were registered in SCRIPT_ERROR_ENTRY_POINTS and wrapped in script_error_boundary. native_topic_test is not registered and main is not wrapped.
  • Why it matters: script-error coverage for topic_test goes from three entry points to zero. Design sections 2.3 and 4.2 require telemetry parity.
  • Suggested fix: register native_topic_test under topic_test, wrap main in script_error_boundary("native_topic_test"), and add it to the registry test.

Minor

  • Unquoted activity text (scripts/debug_plant.py:142): activity: {activity} is written into YAML without quoting or re-parsing. The docs' own example Ready for review: total={Topic.Total} produces a YAML ScannerError. Quote inside the function, re-parse, and raise DebugPlantError on mismatch.
  • Malformed evidence files raise raw KeyError (scripts/native_topic_test.py:331, 437): for example inspection["component"]["id"], execution["prompt"]. main does not catch KeyError, so the user gets a traceback instead of the documented blocked result. Validate shape up front or add KeyError/TypeError to the handled set.
  • Playwright: core or optional? scripts/requirements.txt:14 adds it as a core dependency, while native_topic_test.py:419 says basic installs do not support browser testing. The docs also don't mention the separate playwright install browser step. Pick one model and align both.
  • Docs ask for unsupported CDP calls (src/reference/ess-docs/operations/native-topic-testing.md:97): the session is told to send Emulation.clearDeviceMetricsOverride and refresh the window, which no supported operation does. This conflicts with "capture does not launch, navigate, reset or reload" (design 3.9). Add a narrow operation or have the agent block and report.
  • New chat is not confirmed (scripts/native_topic_test.py:746, 756): only newChatOutcome: clicked is recorded. Design 3.9 asks for the action and its confirmation as separate facts. Record an empty-conversation check after the click.
  • Activation readback compares server-generated fields (scripts/native_topic_test.py:715): other code strips them with _without_server_generated_fields. If activation changes any of them, a successful write is reported as incomplete. Strip the same fields, or confirm against a live tenant.
  • failed mixes two states (scripts/native_topic_test.py:364): an exact reply plus topic drift returns failed. Report drift as incomplete so failed means a wrong reply only.
  • Breaking change has no migration pointer: cdp_driver.py, drive_topic.py, plant_debug.py, strip_debug.py, topic_checker_capture.py, and the live debug_plant API are removed. A short "replaced by" table in native-topic-testing.md would help anyone who scripted the old CLIs.
  • Nit: tests/scripts/test_analytics_pointer.py:207 still says "mirrors drive_topic.py's fallback".

Tests run

python -m pytest -q tests/scripts/test_native_topic_test.py tests/scripts/test_daga_chat_driver.py tests/test_debug_plant.py tests/test_drive_surface.py tests/scripts/test_authoring_guidance.py tests/scripts/test_setup_existing_da.py tests/test_adk_telemetry.py tests/scripts/test_emit_capability.py tests/setup/test_da_setup_router.py: 739 passed, 2 failed. Both failures also fail on the base branch (#504): test_every_canonical_capability_is_actually_emitted (already noted in the PR description) and the Windows line-ending hash test.
ruff check on the changed Python files: passed.

Not verified: live Copilot Studio and browser behavior (Edge launch, New chat and Test pane selectors, streaming replies).

Nkem Dockery and others added 3 commits November 17, 2026 20:26
Preserve both blank-account validation and the testing-layer inventory-reuse coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Use topic_test for the testing CLI and update related guidance, evidence names and current consumers consistently.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Reject malformed saved evidence before browser actions, preserve crash reporting and normalize only generated activation-dialog fields. Handle empty YAML mappings, clarify serialized activity input and retain manual debugging fallback with session-directed recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 45972120-90da-4a86-b7bb-6c8ce20c4261
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants