Skip to content

TEST Use shared watchdog for backend correctness helpers - #3459

Merged
Roman Lutz (romanlutz) merged 4 commits into
microsoft:mainfrom
romanlutz:romanlutz-correctness-test-watchdogs
Oct 23, 2026
Merged

Roman Lutz (romanlutz) merged 4 commits into
microsoft:mainfrom
romanlutz:romanlutz-correctness-test-watchdogs

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Remove two remaining accidental speed requirements from backend correctness tests using the existing finite TEST_WATCHDOG_SECONDS backstop:

  • _settle_send_async now defaults to the shared 300-second watchdog rather than ten seconds. It still polls through finalization to a semantic terminal state, and expiration still leaves the actual send running.
  • The concurrent first-request analytics singleton test uses the same watchdog for report, quick-query, and queue budgets instead of thirty seconds. The backend analytics fixture also references the shared constant instead of repeating its value.
  • Regression checks guard the helper default and the singleton controller's actual budgets. An event-controlled service test expires only the observer with timeout=0, checks that the send has no cancellation request, then releases it and verifies successful completion and reservation cleanup. The existing failure-teardown test also uses deterministic zero-timeout expiry and checks that only lifecycle shutdown cancels the send.

Production deadlines, intentional timeout coverage, and the async helper implementation are unchanged. This completes remaining adoption after #3203, #3268, and #3278. Open #3443 is complementary: its analytics changes cover explicit-expiry checkpoint and cleanup observation, not singleton-controller correctness budgets. This PR does not depend on it.

Merged upstream main ddaa6af996ac6fc1debf2828d0eb050ea509b7b8 without rebasing. Conflict resolution preserved both the combined async helper imports and main's newer message-send ownership tests, plus both adjacent documentation sections.

Tests and Documentation

Validated after merging main on Windows with CPython 3.14.4:

Exact command Result
uv sync --frozen --extra all Passed; synced merged dependency manifests.
uv run --no-sync pytest -q -n 4 --dist=loadfile tests\unit\test_async_utils.py tests\unit\backend\test_message_send_service.py tests\unit\backend\test_conversation_editor.py tests\unit\backend\test_temporary_components.py tests\unit\backend\test_analytics_routes.py tests\unit\backend\test_analytics_lifecycle.py tests\unit\analytics\test_execution.py 458 passed, 79 warnings in 140.60 seconds. Includes helper callers and analytics deadline/cancellation coverage.
uv run --no-sync ruff check tests\unit\backend\mocks.py tests\unit\backend\conftest.py tests\unit\backend\test_analytics_routes.py tests\unit\backend\test_message_send_service.py Passed.
uv run --no-sync ruff format --check tests\unit\backend\mocks.py tests\unit\backend\conftest.py tests\unit\backend\test_analytics_routes.py tests\unit\backend\test_message_send_service.py Passed; four files already formatted.
uv run --no-sync ty check --output-format concise tests\unit\backend\mocks.py tests\unit\backend\conftest.py tests\unit\backend\test_analytics_routes.py tests\unit\backend\test_message_send_service.py Passed.
uv run --no-sync python -m build_scripts.validate_docs Passed; 168 TOC references validated, no orphaned documentation.
git diff --cached --check Passed before the merge commit.

Applicable merge-commit pre-commit hooks passed. A worktree-local pre-commit cache was used to avoid contention on the shared cache lock.

Updated doc/contributing/4_running_tests.md to explain the manual-send watchdog and non-cancelling observer ownership. JupyText is not applicable: no notebooks or executable documentation samples changed. Full-suite and macOS/Linux CI were not run locally. The overloaded-runner failure was not reproduced with real ten/thirty-second delays; regression guards verify the budgets directly without sleeps or global clock patches.

Keep manual-send terminal polling non-cancelling and align the analytics singleton correctness test with the shared deadlock backstop. Guard these budgets and prove a timed-out observer leaves a send able to finish.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1b1dc1a4-802e-4475-9c27-f178cf67a8ab
Preserve both manual-send async helper imports and adjacent watchdog/scorer guidance while retaining main's newer ownership tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1b1dc1a4-802e-4475-9c27-f178cf67a8ab
Comment thread tests/unit/backend/test_message_send_service.py Outdated
Raise async and repeated send class backstops above the shared completion watchdog, guard their effective timeout markers, and document the budgets.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1b1dc1a4-802e-4475-9c27-f178cf67a8ab
Resolve combined async helper imports and retain both manual-send watchdog and target-discovery documentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1b1dc1a4-802e-4475-9c27-f178cf67a8ab
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 23, 2026
Merged via the queue into microsoft:main with commit 88dd1f5 Oct 23, 2026
65 checks passed
@romanlutz
Roman Lutz (romanlutz) deleted the romanlutz-correctness-test-watchdogs branch October 23, 2026 17:28
@richlundeen

Copy link
Copy Markdown
Contributor

Follow-up: make send cleanup ownership consistent across the helper's callers.

tests/unit/backend/mocks.py:20-28 correctly times out the observer without cancelling the send. But the normal message_send_service and real_send_context fixtures (test_message_send_service.py:92-93,118-139) do not shut down accepted sends. For example, test_accepts_before_target_finishes_and_status_cancellation_does_not_cancel_send_async (lines 3114-3151) holds the provider behind release, then waits and asserts without a finally block or lifecycle owner. An assertion failure before release.set() leaves the send blocked as the provider patch and database fixture close. A completion-watchdog failure also leaves the send running beyond those scopes. The new ownership tests use a lifecycle context explicitly, so they do not cover this normal caller path.

Please apply the existing message_send_lifecycle_async ownership pattern to these callers, with shutdown inside the relevant patch scopes and gate release in finally. A fixture can provide a final safety net, but teardown after a test-local patch exits is too late. Add one controlled-failure check through a normal caller to confirm that sends and reservations are drained before mocks and memory close. Keep the observer non-cancelling; it is not the send's owner.

Review authored with GitHub Copilot.

@richlundeen

Copy link
Copy Markdown
Contributor

Follow-up: remove the remaining short correctness deadlines in the same manual-send tests, not only the shared completion helper.

Current main still has these limits:

  • test_message_send_service.py:3948-3952: five seconds for one conversation to finish finalization while sibling sends remain held.
  • test_conversation_editor.py:164: five seconds for the provider to signal entry.
  • test_message_send_service.py:1627,1777-1779,1837,1842: a ten-second concurrency-test class limit, a three-second independent-send limit, and thirty-second readiness waits.

These tests check ownership and independent progress, not a response-time requirement. A busy worker can still fail them after this PR raises the final completion budget. Open #3491 fixes analytics and other watchdogs, but does not cover these cases.

Please handle these together: use shared correctness watchdogs, make the whole-test backstop leave room for cleanup, and observe the owned task as well as its readiness signal so early failure is reported at once. For the per-conversation finalization case, retain polling or a scoped completion signal for that conversation: waiting for the entire send would deadlock against the deliberately held siblings. Keep the existing conflict, held-peer, and independent-progress assertions. Larger backstops need not add sleeps or increase successful test duration.

Review authored with GitHub Copilot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants