Skip to content

fix: shut down JSONRPC cleanly on EOF (Fixes #529) - #559

Merged
Karthik Nadig (karthiknadig) merged 7 commits into
mainfrom
bug/issue-529-shutdown
Sep 25, 2026
Merged

Karthik Nadig (karthiknadig) merged 7 commits into
mainfrom
bug/issue-529-shutdown

Conversation

@karthiknadig

Copy link
Copy Markdown
Member

Treat frame-boundary stdin EOF as normal server shutdown instead of repeatedly logging empty headers. Cancel admitted subprocess probes through ownership cleanup, without waiting indefinitely for blocked transport I/O.

  • Distinguish clean EOF from truncated frames and terminal transport failures; preserve errors that race shutdown.
  • Use bounded FIFO output with a dedicated owned stdout handle, explicit saturation errors, and documented cancellation semantics.
  • Supervise probe admission and cleanup; report cleanup failures or the three-second cleanup deadline explicitly.
  • Close test-client stdin and bound normal exit, forced termination, and reader teardown.
  • Add deterministic transport/output/supervisor coverage and native EOF, unread-output, broken-pipe, and active-descendant regressions.

Validation: Windows and Linux affected suites and CI-feature native tests pass; 80 concurrent lifecycle regression cases pass. Workspace formatting/Clippy and targeted all-target Clippy pass. Independent Reviewer returned no findings. macOS and hosted quality results remain pending.

Multi-header framing and envelope validation remain separate work under #532.

Fixes #529

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 82.658% 80.536% +2.122pp
Functions 85.464% 83.539% +1.925pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 94ms 70ms +24ms +34.3% >100ms and >50% 🔺
Server startup P95 728ms 717ms +11ms +1.5% >750ms and >100% 🔺
Discovery duration P50 163ms 132ms +31ms +23.5% >100ms and >50% 🔺
Discovery duration P95 198ms 190ms +8ms +4.2% >300ms and >100% 🔺
Startup-to-first environment P50 121ms 84ms +37ms +44.0% >150ms and >50% 🔺
Startup-to-first environment P95 169ms 100ms +69ms +69.0% >250ms and >100% 🔺
Cold discovery duration P50 339ms 263ms +76ms +28.9% >250ms and >50% 🔺
Refresh round-trip P50 165ms 133ms +32ms +24.1% >250ms and >50% 🔺
Refresh round-trip P95 199ms 193ms +6ms +3.1% >300ms and >100% 🔺
Request-to-first environment P50 42ms 27ms +15ms +55.6% >50ms and >50% 🔺
Request-to-first environment P95 51ms 34ms +17ms +50.0% >100ms and >100% 🔺
Cold refresh round-trip P50 340ms 264ms +76ms +28.8% >600ms and >50% 🔺
Workload PR Baseline
Environments 10 10
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 8ms 9ms -1ms -11.1% >10ms and >50% ✅
Server startup P95 10ms 13ms -3ms -23.1% >50ms and >100% ✅
Discovery duration P50 107ms 155ms -48ms -31.0% >150ms and >50% ✅
Discovery duration P95 111ms 163ms -52ms -31.9% >250ms and >100% ✅
Startup-to-first environment P50 17ms 23ms -6ms -26.1% >25ms and >50% ✅
Startup-to-first environment P95 23ms 36ms -13ms -36.1% >100ms and >100% ✅
Cold discovery duration P50 106ms 155ms -49ms -31.6% >150ms and >50% ✅
Refresh round-trip P50 108ms 155ms -47ms -30.3% >150ms and >50% ✅
Refresh round-trip P95 112ms 164ms -52ms -31.7% >250ms and >100% ✅
Request-to-first environment P50 9ms 14ms -5ms -35.7% >25ms and >50% ✅
Request-to-first environment P95 15ms 27ms -12ms -44.4% >100ms and >100% ✅
Cold refresh round-trip P50 107ms 155ms -48ms -31.0% >150ms and >50% ✅
Workload PR Baseline
Environments 8 8
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric PR Baseline Delta
Lines 85.154% 83.159% +1.995pp
Functions 88.144% 86.407% +1.736pp

Allowed numerical tolerance: 0.01 percentage points.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric PR Baseline Delta Change Blocking budget Status
Server startup P50 1ms 1ms +0ms +0.0% >5ms and >100% ➖
Server startup P95 1ms 1ms +0ms +0.0% >50ms and >200% ➖
Discovery duration P50 54ms 54ms +0ms +0.0% >25ms and >30% ➖
Discovery duration P95 60ms 56ms +4ms +7.1% >50ms and >50% 🔺
Startup-to-first environment P50 14ms 13ms +1ms +7.7% >20ms and >100% 🔺
Startup-to-first environment P95 16ms 20ms -4ms -20.0% >25ms and >100% ✅
Cold discovery duration P50 137ms 140ms -3ms -2.1% >100ms and >50% ✅
Refresh round-trip P50 55ms 55ms +0ms +0.0% >25ms and >30% ➖
Refresh round-trip P95 60ms 56ms +4ms +7.1% >50ms and >50% 🔺
Request-to-first environment P50 13ms 12ms +1ms +8.3% >20ms and >100% 🔺
Request-to-first environment P95 15ms 19ms -4ms -21.1% >25ms and >100% ✅
Cold refresh round-trip P50 137ms 140ms -3ms -2.1% >100ms and >50% ✅
Workload PR Baseline
Environments 5 5
Managers 1 1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It combines cross-platform process ownership, concurrent transport shutdown, and a new asynchronous writer, with macOS validation still pending.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes JSONRPC EOF shutdown while bounding output and cancelling supervised subprocess probes.

Changes:

  • Treats clean EOF as normal shutdown and malformed/truncated transport as fatal.
  • Adds bounded asynchronous JSONRPC output and subprocess supervision.
  • Adds cross-platform lifecycle and regression coverage.
File Description
docs/​JSONRPC.md Documents transport lifecycle and limits.
crates/​pet/​src/​main.rs Returns server failures via exit status.
crates/​pet/​src/​jsonrpc.rs Coordinates transport and probe cleanup.
crates/​pet-jsonrpc/​src/​lib.rs Routes messages through bounded output.
crates/​pet-jsonrpc/​src/​server.rs Implements EOF-aware framed input.
crates/​pet-jsonrpc/​src/​output.rs Adds bounded FIFO output worker.
crates/​pet-python-utils/​src/​process.rs Integrates supervised cancellation.
crates/​pet-python-utils/​src/​process/​supervisor.rs Tracks probe admission and cleanup.
crates/​pet-python-utils/​src/​env.rs Handles cancelled interpreter probes.
crates/​pet-conda/​src/​conda_info.rs Handles cancelled Conda probes.
crates/​pet-poetry/​src/​environment_locations_spawn.rs Handles cancelled Poetry probes.
crates/​pet/​tests/​process_utils.rs Adds bounded process-test utilities.
crates/​pet/​tests/​jsonrpc_client.rs Supports clean client shutdown.
crates/​pet/​tests/​jsonrpc_server_test.rs Adds shutdown regressions.
crates/​pet/​tests/​e2e_performance.rs Cleans up performance fixtures normally.
crates/​pet/​tests/​fixtures/​shutdown_probe.py Provides descendant-cleanup fixture.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@karthiknadig

Copy link
Copy Markdown
Member Author

Current-head Copilot review completed with no findings. CodeQL's three analyses report zero results/errors/warnings, and Windows/Linux performance improved against the exact base.

Keeping this draft while addressing two quality findings rather than relying on green performance budgets:

  • Linux line coverage improved (+0.345pp), but function coverage decreased by 0.153pp. Native isolated test launchers clear LLVM_PROFILE_FILE, leaving actual server entry points unrecorded despite successful native tests. A narrowly scoped profiling-environment fix is being measured with instrumented subprocess tests.
  • macOS refresh RTT P50 increased from 133ms to 155ms, with larger startup-to-first-environment drift. Preparing a manual-only, read-only same-host base/head control with counterbalanced order and exact inventory identity checks before accepting that drift.

No signing/notarization/release services are involved in these checks.

Keep LLVM_PROFILE_FILE in isolated PET subprocess environments so normal EOF shutdown writes profiles into the coverage collector instead of untracked default files. Preserve fixture overrides and minimal environment isolation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-platform transport, concurrency, and process-ownership changes warrant final human review, especially with macOS validation pending.

Review effort: Balanced
Findings: None

Establish a bounded info exchange before closing the same stdout pipe, so startup is not charged to the unchanged one-second shutdown deadline. Explicitly terminate and bounded-join the readiness helper on setup failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-platform transport and process-lifecycle changes warrant final human review, particularly with macOS validation still pending.

Review effort: Balanced
Findings: None

Concurrent Unix fork/exec can retain another fixture pipe reader until exec, allowing the sole response write to succeed before the final reader disappears. Run the same real-pipe scenario in an isolated test subprocess; keep the one-second measured deadline and bounded cleanup unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@karthiknadig

Copy link
Copy Markdown
Member Author

Quality investigation update:

  • The same-host macOS control completed: https://github.com/microsoft/python-environment-tools/actions/runs/36168462886. Eight counterbalanced passes (B-H-H-B-H-B-B-H), exact base 094217e and production head d9ee6aa, verified binary hashes, and identical full inventories (10 environments, 1 manager). Later commits change tests only.
  • These are medians of per-pass percentiles, not pooled percentiles:
Metric Base P50/P95 Head P50/P95
Refresh round trip 127 / 163.5 ms 118 / 162 ms
Startup to first environment 103 / 132.5 ms 104 / 129.5 ms
Cold refresh round trip 292 / 428 ms 281.5 / 331 ms
Request to first environment 29.5 / 41.5 ms 34 / 49 ms
Cold request to first environment 29 / 39.5 ms 38.5 / 50.5 ms

The large aggregate slowdown in unmatched CI snapshots was not reproduced. The smaller first-notification increase is real, especially the consistent cold P50 difference, not dismissed as noise. Accepted as an explicit small-absolute-latency tradeoff for bounded transport/shutdown, with stable end-to-end first-environment latency and improved complete refresh time. Independent review found no added normal-path timer/batching delay; thread handoffs are plausible, not a proven attribution. Scope: x86_64 under Rosetta on one arm64 macOS host; four passes per revision, so this is not a universal performance claim. Budgets remain unchanged.

The coverage-profile inheritance fix has also been verified by hosted Linux and Windows coverage improvements, without exclusions or budget changes.

Further fixture stress exposed a separate, reproducible Unix fork/exec race: another test's fork can temporarily retain a CLOEXEC pipe reader, so the only response write succeeds before that last reader closes. The server then correctly has no write failure to observe. 60d8d5c isolates the real-pipe scenario in its own test process, retaining the one-second measured shutdown assertion, stdin-open proof, and bounded failure cleanup. Ten full parallel Linux suites (140 cases) passed, as did Windows validation and independent review. The latest test-only head is going through hosted checks; keeping this PR draft until those finish.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-platform process ownership, detached blocked I/O, and concurrent shutdown races warrant final human validation.

Review effort: Balanced
Findings: None

Allow the outer test-process guard to cover readiness, both forced-shutdown waits, reader completion, and fallback cleanup without changing the one-second shutdown assertion. Record the verified subprocess testing pitfalls in the existing Rust skill.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Fixture cleanup has an exit-versus-kill race that can skip reaping and falsely report failure.

Review effort: Balanced
Findings: None

Previously missed (1)

In code that hasn't changed since last review

Medium severity Recheck child exit after kill error to avoid false cleanup failure

crates/​pet/​tests/​process_utils.rs:35

A child can exit after wait_for_exit times out but before kill() runs. On platforms where killing an already-exited process returns an error, ? skips the final reap and falsely reports fixture cleanup failure. Recheck try_wait on a kill error before propagating it so this deadline race still reaps a normally exited child.

Poll the actual descendant lease using only the remainder of the existing four-second shutdown budget instead of requiring an instantaneous lock transition at server exit. Test held-lock timeout and success after release; preserve cancellation and elapsed-time assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@karthiknadig

Copy link
Copy Markdown
Member Author

The review threads are clear, but the final Windows job exposed an instantaneous lease-observation failure in the active-descendant fixture. 22a0e93 now waits for the actual OS-held lease using only the remainder of the original four-second shutdown budget, following the existing subprocess test pattern. No production code or shutdown deadline changed. An independent holder test proves that an unreleased lock times out and succeeds only after release.

Validation: all 15 native cases passed on Windows and Linux; 40 concurrent Windows active-descendant cases passed; formatting/Clippy and independent review are clean. An additional 80-case process-handle diagnostic did not reproduce the immediate lock race locally, so the precise hosted scheduling cause is not claimed as proven. Current-head Copilot review and hosted CI have been requested. Keeping draft until those gates are verified.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-platform process ownership, detached transport threads, and shutdown races warrant final human validation.

Review effort: Balanced
Findings: None

Release the independent holder only after the same polling invocation observes WouldBlock, and assert exactly two real lock attempts. This removes scheduling assumptions and rejects an immediate-timeout mutation without changing shutdown semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The cross-platform transport and process-lifecycle changes are concurrency-sensitive, and macOS validation remains pending.

Review effort: Balanced
Findings: None

@karthiknadig

Copy link
Copy Markdown
Member Author

The corrected Windows job now passes, including the active-descendant test. A new macOS performance run exceeded discovery budgets (253 ms warm / 515 ms cold P50) despite no production changes since the previously passing 124 / 260 ms run. The earlier exact-base/head same-host control did not show this aggregate regression. I am retaining this failed evidence and rerunning only the macOS performance job once to check runner variability; budgets and code remain unchanged. The separate exact combined-protocol same-host control is also running. Auto-merge remains off until the results are inspected.

@karthiknadig

Copy link
Copy Markdown
Member Author

Final validation for 0fee376: all hosted build/test/coverage/performance/CodeQL checks have passed, including Windows active-descendant shutdown and both macOS architectures. Current-head Copilot review has no findings and there are no review threads. Linux/Windows line coverage improves by +1.995/+2.122 percentage points; all three CodeQL analyses report zero results, errors, and warnings. Linux performance is effectively neutral and Windows improves. The single macOS rerun passed without code or budget changes (warm/cold discovery P50 163/339 ms versus the anomalous 253/515 ms); the earlier failed run remains recorded above. Accepting runner-sensitive macOS aggregate variance based on the prior exact-production same-host control, while retaining its explicitly measured small first-notification tradeoff. Ready for protected auto-merge; required human approval must still be satisfied.

@karthiknadig
Karthik Nadig (karthiknadig) marked this pull request as ready for review September 25, 2026 22:20
@karthiknadig
Karthik Nadig (karthiknadig) merged commit de7611f into main Sep 25, 2026
39 of 40 checks passed
@karthiknadig
Karthik Nadig (karthiknadig) deleted the bug/issue-529-shutdown branch September 25, 2026 22:33
Karthik Nadig (karthiknadig) added a commit that referenced this pull request Sep 28, 2026
Merge main after #559 landed. The resolved tree is identical to the reviewed protocol head; the final PR diff now contains only protocol framing, envelope validation, tests, and documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Karthik Nadig (karthiknadig) added a commit that referenced this pull request Sep 28, 2026
Parse bounded multi-header JSONRPC frames and validate request envelopes
before dispatch, while preserving request IDs, notifications, and legacy
method-level error behavior.

- Read headers through the separator; accept case-insensitive names,
optional Content-Type, LF/CRLF, fragmented bytes, and consecutive
frames.
- Enforce exact 8 KiB header and 16 MiB payload limits before
allocation; reject duplicate, missing, invalid, and overflowing lengths.
- Return explicit errors for invalid envelopes/parameter containers and
recover after complete malformed JSON/UTF-8 frames.
- Add deterministic fragmentation, native wire, and
rejection-to-valid-dispatch regression coverage; document the supported
contract.

Validation: 51 library and 19 native cases; workspace formatting/Clippy
and independent code review pass. #559 is merged and main integrated;
only the five protocol files remain in this PR. Exact-main Linux/Windows
line and function coverage now improve, with no budget or exclusion
changes. Current-head Copilot has no code findings and all CodeQL
analyses are clean. The independently reviewed Windows matched-host
control found no material regression; detailed platform snapshots,
accepted tail outliers, and sampling limitations are recorded in the PR
comments. All 35 automated checks now pass, including both native macOS
jobs. Ready for required human approval and protected auto-merge.

Fixes #532

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop the JSONRPC server cleanly on stdin EOF instead of busy-looping

3 participants