Skip to content

Run Python matrix jobs in parallel - #667

Open
Aasav Badera (aasavbadera) wants to merge 4 commits into
masterfrom
aasavbadera/parallelism
Open

Aasav Badera (aasavbadera) wants to merge 4 commits into
masterfrom
aasavbadera/parallelism

Conversation

@aasavbadera

@aasavbadera Aasav Badera (aasavbadera) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Run the Python 3.9-3.13 matrix in parallel, with up to five jobs.
  • Run both unit and Kusto E2E suites with eight workers per version (up to 40 E2E workers per pipeline run), retaining separate merged JUnit reports.
  • Add test-only, per-request exponential backoff with jitter: five attempts including the initial request, a 30-second exponential-delay cap, and support for a longer server Retry-After.
  • Retry throttled requests only when their bodies can be replayed. Retry other transient HTTP/network failures only for queries and read-only management commands; do not blindly replay ambiguous ingestion failures or rerun entire tests.
  • Fail each CI E2E invocation after 25 minutes, terminating its process group and forcing termination after a further 30 seconds if necessary.
  • Propagate exhausted/permanent row-count request failures rather than silently swallowing them. Keep production SDK behavior, Azure authentication, and credential-free fork testing unchanged.

Validation

Latest validated head: 8b9b4ce92d313d7c2aa7c21db529eeb341a69b49.

Azure pipeline 20261116.12 / build 191465162 succeeded against this exact commit.

  • All five Python versions (3.9-3.13) ran concurrently on distinct hosted agents and succeeded on their first job attempt.
  • Per version: 290 unit tests passed, 26 existing unit skips; 31 E2E tests passed with no E2E skips. Both suites used eight workers, verified in the actual command output.
  • All five merged test runs were complete: 321 passed, 26 skipped, zero failures per version (347 cases).
  • Total observed log activity was 6.52 minutes; individual E2E invocations took 3.07-4.81 minutes. Timing uses the log index, from the earliest createdOn to the latest lastChangedOn, not retry-sensitive top-level build timestamps.
  • Formatting, lint, type checking, result publishing, and artifact tasks succeeded. GitHub CodeQL and CLA checks also passed.

The branch was rebased onto updated master containing #666 while the preceding run was executing. All seven files in this retry/concurrency/timeout update were unchanged by that rebase. The current baseline includes #666's local HTTP redirect fixtures, so unit/E2E counts differ from earlier runs. The preceding build 191464067 also passed on bc8f1619fe6813e25caacdf1ded8219ec149f5bb, with 6.84 minutes of log activity; the newer run above validates the actual rebased PR head.

Focused local validation of the retry implementation, before that rebase, also passed:

  • Actual eight-worker unit CI task: 282 passed, 26 existing skips (308 cases), including 19 new retry regressions.
  • The real E2E retry-wiring regression passed with only class setup stubbed, using mocked HTTP 503 responses: exactly five requests and the original failure propagated out of row-count polling.
  • Tests cover sync/async transient failures, a real aiohttp 429 response, Retry-After seconds/dates, exponential delays, exhaustion, permanent errors, unsafe write replay, and rewinding streaming bodies to their original position.
  • GNU timeout probes verified exit 124 and child termination; a TERM-ignoring process group was forcibly killed and returned 137. Both are failures, not successful fallbacks.
  • Full Ruff formatting/lint, type checking (zero diagnostics), YAML/TOML/Bash parsing, task expansion, and git diff --check passed.

Prior observed timings

The original five-job/eight-worker run, build 191449502, passed in 5.32 minutes of log activity. The subsequent one-worker E2E run, build 191454946, passed in 11.82 minutes. This update restores the higher worker count with targeted backoff and an overall deadline instead of the one-worker restriction.

For context, the successful serial baseline build 191431977 took 34.21 minutes. These are observed log-index spans, not controlled benchmarks; the serial master run also generated documentation.

Scope and limitations

Retry limits apply to individual wrapped Kusto transport invocations. Existing higher-level SDK retry/fallback policies remain unchanged; the 25-minute deadline bounds the complete CI E2E invocation. The fixture is scoped to the two E2E modules, including their class setup, and does not change ordinary unit-test or production SDK behavior.

Concurrency is per pipeline run, not a global budget or strict request-rate limit. Concurrent builds still share backend capacity. Hard termination may interrupt fixture cleanup and leave temporary test tables.

@aasavbadera

Copy link
Copy Markdown
Collaborator Author

E2E validation is running on commit 5a7d436357592a246fcf00b6676f5f84d8559eaf from aasavbadera/parallelism:

Azure pipeline 20261116.6 / build 191449502

This run uses the existing pipeline and trusted Azure service connection to test all five Python versions with maxParallel: '5'. Job overlap and the E2E results will be checked before reporting validation complete.

Allow all five Python versions to run concurrently and update the contributor documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 228686cb-da70-40cd-afb9-bb07f8af664f
Keep eight-worker unit testing and five parallel Python matrix jobs, run E2E tests separately with one worker, and publish both JUnit reports together.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 228686cb-da70-40cd-afb9-bb07f8af664f
Retry replayable transient Kusto requests in E2E tests, preserve unsafe-write and permanent-error failures, and bound each CI E2E process group to 25 minutes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 228686cb-da70-40cd-afb9-bb07f8af664f
Comment thread .azuredevops/build.yml Resolved
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 228686cb-da70-40cd-afb9-bb07f8af664f

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the Python CI parallelization and test-only retry changes.

No open issues 🎉

What changed in this PR

This PR reduces Python CI runtime by running the five-version matrix concurrently, with test-only request retries and an E2E timeout to manage transient failures without changing production SDK behavior.

Changes:

  • Run up to five matrix jobs concurrently, using eight workers for each suite and separate JUnit reports merged for publishing.
  • Add replay-aware request retries with exponential backoff, jitter, and Retry-After support, plus a 25-minute CI E2E deadline.
  • Propagate row-count request failures and add regression coverage for retry behavior.
File Description
pyproject.toml Adds an eight-worker E2E task and separate report paths.
e2e_retry.py Implements synchronous and asynchronous replay-aware retries.
CONTRIBUTING.md Documents concurrency, retries, timeouts, and reporting.
conftest.py Applies request retries only to E2E modules, including setup.
azure-kusto-ingest/​tests/​test_e2e_retry.py Tests retry limits, safety, delays, and stream replay.
azure-kusto-ingest/​tests/​test_e2e_ingest.py Propagates row-count failures and tests retry exhaustion.
.gitignore Ignores generated test reports.
.azuredevops/​build.yml Enables parallel jobs, bounds E2E runtime, and merges reports.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants