Skip to content

Use sysmon for coverage on Python 3.12 - #970

Merged
junkmd merged 4 commits into
enthought:mainfrom
junkmd:use_sysmon_coverage_core_on_py312
Oct 19, 2026
Merged

junkmd merged 4 commits into
enthought:mainfrom
junkmd:use_sysmon_coverage_core_on_py312

Conversation

@junkmd

@junkmd junkmd commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Set COVERAGE_CORE=sysmon only in the Python 3.12 coverage test step.
The idea came from python-pillow/Pillow#7820.

Why this is faster

The normal coverage core uses sys.settrace. Python 3.12 makes this kind of tracing very slow. sysmon uses Python's newer sys.monitoring API.
It gets line events from Python with less extra work, so the test job can finish faster.

Coverage safety

This project uses line coverage. It does not enable branch coverage.
sysmon on Python 3.12 records line events, so it checks the same kind of coverage as before.

The test command, the comtypes source scope, and the omitted generated files do not change.
A line that the tests do not run is still reported as missing.
In other words, this change should not make line coverage report a false pass for an untested line.

sysmon cannot measure branch coverage on Python 3.12.
This is safe here because branch coverage is not enabled.
If we enable branch coverage later, we must not use sysmon for the Python 3.12 coverage job.

Scope

  • Python 3.12 coverage tests use sysmon.
  • All other Python versions keep the current coverage core.
  • Other jobs and test commands do not change.

@junkmd junkmd added the ci label Oct 18, 2026
@codecov-commenter

codecov-commenter commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.55%. Comparing base (009c9d1) to head (f026043).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #970   +/-   ##
=======================================
  Coverage   89.55%   89.55%           
=======================================
  Files         143      143           
  Lines       14147    14147           
=======================================
  Hits        12669    12669           
  Misses       1478     1478           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@junkmd
junkmd force-pushed the use_sysmon_coverage_core_on_py312 branch from 4444ab8 to 3609c8c Compare October 18, 2026 08:05
@junkmd

junkmd commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator Author

Tests that sometimes took more than 20 minutes have certainly become faster, but the "memory leak tests" discussed in #971 still fail intermittently.

I don't know whether setting sysmon increases the chance of test failures, but I plan to wait before merging until we understand whether there is a causal link or the "memory leak tests" are fixed.

@junkmd
junkmd marked this pull request as draft October 18, 2026 14:28
@junkmd
junkmd force-pushed the use_sysmon_coverage_core_on_py312 branch from 13877d6 to 911012d Compare October 19, 2026 00:13
Use coverage.py's sys.monitoring-based measurement core only in the
Python 3.12 unit-test matrix jobs. This avoids the significant overhead
of the traditional tracing core on Python 3.12 while preserving the
existing coverage behavior for every other Python version.

The project does not enable branch coverage, which sysmon does not
support on Python 3.12.
Some unit-test combinations are flaky, and canceling the matrix on the first
failure makes reruns harder to complete.
Keep the in-flight variants running so we can collect the remaining signal
and retry only the genuinely broken cases.
Tests that sometimes took more than 20 minutes have certainly become
faster, but the "memory leak tests" discussed in enthought#971 still fail
randomly.

As a temporary workaround, skip `TestInproc_win32com.test_eval` on Python
3.12 in addition to Python >= 3.14.
@junkmd
junkmd force-pushed the use_sysmon_coverage_core_on_py312 branch from 3847c9a to a7372f2 Compare October 19, 2026 12:25
@junkmd
junkmd marked this pull request as ready for review October 19, 2026 12:45
@junkmd junkmd added this to the 1.5.0 / Support Python 3.15 milestone Oct 19, 2026
@junkmd

junkmd commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

The reason for merging this PR, which includes fail-fast: false and skipping tests on a specific Python version, is that the memory leak tests may not be reliable and CI problems are making maintenance harder.

First, the assertions in the memory leak tests may be wrong.
They may report a memory leak even when there is no real leak. The tests may also be liar tests, which means they may not find real memory leaks.
However, we do not know if either of these is true. We also do not know why the tests fail so often, but we have seen many failures on Python 3.12.

Even if the tests are not wrong and are not liar tests, we can still run them on other Python versions.
So, skipping these tests only on Python 3.12 does not mean that we lose all test coverage for them.

Also, these tests are related to a feature that works with a third-party system.
We cannot say for sure that comtypes itself is causing the failures. Rather than spending more time trying to find the cause now, we should first make sure that CI can finish reliably.
Slow CI runs and failures that do not happen every time can block maintenance work, including reviews and releases.

Also, the tests on Python 3.12 with COVERAGE_CORE=sysmon tend to finish sooner than the tests on other Python versions.
If a failure on Python 3.12 stops the tests on other versions, we may never get all the test results.
Setting fail-fast: false makes sure that a failure on one version does not stop the tests on other versions.

These changes do not mean that we have found or fixed the cause of the memory leak test failures.
They are a practical way to keep testing on other Python versions, make CI more reliable, and keep maintenance work moving.

@junkmd
junkmd merged commit bfe275a into enthought:main Oct 19, 2026
149 of 151 checks passed
@junkmd
junkmd deleted the use_sysmon_coverage_core_on_py312 branch October 19, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants