Repository navigation
Improving the memory leak test that fails randomly in CI. #971
Description
Problems
find_memleak in comtypes.test.test_comserver fails randomly in the CI environment.
- The current test checks the difference in the Working Set before and after running the process many times.
- An increase in the Working Set does not always mean an actual memory leak.
- On the other hand, even if the Working Set does not grow, we cannot always find a leak.
- In Local Server tests, measuring memory on the client side cannot directly find leaks on the COM server side running in another process.
Because of this, the test not only fails randomly, but we might also miss real memory leaks.
Technical Solution Candidates: Detect Actual Memory Leaks
- Separate feature tests from memory checks
- Verify COM return values and types in normal feature tests.
- Rethink judgments based only on the difference in the Working Set.
- Directly check the release rules
- Check the transfer of ownership and the matching release process for BSTR, VARIANT, and SAFEARRAY.
- Check not only calls like
VariantClear, but also resource management at the actual COM boundary.
- Look into native memory leaks separately
- Check memory managed by Python using tools like
tracemallocfrom the Python standard library. - If needed, measure both the client and COM server processes, and check for native memory leaks using Windows diagnostic tools.
Completion Condition
Set up a test that reduces random failures caused by memory usage changes in normal CI, while being able to detect actual memory leaks in BSTR, VARIANT, and SAFEARRAY.
Activity
junkmd commentedon Oct 18, 2026
Why Working Set measurements cannot determine memory leaks
See also https://learn.microsoft.com/en-us/windows/win32/psapi/process-memory-usage-information
1. What is the Working Set?
The Working Set in Windows represents the amount of memory pages currently resident in physical memory (RAM) for a process at a given time.
It is neither the total amount of memory allocated by the process nor the amount of memory left unreleased.
Windows loads pages into RAM or pages them out to storage depending on memory usage. Therefore, even if an application performs the exact same tasks, the Working Set can change depending on the running environment and the timing of the measurement.
2. An Increase in the Working Set Does Not Always Mean a Memory Leak
When the Working Set increases, we only know that more memory is resident in RAM at the time of measurement. However, we cannot tell whether that increase is due to a failure to release memory.
For example, consider the following cases:
In such cases, an increase in the Working Set does not necessarily mean an actual memory leak is happening.
Therefore, treating an increase in the Working Set directly as a memory leak leads to false positives.
3. Does No Change in the Working Set Mean No Memory Leak?
No. A lack of change in the Working Set is not proof that there are no memory leaks.
For example, consider the following cases:
In short, the only thing we can learn from a result where the Working Set does not change is that there was no observable difference in the total amount of pages resident in physical memory between the two measured points in time.
It is not a direct indicator of whether a memory leak exists.
4. Limitations of Judgments Based on Working Set Differences
Working Set differences can be used as an indicator to observe changes in memory usage. However, they are insufficient as an indicator to determine memory leaks.
Furthermore, increasing the number of measurements or adjusting thresholds does not solve this fundamental problem.
5. How to Verify Memory Leaks
To verify memory leaks, we need to make clear what should be released and investigate using a method that fits the nature of that resource.
tracemalloc.Note that none of these methods alone can prove all leaks. In particular, we must distinguish between confirming that a release API was called and confirming that the native memory was actually released properly.
Conclusion
The Working Set is the amount of pages resident in physical memory at a given time, not the amount of unreleased memory.
Therefore, an increase in the Working Set does not always mean there is a memory leak, and no change does not mean there is no memory leak.
While a method like
find_memleakthat compares Working Set differences to zero can detect changes in memory usage, it cannot reliably determine the presence or absence of a memory leak.To stabilize tests, rather than simply adjusting measurement thresholds, we need to directly inspect the ownership and release process of the resources we want to test, and complement the investigation of actual native memory leaks with proper diagnostic tools.
junkmd commentedon Oct 18, 2026
@haoxuanjng-lang
I opened this issue based on what I know right now, as mentioned in #962 (comment).
If you can look into this, please leave a comment here.
I recommend resolving
VT_I1issues with a higher priority, because client-side use is more common in this package.This is because the problem in this issue happens in server-side tests, which is complex and used by fewer users.
test: temporarily skip flaky test_eval on Python 3.9
test: temporarily skip flaky test_eval on Python 3.12.
Use `sysmon` for coverage on Python 3.12 (#970)