Performance regression: `asyncio.gather()` is slower and uses more memory since gh-157213

[deadlovelll]

Bug report

Bug description:

Reported by #157213 (comment)

The probable cause of the regression is this line:

        outer.add_done_callback(
            functools.partial(_discard_awaited_by, children, cur_task))

I'll push the PR and send benchmarks results to the comments

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS, Linux, Windows, Other

Linked PRs

• gh-158239: Fix asyncio.gather performance regression #158240

[deadlovelll]

Bench:

import asyncio, pyperf, sys

suffix = '_mem' if '--track-memory' in sys.argv else ''

async def recurse(level):
    if level:
        await asyncio.gather(*[recurse(level - 1) for _ in range(6)])

def eager_loop():
    loop = asyncio.new_event_loop()
    loop.set_task_factory(asyncio.eager_task_factory)
    return loop

runner = pyperf.Runner()
runner.bench_func('gather' + suffix, lambda: asyncio.run(recurse(6)))
runner.bench_func('gather_eager' + suffix, lambda: asyncio.run(recurse(6), loop_factory=eager_loop))

Results:

+------------------+----------+------------------------+
| Benchmark        | before   | after                  |
+==================+==========+========================+
| gather           | 184 ms   | 164 ms: 1.12x faster   |
+------------------+----------+------------------------+
| gather_eager     | 89.8 ms  | 56.5 ms: 1.59x faster  |
+------------------+----------+------------------------+
| gather_mem       | 78.9 MiB | 74.9 MiB: 1.05x faster |
+------------------+----------+------------------------+
| gather_eager_mem | 46.2 MiB | 19.6 MiB: 2.36x faster |
+------------------+----------+------------------------+
| Geometric mean   | (ref)    | 1.45x faster           |
+------------------+----------+------------------------+

[cmaloney]

As a note for the future this could have been a comment + re-open the original issue (it's all the same development and shared discussion space is nice). Separate issue also works :)