Skip to content

Close merge-created iterators throughout initialization and cleanup - #246

Open
mayankdhingra02 wants to merge 2 commits into
maxfischer2781:masterfrom
mayankdhingra02:codex/merge-iterator-cleanup
Open

mayankdhingra02 wants to merge 2 commits into
maxfischer2781:masterfrom
mayankdhingra02:codex/merge-iterator-cleanup

Conversation

@mayankdhingra02

@mayankdhingra02 mayankdhingra02 commented Nov 15, 2026 •

Copy link
Copy Markdown

merge() now records each iterator as soon as it is created and runs its existing try/finally cleanup around initialization as well as iteration. That covers failures while creating an input iterator, reading its first item or computing its key, plus empty and exhausted inputs that never remain in the heap.

Closes #245.

Cleanup runs in reverse creation order and attempts every close. If a close fails, the last cleanup error is propagated after the remaining iterators are closed. Externally scoped iterators retain their ownership behavior.

The regressions live with the existing heapq tests and use explicit async generators and the test suite's sync/awaitify helpers. Seven fail on unchanged source; the interruption and scoped-ownership controls pass there.

Validation:

  • Full tests pass on all configured runtimes: CPython 3.10–3.14 and PyPy 3.8/3.10. Python 3.14 has 457 passes and four existing skips.
  • Black, flake8, mypy, Pyright and a strict Sphinx HTML build pass.
  • The repository's default coverage exclusions include bare ..., which excludes every line. A supplemental untracked configuration escaping that pattern measures 97% overall and 97% in heapq; no coverage configuration is changed here.

These checks ran on macOS; hosted Linux CI remains to run.

@maxfischer2781 maxfischer2781 self-assigned this Nov 16, 2026

@maxfischer2781 maxfischer2781 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. The approach seems sensible but I am concerned about maintainability. Please see comments for details.

Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread asyncstdlib/heapq.py Outdated
) -> "AsyncIterator[_KeyIter[Any]]":
for iterable in iterables:
iterator = aiter(iterable)
iterator = await stack.enter_context(ScopedIter(iterable))

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change and the required infrastructure to support it seem very extensive. Is there a reason why you preferred this approach over just having a finally cleanup harness in this function?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mayankdhingra02 It seems you rolled back this change in favour of a much simpler and less feature rich approach. Which just raises the new question: Why go with so much less now? Why not go with all the features you considered necessary before?

Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Resolved
Comment thread unittests/test_heapq_cleanup.py Outdated


@sync
async def test_merge_closes_exhausted_and_empty_created_iterators():

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need testing?

@maxfischer2781 maxfischer2781 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my earlier comment for the actual code change. I am still rather confused by the testing approach, see comments of this review.

Comment thread asyncstdlib/heapq.py
_KeyIter[Any].from_iters(iterables, reverse, a_key)
)
]
iterators: "list[AsyncIterator[Any]]" = []

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why AsyncIterators are stored here. The further usage only cares about ACloseables. Can you clarify the need for this?

Comment thread asyncstdlib/heapq.py
await itr.tail.aclose()
# Attempt every cleanup, even when another iterator's aclose fails.
close_error: Optional[BaseException] = None
for iterator in reversed(iterators):

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for going through the iterators in reverse?

Comment thread asyncstdlib/heapq.py
Comment on lines +173 to +174
except BaseException as exc: # noqa: B036
close_error = exc

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this explicitly handles BaseExceptions, I am somewhat surprised by how they are handled – namely throwing away any previous exception no matter what. Surely – say – completely discarding a KeyboardInterrupt in favour of a mere ValueError is not appropriate, wouldn't you agree?

Comment thread unittests/test_heapq.py


@sync
async def test_merge_closes_iterators_when_aiter_fails():

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like several tests here merely differ by the iterators, errors and keys. Can't that be parameterized?

Comment thread unittests/test_heapq.py
Comment on lines +63 to +64
yield 1
yield 3

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why 1 and 3 are yielded here. Why not 2? Does that have some meaning?

Comment thread unittests/test_heapq.py
@sync
async def test_merge_closes_iterators_when_initial_key_fails(make_key):
closed = []
started = []

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why this test covers started iterators. Its name and consistency with similar tests implies that it should not. Can you explain this?

Comment thread unittests/test_heapq.py
yield 5

def key(value):
# The first iterator is initialized before the second key fails.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused by this comment. Just a few lines later, all iterators are initialized before any key is computed.

Comment thread unittests/test_heapq.py
finally:
closed.append("second")

async def later():

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All other iterators are named after their position. Why is this called later? later compared to what?

Comment thread unittests/test_heapq.py
async def second():
try:
yield 2
yield 4

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this 4 here? Sorry, but I am utterly confused by the seemingly magic numbers.

Comment thread unittests/test_heapq.py

iterators = [first(), second()]
merged = a.merge(*iterators)
assert await a.anext(merged) == 1

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why the test seemingly arbitrarily tests what happens well before the exception-under-test occurs. Can you clarify?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: merge leaves started sources open when initialization fails

2 participants