Repository navigation
Close merge-created iterators throughout initialization and cleanup - #246
mayankdhingra02 wants to merge 2 commits into
Conversation
maxfischer2781
left a comment
There was a problem hiding this comment.
Thanks for the contribution. The approach seems sensible but I am concerned about maintainability. Please see comments for details.
| ) -> "AsyncIterator[_KeyIter[Any]]": | ||
| for iterable in iterables: | ||
| iterator = aiter(iterable) | ||
| iterator = await stack.enter_context(ScopedIter(iterable)) |
There was a problem hiding this comment.
This change and the required infrastructure to support it seem very extensive. Is there a reason why you preferred this approach over just having a finally cleanup harness in this function?
There was a problem hiding this comment.
@mayankdhingra02 It seems you rolled back this change in favour of a much simpler and less feature rich approach. Which just raises the new question: Why go with so much less now? Why not go with all the features you considered necessary before?
|
|
||
|
|
||
| @sync | ||
| async def test_merge_closes_exhausted_and_empty_created_iterators(): |
There was a problem hiding this comment.
Why does this need testing?
maxfischer2781
left a comment
There was a problem hiding this comment.
See my earlier comment for the actual code change. I am still rather confused by the testing approach, see comments of this review.
| _KeyIter[Any].from_iters(iterables, reverse, a_key) | ||
| ) | ||
| ] | ||
| iterators: "list[AsyncIterator[Any]]" = [] |
There was a problem hiding this comment.
I don't understand why AsyncIterators are stored here. The further usage only cares about ACloseables. Can you clarify the need for this?
| await itr.tail.aclose() | ||
| # Attempt every cleanup, even when another iterator's aclose fails. | ||
| close_error: Optional[BaseException] = None | ||
| for iterator in reversed(iterators): |
There was a problem hiding this comment.
Is there a reason for going through the iterators in reverse?
| except BaseException as exc: # noqa: B036 | ||
| close_error = exc |
There was a problem hiding this comment.
Since this explicitly handles BaseExceptions, I am somewhat surprised by how they are handled – namely throwing away any previous exception no matter what. Surely – say – completely discarding a KeyboardInterrupt in favour of a mere ValueError is not appropriate, wouldn't you agree?
|
|
||
|
|
||
| @sync | ||
| async def test_merge_closes_iterators_when_aiter_fails(): |
There was a problem hiding this comment.
It looks like several tests here merely differ by the iterators, errors and keys. Can't that be parameterized?
| yield 1 | ||
| yield 3 |
There was a problem hiding this comment.
I don't understand why 1 and 3 are yielded here. Why not 2? Does that have some meaning?
| @sync | ||
| async def test_merge_closes_iterators_when_initial_key_fails(make_key): | ||
| closed = [] | ||
| started = [] |
There was a problem hiding this comment.
I don't understand why this test covers started iterators. Its name and consistency with similar tests implies that it should not. Can you explain this?
| yield 5 | ||
|
|
||
| def key(value): | ||
| # The first iterator is initialized before the second key fails. |
There was a problem hiding this comment.
I am confused by this comment. Just a few lines later, all iterators are initialized before any key is computed.
| finally: | ||
| closed.append("second") | ||
|
|
||
| async def later(): |
There was a problem hiding this comment.
All other iterators are named after their position. Why is this called later? later compared to what?
| async def second(): | ||
| try: | ||
| yield 2 | ||
| yield 4 |
There was a problem hiding this comment.
Why is this 4 here? Sorry, but I am utterly confused by the seemingly magic numbers.
|
|
||
| iterators = [first(), second()] | ||
| merged = a.merge(*iterators) | ||
| assert await a.anext(merged) == 1 |
There was a problem hiding this comment.
I don't understand why the test seemingly arbitrarily tests what happens well before the exception-under-test occurs. Can you clarify?
merge()now records each iterator as soon as it is created and runs its existingtry/finallycleanup around initialization as well as iteration. That covers failures while creating an input iterator, reading its first item or computing its key, plus empty and exhausted inputs that never remain in the heap.Closes #245.
Cleanup runs in reverse creation order and attempts every close. If a close fails, the last cleanup error is propagated after the remaining iterators are closed. Externally scoped iterators retain their ownership behavior.
The regressions live with the existing heapq tests and use explicit async generators and the test suite's
sync/awaitifyhelpers. Seven fail on unchanged source; the interruption and scoped-ownership controls pass there.Validation:
..., which excludes every line. A supplemental untracked configuration escaping that pattern measures 97% overall and 97% in heapq; no coverage configuration is changed here.These checks ran on macOS; hosted Linux CI remains to run.