Repository navigation
Conversation
…e offset _get_col_offsets can pass a byte offset of -1 to _PyPegen_byte_offset_to_character_offset_line when a token starts before the current line's span, as an INDENT token does when the tokenizer still holds earlier lines in its buffer. That function then reads data[-1], one byte before the line's UTF-8 buffer. The same-line fast path now only runs when the byte offset is non-negative, so the call never sees a negative offset. The other two calls in the function already pass non-negative offsets, and the callee is shared with the parser, so the guard belongs at this call site rather than in the callee. On main this is hardening: the mismatched bracket that used to reach the over-read is now a TokenError before any INDENT is produced, and no input was found that reaches it, so the new test asserts that TokenError rather than a crash. Verified with an AddressSanitizer build: the repro, the edge cases and 3,000 fuzz inputs run clean both before and after the fix, and the edge cases produce identical tokens.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes gh-162142
_get_col_offsets can pass a byte offset of -1 to _PyPegen_byte_offset_to_character_offset_line when a token starts before the current line's span, as an INDENT token does when the tokenizer still holds earlier lines in its buffer. That function then reads data[-1], one byte before the line's UTF-8 buffer. The same-line fast path now only runs when the byte offset is non-negative, so the call never sees a negative offset. The other two calls in the function already pass non-negative offsets, and the callee is shared with the parser, so the guard belongs at this call site rather than in the callee.
On main this is hardening, and I want to say that plainly: the mismatched bracket that used to reach the over-read is now a TokenError before any INDENT is produced, and no input was found that reaches the over-read on main. The new test asserts that TokenError rather than a crash. Verified with an AddressSanitizer build: the repro, the edge cases and 3,000 fuzz inputs run clean both before and after the fix, and the edge cases produce identical tokens. test_tokenize passes, 157 tests.
Note: this change was written with the help of Muse, Meta's AI assistant powered by Muse Spark 1.3, used as a coding assistant, and the result was fully reviewed and tested.