Skip to content

gh-162142: Guard the same-line fast path against a negative byte offset - #162272

Open
tydavid5 wants to merge 1 commit into
python:mainfrom
tydavid5:fix/tokenize-negative-byte-offset-162142
Open

tydavid5 wants to merge 1 commit into
python:mainfrom
tydavid5:fix/tokenize-negative-byte-offset-162142

Conversation

@tydavid5

Copy link
Copy Markdown

Fixes gh-162142

_get_col_offsets can pass a byte offset of -1 to _PyPegen_byte_offset_to_character_offset_line when a token starts before the current line's span, as an INDENT token does when the tokenizer still holds earlier lines in its buffer. That function then reads data[-1], one byte before the line's UTF-8 buffer. The same-line fast path now only runs when the byte offset is non-negative, so the call never sees a negative offset. The other two calls in the function already pass non-negative offsets, and the callee is shared with the parser, so the guard belongs at this call site rather than in the callee.

On main this is hardening, and I want to say that plainly: the mismatched bracket that used to reach the over-read is now a TokenError before any INDENT is produced, and no input was found that reaches the over-read on main. The new test asserts that TokenError rather than a crash. Verified with an AddressSanitizer build: the repro, the edge cases and 3,000 fuzz inputs run clean both before and after the fix, and the edge cases produce identical tokens. test_tokenize passes, 157 tests.

Note: this change was written with the help of Muse, Meta's AI assistant powered by Muse Spark 1.3, used as a coding assistant, and the result was fully reviewed and tested.

…e offset

_get_col_offsets can pass a byte offset of -1 to _PyPegen_byte_offset_to_character_offset_line when a token starts before the current line's span, as an INDENT token does when the tokenizer still holds earlier lines in its buffer. That function then reads data[-1], one byte before the line's UTF-8 buffer. The same-line fast path now only runs when the byte offset is non-negative, so the call never sees a negative offset. The other two calls in the function already pass non-negative offsets, and the callee is shared with the parser, so the guard belongs at this call site rather than in the callee.

On main this is hardening: the mismatched bracket that used to reach the over-read is now a TokenError before any INDENT is produced, and no input was found that reaches it, so the new test asserts that TokenError rather than a crash. Verified with an AddressSanitizer build: the repro, the edge cases and 3,000 fuzz inputs run clean both before and after the fix, and the edge cases produce identical tokens.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tokenize: _get_col_offsets can pass a negative byte offset to _PyPegen_byte_offset_to_character_offset_line (1-byte heap over-read)

1 participant