Skip to content

PEG generator assigns a new token type to every occurrence of a keyword #160284

Closed

Description

@lysnikolaou

Bug report

KeywordCollectorVisitor.visit_StringLeaf in Tools/peg_generator/pegen/parser_generator.py is meant to give each hard keyword one token type, starting at 500. Its check for "already numbered" compares the keyword with its quotes against dictionary keys stored without them:

val = ast.literal_eval(node.value)
if re.match(r"[a-zA-Z_]\w*\Z", val):  # This is a keyword
    if node.value.endswith("'") and node.value not in self.keywords:
        self.keywords[val] = self.generator.keyword_type()

node.value is "'if'" but the keys are "if", so the check never matches. Every occurrence of a keyword in the grammar takes a fresh number and overwrites the previous one. As a result, the 35 hard keywords in Grammar/python.gram end up numbered between 500 and 721 instead of 500–534.

Parsing is unaffected, since keyword type numbers are internal to the generated parser. But they're sparse and depend on how often each keyword appears in the grammar, which gets in the way of anything that wants to represent keyword types as a bitmask (spoiler alert).

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. added a commit that references this issue on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-parsertype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions