You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PEG generator assigns a new token type to every occurrence of a keyword #160284
KeywordCollectorVisitor.visit_StringLeaf in Tools/peg_generator/pegen/parser_generator.py is meant to give each hard keyword one token type, starting at 500. Its check for "already numbered" compares the keyword with its quotes against dictionary keys stored without them:
val=ast.literal_eval(node.value)
ifre.match(r"[a-zA-Z_]\w*\Z", val): # This is a keywordifnode.value.endswith("'") andnode.valuenotinself.keywords:
self.keywords[val] =self.generator.keyword_type()
node.value is "'if'" but the keys are "if", so the check never matches. Every occurrence of a keyword in the grammar takes a fresh number and overwrites the previous one. As a result, the 35 hard keywords in Grammar/python.gram end up numbered between 500 and 721 instead of 500–534.
Parsing is unaffected, since keyword type numbers are internal to the generated parser. But they're sparse and depend on how often each keyword appears in the grammar, which gets in the way of anything that wants to represent keyword types as a bitmask (spoiler alert).
Activity
gh-160284: Number each keyword once in the PEG generator (#160290)