Repository navigation
Description
Crash report
What happened?
Background
I decided today to experiment with trying to run hypothesmith with hypofuzz on Python's parser. Ironically, I ran into an assertion failure in the interpreter loop before I could get anywhere with testing the parser :)
I apologize in advance I haven't yet been able to make a fully minimal reproducer, but I wanted to open this now that I have narrowed it down to the frame where the assertion failure is happening. I've also tried to give as much relevant information as I can.
Environment
This was produced with a clean build of CPython main. I was also able to reproduce this with 3.14. I used the following configure flags to build:
./configure --with-pydebug CC=clang-20
The tests were run on Ubuntu 24.04 on an x86_64 system.
Initial error message
After running the fuzzer (see linked repo with reproducer below for full details) for a few seconds I got:
python: Python/generated_cases.c.h:12217: PyObject *_PyEval_EvalFrameDefault(PyThreadState *, _PyInterpreterFrame *, int): Assertion `frame->instr_ptr->op.code == INSTRUMENTED_LINE || frame->instr_ptr->op.code == INSTRUMENTED_INSTRUCTION || _PyOpcode_Deopt[frame->instr_ptr->op.code] == SEND || _PyOpcode_Deopt[frame->instr_ptr->op.code] == FOR_ITER || _PyOpcode_Deopt[frame->instr_ptr->op.code] == INTERPRETER_EXIT || _PyOpcode_Deopt[frame->instr_ptr->op.code] == ENTER_EXECUTOR' failed.
Fatal Python error: Aborted
This points to
cpython/Python/generated_cases.c.h
Lines 12212 to 12217 in b13a5df
| assert(frame->instr_ptr->op.code == INSTRUMENTED_LINE || | |
| frame->instr_ptr->op.code == INSTRUMENTED_INSTRUCTION || | |
| _PyOpcode_Deopt[frame->instr_ptr->op.code] == SEND || | |
| _PyOpcode_Deopt[frame->instr_ptr->op.code] == FOR_ITER || | |
| _PyOpcode_Deopt[frame->instr_ptr->op.code] == INTERPRETER_EXIT || | |
| _PyOpcode_Deopt[frame->instr_ptr->op.code] == ENTER_EXECUTOR); |
which is inside of the case for
YIELD_VALUE.
Faulthandler output for the above assertion error
python: Python/generated_cases.c.h:12217: PyObject *_PyEval_EvalFrameDefault(PyThreadState *, _PyInterpreterFrame *, int): Assertion `frame->instr_ptr->op.code == INSTRUMENTED_LINE || frame->instr_ptr->op.code == INSTRUMENTED_INSTRUCTION || _PyOpcode_Deopt[frame->instr_ptr->op.code] == SEND || _PyOpcode_Deopt[frame->instr_ptr->op.code] == FOR_ITER || _PyOpcode_Deopt[frame->instr_ptr->op.code] == INTERPRETER_EXIT || _PyOpcode_Deopt[frame->instr_ptr->op.code] == ENTER_EXECUTOR' failed.
Fatal Python error: Aborted
Thread 0x00007bffe87e16c0 [Thread-5 (_work] (most recent call first):
File "/home/emma/cpython/Lib/pathlib/__init__.py", line 1070 in rename
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/database.py", line 484 in save
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/database.py", line 1132 in _worker
File "/home/emma/cpython/Lib/threading.py", line 1016 in run
File "/home/emma/cpython/Lib/threading.py", line 1074 in _bootstrap_inner
File "/home/emma/cpython/Lib/threading.py", line 1036 in _bootstrap
Thread 0x00007bffe9aeb6c0 [Thread-4 (_work] (most recent call first):
File "/home/emma/cpython/Lib/threading.py", line 362 in wait
File "/home/emma/cpython/Lib/queue.py", line 202 in get
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/database.py", line 1131 in _worker
File "/home/emma/cpython/Lib/threading.py", line 1016 in run
File "/home/emma/cpython/Lib/threading.py", line 1074 in _bootstrap_inner
File "/home/emma/cpython/Lib/threading.py", line 1036 in _bootstrap
Thread 0x00007bffeadf56c0 [Thread-1] (most recent call first):
File "/home/emma/cpython/Lib/threading.py", line 362 in wait
File "/home/emma/cpython/Lib/queue.py", line 202 in get
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/api.py", line 379 in dispatch_events
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/api.py", line 213 in run
File "/home/emma/cpython/Lib/threading.py", line 1074 in _bootstrap_inner
File "/home/emma/cpython/Lib/threading.py", line 1036 in _bootstrap
Thread 0x00007bffec0ff6c0 [Thread-2] (most recent call first):
File "/home/emma/cpython/Lib/threading.py", line 362 in wait
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/utils/delayed_queue.py", line 47 in get
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_buffer.py", line 38 in read_event
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify.py", line 136 in queue_events
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/api.py", line 158 in run
File "/home/emma/cpython/Lib/threading.py", line 1074 in _bootstrap_inner
File "/home/emma/cpython/Lib/threading.py", line 1036 in _bootstrap
Current thread 0x00007bffed4096c0 [Thread-3] (most recent call first):
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_c.py", line 278 in _recursive_simulate
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_c.py", line 381 in read_events
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_buffer.py", line 82 in run
File "/home/emma/cpython/Lib/threading.py", line 1074 in _bootstrap_inner
File "/home/emma/cpython/Lib/threading.py", line 1036 in _bootstrap
Thread 0x00007ffff7e87300 [python] (most recent call first):
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/database.py", line 1234 in choices_to_bytes
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/internal/conjecture/choice.py", line 629 in choices_size
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/internal/conjecture/data.py", line 860 in _draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/internal/conjecture/data.py", line 917 in draw_integer
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/strategies/_internal/strategies.py", line 740 in do_filtered_draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/strategies/_internal/strategies.py", line 715 in do_draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/internal/conjecture/data.py", line 1232 in draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/extra/lark.py", line 184 in draw_symbol
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesmith/syntactic.py", line 74 in draw_symbol
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/extra/lark.py", line 186 in draw_symbol
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesmith/syntactic.py", line 74 in draw_symbol
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/extra/lark.py", line 161 in do_draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesmith/syntactic.py", line 56 in do_draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/internal/conjecture/data.py", line 1237 in draw
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/control.py", line 175 in prep_args_kwargs_from_strategies
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/core.py", line 1038 in run
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/core.py", line 818 in default_executor
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/core.py", line 1135 in execute_once
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypothesis/core.py", line 1195 in _execute_once_for_engine
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypofuzz/hypofuzz.py", line 392 in _execute_once
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypofuzz/hypofuzz.py", line 299 in run_one
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypofuzz/hypofuzz.py", line 595 in start
File "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/hypofuzz/hypofuzz.py", line 711 in _start_worker
File "/home/emma/cpython/Lib/multiprocessing/process.py", line 108 in run
File "/home/emma/cpython/Lib/multiprocessing/process.py", line 320 in _bootstrap
File "/home/emma/cpython/Lib/multiprocessing/spawn.py", line 135 in _main
File "/home/emma/cpython/Lib/multiprocessing/forkserver.py", line 379 in _serve_one
File "/home/emma/cpython/Lib/multiprocessing/forkserver.py", line 339 in main
File "<string>", line 1 in <module>
Current thread's C stack trace (most recent call first):
Binary file "/home/emma/parsefuzz/.venv/bin/python", at ___interceptor_backtrace+0x4a [0x55555581939a]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at _Py_DumpStack+0x109 [0x555555fb8269]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0xab998b [0x55555600d98b]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x45330 [0x7ffff7c45330]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at pthread_kill+0x11c [0x7ffff7c9eb2c]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at gsignal+0x1e [0x7ffff7c4527e]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at abort+0xdf [0x7ffff7c288ff]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x2881b [0x7ffff7c2881b]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3b517 [0x7ffff7c3b517]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at _PyEval_EvalFrameDefault+0x3fceb [0x555555df4a6b]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x8607c9 [0x555555db47c9]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x5182a3 [0x555555a6c2a3]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x515c67 [0x555555a69c67]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x900eff [0x555555e54eff]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x50c313 [0x555555a60313]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at _PyEval_EvalFrameDefault+0x31aca [0x555555de684a]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x8607c9 [0x555555db47c9]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x5182a3 [0x555555a6c2a3]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x515c67 [0x555555a69c67]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0xbb067e [0x55555610467e]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0xa5d80d [0x555555fb180d]
Binary file "/home/emma/parsefuzz/.venv/bin/python", at +0x31d4bb [0x5555558714bb]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x9caa4 [0x7ffff7c9caa4]
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x129c3c [0x7ffff7d29c3c]
Extension modules: psutil._psutil_linux, psutil._psutil_posix (total: 2)
Debugging the issue
I modified the interpreter code locally to the following to debug the issue:
// replaces the assert on line 12212
if(!(frame->instr_ptr->op.code == INSTRUMENTED_LINE ||
frame->instr_ptr->op.code == INSTRUMENTED_INSTRUCTION ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == SEND ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == FOR_ITER ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == INTERPRETER_EXIT ||
_PyOpcode_Deopt[frame->instr_ptr->op.code] == ENTER_EXECUTOR))
{
printf("%d\n", frame->instr_ptr->op.code);
printf("%d\n", _PyOpcode_Deopt[frame->instr_ptr->op.code]);
printf("%d\n", frame->owner);
PyCodeObject *code = PyUnstable_InterpreterFrame_GetCode(frame);
printf("%s", PyUnicode_AsUTF8(code->co_qualname));
exit(1);
}This output the following information about the opcode getting executed.
# frame->instr_ptr->op.code
237 # INSTRUMENTED_FOR_ITER
# _PyOpcode_Deopt[frame->instr_ptr->op.code]
237 # INSTRUMENTED_FOR_ITER
# frame->owner
0 # FRAME_OWNED_BY_THREAD
# code->co_qualname
Inotify.read_events.<locals>._recursive_simulate
It took me a minute to find where this came from, but I tracked it down to the watchdog package, v6.0.0:
Disassmebly of relevant method with dis
-- MAKE_CELL 0 (self)
269 RESUME 0
276 LOAD_CONST 1 (<code object __annotate__ at 0x71835eeda5d0, file "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_c.py", line 276>)
MAKE_FUNCTION
LOAD_FAST_BORROW 0 (self)
BUILD_TUPLE 1
LOAD_CONST 2 (<code object _recursive_simulate at 0x71f35ef2f990, file "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_c.py", line 276>)
MAKE_FUNCTION
SET_FUNCTION_ATTRIBUTE 8 (closure)
SET_FUNCTION_ATTRIBUTE 16 (annotate)
STORE_FAST 2 (_recursive_simulate)
304 LOAD_CONST 3 (b'')
STORE_FAST 3 (event_buffer)
305 L1: NOP
306 NOP
307 L2: LOAD_DEREF 0 (self)
LOAD_ATTR 0 (_lock)
COPY 1
LOAD_SPECIAL 1 (__exit__)
SWAP 2
SWAP 3
LOAD_SPECIAL 0 (__enter__)
CALL 0
L3: POP_TOP
308 LOAD_DEREF 0 (self)
LOAD_ATTR 2 (_closed)
TO_BOOL
POP_JUMP_IF_FALSE 13 (to L6)
NOT_TAKEN
309 BUILD_LIST 0
307 L4: SWAP 3
SWAP 2
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
CALL 3
POP_TOP
L5: RETURN_VALUE
311 L6: LOAD_CONST 4 (True)
LOAD_DEREF 0 (self)
STORE_ATTR 2 (_is_reading)
307 L7: LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
CALL 3
POP_TOP
313 L8: LOAD_DEREF 0 (self)
LOAD_ATTR 7 (_check_inotify_fd + NULL|self)
CALL 0
TO_BOOL
POP_JUMP_IF_FALSE 34 (to L11)
L9: NOT_TAKEN
314 L10: LOAD_GLOBAL 8 (os)
LOAD_ATTR 10 (read)
PUSH_NULL
LOAD_DEREF 0 (self)
LOAD_ATTR 12 (_inotify_fd)
LOAD_FAST_BORROW 1 (event_buffer_size)
CALL 2
STORE_FAST 3 (event_buffer)
316 L11: LOAD_DEREF 0 (self)
LOAD_ATTR 0 (_lock)
COPY 1
LOAD_SPECIAL 1 (__exit__)
SWAP 2
SWAP 3
LOAD_SPECIAL 0 (__enter__)
CALL 0
L12: POP_TOP
317 LOAD_CONST 6 (False)
LOAD_DEREF 0 (self)
STORE_ATTR 2 (_is_reading)
319 LOAD_DEREF 0 (self)
LOAD_ATTR 2 (_closed)
TO_BOOL
POP_JUMP_IF_FALSE 29 (to L15)
NOT_TAKEN
320 LOAD_DEREF 0 (self)
LOAD_ATTR 15 (_close_resources + NULL|self)
CALL 0
POP_TOP
321 BUILD_LIST 0
316 L13: SWAP 3
SWAP 2
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
CALL 3
POP_TOP
L14: RETURN_VALUE
319 L15: NOP
316 L16: LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
CALL 3
POP_TOP
330 L17: NOP
332 LOAD_DEREF 0 (self)
LOAD_ATTR 0 (_lock)
COPY 1
LOAD_SPECIAL 1 (__exit__)
SWAP 2
SWAP 3
LOAD_SPECIAL 0 (__enter__)
CALL 0
L18: POP_TOP
333 BUILD_LIST 0
STORE_FAST 5 (event_list)
334 LOAD_GLOBAL 24 (Inotify)
LOAD_ATTR 27 (_parse_event_buffer + NULL|self)
LOAD_FAST 3 (event_buffer)
CALL 1
GET_ITER
L19: EXTENDED_ARG 2
FOR_ITER 725 (to L47)
UNPACK_SEQUENCE 4
STORE_FAST_STORE_FAST 103 (wd, mask)
STORE_FAST_STORE_FAST 137 (cookie, name)
335 LOAD_FAST 6 (wd)
LOAD_CONST 7 (-1)
COMPARE_OP 88 (bool(==))
POP_JUMP_IF_FALSE 3 (to L20)
NOT_TAKEN
336 JUMP_BACKWARD 16 (to L19)
337 L20: LOAD_DEREF 0 (self)
LOAD_ATTR 28 (_path_for_wd)
LOAD_FAST 6 (wd)
BINARY_OP 26 ([])
STORE_FAST 10 (wd_path)
338 LOAD_FAST 9 (name)
TO_BOOL
POP_JUMP_IF_FALSE 32 (to L23)
L21: NOT_TAKEN
L22: LOAD_GLOBAL 8 (os)
LOAD_ATTR 30 (path)
LOAD_ATTR 33 (join + NULL|self)
LOAD_FAST_LOAD_FAST 169 (wd_path, name)
CALL 2
JUMP_FORWARD 1 (to L24)
L23: LOAD_FAST 10 (wd_path)
L24: STORE_FAST 11 (src_path)
339 LOAD_GLOBAL 35 (InotifyEvent + NULL)
LOAD_FAST_LOAD_FAST 103 (wd, mask)
LOAD_FAST_LOAD_FAST 137 (cookie, name)
LOAD_FAST 11 (src_path)
CALL 5
STORE_FAST 12 (inotify_event)
341 LOAD_FAST 12 (inotify_event)
LOAD_ATTR 36 (is_moved_from)
TO_BOOL
POP_JUMP_IF_FALSE 20 (to L25)
NOT_TAKEN
342 LOAD_DEREF 0 (self)
LOAD_ATTR 39 (remember_move_from_event + NULL|self)
LOAD_FAST 12 (inotify_event)
CALL 1
POP_TOP
EXTENDED_ARG 1
JUMP_FORWARD 385 (to L35)
343 L25: LOAD_FAST 12 (inotify_event)
LOAD_ATTR 40 (is_moved_to)
TO_BOOL
EXTENDED_ARG 1
POP_JUMP_IF_FALSE 367 (to L35)
L26: NOT_TAKEN
344 L27: LOAD_DEREF 0 (self)
LOAD_ATTR 43 (source_for_move + NULL|self)
LOAD_FAST 12 (inotify_event)
CALL 1
STORE_FAST 13 (move_src_path)
345 LOAD_FAST 13 (move_src_path)
LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
CONTAINS_OP 0 (in)
EXTENDED_ARG 1
POP_JUMP_IF_FALSE 288 (to L34)
NOT_TAKEN
346 LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_FAST 13 (move_src_path)
BINARY_OP 26 ([])
STORE_FAST 14 (moved_wd)
347 LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_FAST 13 (move_src_path)
DELETE_SUBSCR
348 LOAD_FAST 14 (moved_wd)
LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_FAST 12 (inotify_event)
LOAD_ATTR 46 (src_path)
STORE_SUBSCR
349 LOAD_FAST 12 (inotify_event)
LOAD_ATTR 46 (src_path)
LOAD_DEREF 0 (self)
LOAD_ATTR 28 (_path_for_wd)
LOAD_FAST 14 (moved_wd)
STORE_SUBSCR
350 LOAD_DEREF 0 (self)
LOAD_ATTR 48 (is_recursive)
TO_BOOL
POP_JUMP_IF_FALSE 188 (to L34)
L28: NOT_TAKEN
351 L29: LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_ATTR 51 (copy + NULL|self)
CALL 0
GET_ITER
L30: FOR_ITER 157 (to L33)
STORE_FAST 15 (_path)
352 LOAD_FAST 15 (_path)
LOAD_ATTR 53 (startswith + NULL|self)
LOAD_FAST 13 (move_src_path)
LOAD_GLOBAL 8 (os)
LOAD_ATTR 30 (path)
LOAD_ATTR 54 (sep)
LOAD_ATTR 57 (encode + NULL|self)
CALL 0
BINARY_OP 0 (+)
CALL 1
TO_BOOL
L31: POP_JUMP_IF_TRUE 3 (to L32)
NOT_TAKEN
JUMP_BACKWARD 73 (to L30)
353 L32: LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_ATTR 59 (pop + NULL|self)
LOAD_FAST 15 (_path)
CALL 1
STORE_FAST 14 (moved_wd)
354 LOAD_FAST 15 (_path)
LOAD_ATTR 61 (replace + NULL|self)
LOAD_FAST_LOAD_FAST 220 (move_src_path, inotify_event)
LOAD_ATTR 46 (src_path)
CALL 2
STORE_FAST 16 (_move_to_path)
355 LOAD_FAST 14 (moved_wd)
LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_FAST 16 (_move_to_path)
STORE_SUBSCR
356 LOAD_FAST 16 (_move_to_path)
LOAD_DEREF 0 (self)
LOAD_ATTR 28 (_path_for_wd)
LOAD_FAST 14 (moved_wd)
STORE_SUBSCR
JUMP_BACKWARD 159 (to L30)
351 L33: END_FOR
POP_ITER
357 L34: LOAD_GLOBAL 8 (os)
LOAD_ATTR 30 (path)
LOAD_ATTR 33 (join + NULL|self)
LOAD_FAST_LOAD_FAST 169 (wd_path, name)
CALL 2
STORE_FAST 11 (src_path)
358 LOAD_GLOBAL 35 (InotifyEvent + NULL)
LOAD_FAST_LOAD_FAST 103 (wd, mask)
LOAD_FAST_LOAD_FAST 137 (cookie, name)
LOAD_FAST 11 (src_path)
CALL 5
STORE_FAST 12 (inotify_event)
360 L35: LOAD_FAST 12 (inotify_event)
LOAD_ATTR 62 (is_ignored)
TO_BOOL
POP_JUMP_IF_FALSE 65 (to L38)
L36: NOT_TAKEN
362 L37: LOAD_DEREF 0 (self)
LOAD_ATTR 28 (_path_for_wd)
LOAD_ATTR 59 (pop + NULL|self)
LOAD_FAST 6 (wd)
CALL 1
STORE_FAST 17 (path)
363 LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_FAST 17 (path)
BINARY_OP 26 ([])
LOAD_FAST 6 (wd)
COMPARE_OP 88 (bool(==))
POP_JUMP_IF_FALSE 14 (to L38)
NOT_TAKEN
364 LOAD_DEREF 0 (self)
LOAD_ATTR 44 (_wd_for_path)
LOAD_FAST 17 (path)
DELETE_SUBSCR
366 L38: LOAD_FAST 5 (event_list)
LOAD_ATTR 65 (append + NULL|self)
LOAD_FAST 12 (inotify_event)
CALL 1
POP_TOP
368 LOAD_DEREF 0 (self)
LOAD_ATTR 48 (is_recursive)
TO_BOOL
L39: POP_JUMP_IF_TRUE 4 (to L40)
NOT_TAKEN
EXTENDED_ARG 2
JUMP_BACKWARD 631 (to L19)
L40: LOAD_FAST 12 (inotify_event)
LOAD_ATTR 66 (is_directory)
TO_BOOL
L41: POP_JUMP_IF_TRUE 4 (to L42)
NOT_TAKEN
EXTENDED_ARG 2
JUMP_BACKWARD 652 (to L19)
L42: LOAD_FAST 12 (inotify_event)
LOAD_ATTR 68 (is_create)
TO_BOOL
L43: POP_JUMP_IF_TRUE 4 (to L44)
NOT_TAKEN
EXTENDED_ARG 2
JUMP_BACKWARD 673 (to L19)
376 L44: NOP
377 L45: LOAD_DEREF 0 (self)
LOAD_ATTR 71 (_add_watch + NULL|self)
LOAD_FAST 11 (src_path)
LOAD_DEREF 0 (self)
LOAD_ATTR 72 (_event_mask)
CALL 2
POP_TOP
381 L46: LOAD_FAST 5 (event_list)
LOAD_ATTR 75 (extend + NULL|self)
LOAD_FAST 2 (_recursive_simulate)
PUSH_NULL
LOAD_FAST 11 (src_path)
CALL 1
CALL 1
POP_TOP
EXTENDED_ARG 2
JUMP_BACKWARD 728 (to L19)
334 L47: END_FOR
POP_ITER
332 L48: LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
LOAD_CONST 5 (None)
CALL 3
POP_TOP
383 LOAD_FAST 5 (event_list)
RETURN_VALUE
307 L49: PUSH_EXC_INFO
WITH_EXCEPT_START
TO_BOOL
POP_JUMP_IF_TRUE 2 (to L50)
NOT_TAKEN
RERAISE 2
L50: POP_TOP
L51: POP_EXCEPT
POP_TOP
POP_TOP
POP_TOP
EXTENDED_ARG 3
JUMP_BACKWARD_NO_INTERRUPT 940 (to L8)
-- L52: COPY 3
POP_EXCEPT
RERAISE 1
316 L53: PUSH_EXC_INFO
WITH_EXCEPT_START
TO_BOOL
POP_JUMP_IF_TRUE 2 (to L54)
NOT_TAKEN
RERAISE 2
L54: POP_TOP
L55: POP_EXCEPT
POP_TOP
POP_TOP
POP_TOP
L56: EXTENDED_ARG 3
JUMP_BACKWARD_NO_INTERRUPT 822 (to L17)
-- L57: COPY 3
POP_EXCEPT
RERAISE 1
L58: PUSH_EXC_INFO
322 LOAD_GLOBAL 16 (OSError)
CHECK_EXC_MATCH
POP_JUMP_IF_FALSE 83 (to L66)
NOT_TAKEN
STORE_FAST 4 (e)
323 L59: LOAD_FAST 4 (e)
LOAD_ATTR 18 (errno)
LOAD_GLOBAL 18 (errno)
LOAD_ATTR 20 (EINTR)
COMPARE_OP 88 (bool(==))
POP_JUMP_IF_FALSE 8 (to L61)
NOT_TAKEN
324 L60: POP_EXCEPT
LOAD_CONST 5 (None)
STORE_FAST 4 (e)
DELETE_FAST 4 (e)
EXTENDED_ARG 4
JUMP_BACKWARD 1080 (to L1)
326 L61: LOAD_FAST 4 (e)
LOAD_ATTR 18 (errno)
LOAD_GLOBAL 18 (errno)
LOAD_ATTR 22 (EBADF)
COMPARE_OP 88 (bool(==))
POP_JUMP_IF_FALSE 8 (to L64)
NOT_TAKEN
327 BUILD_LIST 0
L62: SWAP 2
L63: POP_EXCEPT
LOAD_CONST 5 (None)
STORE_FAST 4 (e)
DELETE_FAST 4 (e)
RETURN_VALUE
329 L64: RAISE_VARARGS 0
-- L65: LOAD_CONST 5 (None)
STORE_FAST 4 (e)
DELETE_FAST 4 (e)
RERAISE 1
322 L66: RERAISE 0
-- L67: COPY 3
POP_EXCEPT
RERAISE 1
L68: PUSH_EXC_INFO
378 LOAD_GLOBAL 16 (OSError)
CHECK_EXC_MATCH
POP_JUMP_IF_FALSE 6 (to L70)
NOT_TAKEN
POP_TOP
379 L69: POP_EXCEPT
EXTENDED_ARG 3
JUMP_BACKWARD 891 (to L19)
378 L70: RERAISE 0
-- L71: COPY 3
POP_EXCEPT
RERAISE 1
332 L72: PUSH_EXC_INFO
WITH_EXCEPT_START
TO_BOOL
POP_JUMP_IF_TRUE 2 (to L73)
NOT_TAKEN
RERAISE 2
L73: POP_TOP
L74: POP_EXCEPT
POP_TOP
POP_TOP
POP_TOP
383 LOAD_FAST_CHECK 5 (event_list)
RETURN_VALUE
-- L75: COPY 3
POP_EXCEPT
RERAISE 1
ExceptionTable:
L2 to L3 -> L58 [0]
L3 to L4 -> L49 [2] lasti
L4 to L5 -> L58 [0]
L6 to L7 -> L49 [2] lasti
L7 to L9 -> L58 [0]
L10 to L12 -> L58 [0]
L12 to L13 -> L53 [2] lasti
L13 to L14 -> L58 [0]
L16 to L17 -> L58 [0]
L18 to L21 -> L72 [2] lasti
L22 to L26 -> L72 [2] lasti
L27 to L28 -> L72 [2] lasti
L29 to L31 -> L72 [2] lasti
L32 to L36 -> L72 [2] lasti
L37 to L39 -> L72 [2] lasti
L40 to L41 -> L72 [2] lasti
L42 to L43 -> L72 [2] lasti
L45 to L46 -> L68 [4]
L46 to L48 -> L72 [2] lasti
L49 to L51 -> L52 [4] lasti
L51 to L53 -> L58 [0]
L53 to L55 -> L57 [4] lasti
L55 to L56 -> L58 [0]
L57 to L58 -> L58 [0]
L58 to L59 -> L67 [1] lasti
L59 to L60 -> L65 [1] lasti
L61 to L62 -> L65 [1] lasti
L62 to L63 -> L67 [1] lasti
L64 to L65 -> L65 [1] lasti
L65 to L67 -> L67 [1] lasti
L68 to L69 -> L71 [5] lasti
L69 to L70 -> L72 [2] lasti
L70 to L71 -> L71 [5] lasti
L71 to L72 -> L72 [2] lasti
L72 to L74 -> L75 [4] lasti
Disassembly of <code object __annotate__ at 0x71835eeda5d0, file "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_c.py", line 276>:
276 RESUME 0
LOAD_FAST_BORROW 0 (format)
LOAD_SMALL_INT 2
COMPARE_OP 132 (>)
POP_JUMP_IF_FALSE 3 (to L1)
NOT_TAKEN
LOAD_COMMON_CONSTANT 1 (NotImplementedError)
RAISE_VARARGS 1
L1: LOAD_CONST 1 ('src_path')
LOAD_CONST 2 ('bytes')
LOAD_CONST 3 ('return')
LOAD_CONST 4 ('list[InotifyEvent]')
BUILD_MAP 2
RETURN_VALUE
Disassembly of <code object _recursive_simulate at 0x71f35ef2f990, file "/home/emma/parsefuzz/.venv/lib/python3.15/site-packages/watchdog/observers/inotify_c.py", line 276>:
-- COPY_FREE_VARS 1
276 RESUME 0
277 BUILD_LIST 0
STORE_FAST 1 (events)
278 LOAD_GLOBAL 0 (os)
LOAD_ATTR 2 (walk)
PUSH_NULL
LOAD_FAST_BORROW 0 (src_path)
CALL 1
GET_ITER
L1: EXTENDED_ARG 1
FOR_ITER 319 (to L8)
UNPACK_SEQUENCE 3
STORE_FAST_STORE_FAST 35 (root, dirnames)
STORE_FAST 4 (filenames)
279 LOAD_FAST 3 (dirnames)
GET_ITER
L2: FOR_ITER 172 (to L5)
STORE_FAST 5 (dirname)
280 LOAD_GLOBAL 4 (contextlib)
LOAD_ATTR 6 (suppress)
PUSH_NULL
LOAD_GLOBAL 8 (OSError)
CALL 1
COPY 1
LOAD_SPECIAL 1 (__exit__)
SWAP 2
SWAP 3
LOAD_SPECIAL 0 (__enter__)
CALL 0
L3: POP_TOP
281 LOAD_GLOBAL 0 (os)
LOAD_ATTR 10 (path)
LOAD_ATTR 13 (join + NULL|self)
LOAD_FAST_BORROW_LOAD_FAST_BORROW 37 (root, dirname)
CALL 2
STORE_FAST 6 (full_path)
282 LOAD_DEREF 11 (self)
LOAD_ATTR 15 (_add_watch + NULL|self)
LOAD_FAST_BORROW 6 (full_path)
LOAD_DEREF 11 (self)
LOAD_ATTR 16 (_event_mask)
CALL 2
STORE_FAST 7 (wd_dir)
283 LOAD_GLOBAL 19 (InotifyEvent + NULL)
284 LOAD_FAST_BORROW 7 (wd_dir)
285 LOAD_GLOBAL 20 (InotifyConstants)
LOAD_ATTR 22 (IN_CREATE)
LOAD_GLOBAL 20 (InotifyConstants)
LOAD_ATTR 24 (IN_ISDIR)
BINARY_OP 7 (|)
286 LOAD_SMALL_INT 0
287 LOAD_FAST_BORROW 5 (dirname)
288 LOAD_FAST_BORROW 6 (full_path)
283 CALL 5
STORE_FAST 8 (e)
290 LOAD_FAST_BORROW 1 (events)
LOAD_ATTR 27 (append + NULL|self)
LOAD_FAST_BORROW 8 (e)
CALL 1
POP_TOP
280 L4: LOAD_CONST 1 (None)
LOAD_CONST 1 (None)
LOAD_CONST 1 (None)
CALL 3
POP_TOP
JUMP_BACKWARD 174 (to L2)
279 L5: END_FOR
POP_ITER
291 LOAD_FAST 4 (filenames)
GET_ITER
L6: FOR_ITER 128 (to L7)
STORE_FAST 9 (filename)
292 LOAD_GLOBAL 0 (os)
LOAD_ATTR 10 (path)
LOAD_ATTR 13 (join + NULL|self)
LOAD_FAST_BORROW_LOAD_FAST_BORROW 41 (root, filename)
CALL 2
STORE_FAST 6 (full_path)
293 LOAD_DEREF 11 (self)
LOAD_ATTR 28 (_wd_for_path)
LOAD_GLOBAL 0 (os)
LOAD_ATTR 10 (path)
LOAD_ATTR 31 (dirname + NULL|self)
LOAD_FAST_BORROW 6 (full_path)
CALL 1
BINARY_OP 26 ([])
STORE_FAST 10 (wd_parent_dir)
294 LOAD_GLOBAL 19 (InotifyEvent + NULL)
295 LOAD_FAST_BORROW 10 (wd_parent_dir)
296 LOAD_GLOBAL 20 (InotifyConstants)
LOAD_ATTR 22 (IN_CREATE)
297 LOAD_SMALL_INT 0
298 LOAD_FAST_BORROW 9 (filename)
299 LOAD_FAST_BORROW 6 (full_path)
294 CALL 5
STORE_FAST 8 (e)
301 LOAD_FAST_BORROW 1 (events)
LOAD_ATTR 27 (append + NULL|self)
LOAD_FAST_BORROW 8 (e)
CALL 1
POP_TOP
JUMP_BACKWARD 130 (to L6)
291 L7: END_FOR
POP_ITER
EXTENDED_ARG 1
JUMP_BACKWARD 322 (to L1)
278 L8: END_FOR
POP_ITER
302 LOAD_FAST_BORROW 1 (events)
RETURN_VALUE
280 L9: PUSH_EXC_INFO
WITH_EXCEPT_START
TO_BOOL
POP_JUMP_IF_TRUE 2 (to L10)
NOT_TAKEN
RERAISE 2
L10: POP_TOP
L11: POP_EXCEPT
POP_TOP
POP_TOP
POP_TOP
EXTENDED_ARG 1
JUMP_BACKWARD 335 (to L2)
-- L12: COPY 3
POP_EXCEPT
RERAISE 1
ExceptionTable:
L3 to L4 -> L9 [6] lasti
L9 to L11 -> L12 [8] lasti
Reproducer repo
I've made a repository with the full reproducer https://github.com/emmatyping/cpython-interpreter-assertion-failure-reproducer, but it's not really minimized yet. I plan on trying to minimize it more tomorrow.
CPython versions tested on:
CPython main branch, 3.14
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a0 (heads/main-dirty:65d2c51c104, Jul 22 2025, 19:19:23) [Clang 20.1.6 (++20250528122018+47addd4540b4-1exp120250528002033.124)]
Activity
gvanrossum commentedon Jul 23, 2025
You realize that generated_cases.c.h is a generated file right? You should probably edit the assert in Python/bytecodes.c.
Do you know what
frame->instr_ptr->op.codeactually is?emmatyping commentedon Jul 23, 2025
Yeah, I only edited generated_cases.c.h locally for faster debug loops (just running
makeinstead of needing to regenerate the file each time). Editing Python/bytecodes.c does work as well.The value at time of assertion is
INSTRUMENTED_FOR_ITERbased on opcode_ids.h. I presume instrumented because hypofuzz uses sys.monitoring.emmatyping commentedon Jul 23, 2025
I updated the issue description to make it clearer what the value of
frame->instr_ptr->op.codeis INSTRUMENTED_FOR_ITER at the time of assertion failure.gvanrossum commentedon Jul 23, 2025
Sigh, sorry for being unhelpful. All I can think of is asking Ken Jin or Mark for help (Ken Jin seems to have touched this more recently than Mark).
The YIELD_VALUE code block looks pretty stable, but that doesn't mean something else couldn't have changed it. What's the behavior without debug mode (so the assert doesn't trigger)?
I'd look for a bug in the instrumentation code (no fun!). How does the hypo* thing use instrumentation? There might be a clue there about what to look for in the instrumentation code.
emmatyping commentedon Jul 23, 2025
No worries! Thank you for the suggestions. I'm planning on trying to minimize the crash reproducer tomorrow. I'll try without the assert as well.
Fidget-Spinner commentedon Jul 23, 2025
I think the following sequence of events might be happening. Though I'm not 100% sure, as Mark and Tian are better instrumentation experts:
I think in this case, changing the assertion to account for INSTRUMENTED_FOR_ITER should be fine? WDY @markshannon ?
emmatyping commentedon Jul 23, 2025
I noticed that
INSTRUMENTED_JUMP_FORWARDis also hit when I update watchdog to the latest git version (trying to work around the crash without much luck). It seems odd that could also occur here, but I guess it could be generated by specialization?The code is refactored a bit to be split into functions but overall rather similar:
https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/observers/inotify.py#L257-L274
emmatyping commentedon Jul 24, 2025
Unfortunately my several attempts to minimize this to just watchdog with hypofuzz's collector have been completely unsuccessful :/
vstinner commentedon Sep 9, 2025
I managed to reproduce the issue using these commands for step 4 and 5:
Without
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1, LibCST fails to build with Python 3.15.emmatyping commentedon Sep 9, 2025
Yeah, this is why I wrote the original instructions for the 3.14 branch, probably should have noted that env variable for 3.15. Thanks!
markshannon commentedon Apr 22, 2026
Hypothesis generating a program or AST to produce this failure, right?
Can you extract the program or AST? That way we have a reproducer that we can look at.
emmatyping commentedon Apr 22, 2026
@markshannon I believe the bug is actually with the watchdog code I linked, not with the actual fuzzing. My recollection is a little fuzzy (pun surprisingly not intended) at this point but I believe the program crashed before any fuzzing started.
vstinner commentedon Apr 27, 2026
I managed to write a reproducer which only uses
sys.monitoringandthreading!Run the script multiple times to get a crash. Example:
vstinner commentedon Apr 27, 2026
By the way, if I try to dump the instrumented bytecode using
dis.dis(iwork, adaptive=True), I get a bug in thedismodule:$ ./python crash.py 2 RETURN_GENERATOR POP_TOP L1: RESUME_CHECK 0 3 LOAD_GLOBAL_BUILTIN 1 (list + NULL) LOAD_GLOBAL_BUILTIN 3 (range + NULL) LOAD_CONST 2 (4096) CALL_BUILTIN_CLASS 1 CALL_BUILTIN_CLASS 1 STORE_FAST 0 (data) 4 LOAD_FAST 0 (data) GET_ITER_VIRTUAL 0 Exception in thread Thread-1 (thread_job): Traceback (most recent call last): File "/home/vstinner/python/main/Lib/threading.py", line 1075, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/home/vstinner/python/main/Lib/threading.py", line 1017, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/crash.py", line 27, in thread_job dis.dis(iwork, adaptive=True) ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 124, in dis _disassemble_recursive(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 842, in _disassemble_recursive disassemble(co, file=file, show_caches=show_caches, adaptive=adaptive, show_offsets=show_offsets, show_positions=show_positions) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 837, in disassemble _disassemble_bytes(_get_code_array(co, adaptive), lasti, linestarts, ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ exception_entries=exception_entries, co_positions=co.co_positions(), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ original_code=co.co_code, arg_resolver=arg_resolver, formatter=formatter) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 916, in _disassemble_bytes print_instructions(instrs, exception_entries, formatter, lasti=lasti) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 920, in print_instructions for instr in instrs: ^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 791, in _get_instructions_bytes argval, argrepr = arg_resolver.get_argval_argrepr(op, arg, offset) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/vstinner/python/main/Lib/dis.py", line 615, in get_argval_argrepr assert lbl is not None ^^^^^^^^^^^^^^^ AssertionError[-]Assertion failure in _PyEval_EvalFrameDefault due to unexpected opcode[/-][+]Assertion failure in _PyEval_EvalFrameDefault() due to unexpected INSTRUMENTED_FOR_ITER opcode[/+]gh-137030: Fix YIELD_VALUE bytecode assertion (#149184)
pythongh-137030: Fix YIELD_VALUE bytecode assertion (python#149184)
[3.14] gh-137030: Fix YIELD_VALUE bytecode assertion (#149184) (#149383)
vstinner commentedon May 4, 2026
I fixed the assertion in 3.14 and main branches. I close the issue. Thanks @emmatyping for the reproducer!