Skip to content

JIT: Segfault from invalid memory read in _PyTier2Interpreter #143092

Closed

Description

@devdanzin

Crash report

What happened?

It's possible to segfault a JIT build by running the code below:

def f1():
    a = "a"
    for i in range(50):
        x = a[i % len(a)]

    s = ""
    for _ in range(10):
        s += ""

    class A: ...
    class B: ...

    match s:
        case int(): ...
        case str(): ...
        case dict(): ...

    (
        u0,
        *u1,
        u2,
        u4,
        u5,
        u6,
        u7,
        u8,
        u9, u10, u11, 
        u12, u13, u14, u15, u16, u17, u18, u19, u20, u21, u22, u23, u24, u25, u26, u27, u28, u29,
    ) = [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,
         None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,
         None, None, None, None, None, None, None, None, None, None, None, None, None, None, None,
         None, None, None, None, None, None, None, None,]

    s = ""
    for _ in range(10):
        s += ""
        s += ""

for i in range(11000):
    f1()

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
_PyTier2Interpreter (current_executor=<optimized out>, frame=<optimized out>, stack_pointer=<optimized out>, tstate=<optimized out>) at Python/executor_cases.c.h:10549
10549               if (Py_TYPE(r) != &PyRangeIter_Type) {

#0  _PyTier2Interpreter (current_executor=<optimized out>, frame=<optimized out>, stack_pointer=<optimized out>, tstate=<optimized out>) at Python/executor_cases.c.h:10549
#1  0x000055555584c1b2 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555d5c9b0 <_PyRuntime+358864>, frame=<optimized out>, frame@entry=0x7ffff7fa7020, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:5348
#2  0x0000555555830e4b in _PyEval_EvalFrame (tstate=0x555555d5c9b0 <_PyRuntime+358864>, frame=0x7ffff7fa7020, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#3  _PyEval_Vector (tstate=tstate@entry=0x555555d5c9b0 <_PyRuntime+358864>, func=func@entry=0x7ffff6bf2390, locals=locals@entry=0x7ffff6c00ad0, args=args@entry=0x0,
    argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:2483
#4  0x0000555555830bde in PyEval_EvalCode (co=co@entry=0x7ffff6c0c3a0, globals=globals@entry=0x7ffff6c00ad0, locals=locals@entry=0x7ffff6c00ad0) at Python/ceval.c:1008
#5  0x000055555599b13a in run_eval_code_obj (tstate=0x555555d5c9b0 <_PyRuntime+358864>, co=co@entry=0x7ffff6c0c3a0, globals=globals@entry=0x7ffff6c00ad0, locals=locals@entry=0x7ffff6c00ad0)
    at Python/pythonrun.c:1366
#6  0x000055555599ac8b in run_mod (mod=mod@entry=0x555555f17e10, filename=filename@entry=0x7ffff6c4e7a0, globals=globals@entry=0x7ffff6c00ad0, locals=locals@entry=0x7ffff6c00ad0,
    flags=0x7fffffffd740, arena=arena@entry=0x7ffff6c711e0, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1469
#7  0x00005555559985ad in pyrun_file (fp=fp@entry=0x555555dd0360, filename=filename@entry=0x7ffff6c4e7a0, start=start@entry=257, globals=globals@entry=0x7ffff6c00ad0,
    locals=locals@entry=0x7ffff6c00ad0, closeit=closeit@entry=1, flags=0x7fffffffd740) at Python/pythonrun.c:1294

ASan output:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==3999430==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x7da56db089a7 bp 0x7ffd350e2e90 sp 0x7ffd350e21a0 T0)
==3999430==The signal is caused by a READ memory access.
==3999430==Hint: address points to the zero page.
    #0 0x7da56db089a7  (<unknown module>)
    #1 0x55577b512f97 in _PyEval_EvalFrame /home/danzin/projects/jit_cpython/./Include/internal/pycore_ceval.h:119:16
    #2 0x55577b512f97 in _PyEval_Vector /home/danzin/projects/jit_cpython/Python/ceval.c:2483:12
    #3 0x55577b5129b4 in PyEval_EvalCode /home/danzin/projects/jit_cpython/Python/ceval.c:1008:21
    #4 0x55577bafc2ce in run_eval_code_obj /home/danzin/projects/jit_cpython/Python/pythonrun.c:1366:12
    #5 0x55577bafb49b in run_mod /home/danzin/projects/jit_cpython/Python/pythonrun.c:1469:19
    #6 0x55577baf5a9c in pyrun_file /home/danzin/projects/jit_cpython/Python/pythonrun.c:1294:15
    #7 0x55577baf35fc in _PyRun_SimpleFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:518:13
    #8 0x55577baf296d in _PyRun_AnyFileObject /home/danzin/projects/jit_cpython/Python/pythonrun.c:81:15
    #9 0x55577bb6ea4a in pymain_run_file_obj /home/danzin/projects/jit_cpython/Modules/main.c:410:15
    #10 0x55577bb6ea4a in pymain_run_file /home/danzin/projects/jit_cpython/Modules/main.c:429:15
    #11 0x55577bb6cb13 in pymain_run_python /home/danzin/projects/jit_cpython/Modules/main.c:691:21
    #12 0x55577bb6cb13 in Py_RunMain /home/danzin/projects/jit_cpython/Modules/main.c:772:5
    #13 0x55577bb6da16 in pymain_main /home/danzin/projects/jit_cpython/Modules/main.c:802:12
    #14 0x55577bb6db87 in Py_BytesMain /home/danzin/projects/jit_cpython/Modules/main.c:826:12
    #15 0x7da56e62a574 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #16 0x7da56e62a627 in __libc_start_main csu/../csu/libc-start.c:360:3
    #17 0x55577aed64f4 in _start (/home/danzin/projects/jit_cpython/python+0x2db4f4) (BuildId: 2bff0bd42ddf0d6a9c5bada7767fdb560f820115)

==3999430==Register values:
rax = 0x0000000000000000  rbx = 0x0000000000000000  rcx = 0x0000000000000000  rdx = 0x000055577c2ad760
rdi = 0x0000000000000000  rsi = 0x00007c356d9e53b8  rbp = 0x00007ffd350e2e90  rsp = 0x00007ffd350e21a0
 r8 = 0x00007c356d9e5408   r9 = 0x0000000000000003  r10 = 0x00000aaaef855aac  r11 = 0x00000aaaef855aac
r12 = 0x00007c356d9e5298  r13 = 0x00007c356d9e5420  r14 = 0x000055577c509830  r15 = 0x0000000000000001
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>)
==3999430==ABORTING

Output from running with PYTHON_LLTRACE=4 PYTHON_OPT_DEBUG=4:
3004_segfault_lltrace_opt_debug.txt

Found using lafleur.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0a3+ (heads/main-dirty:3cc57505e53, Dec 23 2025, 05:19:47) [Clang 21.1.2 (2ubuntu6)]

Linked PRs

Activity

  1. added
    type-crashA hard crash of the interpreter, possibly with a core dump
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Dec 23, 2025
  2. Fidget-Spinner commented on Dec 23, 2025

    @Fidget-Spinner
    Member

    Thanks, verified. The problem is that the first instruction is a super instruction (_BINARY_OP_INPLACE_ADD_UNICODE). The progress requirement forces the trace recorder to record a deopt version of that (BINARY_OP). However, that leads to a wrong trace as the following _STORE_FAST is not recorded, as the trace recorder rightfully thinks that a super instruction has executed.

    I could add a special case for super instructions in the trace recorder, but that would just be error-prone. I'm thinking of removing the progress requirements from the JIT and just make it so ENTER_EXECUTOR executes itself if next_instr == this_instr. WDYT @markshannon ?

  3. Fidget-Spinner commented on Dec 23, 2025

    @Fidget-Spinner
    Member

    The other benefit of removing the progress requirements is we get more speed -- the first instruction can now be specialized. The first instruction is actually a really important one too because it usually corresponds to a loop header!

  4. markshannon commented on Dec 23, 2025

    @markshannon
    Member

    First some thoughts on _BINARY_OP_INPLACE_ADD_UNICODE:

    _BINARY_OP_INPLACE_ADD_UNICODE is a weird special case that only exists because of an historic promise that:

    s = some_string
    for _ in range(1000):
        s += "x"
    

    would not be quadratic.

    If we lower the JIT thresholds a bit (which I think we should anyway) we could do the optimization in the JIT and do away with the super-instruction.

  5. Fidget-Spinner commented on Dec 23, 2025

    @Fidget-Spinner
    Member

    If we lower the JIT thresholds a bit (which I think we should anyway) we could do the optimization in the JIT and do away with the super-instruction.

    I don't think we can do that considering it will remove the quadratic optimization for non-JIT platforms/runtimes, which would be quite bad. Having quadratic behavior on interpreter and non-quadratic on JIT is a weird disparity that I don't want to introduce.

  6. markshannon commented on Dec 23, 2025

    @markshannon
    Member

    I'm thinking of removing the progress requirements from the JIT ...

    We need to be careful to avoid infinite deopt loops that make no progress?

    just make it so ENTER_EXECUTOR executes itself if next_instr == this_instr

    I presume you mean after we return from tier2. How would that work with tailcaliing in the future (when tier2 tailcalls into tier1)

  7. markshannon commented on Dec 23, 2025

    @markshannon
    Member

    I don't think we can do that considering it will remove the quadratic optimization for non-JIT platforms/runtimes, which would be quite bad. Having quadratic behavior on interpreter and non-quadratic on JIT is a weird disparity that I don't want to introduce.

    We don't need to drop it from the interpreter if there is no JIT. We would just don't perform that specialization if the JIT is enabled.

  8. Fidget-Spinner commented on Dec 23, 2025

    @Fidget-Spinner
    Member

    We don't need to drop it from the interpreter if there is no JIT. We would just don't perform that specialization if the JIT is enabled.

    Ok fair enough.

  9. markshannon commented on Dec 23, 2025

    @markshannon
    Member

    Instead of handling the special case in ENTER_EXECUTOR, which doesn't really have enough context to do it properly, we should do it in the jitted code.

    We already have several forms of tier2 -> tier1 transfers: exits, deopts, handle-pending-and-deopt, error.
    Adding another one, exit-but-dont-reenter, wouldn't be too much effort.
    We can then specialize the first uop, but use a different exit uop for it.

  10. Fidget-Spinner commented on Dec 23, 2025

    @Fidget-Spinner
    Member

    How are you envisioning the deopt exit with guranteed progress? in an ideal world, we would use 1 tier 1 op called PROGRESS and execute that which would execute the ENTER_EXECUTOR's original opcode, but I don't want to use up another more tier 1 ops

    I'd prefer we set a bit on the executor that tells it not to be entered, and an extra branch on ENTER_EXECUTOR costs slightly more, but should be negligible as we spend more time in the JIT

  11. added a commit that references this issue on Dec 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-JITtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions