Skip to content

Supporting stack unwinding in the JIT compiler #126910

Open

Description

@pablogsal

TLDR

This is a lot of text because the issue is complex but if you want the gist:

To not break a lot of tools that rely on unwinding, I propose to fix this by compiling the JIT stencils with frame pointers which has a trivial maintenance cost (2 lines) and only involves a 2% hit on speed when using the JIT while making almost all existing debuggers and profilers just work™ in the presence of the JIT.

Although this doesn't fix everything sadly I think is the best compromise that I can find (and we are quite lucky to have it as normally fixing this is a nightmare - as in thousands of complex lines of code and 50% slowdown nightmare).


The issue + proposal

CPython's JIT compiler must provide robust stack unwinding support to maintain compatibility with the Python ecosystem's debugging and profiling tools. This requirement is particularly critical given Python's reliance on native extensions written in C, C++, and Rust. As more performance-critical code moves to these native implementations, the ability to properly unwind through mixed Python and native frames becomes fundamental for effective debugging and profiling. This capability is even more crucial with the introduction of both JIT compilation and free-threaded modes, which significantly increase the complexity of runtime behavior. Without proper unwinding support, debugging te issues that can appear in the presence of these new modes becomes extremely challenging – developers would be unable to generate meaningful stack traces, hampering their ability to understand where and why their programs failed. Consider debugging a deadlock in free threadded code and a Rust extension, or investigating a crash in native code called from JIT-compiled Python – without proper unwinding support, error reports would be incomplete or misleading, making production issues significantly harder to diagnose and fix. The ability to get complete, accurate stack traces in these scenarios is not just a convenience; it's essential for maintaining production applications where Python increasingly interacts with native code through multiple execution modes.

Profiling is equally affected – modern performance analysis tools like py-spy, Austin, or eBPF-based solutions need to understand the complete call stack, including JIT-compiled code, native extensions, and regular Python frames, to provide accurate performance insights.

Some of the popular tools that use native unwinding in some way or form:

  • Austin (statistical profiler). Uses libunwind
  • py-spy (statistical profiler).Uses libunwind
  • memray: uses C++ unwinder for macOS and libunwind for Linux
  • pystack: uses elfutils (as is the only one supporting core files).
  • EBPf unwinders : normally rely on frame pointers or rely on the C++ unwinder (https://israelo.io/blog/ehframes/, https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-based-stack-walking-using-ebpf).
  • Additionally, most JIT compilers implement not only frame pointers to make possible to profile and debug, but also implement either C++ exception unwinder via eh_frames or even more hardcore support for gdb and other debuggers.
  • Gdb and hdb-helpers: a combination of libdw, libunwind or custom
  • Perf: frame pointers, lubunwind or libdw
  • Lldb: libunwind or custom
  • valgrind: uses a custom unwinder based on libdw

This doesn't include the considerable amount of custom tools that are not open source either in the debugging or profiling space.

Unwinding Libraries and Their Capabilities

Several libraries provide stack unwinding capabilities, each with its own strengths and limitations.

Library Platforms Local Unwinding Remote Unwinding Core Files Registration API Remote Registration Support Main Users
libunwind Linux only Yes (fast) Yes No _U_dyn_register No Profilers, debuggers
Native Unwinder (LLVM/GDB) Linux, MacOS Yes (optimized) No No __register_frame No C++ exceptions, backtrace()
libdw (elfutils) Linux only Yes Yes Yes None (ELF parsing) No GDB, debuggers like pystack
GDB/LLDB built-in Cross-platform Yes Yes Yes __jit_debug_register_code No Debug tools

Implementation Plan for Stack Unwinding Support

After extensive experimentation and analysis (believe me this was a lot of very difficult research work, @brandtbucher and I found that since we have preserve_none and LLVM 19 we can just compile the JIT stencils with frame pointers and that makes most tools just work. Based on this I propose implementing frame pointer support as the primary strategy for stack unwinding in CPython's JIT. This approach has proven to be the most pragmatic and effective solution, providing broad compatibility with existing tools while maintaining reasonable performance characteristics. I do think we got very lucky this fixes most of the tools since adding unwinding support for JITs otherwise is very challenging (see following sections).

Frame pointer support can be enabled through a minimal change to the JIT compiler flags:

diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py
index d8dce0a905c..7c3b2e5aab7 100644
--- a/Tools/jit/_targets.py
+++ b/Tools/jit/_targets.py
@@ -135,6 +135,8 @@ async def _compile(
             # Don't call stack-smashing canaries that we can't find or patch:
             "-fno-stack-protector",
             "-std=c11",
+            "-fno-omit-frame-pointer",
+            "-mno-omit-leaf-frame-pointer",
             "-o",
             f"{o}",
             f"{c}",

This simple change enables compatibility with a wide range of tools:

  • libunwind (both local and remote unwinding)
  • libdw (supporting core file analysis, remote and local unwinding)
  • GDB and LLDB
  • eBPF-based unwinders

Most of the popular tools will just work with this change, instead of having to implement tons of different unwinding information support, which is a nightmare)

Performance Impact

https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20241114-3.14.0a1%2B-925b70b-JIT/bm-20241114-linux-x86_64-brandtbucher-justin_frame_pointer-3.14.0a1%2B-925b70b-vs-base.md

The performance impact of this change has been carefully measured:

  • AMD64: Approximately 2% overhead, as shown in recent benchmarks
  • ARM64 and macOS: Even lower overhead due to the presence of dedicated link registers

This modest performance cost is significantly outweighed by the expected JIT performance improvements. Moreover, the maintenance burden is minimal compared to alternative approaches like implementing custom unwinding support for each tool.

Additional Optimization

We can further enhance compatibility by generating eh-frames in the JIT stencils and calling __register_frame_table (a bulk version of __register_frame). Benchmarks show this additional feature has neutral performance impact while enabling native unwinder support. This makes the solution even more robust without additional overhead.

Given these considerations - the minimal implementation complexity, broad tool compatibility, reasonable performance characteristics, and the ability to extend support to native unwinders - frame pointer support represents the optimal path forward for CPython's JIT implementation.

Why this must be activated by default

The decision to enable frame pointers by default is driven by two critical production requirements.

  • When applications crash or hang, tools like pystack need to analyze program state without the luxury of reproduction – you cannot simply "run it again with debug options enabled." You need to unwind the stack either remotely or from a core file.
  • Performance profiling in production requires safely inspecting program state from another process or the kernel. Tools like Austin, py-spy, and eBPF-based profilers need to periodically sample stack traces without interrupting the target process. Both scenarios require frame pointers to work reliably.

This must be enabled by default because most Python users don't compile Python themselves – they install it through package managers or tools like uv. Unless frame pointers are enabled by default, these critical debugging and profiling capabilities won't be available in most Python installations, severely impacting the quality of bug reports to both CPython and C extensions. Given the minimal performance impact of 2% frame pointers should be enabled by default, following the principle of being "safe by default." While we should provide the option to disable them, this should be a conscious decision made by end users who understand they're trading away the ability to properly profile and debug their applications in production – not a choice made by intermediate distributors or package managers.

Additional information

Using frame pointers aligns with a broader industry trend where major tech companies and Linux distributions are reverting previous optimization decisions and re-enabling frame pointers. Companies like Meta and Google now compile their entire software stack, including their Python installations, with frame pointers enabled. This shift is driven by the recognition that observability and debugging capabilities in production environments far outweigh the minor performance impact of frame pointers. Ubuntu has also adopted this approach, now compiling virtually all packages with frame pointers enabled (with Python being a notable exception we aim to address). This industry-wide move reflects a fundamental reality: in modern production environments, the ability to profile, monitor, and debug applications effectively is crucial, and frame pointers provide the most reliable and universal way to achieve this. The performance cost, typically around 1-5%, is normally considered to be a worthwhile trade-off for the improved observability they enable. This is particularly relevant for server workloads where understanding performance characteristics and debugging production issues is far more valuable than the small overhead frame pointers introduce. The fact that major tech companies maintain this policy even for performance-critical services demonstrates that the benefits of comprehensive profiling and debugging support outweigh the minimal performance impact. Some links:

https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default
https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
https://www.polarsignals.com/blog/posts/2023/12/13/embracing-frame-pointers-in-ubuntu-24-04-lts


Background on Stack Unwinding Support Requirements

Adding JIT compilation to CPython requires careful consideration of stack unwinding support. Without proper unwinding capabilities, we risk breaking compatibility with essential development tools that Python developers rely on daily. To understand the scope of this requirement, we need to examine the landscape of tools that depend on stack unwinding.

Tools Requiring Stack Unwinding

Stack unwinding is a critical capability used by three major categories of tools in the Python ecosystem:

  • Debuggers (GDB, LLDB, pystack) rely on unwinding to show developers where their program is currently executing. These tools deal with remote processes or core files.

  • Profilers use unwinding to understand program performance.

  • C++ Exception Handling depends on unwinding information to:

    • Propagate exceptions through the stack
    • Clean up resources during stack unwinding

Unwinding Mechanisms

Stack unwinding occurs in two fundamentally different contexts:

  • In-process unwinding:

    • Code examines its own stack
    • Used by tracing profilers and exception handlers
    • Has direct access to program memory and CPU registers
    • Primarily used for live analysis
  • Out-of-process unwinding:

    • External program examines another program's stack
    • Used by debuggers and statistical profilers
    • Includes core dump analysis

Types of Analysis Tools

The tools that perform unwinding can be categorized by their data collection method:

  • Tracing Profilers:

    • Run inside the target process
    • Instrument function entries and exits
    • Collect detailed execution information
    • Higher overhead but more precise
    • Rely on local unwinding capabilities
    • Examples: cProfile, memray
  • Statistical Profilers:

    • Run outside the target process
    • Sample program state periodically
    • Much lower overhead
    • Suitable for production use
    • Growing eBPF ecosystem:
      • Enables efficient system-wide analysis
      • Becoming standard for production monitoring
      • Requires reliable stack unwinding support
      • Examples: perf, py-spy, bcc tools

Unwinding Libraries and Their Capabilities

Several libraries provide stack unwinding capabilities, each with its own strengths and limitations. The strategies to support unwinding for JIT compiled code are:

  • Allow frame pointers: makes libdw, gdb, lldb and lubinwind work. libunwind and libdw work in remote mode, including core files.

  • Per unwinder support for dynamically generated code using DWARF (only local mode and only if it supports it - libdw has no support):

    • libunwind: Construct ELF .debug_frame data structures, and table_entry data structures, and a unw_dyn_table_info data structure, and a unw_dyn_info_t structure, then call _U_dyn_register.
    • C++ exception unwinder: Construct ELF .eh_frame data structures, then call __register_frame
    • GDB: Construct a full in-memory ELF object, manually maintain a doubly-linked list of all such objects in a global variable called __jit_debug_descriptor, and call a global function called __jit_debug_register_code when this list is changed

Any other library has no support.

Other JITs

Most JIT compilers implement not only frame pointers to make it possible to profile and debug, but also implement either C++ exception unwinder via eh_frames or even more hardcore support for gdb and other debuggers. Some links:

LuaJIT: https://github.com/LuaJIT/LuaJIT/blob/fe71d0fb54ceadfb5b5f3b6baf29e486d97f6059/src/lj_err.c#L620
DotNet: https://github.com/dotnet/coreclr/pull/468/files
DotNet remote unwinding: https://github.com/dotnet/runtime/blob/d9855f27f7d5d5138452009e9f0dd7a81c5c8b74/src/coreclr/pal/src/exception/remote-unwind.cpp
Julia (frame pointers and __register_frame): https://github.com/JuliaLang/julia/blob/caa2f7d52b430f50c8038a7f6766edba28a3fb65/src/debuginfo.cpp#L554
LLVM's ORC jit (frame pointers, __register_frame and much more): https://github.com/llvm/llvm-project/blob/935d753c6dca0cd9bc5ea14fde5b00386ebcc5be/compiler-rt/lib/orc/elfnix_platform.cpp#L191
V8: Frame pointers, __register_frame and much more: https://www.kvakil.me/posts/2022-10-17-ustackjs-unwind-javascript-stacks-in-ebpf.html, https://github.com/v8/v8/blob/b70457462cb22753a011096c9c9be20275dc4437/src/diagnostics/gdb-jit.cc#L1674, https://github.com/v8/v8/blob/b70457462cb22753a011096c9c9be20275dc4437/src/diagnostics/unwinder.cc#L100, https://github.com/v8/v8/blob/b70457462cb22753a011096c9c9be20275dc4437/src/diagnostics/eh-frame.cc

Linked PRs

Activity

  1. cfbolz commented on Nov 18, 2024

    @cfbolz
    Contributor

    Amazing research work for a super important feature. Thank you @pablogsal and @brandtbucher for this effort! I'm a strong +1.

  2. diegorusso commented on Nov 18, 2024

    @diegorusso
    Contributor

    In the text of the issue you state:

    ARM64 and macOS: Even lower overhead due to the presence of dedicated link registers

    and later in the comments:

    aarch64-apple-darwin: 2.1% slower
    aarch64-unknown-linux-gnu: 6.1% slower

    which contradicts what you said earlier. Also that outlier sounds very odd to me as all the AArch64 platforms should behave similarly.

  3. pablogsal commented on Nov 18, 2024

    @pablogsal
    MemberAuthor

    which contradicts what you said earlier. Also that outlier sounds very odd to me as all the AArch64 platforms should behave similarly.

    The original sentence was based on some tests runs I did originally but the second is a full py performance run. I also agree it looks off because macOS is using the same instruction set.

    We need to run again and confirm because it makes no sense to me and also contradicts my own small tests I did separately

  4. brandtbucher commented on Nov 18, 2024

    @brandtbucher
    Member
  5. zooba commented on Nov 18, 2024

    @zooba
    Member

    I'm +1 on this. Frame pointers are basically the only viable way to handle stack unwinding here, and it's absolutely worth the cost. At the very least, we would need a runtime option to enable them (i.e. without recompiling CPython) because there are going to be essential scenarios that need them.

    FWIW, MSVC doesn't allow enabling frame pointers for x64 or ARM64, but the function unwinding tables allow specifying them. I'm not sure what LLVM does, but it should be possible to have unwinding via a frame pointer work on these platforms too.

  6. pablogsal commented on Nov 18, 2024

    @pablogsal
    MemberAuthor

    At the very least, we would need a runtime option to enable them (i.e. without recompiling CPython) because there are going to be essential scenarios that need them.

    The idea is to have them activated by default (safety/debug should be the default) and may be a env variable or some other way to opt-out if you don't care but this should be set by the end user with knowledge of what the consequences are. Another reason we want this by default is that we don't want to make it impossible for users to sent us backtraces when the interpreter or a C extension crashes or hangs because reproducing this can be quite challenging (all of this is covered in the text).

  7. markshannon commented on Nov 18, 2024

    @markshannon
    Member

    (safety/debug should be the default)

    In what way is using frame pointers safer?

  8. markshannon commented on Nov 18, 2024

    @markshannon
    Member

    Then what do you propose?

    The interpreter is compiled without using the frame pointer. So unwinding clearly doesn't need frame pointers.
    So use whatever is used to unwind through the interpreter.

  9. pablogsal commented on Nov 18, 2024

    @pablogsal
    MemberAuthor

    (safety/debug should be the default)

    In what way is using frame pointers safer?

    Because it allows debuggers to work.

  10. markshannon commented on Nov 18, 2024

    @markshannon
    Member

    What do you mean by "safety" in this context?

  11. brandtbucher commented on Nov 18, 2024

    @brandtbucher
    Member

    A GitHub issue isn't good for real-time back-and-forth chat.

  12. pablogsal commented on Nov 18, 2024

    @pablogsal
    MemberAuthor

    What do you mean by "safety" in this context?

    That if you application crashes, hangs or generates a core you can actually use a debugger with that and not getting wrong stacks because the JIT makes them choke.

  13. pablogsal commented on Nov 18, 2024

    @pablogsal
    MemberAuthor

    Then what do you propose?

    The interpreter is compiled without using the frame pointer. So unwinding clearly doesn't need frame pointers.

    It needs frame pointers if you have a JIT compiler in the middle because the JIT doesn't have DWARF (debug information). This is explained in the issue.

    So use whatever is used to unwind through the interpreter.

    You cannot because the JIT doesn't have DWARF and a backing elf file that unwinders can use. It's just a random string of bytes.

  14. pablogsal commented on Nov 18, 2024

    @pablogsal
    MemberAuthor

    A GitHub issue isn't good for real-time back-and-forth chat.

    Agreed, let's chat on Wednesday.

  15. changed the title [-]Unwinding support for the JIT compiler[/-] [+]Supporting stack unwinding in the JIT compiler[/+] on Nov 19, 2024
  16. brandtbucher commented on Nov 20, 2024

    @brandtbucher
    Member

    Some good news: with this fix (that I plan to upstream) for LLVM's existing "reserved frame pointers" functionality...

    diff --git a/llvm/lib/Target/X86/X86RegisterInfo.cpp b/llvm/lib/Target/X86/X86RegisterInfo.cpp
    index 50db211c99d8..9b8652b7e302 100644
    --- a/llvm/lib/Target/X86/X86RegisterInfo.cpp
    +++ b/llvm/lib/Target/X86/X86RegisterInfo.cpp
    @@ -563,7 +563,7 @@ BitVector X86RegisterInfo::getReservedRegs(const MachineFunction &MF) const {
         Reserved.set(SubReg);
     
       // Set the frame-pointer register and its aliases as reserved if needed.
    -  if (TFI->hasFP(MF)) {
    +  if (TFI->hasFP(MF) || MF.getTarget().Options.FramePointerIsReserved(MF)) {
         if (MF.getInfo<X86MachineFunctionInfo>()->getFPClobberedByInvoke())
           MF.getContext().reportError(
               SMLoc(),

    ...and this change to CPython (replacing the 2-line change @pablogsal suggests above)...

    diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py
    index d8dce0a905c..4e898a86f86 100644
    --- a/Tools/jit/_targets.py
    +++ b/Tools/jit/_targets.py
    @@ -121,6 +121,8 @@ async def _compile(
                 f"-I{CPYTHON / 'Python'}",
                 f"-I{CPYTHON / 'Tools' / 'jit'}",
                 "-O3",
    +            "-Xclang",
    +            f"-mframe-pointer={'all' if opname == 'shim' else 'reserved'}",
                 "-c",
                 # This debug info isn't necessary, and bloats out the JIT'ed code.
                 # We *may* be able to re-enable this, process it, and JIT it for a

    ...frame-pointer-based unwinding works, with 0% slowdown on benchmarks!


    More context:

    There are two reasons why frame pointers are slow: you lose a register, and you must save and restore your caller's frame pointer state at the beginning and end of each function. It's the latter that's causing a slowdown for us; we compile each uop as its own function (which tail-calls into the next) and concatenate the bodies. So incrementing a fast local x += 1, should look something like this:

    • push and incref x
    • push 1
    • guard that x is an int
    • add them
    • decref and store x

    With frame pointers, this instead becomes:

    • save and set rbp
    • push and incref x
    • restore rbp
    • save and set rbp
    • push 1
    • restore rbp
    • save and set rbp
    • guard that x is an int
    • restore rbp
    • save and set rbp
    • add them
    • restore rbp
    • save and set rbp
    • decref and store x
    • restore rbp

    All of the frame pointer shuffling obviously isn't necessary; we really only need to do it once at the beginning, and once at the end. But some templates have multiple "returns", so finding and removing all of these is hard. And if we compile without frame pointers, the compiler uses the frame pointer register as scratch space, and clobbers whatever value we put there manually.

    However, LLVM does have (broken on main, seemingly fixable with the change I found above) functionality to "reserve" the frame pointer register, meaning it pretends it isn't even there. This is perfect, since it means that as long as we know the frame pointer register is a valid value on entry to our concatenated sequence of code, it will remain valid throughout.

    Whenever we call into JIT code, we already push a "shim" frame between the interpreter and the JIT code, to convert between the platform calling convention and the one used for the JIT's tail calls. We can compile just this with frame pointers, since it's very cheap to do so, and compile all of the other JIT code with the frame pointer register reserved. So when the shim calls into the JIT code, the frame pointer register remains valid, and the two frames appear as one to unwinders.

  17. zooba commented on Nov 20, 2024

    @zooba
    Member

    and the two frames appear as one to unwinders.

    This may not be the case on Windows (I trust you on other platforms), where the IP is used to look up a table embedded in the executable (or registered dynamically) to decide how to unwind. It looks like the shim function is compiled once, which means it gets its own entry that will unwind correctly, but the jitted code still needs a way to unwind.

    Unless you generate/copy the shim as part of the rest of the function and it's all contiguous, then both frames really would just be a single frame (and you can probably also just copy the function entry from the shim to apply to the rest of the code, since the unwinding procedure will be identical).

    There's some relevant code in https://github.com/microsoft/python-etwtrace/blob/main/src/etwtrace/_etwtrace.c that does this for x64 and ARM64, if you prefer a real example. Note that _thunk is never called directly by this code - that's the most not-obvious part.

  18. brandtbucher commented on Nov 20, 2024

    @brandtbucher
    Member

    Yeah, we haven't checked either approach on platforms other than Linux yet.

    It looks like the shim function is compiled once, which means it gets its own entry that will unwind correctly, but the jitted code still needs a way to unwind.

    Unless you generate/copy the shim as part of the rest of the function and it's all contiguous, then both frames really would just be a single frame (and you can probably also just copy the function entry from the shim to apply to the rest of the code, since the unwinding procedure will be identical).

    It's sort of a mix of the two. We JIT a copy of the shim for each trace, but if one trace jumps into another trace the shim frame from the first remains above it, and the second trace's shim is never used.

    So if trace A side-exits to B which side-exits to C, then the actual stack will be [..., _PyEval_EvalFrameDefault, <shim A>, <trace C>, ...]. With the approach in my comment, an unwinder would just see <shim A>'s frame pointer in rbp upon unwinding to <trace C>. Which means it would see [..., _PyEval_EvalFrameDefault, <shim A>, ...].

    Not sure if that helps at all?

  19. brandtbucher commented on Nov 20, 2024

    @brandtbucher
    Member

    This may not be the case on Windows (I trust you on other platforms), where the IP is used to look up a table embedded in the executable (or registered dynamically) to decide how to unwind.

    Based on our experiments, this seems to be how other unwinding tools work on Linux (@pablogsal can correct me if I'm wrong). If DWARF unwind info for the IP has been registered with the tool (either from loading the executable or through explicit runtime APIs), it will use that. Otherwise, many will attempt to use frame pointers as a fallback just to get through that frame, which is the entire reason why this fix works, even when the rest of the interpreter doesn't have frame pointers.

  20. zooba commented on Nov 20, 2024

    @zooba
    Member

    With the approach in my comment, an unwinder would just see <shim A>'s frame pointer in rbp upon unwinding to <trace C>. Which means it would see [..., _PyEval_EvalFrameDefault, <shim A>, ...].

    This is fine, I'm sure. It makes it a little harder to figure out exactly which Python code led to the code being executed, but stack unwinding should work (and unwinding is far more important).

    Otherwise, many will attempt to use frame pointers as a fallback just to get through that frame, which is the entire reason why this fix works, even when the rest of the interpreter doesn't have frame pointers.

    I'm 95% sure Windows doesn't fall back to frame pointers, because as I mentioned earlier they're not even enabled by the system compilers. I'm only 50% sure that registering (through those runtime APIs) that a function does use a frame pointer will even work, but at least in that case I'm prepared to report it to the OS as a bug and get it fixed.

  21. pablogsal commented on Nov 20, 2024

    @pablogsal
    MemberAuthor

    With the approach in my comment, an unwinder would just see <shim A>'s frame pointer in rbp upon unwinding to <trace C>. Which means it would see [..., _PyEval_EvalFrameDefault, <shim A>, ...].

    This is fine, I'm sure. It makes it a little harder to figure out exactly which Python code led to the code being executed, but stack unwinding should work (and unwinding is far more important).

    Otherwise, many will attempt to use frame pointers as a fallback just to get through that frame, which is the entire reason why this fix works, even when the rest of the interpreter doesn't have frame pointers.

    I'm 95% sure Windows doesn't fall back to frame pointers, because as I mentioned earlier they're not even enabled by the system compilers. I'm only 50% sure that registering (through those runtime APIs) that a function does use a frame pointer will even work, but at least in that case I'm prepared to report it to the OS as a bug and get it fixed.

    I spent some time investigating windows and trying stuff. Seems that there are two options:

    So very similar to __register_frame in Linux.

  22. brandtbucher commented on Nov 20, 2024

    @brandtbucher
    Member

    Our plan now is:

    • I'll open an issue about the reserved frame pointers fix for LLVM's x86 backend, just so we can get an idea of the timeline / likelihood of a fix.
    • We'll test that the reserved frame pointers work on AArch64 Linux with an unmodified LLVM 19. If not, we'll dig into why. If they do work (expected), we'll benchmark it and open a PR with a test (assuming performance doesn't take a hit).
    • In the meantime, don't make any deep changes to the JIT build that may break frame-pointer-based unwinding, since it's untested currently.
  23. zooba commented on Nov 20, 2024

    @zooba
    Member

    I spent some time investigating windows and trying stuff. Seems that there are two options:

    For whatever horrible reason, RtlAddGrowableFunctionTable (and associated RtlGrowFunctionTable ) is actually the only one that works. It was reported over a year ago (after I spent way too much time figuring it out), so it might get fixed, but I'd start with those APIs.

    (And it's probably not safe to actually grow a table, since you can't just append, you have to sort the table yourself. But you can provide the entire table and then never grow it. It's just that this is the only API that actually tells anyone that you added a table.)

  24. diegorusso commented on Jan 14, 2026

    @diegorusso
    Contributor

    Replying to @pablogsal comment on Brandt's PR (#143548) The comment makes more sense here.

    This patch enables frame pointers in JIT code on x86-64 Linux

    What's the situation in aarch64?

    I've been playing with AArch64 in the past days. Currently FP are automatically inserted in JIT stencils, in the shim and every non-leaf uop (that doesn't call any function)
    If we apply this change for AArch64 as well, we remove the x29 dance in the uops that call an external function. This is great becase there is less code to execute for non-leaf uops.

    Does this help the unwinders? Not really. Except the manual unwinding, the others stop at the JIT frame and cannot go past it.
    I would suggest to enable this change (#143548) for AArch64 as well (I can raise it once this goes in). In the meantime I want to write a test for the manual unwinding (with and without JIT) and check that it behaves as expected.

    Later we need to understand what's going on with the different unwinding libraries. Just for reference this is what happens on Linux (with or without this change).

    PYTHON_JIT=0

    [GNU backtrace]
    Total frames: 54
    Showing first 35 frames:
      # 0  /home/dierus01/work/ce-sw/repos/cpython-unwind/stackunwind.cpython-315-aarch64-linux-gnu.so(+0x1ac4) [0xef35ff731ac4]
      # 1  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      # 2  python(_Py_VectorCallInstrumentation_StackRefSteal+0xe8) [0xb05dd50c5178]
      # 3  python(_PyEval_EvalFrameDefault+0x5360) [0xb05dd50cf3c4]
      # 4  python(+0x2b85ac) [0xb05dd50d85ac]
      # 5  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      # 6  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      # 7  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      # 8  python(+0x2b85ac) [0xb05dd50d85ac]
      # 9  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #10  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      #11  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      #12  python(+0x2b85ac) [0xb05dd50d85ac]
      #13  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #14  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      #15  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      #16  python(+0x2b85ac) [0xb05dd50d85ac]
      #17  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #18  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      #19  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      #20  python(+0x2b85ac) [0xb05dd50d85ac]
      #21  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #22  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      #23  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      #24  python(+0x2b85ac) [0xb05dd50d85ac]
      #25  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #26  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      #27  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      #28  python(+0x2b85ac) [0xb05dd50d85ac]
      #29  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #30  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      #31  python(_PyEval_EvalFrameDefault+0x6534) [0xb05dd50d0598]
      #32  python(+0x2b85ac) [0xb05dd50d85ac]
      #33  python(PyObject_Vectorcall+0x50) [0xb05dd4f32fb0]
      #34  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xb05dd50c5634]
      ... (19 more frames)
    
    [libunwind]
    Total frames: 51
    Showing first 35 frames:
      # 0  PyObject_Vectorcall+0x50
      # 1  _Py_VectorCall_StackRefSteal+0xd8
      # 2  _PyEval_EvalFrameDefault+0x54dc
      # 3  _PyEval_Vector+0x20c
      # 4  PyObject_Vectorcall+0x50
      # 5  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      # 6  _PyEval_EvalFrameDefault+0x6534
      # 7  _PyEval_Vector+0x20c
      # 8  PyObject_Vectorcall+0x50
      # 9  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #10  _PyEval_EvalFrameDefault+0x6534
      #11  _PyEval_Vector+0x20c
      #12  PyObject_Vectorcall+0x50
      #13  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #14  _PyEval_EvalFrameDefault+0x6534
      #15  _PyEval_Vector+0x20c
      #16  PyObject_Vectorcall+0x50
      #17  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #18  _PyEval_EvalFrameDefault+0x6534
      #19  _PyEval_Vector+0x20c
      #20  PyObject_Vectorcall+0x50
      #21  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #22  _PyEval_EvalFrameDefault+0x6534
      #23  _PyEval_Vector+0x20c
      #24  PyObject_Vectorcall+0x50
      #25  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #26  _PyEval_EvalFrameDefault+0x6534
      #27  _PyEval_Vector+0x20c
      #28  PyObject_Vectorcall+0x50
      #29  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #30  _PyEval_EvalFrameDefault+0x6534
      #31  _PyEval_Vector+0x20c
      #32  PyObject_Vectorcall+0x50
      #33  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      #34  _PyEval_EvalFrameDefault+0x6534
      ... (16 more frames)
    
    [libdw (DWARF)]
    Total frames: 51
    Showing first 35 frames:
      # 0  PyObject_Vectorcall
      # 1  _Py_VectorCall_StackRefSteal
      # 2  _PyEval_EvalFrameDefault
      # 3  _PyEval_Vector
      # 4  PyObject_Vectorcall
      # 5  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      # 6  _PyEval_EvalFrameDefault
      # 7  _PyEval_Vector
      # 8  PyObject_Vectorcall
      # 9  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #10  _PyEval_EvalFrameDefault
      #11  _PyEval_Vector
      #12  PyObject_Vectorcall
      #13  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #14  _PyEval_EvalFrameDefault
      #15  _PyEval_Vector
      #16  PyObject_Vectorcall
      #17  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #18  _PyEval_EvalFrameDefault
      #19  _PyEval_Vector
      #20  PyObject_Vectorcall
      #21  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #22  _PyEval_EvalFrameDefault
      #23  _PyEval_Vector
      #24  PyObject_Vectorcall
      #25  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #26  _PyEval_EvalFrameDefault
      #27  _PyEval_Vector
      #28  PyObject_Vectorcall
      #29  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #30  _PyEval_EvalFrameDefault
      #31  _PyEval_Vector
      #32  PyObject_Vectorcall
      #33  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      #34  _PyEval_EvalFrameDefault
      ... (16 more frames)
    
    [Manual frame pointers]
    Total frames: 52
    Showing first 35 frames:
      # 0  0xb05dd4f32fb0
      # 1  0xb05dd50c4f48
      # 2  0xb05dd50cf540
      # 3  0xb05dd50d85ac
      # 4  0xb05dd4f32fb0
      # 5  0xb05dd50c5634
      # 6  0xb05dd50d0598
      # 7  0xb05dd50d85ac
      # 8  0xb05dd4f32fb0
      # 9  0xb05dd50c5634
      #10  0xb05dd50d0598
      #11  0xb05dd50d85ac
      #12  0xb05dd4f32fb0
      #13  0xb05dd50c5634
      #14  0xb05dd50d0598
      #15  0xb05dd50d85ac
      #16  0xb05dd4f32fb0
      #17  0xb05dd50c5634
      #18  0xb05dd50d0598
      #19  0xb05dd50d85ac
      #20  0xb05dd4f32fb0
      #21  0xb05dd50c5634
      #22  0xb05dd50d0598
      #23  0xb05dd50d85ac
      #24  0xb05dd4f32fb0
      #25  0xb05dd50c5634
      #26  0xb05dd50d0598
      #27  0xb05dd50d85ac
      #28  0xb05dd4f32fb0
      #29  0xb05dd50c5634
      #30  0xb05dd50d0598
      #31  0xb05dd50d85ac
      #32  0xb05dd4f32fb0
      #33  0xb05dd50c5634
      #34  0xb05dd50d0598
      ... (17 more frames)
    

    PYTHON_JIT=1

    [GNU backtrace]
    Total frames: 8
    Showing first 8 frames:
      # 0  /home/dierus01/work/ce-sw/repos/cpython-unwind/stackunwind.cpython-315-aarch64-linux-gnu.so(+0x1ac4) [0xf3543a591ac4]
      # 1  python(PyObject_Vectorcall+0x50) [0xad493f002fb0]
      # 2  python(_Py_VectorCallInstrumentation_StackRefSteal+0xe8) [0xad493f195178]
      # 3  python(_PyEval_EvalFrameDefault+0x5360) [0xad493f19f3c4]
      # 4  python(+0x2b85ac) [0xad493f1a85ac]
      # 5  python(PyObject_Vectorcall+0x50) [0xad493f002fb0]
      # 6  python(_Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0) [0xad493f195634]
      # 7  [0xf3543ade7520]
    
    [libunwind]
    Total frames: 8
    Showing first 8 frames:
      # 0  PyObject_Vectorcall+0x50
      # 1  _Py_VectorCall_StackRefSteal+0xd8
      # 2  _PyEval_EvalFrameDefault+0x54dc
      # 3  _PyEval_Vector+0x20c
      # 4  PyObject_Vectorcall+0x50
      # 5  _Py_BuiltinCallFastWithKeywords_StackRefSteal+0xd0
      # 6  <unknown>
      # 7  <unknown>
    
    [libdw (DWARF)]
    Total frames: 8
    Showing first 8 frames:
      # 0  PyObject_Vectorcall
      # 1  _Py_VectorCall_StackRefSteal
      # 2  _PyEval_EvalFrameDefault
      # 3  _PyEval_Vector
      # 4  PyObject_Vectorcall
      # 5  _Py_BuiltinCallFastWithKeywords_StackRefSteal
      # 6  <unknown>
      # 7  <unknown>
    
    [Manual frame pointers]
    Total frames: 62
    Showing first 35 frames:
      # 0  0xad493f002fb0
      # 1  0xad493f194f48
      # 2  0xad493f19f540
      # 3  0xad493f1a85ac
      # 4  0xad493f002fb0
      # 5  0xad493f195634
      # 6  0xf3543ade7520 <=== very likely to be the shim JIT
      # 7  0xad493f19eda8
      # 8  0xad493f1a85ac
      # 9  0xad493f002fb0
      #10  0xad493f195634
      #11  0xf3543ade7520 <=== very likely to be the shim JIT
      #12  0xad493f19eda8
      #13  0xad493f1a85ac
      #14  0xad493f002fb0
      #15  0xad493f195634
      #16  0xf3543ade7520 <=== very likely to be the shim JIT
      #17  0xad493f19eda8
      #18  0xad493f1a85ac
      #19  0xad493f002fb0
      #20  0xad493f195634
      #21  0xf3543ade7520 <=== very likely to be the shim JIT
      #22  0xad493f19eda8
      #23  0xad493f1a85ac
      #24  0xad493f002fb0
      #25  0xad493f195634
      #26  0xf3543ade7520 <=== very likely to be the shim JIT
      #27  0xad493f19eda8
      #28  0xad493f1a85ac
      #29  0xad493f002fb0
      #30  0xad493f195634
      #31  0xf3543ade7520 <=== very likely to be the shim JIT
      #32  0xad493f19eda8
      #33  0xad493f1a85ac
      #34  0xad493f002fb0
      ... (27 more frames)
    
  25. added a commit that references this issue on Jan 22, 2026
  26. added a commit that references this issue on Jan 27, 2026
  27. added a commit that references this issue on Feb 15, 2026
  28. ngoldbaum commented on Mar 3, 2026

    @ngoldbaum
    Contributor

    I happened to see this issue today.

    Tools Requiring Stack Unwinding

    There's another use for stack unwinding in the ecosystem besides the uses mentioned here: to enable an optimization in NumPy on Linux. In particular, NumPy will skip creating large temporary arrays if, after scanning the call stack it determines that a temporary array was created by Python or NumPy (e.g. in pure python code) and not inside an extension in some other library.

    See the check_callers function for all the gory details.

  29. pablogsal commented on Mar 3, 2026

    @pablogsal
    MemberAuthor

    I happened to see this issue today.

    Tools Requiring Stack Unwinding

    There's another use for stack unwinding in the ecosystem besides the uses mentioned here: to enable an optimization in NumPy on Linux. In particular, NumPy will skip creating large temporary arrays if, after scanning the call stack it determines that a temporary array was created by Python or NumPy (e.g. in pure python code) and not inside an extension in some other library.

    See the check_callers function for all the gory details.

    Thanks for the comment! We indeed are already familiar with this. We recently added

    PyAPI_FUNC(int) PyUnstable_Object_IsUniquelyReferenced(PyObject *);
    precisely for this use case :)

  30. diegorusso commented on Mar 13, 2026

    @diegorusso
    Contributor

    Hello, here is an update on this.

    I have spent some time investigating and the situation is indeed more complex than it first appeared. The original approach based only on frame pointers (FP) is not sufficient to provide something that works reliably today. The main issue is that profilers and debuggers do not consistently support unwinding using only frame pointers.

    For profiling, frame pointers can often be enough. However, if you want proper debugging information then DWARF data is required.

    For example, @brandtbucher’s PR (#143548) improves the situation on x86_64. On AArch64, however, libunwind also needs changes. I currently have a downstream patch that makes this work. We would still want Brandt’s PR for AArch64 as well because it produces better code generation.

    Thanks to #144137 we know that frame pointers are correctly set and usable. We can manually unwind the stack and walk past the JIT frames. So the issue is not that we are doing something unusual with frame pointers. They are present and valid, but the tooling ecosystem is not fully ready to rely on them. The de facto standard for unwinding remains EH frames.

    I also experimented with embedding the shim directly into the Python binary and generating DWARF information for it. That works while execution stays within the binary, but the debugger still gets lost once execution jumps into JIT allocated memory.
    The core problem is that when we JIT code we do not emit EH frames. As a result, profilers and debuggers cannot determine how to unwind once they reach JIT frames.

    I have also discussed this with the V8 team internally and they confirmed that unwinding through JIT code is generally painful. Their situation is slightly different because in many debugging scenarios they can simply disable the JIT. They rarely use GDB directly and when they do they rely on a dedicated GDB script that knows how to unwind V8 JIT frames: https://chromium.googlesource.com/v8/v8/+/refs/heads/main/tools/gdbinit#803 (This is x86_64 only 😅 )

    For now the plan of action is:

    • get Brandt’s PR merged
    • add support to emit unwind frames for GDB when generating JIT code (PRs coming)
    • propose a libunwind change to use frame pointers on AArch64, similar to how the x86_64 backend already behaves
    • experiment with different tools (samply, perf, GNU backtrace, LLDB, no-perf, remote libunwind)
  31. added 2 commits that reference this issue on Mar 17, 2026
  32. added a commit that references this issue on Mar 24, 2026
  33. added 2 commits that reference this issue on Mar 27, 2026
  34. added 3 commits that reference this issue on Mar 27, 2026
  35. markshannon commented on Mar 28, 2026

    @markshannon
    Member

    Has anyone measured the performance impact of the extra function calls for stack checking?
    What should be a couple of machine instructions seems to have become 2 calls.

  36. Fidget-Spinner commented on Mar 28, 2026

    @Fidget-Spinner
    Member

    @markshannon that's not the implementation on main. It's one call on main #146524 and specifically only for 3 opcodes that use that specific recursion check function: CALL_BUILTIN_O, and two more. Considering only 3 opcodes have an extra call overhead, I highly doubt anything changed. We can wait to see if anything regresses on Savannah's machines tonight.

  37. added a commit that references this issue on Apr 1, 2026
  38. encukou commented on Apr 2, 2026

    @encukou
    Member

    test_perf_profiler (test_python_calls_appear_in_the_stack_if_perf_activated) has been failing on AMD64 CentOS9 NoGIL buildbot since GH-147945 was merged: https://buildbot.python.org/#/builders/1609/builds/5291
    (There were 8 other changes as well; this one looks most suspicious to me.)

  39. added a commit that references this issue on Apr 2, 2026
  40. markshannon commented on Apr 2, 2026

    @markshannon
    Member

    I don't know what to do about that. The test output is tens of thousands of lines long, with no clear hint as to what is wrong.

    Is it failing on any other x86 and/or linux machines? #147945 is hardware specific, not O/S specific.

    p.s.
    I really wish people would not write tests that do regex comparisons on some output, but test the actual thing they care about and report a meaningful error.

  41. added a commit that references this issue on Apr 2, 2026
  42. mcepl commented on Apr 8, 2026

    @mcepl
    Contributor

    What’s the relationship between this and #144741 ?

  43. added 5 commits that reference this issue on Apr 16, 2026
  44. added a commit that references this issue on Apr 23, 2026
  45. added 7 commits that reference this issue on Apr 25, 2026
  46. added 3 commits that reference this issue on Apr 28, 2026
  47. added 3 commits that reference this issue on May 2, 2026
  48. added 2 commits that reference this issue on May 2, 2026
  49. added a commit that references this issue on May 7, 2026
  50. added a commit that references this issue on May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

3.14bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-JIT

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions