Skip to content

test_external_inspection: test_popping_python_frame_is_not_native() fails with a timeout (30 sec) on some buildbots #162251

Open

Description

@vstinner

Bug report

Examples of builds where the test failed and then passed when re-run in verbose mode:

Timeout error:

FAIL: test_popping_python_frame_is_not_native (test.test_external_inspection.TestGetStackTrace.test_popping_python_frame_is_not_native)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_external_inspection.py", line 2089, in test_popping_python_frame_is_not_native
    _get_stack_trace_with_retry(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        unwinder,
        ^^^^^^^^^
    ...<3 lines>...
        ),
        ^^
    )
    ^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_external_inspection.py", line 268, in _get_stack_trace_with_retry
    for _ in busy_retry(timeout):
             ~~~~~~~~~~^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 2853, in busy_retry
    raise AssertionError(msg)
AssertionError: timeout (30.0 seconds)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. added
    type-bugAn unexpected behavior, bug, or error
    testsTests in the Lib/test dir
    on Nov 19, 2026
  2. vstinner commented on Nov 19, 2026

    @vstinner
    MemberAuthor

    The test was modified recently by commit eeafc73 of issue gh-160986 to "Fix flaky test_popping_python_frame_is_not_native". The change adds a new _get_stack_trace_with_retry() function and that's the function which fails with a timeout.

    Before this change, the test failed differently. Example of a RISC-V 64-bit Ubuntu 3.x build, one month ago: https://buildbot.python.org/#/builders/2237/builds/720

    FAIL: test_popping_python_frame_is_not_native (test.test_external_inspection.TestGetStackTrace.test_popping_python_frame_is_not_native)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/var/lib/buildbot/workers/rise-riscv64-4/3.x.rise-riscv64-4/build/Lib/test/test_external_inspection.py", line 1980, in test_popping_python_frame_is_not_native
        self.assertGreater(samples, 1000)
        ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
    AssertionError: 0 not greater than 1000
    
  3. vstinner commented on Nov 19, 2026

    @vstinner
    MemberAuthor

    The "AssertionError: timeout" error is not new. Example of a RISC-V 64-bit Ubuntu 3.x build, 12 days ago: https://buildbot.python.org/#/builders/2237/builds/853. I failed with AssertionError: timeout (60.0 seconds) (and similar logs).

  4. vstinner commented on Nov 19, 2026

    @vstinner
    MemberAuthor

    I failed to reproduce the issue locally on my Fedora 45. I ran a test for 6 minutes (test_external_inspection ran 209 times).

    • Python built in release mode with ./configure --prefix '$(PWD)/target' 'CFLAGS=-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer'
    • Run ./python -m test test_external_inspection -j2 -F -W
  5. tanloong commented on Nov 19, 2026

    @tanloong
    Contributor
    Reproduced on my Arch (screen recording):
    QQ20261119-194551.mp4

    The RemoteUnwinder caches parent interpreter addresses before child has the time to execve() to replace addresses copied from the forked parent address space. After execve() those cached addresses become invalid, from where finding leaf frame fails. Should let the child send a signal when it finishes the execve(), and initialize RemoteUnwinder only after that.

  6. vstinner commented on Nov 19, 2026

    @vstinner
    MemberAuthor

    Oh. It reminds me my commit 6fd15f0 fix which waits until the child process calls execve(). Maybe we need a similar check here?

  7. tanloong commented on Nov 19, 2026

    @tanloong
    Contributor

    OK I have updated in a63e693 to use /proc/<pid>/cmdline to determine whether child has called execve(). Wait until cmdline contains child's script before creating RemoteUnwinder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtopic-profilingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions