Skip to content

os.fork on macOS leaks mach vm region per call (since 3.12) #162266

Open

Description

@karelbilek

Bug report

While running Ansible, I have noticed slowdown of os.fork during relatively long playbooks. Ansible is an orchestration tool that uses os.fork internally very extensively (a lot of short-living fork), and it's not unusual to have hundreds of thousands of os.forks(). While initially, os.fork takes 0.5 ms, over time, it climbs up to 20 ms and taking half of program time. Reverting to python 3.11 "fixes" the issue, as does monkey-patching around os.fork.

With the help of Claude, I have found out the root cause; a memory leak in get_number_of_os_threads. The repros are made by Claude, but they are easy to understand.

If I understand, that whole get_number_of_os_threads is there only to print DeprecationWarning; it causes a memory leak and a significant slowdown even when that DeprecationWarning is not printed; warn_about_fork_with_threads will always be called, even with -w ignore and with no warning printed, and it will always leak.

Reproducible example that works without vmmap

import os, sys, time
heap = [{"k%d" % i: "v" * 20, "n": i} for i in range(1_000_000)]  # ~300 MB
block = []
i = 0
while True:
    i = i+1
    t = time.perf_counter()
    pid = os.fork()
    if pid == 0: os._exit(0)
    block.append(time.perf_counter() - t)
    os.waitpid(pid, 0)
    if i % 5000 == 0:
        b = sorted(block); block = []
        print(f"{sys.version.split()[0]} forks {i:6}: median {b[len(b)//2]*1000:5.2f} ms  p90 {b[int(len(b)*.9)]*1000:5.2f} ms", flush=True)

Root cause

(this is half-written by Claude)

On macOS, every os.fork() call leaks one 16 KB "Mach Message" VM region in the parent process. The regions are never freed; a long-running parent that forks many children (e.g. a multiprocessing fork-per-task pattern) accumulates tens of thousands of them. Because fork() has to walk every VM map entry, the cost of fork() itself then grows linearly with the number of forks done so far.

Measured on macOS 27.2.1 (arm64), 300 MB heap in the parent, children os._exit(0) immediately:

forks done 3.14.7 fork() median 3.11.16 fork() median
5000 1.09 ms 0.58 ms
10000 2.45 ms 0.60 ms
20000 5.57 ms 0.62 ms
30000 9.71 ms 0.60 ms

The issue is in this code

#if defined(__APPLE__) && defined(HAVE_GETPID)
    mach_port_t macos_self = mach_task_self();
    mach_port_t macos_task;
    if (task_for_pid(macos_self, getpid(), &macos_task) == KERN_SUCCESS) {
        thread_array_t macos_threads;
        mach_msg_type_number_t macos_n_threads;
        if (task_threads(macos_task, &macos_threads,
                         &macos_n_threads) == KERN_SUCCESS) {
            num_python_threads = macos_n_threads;
        }
    }

task_threads() returns the thread list as out-of-line Mach message memory that the caller owns; it is never vm_deallocate()d, and the thread ports (and the task port from task_for_pid()) are never deallocated either.

Bigger repro, also Claude-made

"""Repro: os.fork() leaks one "Mach Message" VM region per call in the parent on macOS (CPython 3.12+).

Usage: python3 repro.py [N]      (default N=2000; needs /usr/bin/vmmap, i.e. Xcode command line tools)
"""
import os
import subprocess
import sys
import time


def mach_message_regions():
    out = subprocess.run(["vmmap", "-summary", str(os.getpid())], capture_output=True, text=True).stdout
    for line in out.splitlines():
        if line.startswith("Mach Message"):
            return int(line.split()[-1])  # REGION COUNT column
    return 0


def fork_many(n):
    times = []
    for _ in range(n):
        t = time.perf_counter()
        pid = os.fork()
        if pid == 0:
            os._exit(0)
        times.append(time.perf_counter() - t)
        os.waitpid(pid, 0)
    times.sort()
    return times[len(times) // 2] * 1000


n = int(sys.argv[1]) if len(sys.argv) > 1 else 2000
print(f"python {sys.version.split()[0]}, {n} forks, children exit immediately and are reaped")
print(f"Mach Message regions before: {mach_message_regions()}")
step = n // 4
for i in range(1, 5):
    median = fork_many(step)
    print(f"after {i * step:6} forks: Mach Message regions = {mach_message_regions():6}, fork() median = {median:.2f} ms")

Both repros don't show this issue on 3.11, but show on all 3.12+.

Claude also recommended me a fix, but I am not sure what is the AI copyright policy here; and I will freely admit I don't really understand XNU/Mach; so will probably refrain from doing that.

CPython versions tested on:

3.14

Operating systems tested on:

macOS

Linked PRs

Activity

  1. karelbilek commented on Nov 19, 2026

    @karelbilek
    Author

    the code path

    if (warn_about_fork_with_threads("fork", num_os_threads) < 0)

    if (task_threads(macos_task, &macos_threads,

    https://web.mit.edu/darwin/src/modules/xnu/osfmk/man/task_threads.html

    out pointer to dynamic array

    according to 1992 (!!) MIG PDF, that means you are responsible for deallocating it

    In contrast, the notation “out pointer to dynamic array” means that the kernel will allocate
    space for returned data (as if by vm_allocate) and will modify the pointer named by
    the output parameter (that is, the parameter to the function is a pointer to a pointer) to
    point to this allocated memory. The task should vm_deallocate this space when done referencing
    it.

  2. maurycy commented on Nov 19, 2026

    @maurycy
    Contributor

    @karelbilek I truly love this bug report. I confirm this is a real issue.

    This is the AI policy:

    https://devguide.python.org/getting-started/ai-tools/

    To be honest, I think this is an excellent usage of AI!

    Personally I'd just switch to proc_pidinfo() and call it a day:

    diff --git i/Modules/posixmodule.c w/Modules/posixmodule.c
    index 619154309b4..3e0f2f22ebc 100644
    --- i/Modules/posixmodule.c
    +++ w/Modules/posixmodule.c
    @@ -125,6 +125,10 @@
     
     #if defined(__APPLE__)
     #  include <copyfile.h>           // fcopyfile()
    +#  include <TargetConditionals.h>
    +#  if !defined(TARGET_OS_OSX) || TARGET_OS_OSX
    +#    include <libproc.h>          // proc_pidinfo()
    +#  endif
     #endif
     
     #ifdef HAVE_SCHED_H
    @@ -479,8 +483,6 @@ static const unsigned int _Py_STATX_KNOWN = (STATX_BASIC_STATS | STATX_BTIME
      */
     #if defined(__APPLE__)
     
    -#include <mach/mach.h>
    -
     #if defined(__has_builtin)
     #if __has_builtin(__builtin_available)
     #define HAVE_BUILTIN_AVAILABLE 1
    @@ -8570,16 +8572,11 @@ static Py_ssize_t get_number_of_os_threads(void)
         // of threads in the process. That'd presumably use this platform code but
         // raise an error rather than using the inaccurate fallback.
         Py_ssize_t num_python_threads = 0;
    -#if defined(__APPLE__) && defined(HAVE_GETPID)
    -    mach_port_t macos_self = mach_task_self();
    -    mach_port_t macos_task;
    -    if (task_for_pid(macos_self, getpid(), &macos_task) == KERN_SUCCESS) {
    -        thread_array_t macos_threads;
    -        mach_msg_type_number_t macos_n_threads;
    -        if (task_threads(macos_task, &macos_threads,
    -                         &macos_n_threads) == KERN_SUCCESS) {
    -            num_python_threads = macos_n_threads;
    -        }
    +#if defined(PROC_PIDTASKINFO) && defined(HAVE_GETPID)
    +    struct proc_taskinfo info;
    +    if (proc_pidinfo(getpid(), PROC_PIDTASKINFO, 0, &info, sizeof(info))
    +            == sizeof(info)) {
    +        num_python_threads = info.pti_threadnum;
         }
     #elif defined(__linux__)
         // Linux /proc/self/stat 20th field is the number of threads.

    I can open a PR if you want. Please go ahead, though:)

    My reasoning is that proc_pidinfo() is available since 10.5 (2007), and it's simpler: kernel doesn't pass anything, we don't have to release anything etc.

    cc @gpshead @ned-deily @ronaldoussoren

  3. karelbilek commented on Nov 19, 2026

    @karelbilek
    Author

    Claude wrote me much smaller fix. Just call the vm_deallocate (+ mach_port_deallocate).

            if (task_threads(macos_task, &macos_threads,
                             &macos_n_threads) == KERN_SUCCESS) {
                num_python_threads = macos_n_threads;
                for (mach_msg_type_number_t i = 0; i < macos_n_threads; i++) {
                    mach_port_deallocate(macos_self, macos_threads[i]);
                }
                vm_deallocate(macos_self, (vm_address_t)macos_threads,
                              macos_n_threads * sizeof(thread_act_t));
            }
            mach_port_deallocate(macos_self, macos_task);

    mach_port_deallocate is a slightly different issue, decreases ref counting of the ports (so also potential memory leak).

    I did not try this code. (I don't even know how to compile python.)

  4. karelbilek commented on Nov 19, 2026

    @karelbilek
    Author

    Please open the PR, thanks

  5. karelbilek commented on Nov 19, 2026

    @karelbilek
    Author
  6. karelbilek commented on Nov 19, 2026

    @karelbilek
    Author

    looking just at the source code, your version looks much more faster/cleaner, with less allocation/deallocation. Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-macextension-modulesC modules in the Modules dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions