Skip to content

gh-159986: Avoid in-process multiprocessing manager finalizer deadlocks - #162083

Closed
Jacob-Met wants to merge 2 commits into
python:mainfrom
Jacob-Met:fix/gh-159986-manager-local-decref
Closed

Jacob-Met wants to merge 2 commits into
python:mainfrom
Jacob-Met:fix/gh-159986-manager-local-decref

Conversation

@Jacob-Met

Copy link
Copy Markdown

Problem

A proxy's BaseProxy._decref finalizer synchronously opens a new connection and waits for the manager to answer. When the manager server runs in the same process, cyclic GC can invoke the finalizer on its own accepter thread, or while Server.create holds its mutex. In either case the finalizer waits for work that the blocked server thread cannot complete, permanently deadlocking a live manager. This is reproducible with in-process threaded serve_forever() and is reported in Apache Beam.

Change

Keep a weak registry of Server instances running inside the current process (with PID checking to avoid using a stale registry after fork). In-process proxy finalizers enqueue decref identifiers instead of opening a blocking RPC. The server's existing serve_forever loop consumes these decrements outside the accepter and request threads; remote-manager finalizers retain the existing RPC path. The registry accommodates multiple servers, and entries are retired when servers stop.

Test

Add a deterministic subprocess regression that forces cyclic GC during the accepter's new request-thread creation, both with one and with multiple local manager servers. It asserts the RPC completes rather than deadlocking, and that queued decrements are eventually processed. The earlier isolated run of the exact pushed branch fix/gh-159986-manager-local-decref (tip 4a95ae9) reported 276 manager tests passing. The presenting agent reviewed the deadlock and queue-based fix; no fresh full test suite run is asserted here.

AI disclosure

Implemented and tested with autonomous GPT-6 assistance; reviewed by the presenting assistant. No human review is claimed.

Fixes #159986.

@Jacob-Met
Jacob-Met requested a review from gpshead as a code owner November 16, 2026 05:19
@python-cla-bot

Copy link
Copy Markdown

The following commit authors need to sign the Contributor License Agreement:

CLA not signed

@ZeroIntensity ZeroIntensity added the AI-guidelines-not-met The contribution does not meet our guidelines for using AI tools and is closed automatically. label Nov 16, 2026
@bedevere-app

bedevere-app Bot commented Nov 16, 2026

Copy link
Copy Markdown

Thank you for your interest in contributing to Python. This is a community-driven project, and we want it to be a welcoming place to learn and collaborate.

Contributors are responsible for reviewing and understanding what they submit, checking its claims, and being able to explain changes in their own words. This helps reviewers focus on the contribution and leaves more time to support other contributors.

We believe this contribution does not meet our guidelines for using AI tools, so we're closing it. If you think we've misunderstood your contribution, please let us know. You're welcome to explain your approach or ask questions about the guidelines.

Please read our guidelines for using AI tools.

@bedevere-app bedevere-app Bot closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-guidelines-not-met The contribution does not meet our guidelines for using AI tools and is closed automatically. awaiting review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

multiprocessing.managers proxy finalizer can deadlock a manager server running in the same process

3 participants