[espressolee]

Description

Part of #6159 (item 59).

Under Py_GIL_DISABLED, cpp_function::dispatcher() took the internals lock on every bound call: it reached its function_record through function_record_ptr_from_PyObject(), which calls get_function_record_PyTypeObject() (PYBIND11_LOCK_INTERNALS on internals.mutex).

dispatcher is installed in exactly one place (initialize_generic), and the PyCFunction there is created with m_self set to the function_record_PyObject allocated a few lines above, so self is always that object. The new function_record_ptr_from_dispatcher_self() states that invariant once and casts directly. Its assert compares the versioned tp_name instead of calling is_function_record_PyObject(), which would reacquire the lock in non-NDEBUG builds (the preset default); that costs one or two strcmp per call in debug builds only. is_function_record_PyObject() and function_record_ptr_from_PyObject() are unchanged, and the three call sites that can see a foreign object (the overload chain in initialize_generic, get_function_record, reduce_ex_impl) keep the checked conversion. The exception path (try_translate_exceptions -> with_exception_translators) still takes exception_translator_mutex; that is a different mutex and is not touched here.

Test: test_dispatch_does_not_need_internals_lock in tests/test_thread.*, free-threaded builds only. A native thread holds internals.mutex (through raw PyCFunction helpers, because a pybind11-bound helper would itself go through the dispatcher and deadlock on the lock it is setting up); the verdict is whether the bound call returned before the holder's watchdog (5 s) released the mutex, so on a regression the test fails after 5 s rather than hanging. A positive control (with_internals() with a 1 s watchdog) first proves the held mutex is the one internals users take, so the test cannot go vacuous if the dispatcher's lock ever moves. The body runs in a subprocess with a 60 s timeout. It fails without the fix and passes with it (3/3 each, Debug + PYBIND11_WERROR=ON, CPython 3.14.0rc1t, macOS arm64).

Measured with the reproducer from the issue (one no-arg bound call; Apple M4 Pro, unpinned; CPython 3.14.0rc1 free-threaded; pybind11 3.1.0 headers with and without this patch; -O2 -DNDEBUG -std=c++17; 9 runs each, medians, M calls/s):

threads	before	after
1	36.2	48.9
4	19.6 (x0.54 of 1T)	59.2 (x1.21)
8	14.4 (x0.40)	93.2 (x1.91)

Ranges do not overlap at 4 and 8 threads; at 1 thread they do (one low sample on the patched side).

Suggested changelog entry:

• cpp_function::dispatcher() no longer acquires the internals lock on every call in free-threaded builds: the function record is now read directly from self, which is always the record object created for that dispatcher.

AI assistance

Claude Opus 5 (claude-opus-5[1m]) and Claude Fable 5.1, with review by me: the patch, the test, the builds and the measurements were produced with Claude Code on my machine and reviewed by me before submission.

---

📚 Documentation preview 📚: https://pybind11--6177.org.readthedocs.build/

[Skylion007]

Should be std::strcmp nit

[espressolee]

Thanks — I’ve updated both calls to std::strcmp locally. Formatting, C++11/C++20 compilation, and the full prek check pass. I haven’t pushed the change yet.

[espressolee]

Merged master to resolve the conflict with #6136 (no rebase, so the approved commits are unchanged): the dispatcher now lives in include/pybind11/pybind11-inl.h, so the one-line change moved there, and the diff is otherwise the approved one. The only red job (ubuntu 3.9, FINDPYTHON, C++17) is test_numpy_dtypes::test_str_leak (assert 11 == 12, the refcount went down), which looks like the known flake in #2501: the same job passed on the previous head and on #6189, and it did not reproduce locally with CPython 3.9.25 + numpy 1.21.6 (merge and master, 33 runs each). Could someone re-run it?

[rwgk]

Nice optimization! I see the assertions clearly document the invariant behind the direct cast.