Skip to content

gh-160978: Return the same ctypes array type when created concurrently - #161494

Merged
kumaraditya303 merged 4 commits into
python:mainfrom
thalesmourabh:gh-160978-ctypes-array-cache
Nov 20, 2026
Merged

kumaraditya303 merged 4 commits into
python:mainfrom
thalesmourabh:gh-160978-ctypes-array-cache

Conversation

@thalesmourabh

@thalesmourabh thalesmourabh commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

PyCArrayType_from_ctype() looked the array type up in array_cache, created it on a miss, and only then stored it:

if (_PyDict_GetItemProxy(st->array_cache, key, &result) != 0) {
    Py_DECREF(key);
    return result;
}
/* not found */
...
result = PyObject_CallFunction((PyObject *)st->PyCArrayType_Type, ...);
...
if (PyDict_SetItemProxy(st, st->array_cache, key, result) < 0) { ... }
return result;

Creating the type runs Python code, so two threads can both miss the cache, both create a type, and each return its own. The second one overwrites the first in the cache, and callers end up holding distinct types for the same item type and length — hence the TypeError: expected c_int_Array_1 instance instead of c_int_Array_1 in the issue.

Why not hold a lock around the whole miss path: the type is created by PyObject_CallFunction(), which can run arbitrary Python code — the __set_name__ reproducer in the issue is exactly that. Holding a lock across it invites re-entrancy and deadlock. So this serialises the insertion instead: _PyDict_SetDefaultItemProxy() uses PyDict_SetDefaultRef() so the first type to reach the cache wins, and a thread that loses the race drops the type it built and returns the cached one. Two threads may still both build a type, but they now agree on which one is canonical.

If the cached entry has already been cleared by its DictRemover, the new type is installed instead, as before.

Tests. Two, because they fail on different builds:

default build free-threaded
test_concurrent_creation_with_set_name fails without the fix fails without the fix
test_concurrent_creation_returns_identical_types passes fails without the fix

The first one is the only one that guards the fix on the default build, which is why it is there even though the issue is about free-threading.

Checked. Built and tested on --with-pydebug and --with-pydebug --disable-gil at the same commit: full test_ctypes passes on both (648 tests), and passes under -R 3:3 on the free-threaded build. The reproducer in the issue goes from 397/400 failures to 0/400 there.

Not covered: I did not audit the other proxy caches in _ctypes (st->cache for pointer types, for instance) for the same pattern. They look similar, but I have not reproduced anything there, so this change is limited to the array cache.

…rrently

PyCArrayType_from_ctype() looked the array type up in the cache, created it
on a miss, and only then stored it. Creating the type can run Python code,
so two threads could both miss the cache, both create a type, and each
return its own, leaving callers with distinct types for the same item type
and length.

Insert the new type with PyDict_SetDefaultRef() so that the first type to
reach the cache wins, and return that one.
@python-cla-bot

python-cla-bot Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

thalesmourabh added a commit to thalesmourabh/branches that referenced this pull request Nov 6, 2026
PR aberto com aprovacao explicita:
python/cpython#161494

OPEN, autor thalesmourabh, +122 -3 nos tres arquivos esperados, label
"awaiting review" aplicada pelo bedevere, CI rodando. Conferi antes de abrir
que o corpo nao tem mencao de IA nem trailer de co-autoria.

MAS o check "CLA Signing" falhou. O python-cla-bot comentou nomeando o email
que falta: thalesmoura13@gmail.com. O bot casa pelo EMAIL DO AUTOR DO COMMIT,
nao pela conta do GitHub, e o commit foi autorado com esse email -- entao a
assinatura precisa estar registrada para ele.

Tres explicacoes, por probabilidade: assinou com outro email; a assinatura
ainda nao propagou (o bot re-checa por webhook); ou o email nao esta na lista
de emails da conta em cla.python.org.

Se foi outro email, ha duas saidas: assinar tambem com thalesmoura13@gmail.com,
ou re-autorar o commit com o email usado na assinatura e force-pushar. A
segunda e permitida -- a R2 proibe force-push DURANTE review, e a review nao
comecou.
Emscripten builds without pthreads by default, so ThreadPoolExecutor and
threading.Barrier are unavailable there and the test errored out. Guard the
remaining test with requires_working_threading() as well.
Comment thread Modules/_ctypes/_ctypes.c Resolved
Comment thread Modules/_ctypes/_ctypes.c Resolved
@kumaraditya303
kumaraditya303 merged commit c0f3fcf into python:main Nov 20, 2026
67 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @thalesmourabh for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Nov 20, 2026

Copy link
Copy Markdown

GH-162312 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Nov 20, 2026
@bedevere-app

bedevere-app Bot commented Nov 20, 2026

Copy link
Copy Markdown

GH-162313 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Nov 20, 2026
kumaraditya303 pushed a commit that referenced this pull request Nov 20, 2026
…urrently (GH-161494) (#162313)

gh-160978: Return the same ctypes array type when created concurrently (GH-161494)
(cherry picked from commit c0f3fcf)

Co-authored-by: Thales Moura <thalesmoura13@gmail.com>
kumaraditya303 pushed a commit that referenced this pull request Nov 20, 2026
…urrently (GH-161494) (#162312)

gh-160978: Return the same ctypes array type when created concurrently (GH-161494)
(cherry picked from commit c0f3fcf)

Co-authored-by: Thales Moura <thalesmoura13@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants