Skip to content

UBSan: Calling a function through pointer to incorrect function type is undefined behavior #111178

Closed

Description

@chrstphrchvz

Bug report

Bug description:

UBSan (UndefinedBehaviorSanitizer) in LLVM.org Clang 17 makes -fsanitize=function available for C; previously, it was only for C++. (So it may also be made available in future Apple Xcode clang and GCC.) By default, it is implied by -fsanitize=undefined (which is what ./configure --with-undefined-behavior-sanitizer uses), but it can be disabled using -fno-sanitize=function.

For a project such as CPython, which has long relied on function pointers for callbacks, yet seems to have only required that callbacks behave as expected under typical ABI calling conventions, rather than more strictly be declared/defined as a type compatible with the function pointer they will be called as, this leads to numerous errors from UBSan.

Examples when starting Python REPL:
% ./python.exe           
Objects/object.c:2731:5: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:2731:5 in 
Objects/object.c:878:16: runtime error: call to function long_hash through pointer to incorrect function type 'long (*)(struct _object *)'
longobject.c:3295: note: long_hash defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:878:16 in 
Include/internal/pycore_object.h:365:43: runtime error: call to function type_is_gc through pointer to incorrect function type 'int (*)(struct _object *)'
typeobject.c:5347: note: type_is_gc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/internal/pycore_object.h:365:43 in 
Objects/abstract.c:157:26: runtime error: call to function dict_subscript through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
dictobject.c:2511: note: dict_subscript defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:157:26 in 
Objects/abstract.c:2954:14: runtime error: call to function tupleiter_next through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
tupleobject.c:999: note: tupleiter_next defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:2954:14 in 
Objects/abstract.c:236:19: runtime error: call to function dict_ass_sub through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
dictobject.c:2546: note: dict_ass_sub defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:236:19 in 
Objects/call.c:242:18: runtime error: call to function type_call through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
typeobject.c:1647: note: type_call defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/call.c:242:18 in 
Objects/typeobject.c:10309:24: runtime error: call to function classmethod_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:94: note: classmethod_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:10309:24 in 
Modules/gcmodule.c:493:16: runtime error: call to function list_traverse through pointer to incorrect function type 'int (*)(struct _object *, int (*)(struct _object *, void *), void *)'
listobject.c:2704: note: list_traverse defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/gcmodule.c:493:16 in 
Modules/gcmodule.c:605:20: runtime error: call to function list_traverse through pointer to incorrect function type 'int (*)(struct _object *, int (*)(struct _object *, void *), void *)'
listobject.c:2704: note: list_traverse defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/gcmodule.c:605:20 in 
Objects/dictobject.c:3569:17: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (*)(struct _object *, void *)'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:3569:17 in 
Objects/descrobject.c:694:5: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (*)(struct _object *, void *)'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:694:5 in 
[…]
Objects/typeobject.c:4892:19: runtime error: call to function classmethod_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:94: note: classmethod_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:4892:19 in 
Python/generated_cases.c.h:3859:17: runtime error: call to function func_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
funcobject.c:913: note: func_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3859:17 in 
Objects/object.c:1442:19: runtime error: call to function member_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:160: note: member_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1442:19 in 
Objects/object.c:1503:15: runtime error: call to function method_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:136: note: method_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1503:15 in 
Python/generated_cases.c.h:3857:13: runtime error: call to function meth_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
methodobject.c:160: note: meth_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3857:13 in 
Objects/typeobject.c:2212:19: runtime error: call to function method_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:136: note: method_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:2212:19 in 
Objects/descrobject.c:188:16: runtime error: call to function func_get_name through pointer to incorrect function type 'struct _object *(*)(struct _object *, void *)'
funcobject.c:582: note: func_get_name defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:188:16 in 
Python/generated_cases.c.h:3248:20: runtime error: call to function tupleiter_next through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
tupleobject.c:999: note: tupleiter_next defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3248:20 in 
Objects/object.c:1564:19: runtime error: call to function member_set through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:227: note: member_set defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1564:19 in 
Objects/descrobject.c:241:16: runtime error: call to function func_set_name through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, void *)'
funcobject.c:588: note: func_set_name defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:241:16 in 
Python/generated_cases.c.h:3364:21: runtime error: call to function tupleiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:984: note: tupleiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3364:21 in 
Python/ceval.c:591:5: runtime error: call to function func_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
funcobject.c:913: note: func_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:591:5 in 
Objects/object.c:1031:18: runtime error: call to function _Py_module_getattro through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
moduleobject.c:877: note: _Py_module_getattro defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1031:18 in 
Objects/abstract.c:2895:25: runtime error: call to function dictitems_iter through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
dictobject.c:5180: note: dictitems_iter defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:2895:25 in 
Python/generated_cases.c.h:3193:13: runtime error: call to function dictview_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
dictobject.c:4587: note: dictview_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3193:13 in 
Python/ceval.c:1929:13: runtime error: call to function tupleiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:984: note: tupleiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:1929:13 in 
Objects/abstract.c:2334:19: runtime error: call to function tuplecontains through pointer to incorrect function type 'int (*)(struct _object *, struct _object *)'
tupleobject.c:354: note: tuplecontains defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:2334:19 in 
Objects/typeobject.c:1698:19: runtime error: call to function AttributeError_init through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
exceptions.c:2279: note: AttributeError_init defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:1698:19 in 
Python/generated_cases.c.h:1351:13: runtime error: call to function AttributeError_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
exceptions.c:2315: note: AttributeError_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1351:13 in 
Python/generated_cases.c.h:623:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:623:13 in 
Python/generated_cases.c.h:749:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:749:13 in 
Python/generated_cases.c.h:3809:17: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3809:17 in 
Objects/descrobject.c:393:24: runtime error: call to function dict_pop through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *const *, long)'
dictobject.c.h:138: note: dict_pop defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:393:24 in 
Python/generated_cases.c.h:3260:17: runtime error: call to function dictiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
dictobject.c:4047: note: dictiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3260:17 in 
Objects/object.c:1699:15: runtime error: call to function long_bool through pointer to incorrect function type 'int (*)(struct _object *)'
longobject.c:4890: note: long_bool defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1699:15 in 
Python/generated_cases.c.h:4816:13: runtime error: call to function dict_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
dictobject.c:2373: note: dict_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4816:13 in 
Python/generated_cases.c.h:3896:17: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3896:17 in 
Python/generated_cases.c.h:4625:19: runtime error: call to function dict_pop through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *const *, long)'
dictobject.c.h:138: note: dict_pop defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4625:19 in 
Objects/descrobject.c:467:24: runtime error: call to function list_append through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
listobject.c:842: note: list_append defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:467:24 in 
Include/internal/pycore_object.h:142:9: runtime error: call to function PyObject_Free through pointer to incorrect function type 'void (*)(struct _object *)'
obmalloc.c:829: note: PyObject_Free defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/internal/pycore_object.h:142:9 in 
Objects/call.c:361:18: runtime error: call to function type_call through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
typeobject.c:1647: note: type_call defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/call.c:361:18 in 
Python/generated_cases.c.h:2931:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2931:13 in 
Python/generated_cases.c.h:3362:25: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3362:25 in 
Python/generated_cases.c.h:928:13: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:928:13 in 
Python/ceval.c:1604:5: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:1604:5 in 
Python/bltinmodule.c:378:16: runtime error: call to function gen_iternext through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
genobject.c:609: note: gen_iternext defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/bltinmodule.c:378:16 in 
Objects/object.c:1702:15: runtime error: call to function list_length through pointer to incorrect function type 'long (*)(struct _object *)'
listobject.c:438: note: list_length defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1702:15 in 
Python/generated_cases.c.h:3519:13: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3519:13 in 
Objects/methodobject.c:540:18: runtime error: call to function rlock_acquire through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
_threadmodule.c:314: note: rlock_acquire defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:540:18 in 
Python/generated_cases.c.h:2767:13: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2767:13 in 
Objects/methodobject.c:551:18: runtime error: call to function rlock_release through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
_threadmodule.c:364: note: rlock_release defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:551:18 in 
Objects/typeobject.c:2096:5: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:2096:5 in 
Python/generated_cases.c.h:130:13: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:130:13 in 
Objects/object.c:1181:15: runtime error: call to function type_setattro through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
typeobject.c:4938: note: type_setattro defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1181:15 in 
Python/generated_cases.c.h:1510:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1510:13 in 
Python/generated_cases.c.h:164:13: runtime error: call to function listiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:3222: note: listiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:164:13 in 
[…]

Modules/gcmodule.c:1033:24: runtime error: call to function _list_clear through pointer to incorrect function type 'int (*)(struct _object )'
listobject.c:597: note: _list_clear defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/gcmodule.c:1033:24 in
Objects/abstract.c:62:26: runtime error: call to function list_length through pointer to incorrect function type 'long (
)(struct _object )'
listobject.c:438: note: list_length defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:62:26 in
Objects/abstract.c:270:19: runtime error: call to function dict_ass_sub through pointer to incorrect function type 'int (
)(struct _object *, struct _object *, struct _object *)'
dictobject.c:2546: note: dict_ass_sub defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:270:19 in
Python/bltinmodule.c:329:16: runtime error: call to function gen_iternext through pointer to incorrect function type 'struct _object ()(struct _object *)'
genobject.c:609: note: gen_iternext defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/bltinmodule.c:329:16 in
Objects/abstract.c:1141:18: runtime error: call to function tupleconcat through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object )'
tupleobject.c:443: note: tupleconcat defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:1141:18 in
Objects/abstract.c:440:15: runtime error: call to function bytes_buffer_getbuffer through pointer to incorrect function type 'int (
)(struct _object *, Py_buffer *, int)'
bytesobject.c:1663: note: bytes_buffer_getbuffer defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:440:15 in
Objects/methodobject.c:441:24: runtime error: call to function int_from_bytes through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *const *, long, struct _object )'
longobject.c.h:396: note: int_from_bytes defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:441:24 in
[…]
Python/generated_cases.c.h:4814:13: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4814:13 in
Python/generated_cases.c.h:4815:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object )'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4815:13 in
Python/generated_cases.c.h:1529:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object )'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1529:13 in
Python/generated_cases.c.h:648:17: runtime error: call to function slice_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
sliceobject.c:359: note: slice_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:648:17 in
Objects/abstract.c:953:13: runtime error: call to function long_add through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object )'
longobject.c:3473: note: long_add defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:953:13 in
Python/generated_cases.c.h:3322:21: runtime error: call to function listiter_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
listobject.c:3222: note: listiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3322:21 in
Python/generated_cases.c.h:2821:13: runtime error: call to function PyObject_Free through pointer to incorrect function type 'void (
)(struct _object *)'
obmalloc.c:829: note: PyObject_Free defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2821:13 in
Objects/listobject.c:946:26: runtime error: call to function gen_iternext through pointer to incorrect function type 'struct _object ()(struct _object *)'
genobject.c:609: note: gen_iternext defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/listobject.c:946:26 in
Python/generated_cases.c.h:4503:19: runtime error: call to function list_extend through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object )'
listobject.c:861: note: list_extend defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4503:19 in
Python/generated_cases.c.h:4507:13: runtime error: call to function gen_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
genobject.c:128: note: gen_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4507:13 in
Python/generated_cases.c.h:2378:17: runtime error: call to function structseq_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
structseq.c:119: note: structseq_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2378:17 in
Python/ceval.c:1605:5: runtime error: call to function dict_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
dictobject.c:2373: note: dict_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:1605:5 in
Python/generated_cases.c.h:1548:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1548:13 in
Objects/descrobject.c:439:24: runtime error: call to function _io_FileIO_isatty through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *)'
fileio.c.h:532: note: _io_FileIO_isatty defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:439:24 in
Objects/methodobject.c:484:24: runtime error: call to function _io_FileIO_readall through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *)'
fileio.c.h:284: note: _io_FileIO_readall defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:484:24 in
Objects/descrobject.c:372:24: runtime error: call to function _io_FileIO_close through pointer to incorrect function type 'struct _object ()(struct _object *, struct _typeobject *, struct _object *const *, unsigned long, struct _object *)'
fileio.c.h:29: note: _io_FileIO_close defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:372:24 in
Objects/object.c:783:15: runtime error: call to function bytes_richcompare through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object , int)'
bytesobject.c:1524: note: bytes_richcompare defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:783:15 in
Python/generated_cases.c.h:650:13: runtime error: call to function memory_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
memoryobject.c:1141: note: memory_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:650:13 in
Objects/abstract.c:804:9: runtime error: call to function memory_releasebuf through pointer to incorrect function type 'void (
)(struct _object *, Py_buffer *)'
memoryobject.c:1593: note: memory_releasebuf defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:804:9 in
Objects/abstract.c:1950:25: runtime error: call to function list_item through pointer to incorrect function type 'struct _object ()(struct _object , long)'
listobject.c:460: note: list_item defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:1950:25 in
Modules/timemodule.c:2087:5: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (
)(struct _object *, void )'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/timemodule.c:2087:5 in
Python/generated_cases.c.h:2660:21: runtime error: call to function set_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
setobject.c:489: note: set_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2660:21 in
Python/generated_cases.c.h:4440:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object )'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4440:13 in
Python/generated_cases.c.h:2822:13: runtime error: call to function PyObject_Free through pointer to incorrect function type 'void (
)(struct _object )'
obmalloc.c:829: note: PyObject_Free defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2822:13 in
Python/generated_cases.c.h:2157:13: runtime error: call to function dict_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
dictobject.c:2373: note: dict_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2157:13 in
Objects/typeobject.c:4872:19: runtime error: call to function getset_get through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *, struct _object )'
descrobject.c:180: note: getset_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:4872:19 in
Python/generated_cases.c.h:4730:17: runtime error: call to function func_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
funcobject.c:913: note: func_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4730:17 in
Objects/typeobject.c:4905:15: runtime error: call to function method_get through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *, struct _object )'
descrobject.c:136: note: method_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:4905:15 in
Objects/typeobject.c:1863:16: runtime error: call to function tupletraverse through pointer to incorrect function type 'int (
)(struct _object , int ()(struct _object *, void *), void *)'
tupleobject.c:606: note: tupletraverse defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:1863:16 in
[…]
Include/internal/pycore_call.h:187:11: runtime error: call to function range_vectorcall through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *const *, unsigned long, struct _object )'
rangeobject.c:148: note: range_vectorcall defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/internal/pycore_call.h:187:11 in
Python/generated_cases.c.h:2878:13: runtime error: call to function mappingproxy_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
descrobject.c:1160: note: mappingproxy_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2878:13 in
Python/generated_cases.c.h:3320:25: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3320:25 in
Python/generated_cases.c.h:2766:13: runtime error: call to function set_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
setobject.c:489: note: set_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2766:13 in
Modules/_abc.c:52:5: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (
)(struct _object *, void *)'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/_abc.c:52:5 in
Python/bltinmodule.c:1408:25: runtime error: call to function tupleiter_next through pointer to incorrect function type 'struct _object ()(struct _object )'
tupleobject.c:999: note: tupleiter_next defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/bltinmodule.c:1408:25 in
Python/generated_cases.c.h:673:17: runtime error: call to function slice_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
sliceobject.c:359: note: slice_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:673:17 in


(Omitting remaining errors due to GitHub comment length limit.)


Python 3.13.0a1+ (heads/patch-103194-dirty:0a6e69f9a2, Oct 21 2023, 14:45:09) [Clang 17.0.3 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Example workaround for the first error and likely many others, where instead of casting functions to incompatible pointers, the functions use compatible signatures and cast their parameter(s):

--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -343,8 +343,9 @@ PyList_Append(PyObject *op, PyObject *newitem)
 /* Methods */
 
 static void
-list_dealloc(PyListObject *op)
+list_dealloc(PyObject *self)
 {
+    PyListObject *op = (PyListObject *)self;
     Py_ssize_t i;
     PyObject_GC_UnTrack(op);
     Py_TRASHCAN_BEGIN(op, list_dealloc)
@@ -3104,7 +3105,7 @@ PyTypeObject PyList_Type = {
     "list",
     sizeof(PyListObject),
     0,
-    (destructor)list_dealloc,                   /* tp_dealloc */
+    list_dealloc,                               /* tp_dealloc */
     0,                                          /* tp_vectorcall_offset */
     0,                                          /* tp_getattr */
     0,                                          /* tp_setattr */

In other cases, it may be less disruptive to introduce a wrapper function with the correct signature:

@@ -615,6 +617,13 @@ _list_clear(PyListObject *a)
     return 0;
 }
 
+static int
+_list_clear_wrap(PyObject *self)
+{
+    PyListObject *a = (PyListObject *)self;
+    return _list_clear(a);
+}
+
 /* a[ilow:ihigh] = v if v != NULL.
  * del a[ilow:ihigh] if v == NULL.
  *
@@ -3123,7 +3133,7 @@ PyTypeObject PyList_Type = {
         _Py_TPFLAGS_MATCH_SELF | Py_TPFLAGS_SEQUENCE,  /* tp_flags */
     list___init____doc__,                       /* tp_doc */
     (traverseproc)list_traverse,                /* tp_traverse */
-    (inquiry)_list_clear,                       /* tp_clear */
+    _list_clear_wrap,                           /* tp_clear */
     list_richcompare,                           /* tp_richcompare */
     0,                                          /* tp_weaklistoffset */
     list_iter,                                  /* tp_iter */

Likely instances of this can be found at compile time using e.g. -Wcast-function-type (although this emits false positives for when the function pointer is cast back to the correct type before called, and this warning is suppressed by intermediate casts through (void *)):

Objects/listobject.c:3162:5: warning: cast from 'void (*)(PyListObject *)' to 'destructor' (aka 'void (*)(struct _object *)') converts to incompatible function type [-Wcast-function-type-strict]
 3162 |     (destructor)list_dealloc,                   /* tp_dealloc */
      |     ^~~~~~~~~~~~~~~~~~~~~~~~

I would be interested in combing through and replacing similar instances. But I would not be surprised if sooner or later I encounter an instance that is won’t-fix because it involves a stable API, or if I am told that this problem should be ignored because fixing it is too disruptive or requires disproportionate review effort. I am not aware how immediate any danger is from optimizing compilers exploiting this type of undefined behavior.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS


Written by @picnixz:

See #111178 (comment) for the remaining files to fix.

For detecting the UBSan failures, contributors may configure Python with:

./configure                                                 \
    --with-pydebug                                          \
    --prefix="$(pwd)/build"                                 \
    CC=clang LD=clang                                       \
    CFLAGS="-fsanitize=function     -fsanitize-recover"     \
    LDFLAGS="-fsanitize=function    -fsanitize-recover"

The complete list of failures can be retrieved as follows:

PAT='runtime error: call to function (\w+) through pointer' && \
make -j12 2>&1 >/dev/null | \
    grep -E "$PAT" | \
    sed -r "s#^(.+): $PAT.+#\1@\2#g" | \
    sort -k1,2 -t@ -u | \
    awk 'BEGIN {
            PROCINFO["sorted_in"]="@ind_num_asc";
            FS=SUBSEP="@"
        } {
            A[$1][length(A[$1])+1]=$2
        } END {
            for (m in A) {
                for (i in A[m]) {
                    if (i == 1) printf "%s\n", m;
                    print "#", A[m][i]
                }
                print ""
            }
        }' | \
    sed -r "s#"$(pwd)"/?(\./)?(.+)#\2#g"

Note that different builds should be configured in order to hunt all UBSan failures (e.g., --with-trace-refs or --disable-gil to expose conditional compiled code guarded by macros).


Note by @encukou:

We use the macro _Py_NO_SANITIZE_UNDEFINED to disable the UB sanitizer in some hard-to-fix cases, so that we can get a stable, regression-monitoring checker sooner. This issue should only be closed after the macro is removed.


Linked PRs

Activity

  1. added 6 commits that reference this issue on Jan 27, 2025
  2. added 2 commits that reference this issue on Feb 7, 2025
  3. picnixz commented on Feb 7, 2025

    @picnixz
    Member

    So I managed to get roughly 31 files done. Here are the files left where UBSan failures are still there (note that it was a crude grep that I did, we might have more, but once we merge the big ones, it will be easier to check each file...).

    Modules/_asynciomodule.c
    
    Modules/_testbuffer.c
    Modules/_testcapimodule.c (some types, and some redundant casts)
    Modules/_testlimitedcapi.c
    Modules/_testmultiphase.c
    Modules/_testsinglephase.c
    
    Modules/_testcapi/buffer.c
    Modules/_testcapi/heaptype.c
    Modules/_testcapi/monitoring.c
    
    Objects/fileobject.c
    Objects/genericaliasobject.c (gaiterobject)
    Objects/genobject.c
    Objects/iterobject.c
    Objects/longobject.c (long_long_meth)
    Objects/namespaceobject.c
    Objects/odictobject.c
    Objects/picklebufobject.c
    Objects/rangeobject.c
    Objects/sliceobject.c
    Objects/structseq.c
    Objects/unicodeobject.c
    
    Python/instruction_sequence.c
    Python/instrumentation.c
    Python/legacy_tracing.c
    Python/optimizer.c
    Python/symtable.c
    

    There might be more Objects to be fixed as I only searched for files that had an explicit PyCFunction cast, or something with setter/getter casts. I'm actually however worried about something: unless we make some cast, we do have functions that perform casts to (void)(*)(void) and then to PyCFunction (essentially, all methods that have keyword arguments).

    Ideally, we need to come up with a new way to represent them in the future (ideally we should have a union where we would assign the correct member, but that would be a great breaking change which I'm not even sure we want to go down that road. For now, those are only warnings, but if by any chance, they become unsuppressible errors, then we're in for a veeeery bad road).

    At least, once we're done with the explicit casts to PyCFunction and getter/setter that hide the most common issues, we will be able to deal with future issues much faster and much easier (at least, the fixed code won't be a UB).

  4. encukou commented on Feb 7, 2025

    @encukou
    Member

    The types only need to be compatible when the function is called. The cast itself is not undefined behaviour (as long as it's to another function pointer).

  5. picnixz commented on Feb 7, 2025

    @picnixz
    Member

    The cast itself is not undefined behaviour (as long as it's to another function pointer).

    Ok! so nothing to worry about this. At least we won't have raw casts that may sometimes be un-necessary. With a _PyCFunction_CAST usage, at least, it convenes the intent that the cast is necessary (I found lots of useless casts that sometimes lead me to think that there could be a UB but there was actually none)

  6. added 4 commits that reference this issue on Feb 8, 2025
  7. added a commit that references this issue on Feb 10, 2025
  8. added 2 commits that reference this issue on Feb 17, 2025
  9. added a commit that references this issue on Feb 17, 2025
  10. added 12 commits that reference this issue on Feb 17, 2025
  11. added 2 commits that reference this issue on Feb 22, 2025
  12. added 5 commits that reference this issue on Feb 24, 2025
  13. added a commit that references this issue on Feb 25, 2025
  14. added 6 commits that reference this issue on Feb 25, 2025
  15. encukou commented on Feb 26, 2025

    @encukou
    Member

    97 PRs merged. Thank you for the massive amount of work, @chrstphrchvz, @vstinner & @picnixz!

    I'm not sure how much is left, but the buildbot now complains about inst_seq_dealloc in Python/instruction_sequence.c

  16. added 10 commits that reference this issue on Feb 26, 2025
  17. picnixz commented on Feb 28, 2025

    @picnixz
    Member

    There are a bunch of files in Python and Objects I think. For modules only some are left

  18. added 2 commits that reference this issue on Feb 28, 2025
  19. vstinner commented on Feb 28, 2025

    @vstinner
    Member

    Argument Clinic generates the "wrong" signature. Example with list.append(): list_append(PyListObject *self, PyObject *object). I wrote PR gh-130682 to change the signature to: list_append(PyObject *self, PyObject *object). It changes Argument Clinic.

    I also wrote PR gh-130683 (structseq) and PR gh-130684 (unicodeiter) for the next issues.

  20. picnixz commented on Mar 1, 2025

    @picnixz
    Member

    @vstinner Are you planning to fix the remaining files in #111178 (comment)?

  21. vstinner commented on Mar 1, 2025

    @vstinner
    Member

    Yes, it is already implemented. Please test any 3.14 alpha version, your feedback is important for us.

    I build Python with ./configure --prefix '$(PWD)/target' CC=clang LD=clang CFLAGS=-fno-sanitize-recover --with-undefined-behavior-sanitizer && make and then fix the new errors. That's how I pick files/objects.

  22. added 2 commits that reference this issue on Mar 4, 2025
  23. added 7 commits that reference this issue on Mar 11, 2025
  24. added a commit that references this issue on Mar 12, 2025
  25. added 14 commits that reference this issue on Mar 12, 2025
  26. vstinner commented on Mar 13, 2025

    @vstinner
    Member

    It's now possible to build Python with the Undefined Behavior Sanitinizer (UBsan) on my Fedora 41 without errors:

    ./configure CC=clang LD=clang CFLAGS=-fno-sanitize-recover --with-undefined-behavior-sanitizer
    make
    
  27. added 8 commits that reference this issue on Mar 13, 2025
  28. added a commit that references this issue on Mar 14, 2025
  29. added a commit that references this issue on Mar 14, 2025
  30. added a commit that references this issue on Mar 17, 2025
  31. added 13 commits that reference this issue on Mar 17, 2025
  32. added 11 commits that reference this issue on Mar 19, 2025
  33. added a commit that references this issue on Mar 20, 2025
  34. picnixz commented on Mar 22, 2025

    @picnixz
    Member

    I have more local PRs for UBSan that happen at runtime and not at compile time. Some are quite easy to fix (always the same thing we do) but some are harder, and those are especially those related to callbacks in the parser, tokenizer, builder and expat API. I also found a bunch of useless casts that we can remove. I'll try to push something tomorrow.

  35. added a commit that references this issue on Mar 23, 2025
  36. picnixz commented on Mar 23, 2025

    @picnixz
    Member

    So the PRs I have pushed are taking care of the "usual" UBSan, except for #131605 which removes casts and fix a test at the same time. I have one other local branch with the unnecessary casts. However, after that, I need to find a way to fix PyArg_*, pegen, and Expat (namely, the runtime callbacks and those auto-generated).

    To check the runtime UBs, I'm just compiling with the sanitizer and run the test suite.

  37. added 10 commits that reference this issue on Mar 24, 2025
  38. vstinner commented on Mar 24, 2025

    @vstinner
    Member

    Current status with merged changes.

    • Python built on Linux with: ./configure CC=clang LD=clang CFLAGS="-fno-sanitize-recover -O0" --with-undefined-behavior-sanitizer.
    • Test suite run with ./python -m test -j0 -r

    Only 37 tests are failing which is quite good!

    37 tests failed:
        test.test_multiprocessing_fork.test_manager
        test.test_multiprocessing_fork.test_processes
        test.test_multiprocessing_fork.test_threads
        test.test_multiprocessing_forkserver.test_manager
        test.test_multiprocessing_forkserver.test_processes
        test.test_multiprocessing_forkserver.test_threads
        test.test_multiprocessing_spawn.test_manager
        test.test_multiprocessing_spawn.test_processes
        test.test_multiprocessing_spawn.test_threads test_asyncgen
        test_bytes test_capi test_clinic test_code test_coroutines
        test_ctypes test_embed test_frame test_fstring test_functools
        test_minidom test_plistlib test_pulldom test_pyexpat test_regrtest
        test_sax test_socket test_stable_abi_ctypes test_syntax
        test_threading test_userdict test_xml_dom_xmlbuilder
        test_xml_etree test_xml_etree_c test_xmlrpc test_xxtestfuzz
        test_zoneinfo
    
    423 tests OK.
    
  39. picnixz commented on Mar 24, 2025

    @picnixz
    Member

    Btw, I've already fixed "some" of those runtime failures but I didn't push anything yet as I want to present first my analysis. I didn't manage to change the pegen tool so that it doesn't fail miserably and I don't know how to deal with expat yet :/

  40. added 11 commits that reference this issue on Mar 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions