JIT (opcache.jit=1235): hang in ir_schedule_topsort() and segfaults in zend_build_call_map() / ir_fix_bb_order()

[camweb21]

Description

With opcache.jit=1235 and opcache.jit_buffer_size=256M, workers of the built-in server intermittently hang or segfault while JIT-compiling a hot function (zend_jit_hot_func, trigger 3) under concurrent load.

There is no minimal reproducer. The setup is WordPress 7.1.2 and WooCommerce 11.1.2 (plus a private plugin), served by php -S with PHP_CLI_SERVER_WORKERS=32 against MySQL 9.7.2, taking 200 concurrent Store API requests (add-to-cart, then checkout). Over 10 runs with opcache.jit=1235 (two batches of five, the second with debug symbols), 6 had a hang and 2 of those also had segfaults. Over 6 runs with opcache.jit=disable, on the same commit, none did.

Hang. In each of the 6, one worker ran at 100% CPU while the other workers, apart from any that were segfaulting, waited in fcntl (kernel wchan fcntl_setlk). After max_execution_time the busy worker exited with code 124, printing Maximum execution time of 30+2 seconds exceeded (terminated) in .../wp-includes/rest-api.php on line 2820 (the first line of WordPress's rest_sanitize_value_from_schema()), and the others carried on. 7 workers were killed that way in all. Two of the three symbolised hangs look like this, and the three hangs from the first batch have the same return addresses from zend_jit_ir_compile up:

#0  ir_schedule_topsort (...) at ext/opcache/jit/ir/ir_gcm.c
#1  ir_schedule (ctx=...) at ext/opcache/jit/ir/ir_gcm.c:1255
#2  zend_jit_ir_compile (...) at ext/opcache/jit/zend_jit_ir.c:2908
#3  zend_jit_finish (...) at ext/opcache/jit/zend_jit_ir.c:16802
#4  zend_jit (op_array=..., ssa=..., rt_opline=...) at ext/opcache/jit/zend_jit.c:2964
#5  zend_real_jit_func (..., trigger=3) at ext/opcache/jit/zend_jit.c:3086
#6  zend_jit_hot_func (...) at ext/opcache/jit/zend_jit.c:3221
#7  ?? ()
...
#14 zend_call_function (...) at Zend/zend_execute_API.c:1010
#15 zif_array_reduce (...) at ext/standard/array.c:6601

In the third symbolised hang, the busy worker was in zend_build_call_map(), called from zend_real_jit_func() at zend_jit.c:3072.

Segfaults. 9 workers segfaulted, in 2 of the 6 runs (8 in one, 1 in the other). By faulting address, 8 were in zend_build_call_map() and 1 in ir_fix_bb_order(). Backtraces from those two cores, with debug symbols:

#0  zend_build_call_map (arena=<compiler_globals+360>, info=..., op_array=...) at Zend/Optimizer/zend_call_graph.c:276
#1  zend_real_jit_func (..., trigger=3) at ext/opcache/jit/zend_jit.c:3072
#2  zend_jit_hot_func (...) at ext/opcache/jit/zend_jit.c:3221

#0  ir_fix_bb_order (...) at ext/opcache/jit/ir/ir_gcm.c:955
#1  ir_schedule (...) at ext/opcache/jit/ir/ir_gcm.c:1133
#2  zend_jit_ir_compile (...) at ext/opcache/jit/zend_jit_ir.c:2908
#3  zend_jit_finish (...) at ext/opcache/jit/zend_jit_ir.c:16802
#4  zend_jit (...) at ext/opcache/jit/zend_jit.c:2964
#5  zend_real_jit_func (..., trigger=3) at ext/opcache/jit/zend_jit.c:3086
#6  zend_jit_hot_func (...) at ext/opcache/jit/zend_jit.c:3221

We can re-run the same CI setup with other opcache.jit modes, worker counts or buffer sizes, and share the full logs, or core dumps from a re-run.

PHP Version

PHP 8.5.11 (NTS), installed by shivammathur/setup-php

Operating System

Ubuntu 24.04, x86-64 (GitHub Actions runner)

[camweb21]

Follow-up: the same happens on PHP 8.4.26 (NTS, shivammathur/setup-php, debug symbols), so it doesn't look like an 8.5 regression. Same setup and settings (opcache.jit=1235, opcache.jit_buffer_size=256M), 5 runs: 2 had it, 3 were clean.

• One run hung. The busy worker was in ir_schedule_topsort(), from ir_schedule() at ir_gcm.c:1255, and was killed by the hard timeout with the same message (rest-api.php line 2820).
• One run had 5 workers segfault and no hang. By faulting address, 4 were in zend_build_call_map() and 1 in ir_fix_bb_order().

The two cores from that run:

#0  ir_fix_bb_order (...) at ext/opcache/jit/ir/ir_gcm.c:955
#1  ir_schedule (...) at ext/opcache/jit/ir/ir_gcm.c:1133
#2  zend_jit_ir_compile (...) at ext/opcache/jit/zend_jit_ir.c:2865
#3  zend_jit_finish (...) at ext/opcache/jit/zend_jit_ir.c:16557
#4  zend_jit (...) at ext/opcache/jit/zend_jit.c:2821
#5  zend_real_jit_func (..., trigger=3) at ext/opcache/jit/zend_jit.c:2943
#6  zend_jit_hot_func (...) at ext/opcache/jit/zend_jit.c:3064

#0  zend_build_call_map (arena=<compiler_globals+360>, info=..., op_array=...) at Zend/Optimizer/zend_call_graph.c:276
#1  zend_real_jit_func (..., trigger=3) at ext/opcache/jit/zend_jit.c:2929
#2  zend_jit_hot_func (...) at ext/opcache/jit/zend_jit.c:3064

[ndossche]

Thanks for the information. I was able to reproduce this after a while and come up with a PR that fixes this: #24001
I believe this suffices. If possible, can you please check whether this patch resolves your issue?