Repository navigation
Description
During GC we can use this struct to buffer in traversed objects if total object count is over 200k.
cpython/Python/gc_free_threading.c
Lines 594 to 601 in f3c11aa
| typedef struct { | |
| unsigned int in; | |
| unsigned int out; | |
| _PyObjectStack stack; | |
| gc_span_stack_t spans; | |
| PyObject *buffer[BUFFER_SIZE]; | |
| bool use_prefetch; | |
| } gc_mark_args_t; |
On M5 and other arm platforms a paired load is emitted due to the location of the in and out integer fields for tracking the buffers positions. If we instead adjust the struct such that it those two fields no longer sit next to each other like below. We instead have a single load as we iterate through the GC heap and store + load in. This seems to benefit from various load predictions vs the previous paired load.
--- a/Python/gc_free_threading.c
+++ b/Python/gc_free_threading.c
@@ -592,11 +592,15 @@ typedef struct {
} gc_span_stack_t;
typedef struct {
+ // Keep 'in' and 'out' apart. If they are adjacent, AArch64 compilers
+ // read both with one paired load (ldp)
unsigned int in;
- unsigned int out;
_PyObjectStack stack;
gc_span_stack_t spans;
PyObject *buffer[BUFFER_SIZE];
+ unsigned int out;
bool use_prefetch;
} gc_mark_args_t;
This can be shown with this micro benchmark timing gc collect.
import gc, statistics, time
gc.disable()
heap = [[i] * 5 for i in range(1_000_000)]
gc.collect() # the first collection doesn't use the prefetch buffer yet
times = []
for _ in range(9):
t0 = time.perf_counter()
gc.collect()
times.append(time.perf_counter() - t0)
print(f"{statistics.median(times) * 1e3:.1f} ms per gc.collect()")
| CPU | Platform | Compiler | main (ms) | fix (ms) | fix / main |
|---|---|---|---|---|---|
| Apple M5 | macOS 27 | Apple clang 21 | 19.2 | 12.5 | 0.65 |
| Apple M5 | macOS 27 | Apple clang 21, PGO+LTO | 18.6 | 13.2 | 0.71 |
| Neoverse N3 (GCP n4a-standard-4) | Ubuntu 24.04 | clang 18.1 | 47.9 | 35.2 | 0.74 |
| Neoverse N3 (GCP n4a-standard-4) | Ubuntu 24.04 | GCC 13.3 | 32.4 | 30.6 | 0.94 |
| Neoverse V2 (GCP c4a-standard-4) | Ubuntu 24.04 | clang 18.1 | 33.8 | 33.3 | 0.98 |
| Neoverse V2 (GCP c4a-standard-4) | Ubuntu 24.04 | GCC 13.3 | 30.7 | 31.5 | 1.03 |
| Neoverse N1 (GCP t2a-standard-8) | Ubuntu 24.04 | clang 18.1 | 49.5 | 48.0 | 0.97 |
| Neoverse N1 (GCP t2a-standard-8) | Ubuntu 24.04 | GCC 13.3 | 43.8 | 43.7 | 1.00 |
For M5 the improvement was clear from the load speculation of the in counters as the tree is traversed as it possible to explicitly disable that optimisation and see any performance gain be removed.
For the other Arm CPUs on the cloud platforms I tested with I was able to see benefit from the same patch but seemed be to different root cause than for M5, I'm unable yet to find a concrete conclusion.
For completeness some other workloads were tested which seemed to optimise better under different scenarios.
| Workload | Apple M5 Apple clang 21 |
Neoverse N3 clang 18 |
Neoverse N3 GCC 13 |
Neoverse V2 clang 18 |
Neoverse V2 GCC 13 |
Neoverse N1 clang 18 |
Neoverse N1 GCC 13 |
|---|---|---|---|---|---|---|---|
inst: linked instances |
0.72 | 0.79 | 0.90 | 0.97 | 1.00 | 1.01 | 1.00 |
dict: small dicts |
0.73 | 0.78 | 0.96 | 0.97 | 0.91 | 1.00 | 0.98 |
list: small lists |
0.68 | 0.74 | 0.94 | 0.99 | 1.03 | 0.97 | 1.00 |
shuffled: dicts in random order |
0.89 | 0.87 | 0.62 | 0.99 | 0.56 | 1.02 | 1.00 |
inst, 100k objects (below the 200k threshold) |
0.97 | 0.99 | 1.00 | 0.99 | 1.00 | 0.98 | 1.00 |
- workload being gc collecting a heap filled with X type of objects.
Wondering if there's any other insight into this and if it's worth this small change. x86 doesn't emit paired in load instructions so is unaffected either way.
Note I did try one real benchmark in building Python's Sphinx docs which did show GC improvements but total time was within noise due to already low amount.
| main | fix | fix / main | |
|---|---|---|---|
| Total GC pause (median) | 3.51 s | 3.29 s | 0.94 |
| Total GC pause (range) | 3.46–3.51 s | 3.28–3.32 s | |
| Total build time (median) | 38.6 s | 38.2 s | 0.99 (within noise) |
| GC share of build time | 9.1% | 8.6% |
Activity