Skip to content

gh-149584: _remote_debugging: Read stack chunks directly instead of through the page cache - #160888

Merged
pablogsal merged 2 commits into
python:mainfrom
maurycy:rd-chunk-direct
Nov 18, 2026
Merged

pablogsal merged 2 commits into
python:mainfrom
maurycy:rd-chunk-direct

Conversation

@maurycy

@maurycy maurycy commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

process_single_stack_chunk would read the chunk with _Py_RemoteDebug_ReadRemoteMemory instead of going through the page cache. Reading chunks through the page cache doesn't really do anything:

  • on Linux pages are 4KB, so it goes straight to process_vm_readv
  • on macOS the chunk is exactly one page, the sampler keeps its own cache and nothing else in the sample reads that page, so all the paged read adds is a retry after the first KERN_INVALID_ADDRESS

That retry can only succeed after the target has mapped a new chunk at the same address, so the sampler ends up parsing a chunk that is not the one the thread state pointed at. Invalid stack chunk size 0 is the symptom (a live chunk never has size 0):

maurycy@gimel cpython-rd-base (main 312222f) % sudo dtrace -q -Z -x dynvarsize=64m -n '
  pid$target:_remote_debugging*:copy_stack_chunks:entry { self->in_copy = 1; self->fails = 0; }
  pid$target:_remote_debugging*:copy_stack_chunks:return /self->in_copy/ { @per_copy[self->fails] = count(); self->in_copy = 0; }
  pid$target::mach_vm_read_overwrite:entry /self->in_copy/ { self->addr = arg1; }
  pid$target::mach_vm_read_overwrite:return /self->in_copy && self->addr/ { @kr[arg1] = count(); self->fails += arg1 != 0; self->addr = 0; }
  END { printa("  chunk read kr=%d: %@d\n", @kr); printa("  copies with %d failed reads: %@d\n", @per_copy); }' \
  -c "./python.exe repro_chunk_retry.py 5000"
  chunk read kr=1: 954
  chunk read kr=0: 10607
  copies with 2 failed reads: 315
  copies with 1 failed reads: 324
  copies with 0 failed reads: 4361

324 copies got KERN_INVALID_ADDRESS and then a successful read of the same address a few microseconds later.

deep_recursion oracle target at 100kHz, cache_frames=True, 3s:

error main direct chunk read
chunk read failed (size 16384, kern_return_t 1) 41575 35592
frame read failed (size 88, kern_return_t 1) 24037 37221
Incomplete sample: did not reach base frame 8659 7027
Invalid stack chunk size 0 669 17
total failed samples 74940 (25.0%) 79857 (26.6%)

On Linux perf stat counts the same number of process_vm_readv calls with both builds. The retried reads are reported as the original read failure now, so the error rate goes up on Oracle.

Not newsworthy IMO. I don't know how to create a not racy test here.

Context: I've found this one while reviving maurycy#3 and investigating some Oracle numbers when aliasing.

@maurycy

maurycy commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@pablogsal Thank you. Do we want NEWS here?

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I reviewed and found no problems! Amazing work

@pablogsal
pablogsal disabled auto-merge November 18, 2026 12:20
@pablogsal
pablogsal enabled auto-merge (squash) November 18, 2026 12:20
@pablogsal
pablogsal merged commit 355486e into python:main Nov 18, 2026
153 of 157 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @maurycy for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Nov 18, 2026

Copy link
Copy Markdown

GH-162211 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Nov 18, 2026
@maurycy
maurycy deleted the rd-chunk-direct branch November 18, 2026 15:06
pablogsal pushed a commit that referenced this pull request Nov 18, 2026
…-160888) (GH-162211)

Read stack chunks directly in the sampling profiler. Avoid the page cache retry that can read a replacement chunk after the target process reuses the same address.

(cherry picked from commit 355486e)

Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants