Repository navigation
feat: minidump scope - #1340
feat: minidump scope#1340timfish wants to merge 12 commits into
Conversation
…t hub Behind the thread-registry feature. Each thread registers a slot on first use of its hub, SwitchGuard keeps the slot pointing at the hub current on that thread, and the slot is removed when the thread exits. Hub::for_os_thread looks a hub up by the id current_os_thread_id returns, which is the id a crash handler sees for the crashing thread.
Nothing is sent to the crash reporter until the crash. The on_crash hook in minidumper-child names the crashing OS thread; a parked helper thread serializes the scope of the hub current on that thread (inline on macOS, where other threads are suspended) and the handler sends it before the dump request. The reporter merges it with the fatal event. This replaces the manual set_user/set_tag/set_extra/add_breadcrumb mirror, which cost one message per scope write. Depends on the on_crash hook from the feat/on-crash-hook branch of minidumper-child via a crates-io patch until it is released.
|
👋 Thanks for sending this our way! Before a maintainer reviews the code, we ask community contributors to align with us on the approach first — it keeps your time pointed at changes we can land. The easiest way is to open or find a GitHub issue and discuss the approach with a maintainer there, then link that issue from this PR. If the issue is already assigned to someone else, please check in with them (or with us) before continuing — otherwise two people may end up working on the same task. See our contributing guidelines for the full picture. |
This comment was marked as outdated.
This comment was marked as outdated.
0.6 has the on_crash hook, so the git patch is no longer needed.
On macOS the crash handler serialized the scope inline, with no timeout. A crash while the crashing thread held the hub lock, the registry mutex or the allocator lock blocked the handler forever, so no minidump was written. The helper thread now runs on every platform. On macOS the handler resumes it with thread_resume, because crash-handler suspends every other thread before the callback. The handler then waits at most scope_timeout, as on Linux and Windows, and requests the dump without scope if the helper is blocked.
It copies the hub's current scope under the read lock only. configure_scope also writes the scope back under the write lock, so a reader that only needs a copy can block on another thread's read lock.
The helper now copies the scope with Hub::scope_snapshot. With configure_scope it took the hub's write lock, and blocked until scope_timeout if the crashing thread held the read lock. On macOS, start now waits until the helper has stored its Mach port. A crash right after init could otherwise find no port, leave the helper suspended and send the event without scope.
szokeasaurusrex
left a comment
There was a problem hiding this comment.
Thanks for this!
I found a few potential issues that seem concerning, especially the potential deadlock while crashing.
Besides that, since the change is breaking, I updated the PR title to feat!. I would also like to wait to merge this until we plan to make a breaking release (likely within a few months). However, if it would be possible to make this change backwards compatible, and defer the breaking changes to a later release, that would be even better.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1340 +/- ##
==========================================
+ Coverage 73.81% 74.13% +0.32%
==========================================
Files 64 81 +17
Lines 7538 10278 +2740
==========================================
+ Hits 5564 7620 +2056
- Misses 1974 2658 +684 |
current_os_thread_id now returns Option<u64> and exists only on Linux, Android, macOS and Windows. The pthread_self fallback also compiled on non-unix targets, where libc is not a dependency. On Linux a seccomp filter can make gettid return -1 without running it. Cast to u64, every such thread registered under u64::MAX and overwrote the others. A failed gettid now gives None, and a thread without an id is not registered.
|
I'll look into make this non-breaking! |
set_user, set_tag, set_extra and add_breadcrumb on MinidumpIntegration work as in 0.49.3 again, so this change is not breaking. Each call still sends a scope update to the reporter, which applies it to its hub. The crash event already passes through that hub, so these values merge with the crashing thread's scope.
|
This is now non-breaking and the manual scope sync methods are marked as deprecated. |
szokeasaurusrex
left a comment
There was a problem hiding this comment.
I found a few more things.
Most important would be to reduce the chunk size, reduce the additional public api surface (also here), and possibly to benchmark whether tracking the hub adds significant overhead, so we can reduce or document this overhead.
Address review feedback and make the thread registry opt-in, since it adds a few nanoseconds to every hub switch. By default the crash event carries the main hub's scope.
|
I think addressed everything in 45f3390:
The change is still non-breaking, and CI now runs the minidump tests with and without |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 45f3390. Configure here.

The minidump crash event now carries the main hub's scope. With the new
minidump-scopefeature it carries the scope of the thread that crashed instead, including hubs bound withHub::run, so a server with one hub per request reports the request that crashed.Nothing is sent to the crash reporter until the crash. The new
on_crashhook inminidumper-childnames the crashing OS thread, a parked helper thread serializes the scope (the main hub's, or that thread's withminidump-scope) withScope::apply_to_event, and the handler sends it before the dump request. The reporter merges it with the fatal event. This makes the manualset_user/set_tag/set_extra/add_breadcrumbonMinidumpIntegrationunnecessary, so they are deprecated. They still work as before: each call sends a scope update to the reporter, which applies it to its hub, and the crash event merges those values with the scope it carries. This keeps the change non-breaking.Features and cost
sentry-coregainsHub::scope_for_os_thread, which returns the scope of the hub current on a thread, given its OS thread id, under the read lock only. Threads are tracked only with the newthread-registryfeature; without it, the function returns the main hub's scope.thread-registrykeeps a registry of OS thread ids, andSwitchGuardupdates it on every hub switch. Hub switches are frequent:sentry-tracingswitches on every span enter and exit, andbind_hubfutures switch on every poll. The newhub-switchbenchmark putsHub::runat 3.6 ns without the registry and 11.1 ns with it. So the registry is opt-in:minidump-scopeonsentryturns onscopeonsentry-minidump, which turns onthread-registry. With plainminidump, hub switches cost the same as before.Crash handler safety
When the handler runs, the crashed thread may hold the allocator, registry or scope lock. So on every platform the handler allocates nothing and takes no lock that the app can hold: it sets atomics,
unparks the helper (one wake syscall, no allocation), polls an atomicdoneflag with 1 ms sleeps, then reads the result withtry_lock(which cannot block, because the helper unlocks before it setsdone) and writes it to the socket.Everything else runs on the
sentry-minidump-scopehelper thread: registry lookup, scope copy, event processors, JSON. It gets the scope withHub::scope_for_os_thread, notconfigure_scope, becauseconfigure_scopealso writes the scope back under the write lock and would block if the crashed thread held the read lock. It starts at init, because spawning a thread in the handler would allocate. If the crashed thread holds a lock the helper needs, the helper blocks, the handler's wait hitsscope_timeout(default 2s) and the minidump event goes out without scope. The worst case is a missing scope, never a missing minidump.On Linux and Windows the handler runs on the crashing thread (signal handler / exception handler).
macOS: resuming the helper
crash-handleruses a Mach exception port on macOS. The callback runs on a dedicated handler thread, not the crashing thread, andSIGABRTtakes the same path. Before it calls the callback, the handler suspends every other thread, the helper included. So the helper stores its Mach port (pthread_mach_thread_np) at startup,startwaits until it has, andon_crashcallsthread_resumeon it before it wakes it. After that, macOS uses the same wait and timeout as Linux and Windows.These are the crate's only
unsafecalls:thread_resumethroughmach2(already in the lockfile throughcrash-handler) andpthread_mach_thread_npthroughlibc. Whencrash-handlerresumes all threads after the callback, its call on the helper fails harmlessly.Tested on macOS: a crash inside
add_breadcrumb, which holds the hub write lock, hung forever with the earlier inline serialization. Now the minidump arrives afterscope_timeout, without scope. A crash right aftersentry::initgets its minidump in 0.1 to 0.5 s, so the helper is resumed without waiting for the timeout.Details
minidump-scope, thread ids match the handler's:gettid/tidon Linux,GetCurrentThreadId/thread_idon Windows, and the Mach port name (pthread_mach_thread_np/thread) on macOS. Threads that never used Sentry fall back to the main hub. A thread whosegettidfails, for example under seccomp, is not registered.Eventin 4 KiB chunks plus an end message, because the reporter reads one message per call and a local socket on macOS buffers at most 8 KiB. An e2e test sends a 32 KiB scope. Bad or missing JSON gives an empty event. Attachments on the scope are not carried.CI runs the e2e tests on Linux, macOS and Windows, with and without
minidump-scope.