Skip to content

feat: minidump scope - #1340

Open
timfish wants to merge 12 commits into
getsentry:masterfrom
timfish:feat/minidump-scope
Open

timfish wants to merge 12 commits into
getsentry:masterfrom
timfish:feat/minidump-scope

Conversation

@timfish

@timfish timfish commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

The minidump crash event now carries the main hub's scope. With the new minidump-scope feature it carries the scope of the thread that crashed instead, including hubs bound with Hub::run, so a server with one hub per request reports the request that crashed.

Nothing is sent to the crash reporter until the crash. The new on_crash hook in minidumper-child names the crashing OS thread, a parked helper thread serializes the scope (the main hub's, or that thread's with minidump-scope) with Scope::apply_to_event, and the handler sends it before the dump request. The reporter merges it with the fatal event. This makes the manual set_user/set_tag/set_extra/add_breadcrumb on MinidumpIntegration unnecessary, so they are deprecated. They still work as before: each call sends a scope update to the reporter, which applies it to its hub, and the crash event merges those values with the scope it carries. This keeps the change non-breaking.

Features and cost

sentry-core gains Hub::scope_for_os_thread, which returns the scope of the hub current on a thread, given its OS thread id, under the read lock only. Threads are tracked only with the new thread-registry feature; without it, the function returns the main hub's scope.

thread-registry keeps a registry of OS thread ids, and SwitchGuard updates it on every hub switch. Hub switches are frequent: sentry-tracing switches on every span enter and exit, and bind_hub futures switch on every poll. The new hub-switch benchmark puts Hub::run at 3.6 ns without the registry and 11.1 ns with it. So the registry is opt-in: minidump-scope on sentry turns on scope on sentry-minidump, which turns on thread-registry. With plain minidump, hub switches cost the same as before.

Crash handler safety

When the handler runs, the crashed thread may hold the allocator, registry or scope lock. So on every platform the handler allocates nothing and takes no lock that the app can hold: it sets atomics, unparks the helper (one wake syscall, no allocation), polls an atomic done flag with 1 ms sleeps, then reads the result with try_lock (which cannot block, because the helper unlocks before it sets done) and writes it to the socket.

Everything else runs on the sentry-minidump-scope helper thread: registry lookup, scope copy, event processors, JSON. It gets the scope with Hub::scope_for_os_thread, not configure_scope, because configure_scope also writes the scope back under the write lock and would block if the crashed thread held the read lock. It starts at init, because spawning a thread in the handler would allocate. If the crashed thread holds a lock the helper needs, the helper blocks, the handler's wait hits scope_timeout (default 2s) and the minidump event goes out without scope. The worst case is a missing scope, never a missing minidump.

On Linux and Windows the handler runs on the crashing thread (signal handler / exception handler).

macOS: resuming the helper

crash-handler uses a Mach exception port on macOS. The callback runs on a dedicated handler thread, not the crashing thread, and SIGABRT takes the same path. Before it calls the callback, the handler suspends every other thread, the helper included. So the helper stores its Mach port (pthread_mach_thread_np) at startup, start waits until it has, and on_crash calls thread_resume on it before it wakes it. After that, macOS uses the same wait and timeout as Linux and Windows.

These are the crate's only unsafe calls: thread_resume through mach2 (already in the lockfile through crash-handler) and pthread_mach_thread_np through libc. When crash-handler resumes all threads after the callback, its call on the helper fails harmlessly.

Tested on macOS: a crash inside add_breadcrumb, which holds the hub write lock, hung forever with the earlier inline serialization. Now the minidump arrives after scope_timeout, without scope. A crash right after sentry::init gets its minidump in 0.1 to 0.5 s, so the helper is resumed without waiting for the timeout.

Details

  • With minidump-scope, thread ids match the handler's: gettid/tid on Linux, GetCurrentThreadId/thread_id on Windows, and the Mach port name (pthread_mach_thread_np/thread) on macOS. Threads that never used Sentry fall back to the main hub. A thread whose gettid fails, for example under seccomp, is not registered.
  • The scope travels as a serialized Event in 4 KiB chunks plus an end message, because the reporter reads one message per call and a local socket on macOS buffers at most 8 KiB. An e2e test sends a 32 KiB scope. Bad or missing JSON gives an empty event. Attachments on the scope are not carried.

CI runs the e2e tests on Linux, macOS and Windows, with and without minidump-scope.

…t hub

Behind the thread-registry feature. Each thread registers a slot on first use of its hub, SwitchGuard keeps the slot pointing at the hub current on that thread, and the slot is removed when the thread exits. Hub::for_os_thread looks a hub up by the id current_os_thread_id returns, which is the id a crash handler sees for the crashing thread.
Nothing is sent to the crash reporter until the crash. The on_crash hook in minidumper-child names the crashing OS thread; a parked helper thread serializes the scope of the hub current on that thread (inline on macOS, where other threads are suspended) and the handler sends it before the dump request. The reporter merges it with the fatal event. This replaces the manual set_user/set_tag/set_extra/add_breadcrumb mirror, which cost one message per scope write.

Depends on the on_crash hook from the feat/on-crash-hook branch of minidumper-child via a crates-io patch until it is released.
@sdk-maintainer-bot

Copy link
Copy Markdown

👋 Thanks for sending this our way! Before a maintainer reviews the code, we ask community contributors to align with us on the approach first — it keeps your time pointed at changes we can land.

The easiest way is to open or find a GitHub issue and discuss the approach with a maintainer there, then link that issue from this PR. If the issue is already assigned to someone else, please check in with them (or with us) before continuing — otherwise two people may end up working on the same task.

See our contributing guidelines for the full picture.

@timfish
timfish marked this pull request as ready for review October 5, 2026 08:27
@timfish
timfish requested a review from a team as a code owner October 5, 2026 08:27
@timfish

This comment was marked as outdated.

@szokeasaurusrex szokeasaurusrex changed the title feat: minidump scope feat!: minidump scope Oct 8, 2026
0.6 has the on_crash hook, so the git patch is no longer needed.
Comment thread sentry-minidump/src/scope_sync.rs Outdated
On macOS the crash handler serialized the scope inline, with no timeout. A crash while the crashing thread held the hub lock, the registry mutex or the allocator lock blocked the handler forever, so no minidump was written.

The helper thread now runs on every platform. On macOS the handler resumes it with thread_resume, because crash-handler suspends every other thread before the callback. The handler then waits at most scope_timeout, as on Linux and Windows, and requests the dump without scope if the helper is blocked.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread sentry-minidump/src/scope_sync.rs
Comment thread sentry-minidump/src/scope_sync.rs
Comment thread sentry-minidump/src/scope_sync.rs Outdated
It copies the hub's current scope under the read lock only. configure_scope also writes the scope back under the write lock, so a reader that only needs a copy can block on another thread's read lock.
The helper now copies the scope with Hub::scope_snapshot. With configure_scope it took the hub's write lock, and blocked until scope_timeout if the crashing thread held the read lock.

On macOS, start now waits until the helper has stored its Mach port. A crash right after init could otherwise find no port, leave the helper suspended and send the event without scope.

@szokeasaurusrex szokeasaurusrex left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this!

I found a few potential issues that seem concerning, especially the potential deadlock while crashing.

Besides that, since the change is breaking, I updated the PR title to feat!. I would also like to wait to merge this until we plan to make a breaking release (likely within a few months). However, if it would be possible to make this change backwards compatible, and defer the breaking changes to a later release, that would be even better.

Comment thread CHANGELOG.md
Comment thread sentry-core/src/thread_registry.rs Outdated
Comment thread sentry-core/src/thread_registry.rs Outdated
Comment thread sentry-minidump/src/scope_sync.rs Outdated
@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.71004% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.13%. Comparing base (a57b91c) to head (7773d58).
⚠️ Report is 188 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1340      +/-   ##
==========================================
+ Coverage   73.81%   74.13%   +0.32%     
==========================================
  Files          64       81      +17     
  Lines        7538    10278    +2740     
==========================================
+ Hits         5564     7620    +2056     
- Misses       1974     2658     +684     

current_os_thread_id now returns Option<u64> and exists only on Linux, Android, macOS and Windows. The pthread_self fallback also compiled on non-unix targets, where libc is not a dependency.

On Linux a seccomp filter can make gettid return -1 without running it. Cast to u64, every such thread registered under u64::MAX and overwrote the others. A failed gettid now gives None, and a thread without an id is not registered.
@timfish

timfish commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

I'll look into make this non-breaking!

set_user, set_tag, set_extra and add_breadcrumb on MinidumpIntegration work as in 0.49.3 again, so this change is not breaking. Each call still sends a scope update to the reporter, which applies it to its hub. The crash event already passes through that hub, so these values merge with the crashing thread's scope.
@timfish timfish changed the title feat!: minidump scope feat: minidump scope Oct 8, 2026
@timfish

timfish commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

This is now non-breaking and the manual scope sync methods are marked as deprecated.

@szokeasaurusrex szokeasaurusrex left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a few more things.

Most important would be to reduce the chunk size, reduce the additional public api surface (also here), and possibly to benchmark whether tracking the hub adds significant overhead, so we can reduce or document this overhead.

Comment thread sentry-core/src/thread_registry.rs Outdated
Comment thread sentry-core/src/thread_registry.rs Outdated
Comment thread sentry-core/src/thread_registry.rs Outdated
Comment thread sentry-core/src/hub_impl.rs Outdated
Comment thread sentry-minidump/src/scope_sync.rs
Comment thread sentry-minidump/src/scope_sync.rs Outdated
Comment thread sentry-minidump/src/lib.rs Outdated
Comment thread sentry-core/src/hub_impl.rs
Address review feedback and make the thread registry opt-in, since it adds a few nanoseconds to every hub switch. By default the crash event carries the main hub's scope.
@timfish

timfish commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

I think addressed everything in 45f3390:

  • Public API: current_os_thread_id is now private. Hub::for_os_thread and Hub::scope_snapshot are replaced by a single Hub::scope_for_os_thread, as you suggested, and threads without an OS thread id no longer get a slot.
  • Chunk size: reduced from 16 KiB to 4 KiB, since macOS buffers at most 8 KiB on a local socket. A new e2e test sends a 32 KiB scope, so CI checks this on all three platforms.
  • MSG_SCOPE_UPDATE now lives in scope_sync.rs with the other message kinds, and the docs no longer list the scope's level.
  • Overhead: I added a hub-switch benchmark. Hub::run goes from 3.6 ns to 11.1 ns with the registry, and sentry-tracing and bind_hub futures switch hubs often, so the registry is now opt-in. By default the crash event carries the main hub's scope, with no extra cost per hub switch. The new minidump-scope feature turns on the registry and sends the crashing thread's scope instead. Because we always include main hub scope, should this opt-in feature be called minidump-thread-scope?

The change is still non-breaking, and CI now runs the minidump tests with and without minidump-scope.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 45f3390. Configure here.

Comment thread sentry-core/src/thread_registry.rs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minidump scope sync

2 participants