Skip to content

xds: Use real ScheduledExecutorService in ext_proc draining test to avoid FakeClock TSAN race - #13091

Open
kannanjgithub wants to merge 5 commits into
grpc:masterfrom
kannanjgithub:tsan_ext_proc
Open

kannanjgithub wants to merge 5 commits into
grpc:masterfrom
kannanjgithub:tsan_ext_proc

Conversation

@kannanjgithub

@kannanjgithub kannanjgithub commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

In ExternalProcessorClientInterceptorTest.givenDrainingStream_whenExtProcStreamCompletes_thenOnReady(),
the mock ext-proc server spawns a background thread that invokes responseObserver.onNext(),
scheduling completion tasks on scheduler. Concurrently, the test runner thread was
advancing fakeClock.forwardTime(...). Because FakeClock is not thread-safe, this
resulted in a TSAN data race on FakeClock.currentTimeNanos.

This change replaces the FakeClock-backed scheduler with a dedicated single-thread
ScheduledExecutorService (realScheduler), configures the in-process server/channel with
directExecutor(), and relies on explicit latch synchronization instead of FakeClock
ticks, eliminating the data race.

Fixes http://shortn/_IBJaU5B59O.

Only add RawMessageClientInterceptor to the interceptor chain in XdsNameResolver
when either GRPC_EXPERIMENTAL_XDS_EXT_PROC_ON_CLIENT or
GRPC_EXPERIMENTAL_XDS_EXT_PROC_ON_SERVER is true. This prevents
RawMessageClientInterceptor from corrupting method descriptors and causing
empty payloads on retry attempts for regular xDS channels.

Fixes grpc#13010

TAG=agy
CONV=56c87717-243d-4f12-af9e-c532e509892a
…void FakeClock TSAN race

In ExternalProcessorClientInterceptorTest.givenDrainingStream_whenExtProcStreamCompletes_thenOnReady(),
the mock ext-proc server spawns a background thread that invokes responseObserver.onNext(),
scheduling completion tasks on scheduler. Concurrently, the test runner thread was
advancing fakeClock.forwardTime(...). Because FakeClock is not thread-safe, this
resulted in a TSAN data race on FakeClock.currentTimeNanos.

This commit replaces the FakeClock-backed scheduler with a dedicated single-thread
ScheduledExecutorService (realScheduler), configures the in-process server/channel with
directExecutor(), and relies on explicit latch synchronization instead of FakeClock
ticks, eliminating the data race.
@kannanjgithub
kannanjgithub requested a review from sauravzg October 6, 2026 11:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant