Repository navigation
Trace Lettuce batch writes, stop orphan reactive traces and unfinished spans of done commands - #843
Merged
Merged
Conversation
…d spans of done commands - Match RedisChannelWriter#write(Collection) instead of write(List), so commands sent as a batch get one Lettuce/BATCH_WRITE span, carried by the last command that is not done. - Do not create the Lettuce/Reactive/createMono|createFlux local span when no context is active. It started a trace of its own for every reactive command created in WebFlux; the snapshot in the Reactor Context is still continued. - Do not create a span for a command that is already done when written, as the CommandHandler drops it without another terminal call. Re-check after setting the span, and take the span atomically, so it is finished exactly once.
wankai123
approved these changes
Oct 31, 2026
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix untraced Lettuce batch writes, orphan reactive traces, and unfinished spans of done commands
Follow-up to #841. Fixes three Lettuce plugin problems found while reviewing it. The goal: every Redis
server access gets one exit span. Linking it to a parent stays best effort and depends on upstream plugins.
Bug fixes
1. Batch writes were never traced.
RedisChannelWriterInstrumentationV5/V65matchedwrite(List). Lettuce declareswrite(Collection<? extends RedisCommand>)in everyRedisChannelWriter, so commands sent withStatefulConnection#dispatch(Collection)(also used by Redis command interfaces with@BatchSize) produced no span.write(Collection). The existing interceptor then builds oneLettuce/BATCH_WRITEexit span for thewhole flush.
cache.typeandcache.cmd=BATCH_WRITE. Like the Jedis pipeline span, it has noper-command details.
2. Every reactive command created without an active trace started a trace of its own.
createMono/createFluxinterceptor runs on a thread with no active context, because thewebflux plugin hands its entry span over to the Reactor Context. So
ContextManager.createLocalSpanstarted a newtrace: one orphan
Lettuce/Reactive/createMonoroot segment per command. The snapshot it captured was then ignored,because webflux's snapshot was already in the Reactor Context.
Context is still continued when the command is written. A command with no parent becomes a root trace holding only
its exit span.
3. A command that was already done when written got a span that never finished.
CommandHandlerdrops a done command without another terminal call. A command can be done by then ifit was cancelled or timed out while its write was delayed, for example by
MasterReplicaChannelWriterwaiting for aconnection. A span created for it never finished, so its segment was never reported.
RedisCommandEnhanceInfo#takeSpan), so concurrentcomplete/cancel/re-check finish it exactly once. Before, they could finish it twice.
Known limitations
spans. Closing this needs one span shared by all batch members, which is left for a later change.
Command#completeExceptionallydoes not set the command's status in those versions. A reactivecommand that times out while a master/slave write waits for a connection can therefore still leave a span unfinished.
RedisHandshakewrites HELLO/AUTH/SELECT/CLIENT straight to thechannel) stays untraced. Manual pipelining (
setAutoFlushCommands(false)+flushCommands()) keeps one span percommand.
Tests
lettuce-scenarioandlettuce-6.5.x-scenariodispatch a two-command batch and expect oneLettuce/BATCH_WRITEspan.lettuce-webflux-5x-scenarioandlettuce-webflux-6x-scenarioassert the exact segment count: 5 (health check, entry,three Redis commands). Before this fix there were 8.
RedisChannelWriterInterceptorTestadds unit tests for the cases a scenario cannot reproduce: a done command,cancellation while the span is created, two terminal interceptors racing for one span, and a done last command in a
batch.
Ran locally for every version in the four
support-version.listfiles (16 versions): mock collector, agent,redis:3.2.9-alpine, validator. All pass with this change. All fail withmain's Lettuce plugin jars: noBATCH_WRITEspan, and 8 webflux segments instead of 5. The new unit tests also fail againstmain's code.Add a unit test to verify that the fix works.
Explain briefly why the bug exists and how to fix it.
Update the
CHANGESlog.