Skip to content

Trace Lettuce batch writes, stop orphan reactive traces and unfinished spans of done commands - #843

Merged
wu-sheng merged 1 commit into
mainfrom
lettuce-batch-and-orphan-spans
Oct 31, 2026
Merged

wu-sheng merged 1 commit into
mainfrom
lettuce-batch-and-orphan-spans

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

Fix untraced Lettuce batch writes, orphan reactive traces, and unfinished spans of done commands

Follow-up to #841. Fixes three Lettuce plugin problems found while reviewing it. The goal: every Redis
server access gets one exit span. Linking it to a parent stays best effort and depends on upstream plugins.

Bug fixes

1. Batch writes were never traced.

  • Cause: RedisChannelWriterInstrumentationV5/V65 matched write(List). Lettuce declares
    write(Collection<? extends RedisCommand>) in every RedisChannelWriter, so commands sent with
    StatefulConnection#dispatch(Collection) (also used by Redis command interfaces with @BatchSize) produced no span.
  • Fix: match write(Collection). The existing interceptor then builds one Lettuce/BATCH_WRITE exit span for the
    whole flush.
    • The span is carried by the last command that is not done, since Lettuce drops done commands but still sends the rest.
    • Nested writers (expiry, listener, master/replica) reuse that span.
    • A cluster batch gets one span per node write, the same way Lettuce splits it.
  • Scope: the span records only cache.type and cache.cmd=BATCH_WRITE. Like the Jedis pipeline span, it has no
    per-command details.

2. Every reactive command created without an active trace started a trace of its own.

  • Cause: in WebFlux, the createMono/createFlux interceptor runs on a thread with no active context, because the
    webflux plugin hands its entry span over to the Reactor Context. So ContextManager.createLocalSpan started a new
    trace: one orphan Lettuce/Reactive/createMono root segment per command. The snapshot it captured was then ignored,
    because webflux's snapshot was already in the Reactor Context.
  • Fix: the interceptor does nothing when no context is active. A snapshot that another plugin put into the Reactor
    Context is still continued when the command is written. A command with no parent becomes a root trace holding only
    its exit span.

3. A command that was already done when written got a span that never finished.

  • Cause: the CommandHandler drops a done command without another terminal call. A command can be done by then if
    it was cancelled or timed out while its write was delayed, for example by MasterReplicaChannelWriter waiting for a
    connection. A span created for it never finished, so its segment was never reported.
  • Fix: the writer interceptor skips a command when either the written command or its span carrier is done.
    • It re-checks after setting the span, for a cancellation that ran while the span was created.
    • Whoever finishes the span now takes it atomically (RedisCommandEnhanceInfo#takeSpan), so concurrent
      complete/cancel/re-check finish it exactly once. Before, they could finish it twice.

Known limitations

  • A batch member cancelled while the batch is being written can leave that batch without a span, or with two finished
    spans. Closing this needs one span shared by all batch members, which is left for a later change.
  • Lettuce 5.0/5.1 only: Command#completeExceptionally does not set the command's status in those versions. A reactive
    command that times out while a master/slave write waits for a connection can therefore still leave a span unfinished.
  • Not changed, by decision: the connection handshake (RedisHandshake writes HELLO/AUTH/SELECT/CLIENT straight to the
    channel) stays untraced. Manual pipelining (setAutoFlushCommands(false) + flushCommands()) keeps one span per
    command.

Tests

  • lettuce-scenario and lettuce-6.5.x-scenario dispatch a two-command batch and expect one Lettuce/BATCH_WRITE span.

  • lettuce-webflux-5x-scenario and lettuce-webflux-6x-scenario assert the exact segment count: 5 (health check, entry,
    three Redis commands). Before this fix there were 8.

  • RedisChannelWriterInterceptorTest adds unit tests for the cases a scenario cannot reproduce: a done command,
    cancellation while the span is created, two terminal interceptors racing for one span, and a done last command in a
    batch.

  • Ran locally for every version in the four support-version.list files (16 versions): mock collector, agent,
    redis:3.2.9-alpine, validator. All pass with this change. All fail with main's Lettuce plugin jars: no
    BATCH_WRITE span, and 8 webflux segments instead of 5. The new unit tests also fail against main's code.

  • Add a unit test to verify that the fix works.

  • Explain briefly why the bug exists and how to fix it.

  • Update the CHANGES log.

…d spans of done commands

- Match RedisChannelWriter#write(Collection) instead of write(List), so commands sent as a batch get one
  Lettuce/BATCH_WRITE span, carried by the last command that is not done.
- Do not create the Lettuce/Reactive/createMono|createFlux local span when no context is active. It started a trace
  of its own for every reactive command created in WebFlux; the snapshot in the Reactor Context is still continued.
- Do not create a span for a command that is already done when written, as the CommandHandler drops it without
  another terminal call. Re-check after setting the span, and take the span atomically, so it is finished exactly once.
@wu-sheng wu-sheng added this to the 9.9.0 milestone Oct 31, 2026
@wu-sheng wu-sheng added plugin bug Something isn't working labels Oct 31, 2026
@wu-sheng
wu-sheng merged commit d78ab34 into main Oct 31, 2026
223 checks passed
@wu-sheng
wu-sheng deleted the lettuce-batch-and-orphan-spans branch October 31, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants