Repository navigation
🍒 12898 - Fix trace and StatsD connections over Windows named pipes - #13241
Open
apiarian-datadog wants to merge 1 commit into
Open
apiarian-datadog wants to merge 1 commit into
apiarian-datadog wants to merge 1 commit into
Conversation
Warn when the StatsD client gives up connecting When the StatsD connection ran out of connect retries it logged at debug and stopped for the life of the JVM, so default customer logs showed no sign that tracer metrics were never sent. Warn instead, with the StatsD address, the number of attempts, and the last error. Keep retrying the first StatsD connection When the DogStatsD pipe or port was missing at startup, the StatsD connection retried 20 times, 10 seconds apart, then gave up for the life of the JVM, so a DogStatsD that came up later was never used. In Azure App Service this happens when the tracer starts before DogStatsD listens. - Retry forever, doubling the delay from 10 seconds to a cap of 60 seconds. Drop the jitter on retries so the delay stays within the cap. - Warn on the first failure, and again at most every 5 minutes once the delay reaches the cap, so a lasting failure stays visible without debug logs. Log other attempts at debug, and log once at info when a connection succeeds after failures. Recreate the StatsD client after writes keep failing The StatsD client opens its named pipe once and never reopens it. After the DogStatsD it was writing to went away, every write failed for the life of the JVM, even once a new DogStatsD served the same pipe name. In an Azure App Service overlapping restart, tracer metrics were still missing 50 minutes after the old worker's DogStatsD exited. - Track the client's own send failures, for named pipes only. They use a separate error handler, so errors reported through StatsDClient.error() are still logged and counted by handle() but never trigger a recreate. The UDP and Unix datagram clients send to their address on every write, and the Unix stream client reconnects itself, so recreating them would only drop queued metrics, for example while an overloaded agent fills the socket buffer. - Errors more than 60 seconds apart start a new streak; a streak of at least 3 errors spanning at least 30 seconds triggers a recreate. The 60 second gap is longer than the 30 second tracer health metrics flush, so a dead pipe keeps one streak going, while brief bursts and isolated timeouts under load do not trigger it. - Recreate on the task scheduler, not the client's own thread: close the client and reconnect through the existing connect path, which retries with backoff if the pipe is still missing. - Warn on recreate, rate limited to once every 5 minutes. Open a new named pipe connection for each socket NamedPipeSocketFactory kept one NamedPipeSocket per pipe and returned it to every OkHttp connection (#3493). NamedPipeSocket ignores read timeouts, so when OkHttp checks a pooled connection's health it blocks on an unbounded pipe read, and every other agent request (traces, stats, remote config, telemetry) waits behind it on the shared handle. On Azure Functions with DD_TRACE_PIPE_NAME set, the tracer queued spans but never sent or failed a trace upload until the JVM restarted. Open a new pipe handle for each socket, as the .NET tracer does for each request. #3493 shared the handle because a named pipe was thought to allow only one client at a time; that change was untested on Windows, and the .NET tracer's per-request pipe connections show the agent's pipe server accepts several clients. The agent's pipe servers create their next pipe instance only after a client connects, so a request that opens the pipe in between fails with "All pipe instances are busy". Like .NET's NamedPipeClientStream.Connect, retry opening the pipe for up to 500 ms, the .NET tracer's default DD_TRACE_PIPE_TIMEOUT_MS. Java reports a busy pipe, a missing pipe, and other open failures with the same exception, so all of them are retried. Don't reuse named pipe connections NamedPipeSocket can't time out a read, so when OkHttp checks a pooled connection's health before reusing it, the check can block forever on the pipe. Give named-pipe clients a connection pool that keeps no idle connections, so each request opens and closes its own pipe connection, as the .NET tracer does. For named pipes this replaces the maxRunningRequests connection pool; the dispatcher limits are unchanged. Connect tracer StatsD over a named pipe without a port When a DogStatsD named pipe was set and no port was configured, DDAgentStatsDConnection left the port null, and building the client threw NullPointerException on .port(port). Tracer health metrics and JMXFetch JVM metrics were never sent over a pipe. Use port 0 when a named pipe is set and no port is configured. The client connects to the pipe and ignores the port; port 0 also selects the same packet size and socket settings the tracer uses for Unix domain sockets. Close named pipe connections after each exchange Review follow-ups for named pipes: - An empty connection pool did not stop reuse: OkHttp sends a retry after HTTP 408, or 503 with Retry-After: 0, on the same connection without returning it to the pool, and the health check before that reuse can block forever on the pipe. Send "Connection: close" on every HTTP/1.1 exchange instead, which covers retries too. The named pipe client goes back to the usual connection pool, which never holds these connections. - Resolve the agent host to one loopback address. The socket factory opens the pipe whatever the address, so with both IPv4 and IPv6 addresses OkHttp waited out the pipe connect timeout once per address. - Only an IOException advances the StatsD write failure streak. The client also reports encoding errors, such as a metric that is too long, which a new client would not fix. Test that named pipe retries open a new connection OkHttp sends a retry after HTTP 408, or 503 with Retry-After: 0, on the connection it already holds, and the health check before that reuse can block forever on a named pipe. Check that both attempts send "Connection: close" and each is the first request on its connection. A TCP socket stands in for the pipe, so the tests run on any OS. Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io> (cherry picked from commit 17c6cd1)
apiarian-datadog
requested review from
khanayan123
and removed request for
a team
November 5, 2026 17:33
apiarian-datadog
added
type: bug fix
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
More details
Source inspection found no reportable defects in HTTP named-pipe connection handling or StatsD reconnection. Actual Windows named-pipe behavior remains unverified at runtime.
🤖 Bits Code Review · Commit d11134e
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Contributor
Author
|
Checked with a test app derived from our serverless Azure App Service windows extension test suite. Confrirmed that traces and trace stats now flow correctly. |
Lewis-E
approved these changes
Nov 5, 2026
mhlidd
approved these changes
Nov 6, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport #12898 to release/v1.68.x