[timtofan]

What issue is this change attempting to solve?

Why

HTTP render coalescing reduces frontend wire overhead but can regress ordinary read latency by defeating go-carbon's response and expanded-glob caches. Enabling it by default in #567 exposed that interaction: the caches identify the complete request, rather than each logical metric. A changing batch creates unrelated cache entries even when its metrics were already read individually.

How does this change solve the problem? Why is this the best approach?

Architecture

Expanded metrics are still routed separately to each backend and retain their existing per-metric replica-majority handling. This change sets renderBatchSize to 1 in the default configuration and example, so the backend queue sends individual requests unless an operator explicitly opts in. It leaves the coalescer implementation, routing, concurrency accounting and the allocation/compression improvements from #566 intact. There is no new transport or compatibility path.

The backend cache boundary matters here. go-carbon's render-cache key contains the entire target list in request order (getRenderCacheKeyAndSize). Its expanded-glob key joins the metric-name list (getExpandedGlobsWithCache); getUniqueMetricNames supplies that list through unsorted map iteration. Queue-dependent batch membership/order therefore loses response-cache reuse, and map iteration further fragments glob-cache reuse. Sorting alone would not fix changing batch membership.

Before / after

Before	After
Omitted renderBatchSize enables batches up to 32 metrics.	Omitted renderBatchSize sends individual metric requests.
Expanded reads can lose backend cache reuse without an explicit rollout decision.	Existing per-metric cache keys remain reusable by default.
Explicit values above 1 enable coalescing.	Explicit values above 1 continue to enable coalescing.
Rolling back the release also removes unrelated allocation improvements.	The default mitigates cache churn while retaining those improvements.

Usage and rollout

For an existing binary containing #567, set renderBatchSize: 1 explicitly to disable coalescing. Updating to this change makes that the default when the field is omitted; an existing explicit value above 1 must still be changed. The example configuration uses 1 too.

Keep coalescing opt-in until backend cache reuse is independent of dynamic wire batches, and a representative production comparison passes backend CPU/allocation, cache-miss and simple-request tail-latency checks. This is a mitigation, not a cache-granularity fix; no post-mitigation production improvement is claimed here.

How can we be sure this works as expected?

• go test ./pkg/cfg ./pkg/backend ./pkg/app/carbonapi ./pkg/util passed.
• go test -race ./pkg/backend ./pkg/app/carbonapi ./pkg/util passed, including existing coalescing output and queue-liveness tests.
• A bounded diagnostic replay of four captured queries produced identical decoded responses for all 72 requests across the previous release, the current release, and the current release with batching disabled. These queries establish the unbatched-path control; they do not measure the mitigation's production impact.
• Production store logs show the newly coalesced requests losing cache reuse, while profiles attribute substantial allocation/CPU to glob expansion and GC. Validation of the rollout must include those shared backend costs, not only frontend CPU.