perf: coalesce expanded reads in HTTP backend queues

[timtofan]

Why

Large resolved globs currently create one HTTP render request per metric and replica. This repeats HTTP, compression and protobuf framing work thousands of times while fetching the same underlying series. Reduce that fixed request overhead without changing metric routing, replica-majority selection or the number of metrics admitted concurrently.

Architecture

Coalescing happens inside each backend's existing render queue, after TLD/path-cache routing has independently selected the backend for each metric:

flowchart LR
    Expansion[Glob expansion] --> Routing[Existing per-target routing]
    Routing --> Queue[Per-backend render queue]
    Queue --> Wire[Bounded multi-target HTTP request]
    Wire --> Replies[Separate replies for original targets]
    Replies --> Merge[Existing per-metric replica merge]

Loading

One glob expansion carries a unique context marker. A backend coalesces only immediately queued, unique literal targets carrying that exact context and time window. Batches default to at most 32 targets and stay below an 8KiB encoded URL budget. There is no batching timer or sharing across independent client requests.

The backend's shared semaphore reserves one slot per original metric in a batch. Find/info requests still share the same capacity. Admission honours cancellation, and a carried slow request cannot starve the fast queue.

Replies are delivered to the original per-target channels. Missing names become the HTTP client's existing not-found error; a failed or slow replica reports its error once to each original replica fan-in. Unexpected renamed/duplicate replies use concurrent individual reads within the already reserved slots. Routing-cache miss fallback and the existing majority merger remain unchanged.

HTTP backends explicitly declare the required named-response/missing-series contract. gRPC retains individual calls because empty stream EOF and per-request NotFound have different semantics. Supporting gRPC coalescing requires defining and testing its per-target error contract first. This is a transport capability boundary, with no new fork or wire format.

Before / after

Before	After
Every expanded metric creates a separate HTTP request to each selected backend.	Compatible queued metrics share a bounded request to the backend already selected for each metric.
Backend capacity counts individual render requests.	A batch of k original metric requests reserves k slots; capacity does not multiply by batch size.
Per-target result/error channels feed replica selection.	The same channels and per-target replica merger remain in use.
Each HTTP render is recorded by the backend duration histogram.	Each physical batch or compatibility retry is recorded; logical enqueue/subrequest counts retain their original meaning.
All implementations use individual calls.	HTTP opts in; gRPC, unmarked contexts, globs and single-target work retain individual calls.

Usage

The existing glob-expansion call site marks eligible requests, and both the default API configuration and example enable:

renderBatchSize: 32

Set renderBatchSize: 1 and restart carbonapi to restore individual HTTP requests. Existing resolveGlobs, queue classification and routing configuration remain in force.

Validation

• The real backend queue tests cover context/time/URL/count boundaries, selected backend sets, missing series, a slow failed replica, majority repair, cancellation under small capacity, value ownership, renamed replies, fast-queue progress and an actual HTTP/protobuf round trip with escaped names and floating-point bit checks.
• The full race suite with Cairo, targeted transport/queue tests, changed-code lint and govulncheck pass on the rebased branch (Go 1.27.2; golangci-lint 2.14.0).
• An offline oracle uses two frozen production snapshots across three HTTP replica fixtures: 14,133 series and 5,087,880 returned points. The reference is freshly fetched master at d651603c, including perf: reduce carbonapi CPU, allocations and retained cache memory #566 and the current dependency/toolchain updates. Binary values and JSON match exactly. In one replay, physical backend requests fell from 42,399 to 13,370. Coalescing is opportunistic, so the request count varies with scheduling; this is not a live production CPU result.
• Module/vendor metadata is regenerated with golang.org/x/sync v0.23.0, matching the current base branch's Go 1.26 minimum. No additional Go support change is introduced by this PR.

Rollout and rollback

A live canary is still required before rollout: compare CPU per returned point, p99/max read latency, peak RSS, allocations and backend errors under matched production traffic. Observe physical HTTP counts separately from logical metric subrequests. Restore renderBatchSize: 1 and restart, or revert the release, if the canary does not improve the production result.