Skip to content

perf: avoid copying aligned aggregation and arithmetic inputs - #571

Closed
timtofan wants to merge 2 commits into
masterfrom
atimofieiev/carbonapi-reduce-profile-allocs
Closed

timtofan wants to merge 2 commits into
masterfrom
atimofieiev/carbonapi-reduce-profile-allocs

Conversation

@timtofan

@timtofan timtofan commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A live frontend profile shows MetricData.Consolidate allocating point arrays for aggregation inputs that already have the common resolution. Aggregation and arithmetic combination only read those arrays, so copying them increases allocation and GC cost without changing the result.

Architecture

The change is local to expression evaluation. AggregateSeries and CombineSeries normalize their inputs through one internal read-only consolidation helper and write a separate output. Inputs that need resampling still use Consolidate; inputs already at the common step borrow their point slices while retaining separate metadata. The exported Normalize behavior stays unchanged for callers that may modify its returned data.

Backend routing, replica-majority merging, caching, response serialization, and admission limits stay at their existing boundaries.

Before / after

Before After
Aggregation copies every multi-series input during normalization and both arithmetic operands, including series at the common step. Aggregation and arithmetic borrow unchanged point slices and allocate only the metadata and any resampled points.
Other normalization consumers receive the existing copy behavior. Their behavior is unchanged.
Aggregation writes a detached result. The result remains detached, including when an aggregate function reorders its scratch values.

Validation

  • The final patch passed the complete local race suite and the complete native production-host race suite with Go 1.27 and Cairo.
  • Tests compare borrowed normalization against the existing copying path for equal/mixed steps, missing values, start offsets, and existing consolidation metadata. They verify that aggregate callbacks and result mutation cannot alter shared inputs.
  • The final evaluator and serializer matched the reference in all 24 comparisons over two frozen production inputs (14,133 series and 5,087,880 points each), four expressions (including asPercent/divideSeries), and three input permutations; natural response byte order also matched. The initial aggregation-only patch also passed 40 comparisons across ten permutations.
  • A separate replay of those captured inputs reduced median evaluator allocation from 110.9 MB to 62.1 MB for grouping, 210.6 MB to 60.5 MB for asPercent, and 211.5 MB to 61.3 MB for division. These are captured-query results, not fleet CPU or latency claims.
  • The exact final head ffd97cb5 completed a bounded production A/B/A test on one frontend with three unchanged controls. Candidate and restored-stock measurements each cover ten minutes after five minutes of warm-up. No profiling or build work ran inside either measured window. Both phases recorded zero HTTP 5xx and no unexpected restarts; stock executable/configuration checksums, health and traffic registration were verified afterward. Temporary binaries, overrides, timers and build/profile files have been removed.

Production result and remaining gate

This PR remains draft: allocation efficiency improved, but this run does not establish the required absence of a CPU/RSS peak or latency regression. These are operational measurements under changing real query traffic, not an isolated causal estimate.

Production measurement Stock reference Final candidate
Maximum one-minute process CPU, cores 2.02 2.49
CPU per merged upstream point, p99 of ten one-minute samples, µs 1.122 1.168
Maximum allocation rate, GB/s 0.489 0.458
Allocated bytes per merged upstream point, p99 of ten one-minute samples 160.3 164.0
Maximum sampled RSS, GiB 4.20 4.63
Maximum sampled live heap, GiB 3.05 1.66
Render-request p99 over the full window, ms 397 472
Maximum one-minute render-request p99 estimate, ms 720 902
Render HTTP 5xx 0 0
Pre-existing partial-response fraction 19.36% 19.95%

Candidate: 17 October 2026, 15:02:46–15:12:46 CEST. Restored stock: 15:18:19–15:28:19 CEST. Request counts are 24,591 and 24,592 respectively. The candidate received 28.7% more merged upstream points in total and its peak point rate was 11.70M/s versus 4.00M/s. Partial responses already occur on all compared frontends; this test does not establish an improvement there.

Over each complete window, CPU per merged point is 0.606 → 0.427 µs and allocation is 120.7 → 80.3 bytes/point (stock → candidate). Adjusting for the three controls gives candidate/stock ratios of 0.652 for CPU/point, 0.639 for bytes/point and 0.943 for request p99. These comparisons support a possible efficiency improvement, but do not remove the query-mix confounder or clear the peak/latency gate. A better matched production comparison is required before making this ready or rolling it out.

A separate warm-up-only profile of the final candidate contained no allocation samples in MetricData.Consolidate. Remaining allocation is dominated by protobuf decoding, expression result arrays and response-body reads; percentages vary substantially with query mix.

@timtofan timtofan changed the title perf: avoid copying aligned series for aggregation perf: avoid copying aligned aggregation and arithmetic inputs Oct 17, 2026
@timtofan timtofan closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant