Skip to content

Avoid quadratic copies in curriculum cluster selection - #9046

Open
Hang-get wants to merge 1 commit into
deepspeedai:masterfrom
Hang-get:perf/curriculum-cluster-concatenation
Open

Hang-get wants to merge 1 commit into
deepspeedai:masterfrom
Hang-get:perf/curriculum-cluster-concatenation

Conversation

@Hang-get

Copy link
Copy Markdown

Problem

Schedule-based curriculum learning rebuilds a sample-ID array whenever the accepted difficulty changes. Rank 0 completes this work before a distributed barrier, so every rank waits for cluster construction.

Both the value and percentile selectors currently call np.concatenate after every matching index row. Each call recopies all IDs accumulated so far, making large selections quadratic and retaining an extra full-size temporary array.

Change

  • Collect matching mmap rows and concatenate them once.
  • Preserve the existing order, dtype, None result for no matches, and independent copy for a single matching row.
  • Cover both selectors plus empty and single-row behavior.

This changes no config, checkpoint format, public API, RNG use, or sampling order.

Benchmark

CPU-only benchmark on a two-socket Intel Xeon Platinum 8352Y host (128 logical CPUs), pinned to one logical CPU, with Python 3.11.18, PyTorch 2.15.0+cpu, and NumPy 2.4.6. The input has 2,048 metric rows with 2,048 int64 sample IDs per row (4,194,304 IDs). Each revision received one untimed output-equivalence call followed by five timed calls to the real DeepSpeedDataSampler method in a fresh process.

Revision Median Timed range
master (6f96b4a0) 4.846934 s 4.746916–5.668030 s
master + this patch 0.022617 s 0.021848–0.043923 s

The median speedup is 214.3x, saving 4.824 seconds during this cluster rebuild. Outputs were bitwise equal with np.array_equal.

For 2,048 rows with 4,096 IDs per row (8,388,608 IDs), process peak RSS fell from 443.2 MiB to 379.1 MiB, a 64.1 MiB (14.5%) reduction.

Commands, run once in each checkout:

taskset -c 0 python ai_script/b6_sampler_benchmark.py --rows 2048 --samples-per-row 2048 --repeats 5
taskset -c 0 python ai_script/b6_sampler_benchmark.py --rows 2048 --samples-per-row 4096 --mode memory
Benchmark driver
# SPDX-License-Identifier: Apache-2.0
# DeepSpeed Team

import argparse
import gc
import resource
import statistics
import time

import numpy as np

from deepspeed.runtime.data_pipeline.data_sampling.data_sampler import DeepSpeedDataSampler


def concatenate_once(sampler, metric, value_start, value_end):
    rows = []
    for row in range(len(sampler.curriculum_index_to_sample[metric])):
        value = sampler.curriculum_index_to_metric[metric][row]
        if value <= value_end and value > value_start:
            rows.append(sampler.curriculum_index_to_sample[metric][row])
    if not rows:
        return None
    if len(rows) == 1:
        return np.copy(rows[0])
    return np.concatenate(rows, axis=None)


def measure(fn, repeats):
    times = []
    for _ in range(repeats):
        gc.collect()
        start = time.perf_counter()
        result = fn()
        times.append(time.perf_counter() - start)
        del result
    return times


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--rows", type=int, default=2048)
    parser.add_argument("--samples-per-row", type=int, default=2048)
    parser.add_argument("--repeats", type=int, default=5)
    parser.add_argument("--mode", choices=("timing", "memory"), default="timing")
    args = parser.parse_args()

    sample_rows = [
        np.arange(i * args.samples_per_row, (i + 1) * args.samples_per_row, dtype=np.int64)
        for i in range(args.rows)
    ]
    metric_rows = [np.array([i], dtype=np.int64) for i in range(args.rows)]
    sampler = DeepSpeedDataSampler.__new__(DeepSpeedDataSampler)
    sampler.curriculum_index_to_sample = {"metric": sample_rows}
    sampler.curriculum_index_to_metric = {"metric": metric_rows}

    actual = lambda: sampler.get_sample_based_on_metric_value("metric", -1, args.rows - 1)
    reference = lambda: concatenate_once(sampler, "metric", -1, args.rows - 1)
    if args.mode == "memory":
        result = actual()
        peak_mib = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
        print(f"size={result.size} first={result[0]} last={result[-1]} peak_mib={peak_mib:.1f}")
        return

    actual_output = actual()
    reference_output = reference()
    assert np.array_equal(actual_output, reference_output)
    del actual_output, reference_output
    times = measure(actual, args.repeats)
    print(f"rows={args.rows} samples_per_row={args.samples_per_row} total={args.rows * args.samples_per_row}")
    print("times_s=" + ",".join(f"{value:.6f}" for value in times))
    print(f"median_s={statistics.median(times):.6f}")


if __name__ == "__main__":
    main()

Validation

DS_ACCELERATOR=cpu pytest --forked -q tests/unit/runtime/test_data_efficiency.py
# 16 passed, 2 skipped

pre-commit run --files deepspeed/runtime/data_pipeline/data_sampling/data_sampler.py tests/unit/runtime/test_data_efficiency.py
# all hooks passed

python ci/tests_fetcher.py --base origin/master
# subset mode: 90 tests/unit/v1 targets selected

The selected v1 tests were not run locally. The two skips are the existing accelerator curriculum integration cases, which do not support the CPU accelerator. No GPU training, convergence run, or distributed barrier timing was run. The change only affects CPU integer-index construction; ZeRO stage, offload, precision, and parallelism paths are unchanged.

Duplicate check

I searched open and closed issues and pull requests for the function name and combinations of curriculum, sampler, cluster, concatenate, performance, slow, speed, and memory. No existing work targets these repeated concatenations. #8520 fixes repeated wraparound of small clusters, and #8788 restores sampler RNG state; neither overlaps this change.

AI assistance (Cursor) was used for this change.

Collect matching sample rows and concatenate them once to avoid quadratic copying while preserving selection order and single-row copy semantics.

AI assistance (Cursor) was used for this change.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: YangHaoooooo <233765393+Hang-get@users.noreply.github.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T08:31:47.209980Z e92ec23 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant