Repository navigation
Conversation
Collect matching sample rows and concatenate them once to avoid quadratic copying while preserving selection order and single-row copy semantics. AI assistance (Cursor) was used for this change. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: YangHaoooooo <233765393+Hang-get@users.noreply.github.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Schedule-based curriculum learning rebuilds a sample-ID array whenever the accepted difficulty changes. Rank 0 completes this work before a distributed barrier, so every rank waits for cluster construction.
Both the value and percentile selectors currently call
np.concatenateafter every matching index row. Each call recopies all IDs accumulated so far, making large selections quadratic and retaining an extra full-size temporary array.Change
Noneresult for no matches, and independent copy for a single matching row.This changes no config, checkpoint format, public API, RNG use, or sampling order.
Benchmark
CPU-only benchmark on a two-socket Intel Xeon Platinum 8352Y host (128 logical CPUs), pinned to one logical CPU, with Python 3.11.18, PyTorch 2.15.0+cpu, and NumPy 2.4.6. The input has 2,048 metric rows with 2,048
int64sample IDs per row (4,194,304 IDs). Each revision received one untimed output-equivalence call followed by five timed calls to the realDeepSpeedDataSamplermethod in a fresh process.master(6f96b4a0)master+ this patchThe median speedup is 214.3x, saving 4.824 seconds during this cluster rebuild. Outputs were bitwise equal with
np.array_equal.For 2,048 rows with 4,096 IDs per row (8,388,608 IDs), process peak RSS fell from 443.2 MiB to 379.1 MiB, a 64.1 MiB (14.5%) reduction.
Commands, run once in each checkout:
Benchmark driver
Validation
The selected v1 tests were not run locally. The two skips are the existing accelerator curriculum integration cases, which do not support the CPU accelerator. No GPU training, convergence run, or distributed barrier timing was run. The change only affects CPU integer-index construction; ZeRO stage, offload, precision, and parallelism paths are unchanged.
Duplicate check
I searched open and closed issues and pull requests for the function name and combinations of curriculum, sampler, cluster, concatenate, performance, slow, speed, and memory. No existing work targets these repeated concatenations. #8520 fixes repeated wraparound of small clusters, and #8788 restores sampler RNG state; neither overlaps this change.
AI assistance (Cursor) was used for this change.