[micahtyong]

Summary

Stacked on #146.

A GRPO step with the tinker-cookbook defaults sends 128 samples to the sampler at once. The DeepSeek-V4.1-Flash recipes let 32 requests run at a time, so a step ran in four waves: 32 generate while 96 wait, and once the first wave starts finishing, each freed slot admits one request on its own. Reading that one prompt (about 160 tokens) stalls the other 31 streams, and it happened about a hundred times per step. The GPU had room for more: generating one token for 32 streams took about 65 ms, against 16 ms for one stream, and the KV cache holds 36M tokens.

With 128 running requests the whole step is admitted in four prompt batches of 32 and all 128 requests generate together.

E2E results

Setup: one 8×H200 sampler replica of deepseek_v41_flash_1017_lora_8k, no trainer, the stock tinker SDK sampling client sending 128 requests at once (16 GSM8K prompts × 8 samples, up to 512 tokens each), three rounds; steady state is the mean of rounds 1 and 2. Before / After

Per round of 128 samples	Before	After
Time until all 128 are done	37 s	19.7 s
Mean time per sample	32.8 s	11.7 s
Errors or 503s on /generate	0	0

In the 5-step GRPO run of the whole stack (#146–#149; before / after) the client's sampling time per step went from 42.3 s to 22.9 s, with the KL between sampler and trainer unchanged (0.0032 → 0.0024).

🤖 Generated with Claude Code

[micahtyong]

Hold: on B200 and B300 the sampler crashes while capturing the decode CUDA graph for batch 128 (CUDA error: unspecified launch failure, right after the Triton MoE and LoRA kernels report no tuned config for Blackwell). The earlier B300 smoke test captured up to batch 32 and passed, and H200 is fine at 128. Bisecting on a Blackwell sampler now; this PR should not merge until the recipe runs on its preferred GPUs.

[micahtyong]

Diagnosed. The Blackwell crash is a race in SGLang's Blackwell-only multi-stream decode overlap, not a batch-size or kernel-config problem: the same batch-128 capture passes with CUDA_LAUNCH_BLOCKING=1 on B200 and passes on B300 with SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=0, while H200 never takes that path. This PR now adds that env var to the recipe (commit 3dd5e4e) until the fork carries the missing stream dependency. Decode once it runs: B300 ~5,200 tok/s at batch 128 against H200's ~2,900. A B200 run of the workaround and an H200 no-regression check are in progress; results to follow here.

[micahtyong]

Validation of the overlap-off env: on H200 it changes nothing (7 steady rounds each, A/B on the same node type: 866 vs 881 tok/s, 12.0 vs 11.3 s mean latency, capture 188 vs 183 s, 0 errors either way; runs nq8vwxxw / y3opy1sb vs 2hp2q89i). On Blackwell the evidence is the earlier B300 run with the env (ig61ncu0: batch-128 capture in 313 s, 3 × 128 requests, 0 errors) plus B200 passing the same capture under launch blocking (it4spjvk); a plain B200 rerun waited 1 h 52 min for a node today without getting one and will be repeated when Blackwell capacity returns.