[micahtyong]

Summary

Samplers now keep their compiled kernels on the shared spindle-kernel-cache volume, the way trainers already do (#82). Before, every sampler start compiled its kernels again on the container's local disk. For DeepSeek-V4.1-Flash that took about five minutes of a sixteen-minute start.

SGLang keeps all of its compile caches under one folder, set by SGLANG_CACHE_DIR. Spindle now points that folder at the volume for samplers. A recipe can still override it in inference_env. Trainers don't get it: the trainer imports SGLang too, and the setting would move its CUDA cache onto the network volume.

SGLang also looks in that folder for a small Rust extension it uses to index cached prompts. When it doesn't find it there, it compiles it with cargo at startup. The sampler image now ships the extension prebuilt and tells SGLang never to build it.

The compiled kernels in the cache are keyed by library version and GPU architecture, so H200 and Blackwell samplers can share the volume.

This branch also carries #147, which was merged into it. DeepSeek-V4.1-Flash samplers now run up to 128 requests at once instead of 32, so the 128 samples of a GRPO step generate together instead of in four waves. At 128 requests, an SGLang optimization that splits decode work across several CUDA streams crashed on Blackwell while the sampler recorded its decode graphs. The recipe turns that optimization off. H200 uses less of it and never crashed.

E2E results

Setup: stock tinker-cookbook math_rl (GSM8K) on deepseek_v41_flash_1017_lora_8k with one 8×H200 sampler replica. Before is main. After is the whole stack (#146, #148, #149) with this model's kernels already on the volume. Before / After

	Before	After
Kernel compile at sampler start	304 s	13 s
Sampler start to ready	985 s	438 s
Sampling time per GRPO step	42.3 s	22.9 s

KL between sampler and trainer stayed flat (0.0032 before, 0.0024 after).

The 128-request change on its own, on one sampler with 128 requests sent at once (16 GSM8K prompts × 8 samples, up to 512 tokens). Before / After

Per round of 128 samples	Before	After
Time until all 128 finish	37 s	19.7 s
Mean time per sample	32.8 s	11.7 s

With the multi-stream optimization off, an 8×B300 sampler recorded its decode graphs up to batch 128 and served three rounds of 128 requests with no errors. With it on, B200 and B300 samplers crashed at that step. On H200 the setting makes no measurable difference: 866 tok/s with it off and 881 tok/s with it on, within run-to-run noise.

🤖 Generated with Claude Code