Skip to content

NOTES.md

Latest commit

 

History

History
238 lines (221 loc) · 14.2 KB

NOTES.md

File metadata and controls

238 lines (221 loc) · 14.2 KB
Raw

Data caveats

What someone reusing results/ or archive/results/ must know: which data are invalid or superseded, the limits of each benchmark mode, and host effects. Platform setup traps are in PLATFORMS.md. Files over 64 KB in archive/results/ are gzipped.

Invalid, superseded or partial data

mini-gda

  • The first-pass mini-gda data is void (a ring-index bug let warm-up CQEs satisfy the timed loop). Only archive/results/minigda/run3 is kept; its 2.3 µs "issue" in axis1b_uar.csv is 1 / (batch-1 rate), measured issue is 0.42–1.63 µs.
  • Builds before 2026-09-11 16:00 UTC also stored the CQ consumer index, which shifts issue cost and one-thread ladder rows by 6–18 %. Use results/sec62/20260911T160124Z (A/B in results/sec62/cq_ab_*). The paper floor is o5p2d (make mgda_bench_o5p2d).
  • Ordering and tax tables from before §4.1's rework are superseded by results/sec62 and sec62b (and the archived sec62c): the fence rows of archive/results/minigda/run3, archive/results/ordering/20260831T173131Z{,b}, archive/results/session20260831/tab_fence.tex (mixed pairs and builds), and the three-stack tax / "GDAKI at or below floor" in archive/results/lattax/lat1 and archive/results/a1/a1pass1 (system-scope mini-gda).
  • In the 2026-09-02 sec62 pass, the ordering knob does not move only one stage (at system scope the wait grows too), and its "warpcheck +1 %" is corrected in sec62b. Its p_lat mismatched=8 is a validator bug, and its cross-switch wire brackets read mlx5_0; both are fixed in sec62b/sec62c.
  • mini-gda DC mode is LID-only. DC stamps before results/dc/20260831T192501Z have an SRQ bug (srqn=0); that stamp's "destination switching is free" rests on one 0.38 M ops/s single-thread point.

GDAKI / GIN

  • gin_bench once reset only context 0, so every wait on another context timed issue only. Invalid: all _other rows in archive/results/proxies_pib/20260831T185424Z (the "1.9 µs RTT"), latctx1 in archive/results/e3e9/20260901T100519Z, the GDAKI/GIN Proxy rows in archive/results/e3e9/20260831T180323Z. Valid: archive/results/gdaki_audit/20260901T141058Z (RTT 12.2–12.3 µs on any context).
  • The earlier "same-context starvation" timeouts were a misconfiguration (thread-exclusive sharing declared on a shared context). With --probe-scope gpu the probe gives ~350 µs at 64 CTAs (results/sec43/20260910T153220Z).
  • The load sweep in archive/results/proxies_pib/20260831T185424Z set NUM_RC_PER_PE = bg_ctas; its load rows are historical only.

Ladder and §4.2 (sec43)

  • results/ladder/20260910T201423Z and 20260910T214725Z are superseded. One-thread-per-QP rows come from 20260911T185327Z; warp, NVSHMEM and GDAKI rows from 20260910T225741Z. packed_q64_c4_b16 and c8_b16 are bimodal (~125 vs ~225 M/s); report a range (six runs with archive/results/ladder_placement/20260916T113400Z).
  • results/sec43/20260911T100041Z (pair 59→58) is valid for idle, rate, payload and 0/1/4-CTA load. Loaded curves at 16/64 CTAs are valid only from 20260910T153220Z (41↔59).
  • mini-proxy cells with --bg-pace-ns 4000000 are invalid (__nanosleep saturates).
  • Tuned mini-proxy RTT p99 before 2026-09-11 (21–23 µs) came from one GDRCopy publish per slot; after the fix it is 9.8 µs.

fabric-lib

  • Loaded latency is valid only with fabriclib_native_bin2 (md5 a6a78942): in results/sec43/20260911T100041Z only makeup cells c1218–c1262; the r2 fabric-lib latload cells in results/bigrerun/sec42_20260915T095029Z are empty. Otherwise the two builds agree (results/fabriclib_prov/20260916T111532Z).
  • The Python-client numbers in archive/results/proxies_mn5/20260824T165837Z include ~16 µs of Python/GIL cost; retired.

bigrerun, validation, kernel cost

  • results/bigrerun/ is all warm (BENCH_BOOST_MS=20000, pair 59→58). Do not use aborted-first_20260914T084919Z, first_20260914T100926Z (empty), first_20260914T202456Z (stopped) or aborted-warmcheck_*. sec41_20260915T005320Z had makeups appended in place: read summary_*_postmakeup.csv. The GDAKI private-context 64-CTA cells of the sec42 r2 pass are suspect; use first_20260914T220735Z.
  • results/sec432_validation/20260914T102042Z: the NVSHMEM producer/consumer leg has no data (co-residency deadlock). Use 20260916T081500Z-pc264 and the lean builds; the instrumented GDAKI 32-shared-context 23 % loss is an artifact (lean: 3.2 %).
  • results/regcost/{20260901T164505Z,20260911T013855Z}: blocks_achieved is peak concurrent blocks, not occupancy; persistent-workload curves are invalid; plot against nlive, not registers (baselines cap at 255).
  • The DeepEP "4 % of kernel / 19 ns per put" in results/regcost_review/20260913T173550Z/r3_deepep is superseded by results/deepep_runtime/20260914T092939Z; use exposure_cd8_final.csv, not exposure.csv.

MN5 proxies and knee

  • Discard archive/results/proxies_mn5/20260824T112344Z; in 20260824T105204Z rows ≥90 are contaminated (both carry CONTAMINATED.txt). UCCL-EP T1 rows are missing in 20260824T131844Z; use the -t1patch2 / -patch3 stamps.
  • results/knee/knee_45375994_degraded_as07r4b01 (one 30 Gb/s uplink) and the two *_preflight_failed stamps are not usable for numbers.
  • The NIC-cache reading of results/axisICM_mn5 and results/knee is superseded by results/knee_review. In k4pktpay_45825969, _45828303 and _45829033 the 7 KiB "all" cells fail on PE 5 (bench limit).

Older

  • Loaded hazard runs all livelock: archive/results/axis6b/*/axis6b_hazard_loaded.csv is header-only; the P-RoCE B6 part-4 rows are void.
  • archive/results/axis5b ("arming is free") is superseded by results/regcost.
  • results/e8_inline/20260831T182408Z: the 128 B cells of the first sweep are corrupt; use the inl_rerun rows (pair 59↔60).
  • archive/results/hostknee: only 20260824T090731Z is usable.

Benchmark and mode limits

  • ibgda_bench rtt and latload are valid only at npes = 2: the default peer (mype+1) % npes forms a ring. This voids the rtt column above one HCA in archive/results/axis9t. --peer-xor pairs PEs but is not the default.
  • Only RTT has matched semantics across stacks. Tuned mini-proxy "issue" is one posted store; mini-gda issue is a steady-state loop value; put+completion mixes GDAKI flush, NVSHMEM quiet and per-op counters. %globaltimer has a 32 ns quantum.
  • nvshmem_quiet walks every QP in scope: 21.9 µs put+completion at 16 RC vs 13.1 µs at 2 RC. Quote idle latency from 2 RC. ibgda anchor_rc16 put_lat does not respond to payload; use it only as a state control.
  • NVSHMEM 3.7.2 round-robins puts over RC QPs: internal_vlat/sqlat are valid only at RC 1 (use internal_uvlat/uqlat), and the 3.7.2 latload probe shares QPs with the background.
  • mini-proxy: msgrate mode under-reports (the latload background rate is the real ceiling); B1 rows are host-post-bound and differ between nodes, B16 rows agree; tuned VALIDATE checks one slot only.
  • Expected nonzero exits: mini-proxy T4_R1_B16 (rc=1) and T1/R8 at 64-CTA latload (rc=5, GEOM OVERSUBSCRIBED). scripts/proxies/run_sec43.sh keeps cells with rc≠0 when rows exist, so also filter on the flags column.
  • --stamp on detects replay only for inline payloads; pointer payloads are checked for loss and misdirection only. mini-gda rate cells need --uar-per-qp on, or same-warp doorbells on a shared UAR are lost.
  • GDAKI --bg-pace-ns only lands on plateaus (58–69 / 31 / 17 / 10 / 5 M msg/s), so equal-load comparisons with a proxy work only near 30 M. In gin_bench, thread sharing with 256 threads loses payloads, and --disc 2 can livelock unless threads_per_ctx × 16 ≤ qdepth / 2.
  • Counters: wire.csv and port counters include warm-up; port_xmit_data is in 4-byte units; archive/results/e7_pib/20260901T093234Z reads 2.0× at ≥128 KiB because of warm-up. The RoCE wire constant (1.005 × payload + 335 B) applies to put_lat only.
  • All knee data are MN5 with NIC_HANDLER=cpu. Read all-to-all cells with the per-PE min/median. perftest -q, the host bench and GIN Proxy never reach the knee regime; k2a2a_45819747 is not a matched control.

Hosts and clocks

  • Never pool latency across P-IB node pairs. kolyoz41 is slow in both roles (~0.9 µs one-way). Put+CQE ≈ sender base + 1.9 µs when 41 receives, sender bases 59 = 6.30, 60 = 7.37, 58 = 7.39, 41 = 8.25 µs; issue cost does not move. Pair offsets: 59→58 4.03 µs, 58→59 5.06 µs, any direction with 41 5.92–6.98 µs (results/bigrerun/paircheck-*).
  • kolyoz58 collapses as a loaded responder: CPU-proxy backgrounds drop to 0.5 M msg/s at 16–64 CTAs, and mini-proxy T4/R32/B16 at 64 CTAs never passes BGREADY. Warm-up does not fix it.
  • CPU-proxy numbers depend on the worker core's clock (IBRC, GIN Proxy, mini-proxy, UCCL-EP, MSCCL++, fabric-lib): ~2.2–2.5 GHz cold vs 4.04 GHz warm. IBRC 8 B rate 1.9 vs 3.8 M msg/s; tuned mini-proxy put+completion 4.2 vs 3.6 µs. GPU-submitted paths barely move. Never mix cold stamps with the warm bigrerun stamps; fabric-lib has no boost hook. Data: results/validation/pib/20260915T203000Z-pibB/W1-warmstate, results/ibrc_cold/20260916T094608Z, results/oneop_states/20260916T081315Z.
  • CPU frequency is measured as perf cycles:u / task-clock; cold latency cells shorter than 10 s give a meaningless 0.5–0.8 GHz. IBRC latency swings (up to 25 % across repeats in archive/results/lat1a/lat1) are partly this effect.
  • A few percent of srun/PMIx steps on kolyoz stall before MPI_Init. Launch with </dev/null and a bounded retry. Earlier "GIN Proxy putc hangs" and "cross-switch rtt fails" were this.
  • Forcing the cross-socket NIC (mlx5_4) adds 1.3–1.7 µs put+completion and 1.8–2.2 µs RTT, nothing on issue. The P-IB ~515 M msg/s per-pair wall is on the sender side (archive/results/a5/a5pass1).
  • MN5: mini-proxy T ≥ 16 rates in archive/results/proxies_mn5/{20260824T115034Z,20260824T135808Z,20260824T153414Z} and archive/results/e7_mn5/20260824T185109Z are lower bounds (20-CPU cgroup). A cross-rack pair is 2.5–3 µs slower. port_xmit_wait is pegged except on as01r5b.
  • P-RoCE: the rate ceiling is per NIC, ~53 M msg/s (archive/results/axisB7/b200mn1); relaxed ordering helps only within the GPU's own PCIe switch. The 2026-09-16 results/proce data are single-node (GPU4/NIC4 ↔ GPU5/NIC5, released 3.7.2 wheels); do not mix them with the August two-node data.

Library behaviour (corrections to early notes)

  • NVSHMEM's shipped default is 2 RC QPs per PE (3.7.2: DCI 1 / shared 1 / RC 2), not one shared DCI. The ~50 M msg/s default cap is the RC 2 cap. NUM_DCI=0 means automatic sizing.
  • NVSHMEM ≥ 3.5.19 ignores IBGDA_RC_MAP_BY and round-robins. The RC_MAP_BY=cta results (archive/results/latll/lat1, archive/results/axis3) are 3.4.5 only.
  • NVSHMEM's default build orders with __threadfence() (GPU scope); st.release.sys (mini-gda o2p1 / "f2") is not NVSHMEM-equivalent. The system-scope fence row is o1p1, not o8p1. At system scope a trailing __threadfence_system() drains the posted doorbell and runs 0.67–0.80 µs faster end to end.
  • NVSHMEM_IBGDA_NIC_HANDLER=cpu does engage on P-IB. 8 B rate cells sit at the NIC ceiling and hide it; put+completion shows it (10.88 → 13.47 µs on 3.7.2, 11.78 → 17.54 µs on 3.4.5; results/validation/pib/20260915T203000Z-pibB/H2-handler).
  • GDAKI's peak at 8 threads per context is lane divergence at queue publication: __syncwarp (gin_bench_sync.cu) lifts 16×32 from 34 to 106–110 M msg/s; the same mechanism explains the GB200 drop. The signal wait is >= and per context, so every context must be reset before each phase.
  • mini-gda host-memory queues: at GPU scope (o5p2) +0.80 µs put+CQE and −0.86 µs RTT; at o2p1 +0.93 µs. DeepEP's lower issue cost comes from its CTA-scope CQ poll, not its doorbell.
  • Shipped MSCCL++ is flat at ~1.3–2.8 M msg/s from RegisteredMemory refcount contention; the const-ref patch (stack=mscclppt) scales. UCCL-EP's D2H ring is MSCCL++'s FIFO. UCCL/MSCCL++ commits differ between MN5 (a8c94e4 / b546f39) and P-IB (a3d520e / v0.10.0).
  • NCCL 2.30.7 and 2.31.2 GDAKI numbers differ; NCCL_GIN_PROXY_NTHREADS needs ≥ 2.31.
  • DeepEP: messages are 7,408 B (dispatch) and 14,336 B (combine); the instrumented build has no in-loop spills and can run slightly faster than production; per-rank times are coupled through receive waits. archive/results/deepep_causal/20260903T175557Z used the 3.4.5 / NCCL 2.30.7 wheels (a 3.7.2 rebuild gives the same register counts).

Build and tooling traps

  • Stale binary names: gin_bench_231 predates the context-reset fix (use 231f or later); ibgda_bench_372 from the 2026-09-02 pass predates the p_lat fix. Check md5s against each stamp's versions.txt (kolyoz59 once served a stale Lustre view). Post-09-13 mini-gda builds give a harmless false FAIL ("extra ld_sys=12") at the scripts/oneop/run_sec62.sh G4 SASS gate.
  • ibgda_bench.cu AQ_MAXN defaulted to 16,384; larger QP pools fail every cell as touched != active (fixed from ibgda_bench_mn5k4e on).
  • Some analyzers (analysis/analyze_regcost.py, the regcost figure) write into the stamp they read. Copy a runner or analyzer before changing one that produced archived data.
  • mlx5 hw_counters never count IBGDA/DEVX QPs, and rdma res show stops at 255. Use port counters (packets = data + ACKs) and per-QP library counters.
  • On P-GB200 apply IBRC overrides after sourcing the env file. env/b200_hosts with slots=4 once gave zero rows silently: check row counts, not exit codes.
  • P-RoCE host: sudo strips LD_LIBRARY_PATH (RUNPATH decides the library); gdrdrv does not survive a reboot; reset clocks with nvidia-smi -rgc after an interrupted clock sweep.
  • report_sec432.py needs system python3 (the venv's pandas segfaults on deepep_runtime validation.csv). The sec432 harness block-buffers stdout; use stdbuf -oL. pgrep -f guards match their own command line.
  • fabric-lib on MN5 needs CUDA 12.8, LIBCLANG_PATH and BINDGEN_EXTRA_CLANG_ARGS (env/fabriclib_mn5.sh), with torch libraries on LD_LIBRARY_PATH at run time.