EPIC: 5G-NR-style scheduled RAN on devourer (multi-user + multi-cell) #254

Umbrella / tracking issue for building a 5G-NR-style scheduled Radio Access Network on devourer: time-synced AP cells serving multiple scheduled UEs (robots/drones), with per-UE timing advance, coordinated scheduling, and make-before-break handover — riding the bench-proven per-link primitives and modelled on 5G NR design decisions (not LTE).

docs/multi-ap-cellular.md promotes the architecture; this EPIC turns it into a step-by-step build. The per-milestone write-ups are docs/scheduled-mac.md (M0) and docs/cell-m1.md … docs/cell-m6.md. Rig orchestration and results are in the testrig repo (scripts/m*_*.sh, results/).

Status (2026-11-12)

Built and run on the testrig: three Topton J6412 boxes, each with two RTL8852CE (Kestrel) over PCIe/vfio, plus an N300 hub that is PTP grandmaster and CU host (a star of direct i226 links, no switch in the timing path). On the rig:

• three PTP-locked cells, on three channels, with UEs scheduled in fixed slots;
• HARQ and link adaptation;
• RACH join and departure;
• CU-coordinated schedules and phase planning;
• measurement gaps;
• conditional handover, and DAPS on one channel and across channels;
• per-cell keys;
• UL and DL sequence handling across handovers.

What remains:

• the QoS scheduler (M1);
• runtime channel moves (M5);
• the coarse/USB tier (Coarse-tier (USB) DU on a PTP node — the M4/M5 items split out of the PCIe tier #565);
• M6 follow-ups (Follow-ups left open by M6 (#260): EVM, roam baseline, the named test, residuals #566);
• a moving-UE bench (M6 test realism — a moving-UE handover bench (conducted path-loss matrix, then an over-the-air walk) #563).

The primitives (this is the multi-user/multi-cell layer)

Bench-proven and library-stable:

• StartBeacon and UpdateBeaconPayload, a beacon body update measured frame-atomic on air (docs/scheduled-mac.md);
• PinBeaconTbtt, the absolute TSF-preserving TBTT actuator that closed an AP↔PTP loop to ~0.67 µs on the 8821CE;
• on TSF-locked-TBTT parts (Kestrel/AX), TSF steps against the free-running counter (kestrel: multi-cell beacon sync on one PTP timebase — free-running-counter TSF discipline + on-air observer (#258) #513; TBTT p95 3–4 µs);
• a deterministic beacon departure, SetBeaconBackoff (kestrel: IRadio::SetBeaconBackoff — deterministic hardware-beacon departure #514);
• TsfSync, WriteTsf/ReadTsfAndFreeRun, Packet::TxEgressTsf;
• hardware ACK (SetAckResponder, the 8852C responder included), A-MPDU (SetAmpduMode), per-frame TX status (TxReport, where the chip has it; Kestrel does not), per-packet rate/power, GetRxQuality;
• FastRetune/FastSetBandwidth/SetCcaMode.

5G-NR mapping

Architecture boundary (enforced in every milestone)

• src/cell/ — the per-cell (DU) scheduled-MAC library (reusable):

  • SlotGrid, GrantMap, UeRegistry, UeRxAttribution, Harq, LinkAdapt, Rach (M1–M3);
  • TsfDiscipline, SyncHealth (M4);
  • F1, Coord, PhasePlan (M5);
  • Handover, HoCommand, MeasGap, Keys (M6).

  It adds nothing to the radio core unless a bench test proves userspace can't. (CellScheduler: 5QI classes; TimingAdvance: the per-UE TA loop, held per UE in UeRegistry's UeContext.)

• examples/cell/ — a reference gNB-DU + UE node.

• tools/ran-controller/ — the CU / backhaul app (multi-cell scheduler, coordination, CHO/DAPS). Distributed software over the F1 control link (with direct Xn DU↔DU coordination where lower latency helps); never touches the radio. (Note: this is an NR-inspired centralized facility scheduler, not a faithful 3GPP CU — it owns scheduling, not PDCP/RRC.)

Rule: logic that would still make sense with a different radio underneath is NOT in the radio core.

Milestones (coarse; each demoable, implemented step by step)

Phase 0 — feasibility contracts (measure, don't assume):

• M0 M0 — Feasibility contracts (measure before building the cell) #261 — Feasibility contracts (DL departure jitter, dynamic grant delivery, unicast-ACK/TxReport matrix, per-UE RX attribution) — blocks M1

Phase A — single gNB-DU (one cell, N UEs):

• M1 M1 — Single-cell scheduled MAC (src/cell/ DU library + examples/cell node) #255 — Single-cell scheduled MAC (src/cell/ library + demo)
• M2 M2 — Reliability & link adaptation (per-UE ACK, HARQ, rate/power) #256 — Reliability & link adaptation (per-UE ACK, HARQ, rate/power)
• M3 M3 — Admission: association + 2-step RACH #257 — Admission: 2-step RACH and departure (association descoped)

Phase B — multi-cell RAN (CU + mobility):

• M4 M4 — Network-wide time sync across cells (TSN/802.1AS-style) #258 — Network-wide time sync across cells (TSN/802.1AS-style) — closed on the PCIe tier
• M5 M5 — CU + coordinated scheduling (tools/ran-controller/) #259 — CU + coordinated scheduling — closed on the PCIe tier; runtime channel moves in cell M5: move a running cell to another channel (runtime channel separation) #567
• M6 M6 — Mobility: CHO + DAPS handover #260 — Mobility: CHO + DAPS handover
• M6 test realism M6 test realism — a moving-UE handover bench (conducted path-loss matrix, then an over-the-air walk) #563 — Moving-UE handover bench: a conducted path-loss matrix (UL and DL follow a trajectory, fading, several UEs), then an over-the-air walk
• Coarse-tier (USB) DU on a PTP node — the M4/M5 items split out of the PCIe tier #565 — Coarse-tier (USB) DU on a PTP node (split out of M4/M5)
• Follow-ups left open by M6 (#260): EVM, roam baseline, the named test, residuals #566 — Follow-ups left open by M6 (EVM, roam baseline, the named test, residuals)

Dependency chain: M0 → M1; M1 → M2, M3, M4; M4 → M5 → M6 → #563 (test realism).

Cross-cutting deliverables (from the architecture review)

• Slot-budget accounting — a valid slot = PSDU airtime + PHY preamble + SIFS/ACK + retry allowance + A-MPDU duration + radio turnaround + retune/settling + timing guard. Payload size alone is not a budget. Done in src/cell/SlotGrid.h (A-MPDU airtime not modelled); a DL frame that would start past its slot's guard is not aired.
• A versioned on-air protocol envelope — protocol version, cell+UE identity, superframe/slot number, schedule epoch, packet/HARQ sequence, integrity + replay protection. Done: kCellProtoVersion (v5), cell-key tags on every cell frame, area tags for neighbours, replay floors. Integrity is on only with a key configured.
• Fault detection — watchdogs for lost PTP sync, stalled beacon engine, missed schedule epochs, CU/backhaul outage, excessive TX-queue delay, over-budget retunes. Partly done:
  • loss of PTP sync (SyncHealth, auto-suspend), CU outage (DUs go autonomous), stale or late schedules (acks), and late DL frames are handled;
  • a stalled beacon engine is detected only indirectly, through the UEs' uplinks, and there is no general over-budget-retune alarm.
• KPI-based validation — report distributions not averages: deadline-miss ratio, slot-arrival phase, collision rate, handover interruption, DAPS duplicate gain, per-UE fairness, clock holdover, control-plane update latency. Mostly done in the docs/cell-m*.md tables; still missing: per-UE fairness, a stated DAPS duplicate gain and clock holdover.

Out of scope / relationships

• OFDMA / UL-MU / MU-MIMO (5G air interface) = Kestrel 11ax, 802.11ax trigger-based UL + TWT (Kestrel / RTL8852) — scheduled contention-free TX, the standards answer to the TX-egress gap #236. OUT. UL is TDMA-only throughout; numerology/BWP are time/bandwidth analogs, not real SCS.
• PCIe: RTL8822BE (Jaguar2) + RTL8822CE (Jaguar3) support — bring-up recipe for when lab hardware arrives #214 (J3 PCIe) = optional tighter TBTT actuator for a production cell.
• Timing benchmark: SDIO vs USB (RTL8733BS vs 8733BU) — confirm/refute the transport-latency hypotheses #235 (SDIO) = the FPV-transport path.
• SPIKE: ARM cell-node feasibility for warehouse-scale RAN (PTP quality, vfio/SMMU, USB-tier PinBeaconTbtt discipline) #381 (ARM cell-node spike) — its USB-tier gate is the same question as Coarse-tier (USB) DU on a PTP node — the M4/M5 items split out of the PCIe tier #565.
• Multi-receiver ground station: N radios / N channels / one timebase — nothing measured beyond 2 TX → 1 RX #457 (multi-receiver ground station: N radios / N channels / one timebase) — the many-transmitters-into-one-site side of the same problem.
  • It shares M4's single PTP timebase (its "N nodes over Ethernet" shape is the testrig fleet) and M5's per-channel planning.
  • It extends M0's per-UE RX attribution from 2 transmitters to 10–25.
  • M6 test realism — a moving-UE handover bench (conducted path-loss matrix, then an over-the-air walk) #563's attenuator matrix would serve it too.

Standing risk register (status 2026-11-12)

• Deterministic downlink (M0/M1, headline): retired. Kestrel's DL departure was measured in M0, and in M1 the DL air start was p50/p99 31/41 µs after its slot start. A rare multi-ms send_packet tail remains, so a RACH slot is kept away from the next beacon.
• No atomic beacon update (M0/M1): retired. UpdateBeaconPayload exists on every family and is frame-atomic on air (not atomic against the TBTT; it airs on the next or next-but-one TBTT). The grant map rides the beacon with an epoch and its effective superframe.
• Unicast ACK per node, not multi-MACID (M0/M2): retired. One responder per node; the 8852CE responder measured 1036/1036. Kestrel has no per-frame TxReport, so HARQ runs on explicit UE feedback.
• GetRxQuality is device-wide: addressed differently. UeRxAttribution exists, but link adaptation uses the SNR the UE reports for the downlink.
• send_packet UL has no sub-slot air-departure control: followed. The UL is the UE's hardware beacon, steered through its TSF: 1.4–1.8 µs rms.
• ~ms FastRetune: measured, 23.6 ms on the 8852CE. Different-channel DAPS is sequential (visits), with measurement gaps budgeted at 26 + 2 ms.
• Timing advance corrects clock/actuation phase, not propagation: as designed.
• PCIe vs USB sync tiers: the tight tier is done, with loss-of-sync auto-suspend; the coarse tier is unmeasured (Coarse-tier (USB) DU on a PTP node — the M4/M5 items split out of the PCIe tier #565).
• A3/CHO tuning: hysteresis and time-to-trigger on filtered quality. In the selftest the filtered A3 made one handover per crossing (raw RSSI: 30.6); on the rig, 0 in the hover case. Fading is still untested (M6 test realism — a moving-UE handover bench (conducted path-loss matrix, then an over-the-air walk) #563).
• Half-duplex UE turnaround: budgeted in the grid (turnaround_us).

Architecture review folded in: this epic and its milestones were revised per an external review — M0 added, terminology corrected (F1/Xn), the multi-MACID framing corrected, M1 numerology restricted to fixed, and cross-cutting deliverables added. Refreshed 2026-11-12 against what was built.