[Nabla7]

Why do we need this?

Use the enterprise Pico browser's full-body tracking to drive G1 SONIC, without the native Pico/XRoboToolkit workaround and without a second controller owning the motors.

What is this feature?

Stack: #4304 -> #4305 -> #3557 -> #4783 -> #4784.

Controller UI companion: jeff-hykin/dim-controller#1.

Layer 5, based on #4783. 17 files, +1500/-60 relative to that base. Builds on Tom's WebXR/SMPL retargeting work in #3721 (cc/feat/pico-teleop-sonic), adapted to the standalone SONIC controller.

• Reuse the existing WebXR body/controller transport. Require the browser's body-tracking feature, all 24 body joints and both controllers.
• Retarget and resample body poses at 50 Hz into ten-frame SONIC reference windows.
• Release, then A+X toggles full-body pose mode. It never arms the robot.
• Blend planner and tracked-pose encoder references over one second while decoder and measured-state history continue.
• Tracking loss, recentering or stale capture exits pose mode. Keep the last valid pose until a new stationary plan from the measured robot pose is ready; retain the existing planner timeout and damping watchdog.
• Left stick drives through MovementManager; right stick turns or adjusts reference heading. Pose mode blocks competing clip/arm/three-point references.
• A+B+X+Y requests latched software damping. This is not physical power isolation.
• Add unitree-g1-sonic-webxr-teleop by composing the existing SONIC stack. One policy controller, one motor-command owner.

No APK, PC Service, gRPC, native video, sim2 dependency or ControlCoordinator task. Unsupported browsers fail rather than falling back to head/wrist-only poses.

How to manually run this

On macOS or a CUDA-enabled Linux host with SONIC dependencies installed. On a Mac, first run uv sync --extra control --extra sim; the blueprint selects CPU inference automatically. Real hardware still requires CUDA:

uv run --no-sync dimos-sonic-models
uv run --no-sync dimos --simulation mujoco --transport zenoh --viewer rerun run unitree-g1-sonic-webxr-teleop

Open https://<simulation-host>:8443/teleop in the enterprise Pico browser using a trusted host certificate and enter the immersive session. The existing WebXR HTTPS setup applies.

In simulation, check walking, A+X pose entry, asymmetric arm motion, return to walking, recenter, headset/controller disconnect and reconnect, Desktop Stop motion, and the four-button damping stop. Tracking recovery must not automatically re-enter pose mode. Inspect status().webxr plus planner/decoder timing.

Controls and acceptance checks.

Verification

On the final branch:

• SONIC, WebXR and MovementManager tests: 173 passed, 5 deselected.
• Gateway, G1 hardware-safety/clock checks and existing simulation adapter: 48 passed, 1 skipped.
• Real/sim Mac/Linux composition and WebXR message serialization: 5 passed. Body snapshots use pickle Zenoh; button messages retain their typed UInt32 encoding.
• Generated blueprint registry: 5 passed.
• New WebXR simulation blueprint imports with exactly one SONIC policy; targeted mypy (10 source files), Ruff and commit hooks pass.

Unit tests use synthetic tracking and fake model/motor IO. Separately, native Mac CPU tests ran the actual models with both full blueprints in MuJoCo, with Rerun disabled: walking, stopping and staying upright without a policy fault. The WebXR HTTPS server started successfully. Legacy shared-memory resource-tracker cleanup warnings remain. A later repeat under severe host load (load average 150.8; 18.5 GB swap in use) hit the existing planner timeout and latched damping. The CPU path is functional, not a hard-real-time guarantee; no watchdog threshold was relaxed. Enterprise-Pico tracking and physical-robot acceptance remain pending; this PR stays draft. No robot deployment or actuation was performed.

Which issue(s) does this PR close?

Follow-up to #3557 and browser retargeting work from #3721; no separate issue closed.

Checklist

• I have read and approved the CLA.

[github-actions]

go2 replay realtime (arm64)

Details

Benchmark suite	Current: 1461538	Previous: c3f0add	Ratio
peak memory	1827.445 MB	1762.234 MB	1.04
peak threads	372 threads	359 threads	1.04
network (transport)	2730.966 MB	2735.794 MB	1.00
disk write	1.711 MB	1.699 MB	1.01
instructions	137.526 G	137.654 G	1.00

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
6706	5	6701	47

View the top 3 failed test(s) by shortest run time

dimos.hardware.sensors.lidar.pointlio.test_wiring::test_every_pointlio_input_has_one_producer

Stack Traces | 0.01s run time

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xff63f0a120c0>

    def test_every_pointlio_input_has_one_producer(monkeypatch: pytest.MonkeyPatch) -> None:
        monkeypatch.setattr(global_config, "robot_ips", "192.0.2.10,192.0.2.11")
        checked = []
        skipped = []
        for name in UBUNTU_BLUEPRINTS:
            try:
                blueprint = load_blueprint(name)
            except OptionalDependencyError as e:
                skipped.append(f"{name} ({e})")
                continue
            lio = [atom for atom in blueprint.active_blueprints if atom.module is PointLio]
            for atom in lio:
                for port in POINTLIO_INPUTS:
                    producers = _producers(blueprint, atom.name, port)
                    names = [f"{p.name}.{port}" for p in producers]
>                   assert len(producers) == 1, f"{name}: {atom.name}.{port} fed by {names}"
E                   AssertionError: unitree-g1-sonic-webxr-teleop: pointlio.lidar_raw fed by []
E                   assert 0 == 1
E                    +  where 0 = len([])

atom       = BlueprintAtom(kwargs={'lidar_ip': '192.168.123.120'}, module=<class 'dimos.hardware.sensors.lidar.pointlio.module.Poin...me='tf', type=<class 'dimos.msgs.tf2_msgs.TFMessage.TFMessage'>, direction='out')), module_refs=(), instance_name=None)
blueprint  = Blueprint(blueprints=(BlueprintAtom(kwargs={'command_timeout_seconds': 0.1}, module=<class 'dimos.robot.unitree.g1.who...ts'}), requirement_checks=(<function _require_zenoh at 0xff6449ab1b20>,), configurator_checks=(), hosted_placements=())
checked    = ['alfred-keyboard-teleop', 'alfred-nav', 'demo-virtual-mid360-pointlio', 'go2-nav', 'go2-nav-hosted', 'go2-nav-l1', ...]
driver     = BlueprintAtom(kwargs={'frame_id': 'mid360_link', 'point_format': 'full'}, module=<class 'dimos.hardware.sensors.lidar...., type=<class 'dimos.msgs.sensor_msgs.PointCloud2.PointCloud2'>, direction='out')), module_refs=(), instance_name=None)
driver_frame = 'mid360_link'
lio        = [BlueprintAtom(kwargs={'lidar_ip': '192.168.123.120'}, module=<class 'dimos.hardware.sensors.lidar.pointlio.module.Poi...e='tf', type=<class 'dimos.msgs.tf2_msgs.TFMessage.TFMessage'>, direction='out')), module_refs=(), instance_name=None)]
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xff63f0a120c0>
name       = 'unitree-g1-sonic-webxr-teleop'
names      = []
port       = 'lidar_raw'
producers  = []
sensor_frame = 'mid360_link'
skipped    = ['demo-object-scene-registration (ZED SDK not installed. Please install pyzed package to use ZED camera functionality.)', 'unitree-g1-nav-simple (unitree_sdk2py)', 'unitree-g1-record (unitree_sdk2py)']

.../lidar/pointlio/test_wiring.py:67: AssertionError

dimos.codebase_checks.test_blueprint_kwargs::test_blueprint_atom_kwargs_match_module_config[unitree-g1-sonic-webxr-teleop]

Stack Traces | 0.014s run time

blueprint_name = 'unitree-g1-sonic-webxr-teleop'
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xff6449ac6780>

    @pytest.mark.parametrize("blueprint_name", _blueprint_params())
    def test_blueprint_atom_kwargs_match_module_config(
        blueprint_name: str, monkeypatch: pytest.MonkeyPatch
    ) -> None:
        """Fail when blueprint kwargs cannot be consumed by their target module."""
        # The multi-robot blueprints read ROBOT_IPS at import time.
        monkeypatch.setattr(global_config, "robot_ips", "192.0.2.10,192.0.2.11")
        blueprint = _get_blueprint_or_skip(blueprint_name)
    
        violations: list[str] = []
        for atom in blueprint.blueprints:
            unknown_kwargs = sorted(set(atom.kwargs) - _allowed_kwarg_names(atom.module))
            if unknown_kwargs:
                violations.append(
                    f"{atom.module.__module__}.{atom.module.__name__}: unknown kwargs {unknown_kwargs}"
                )
    
        if violations:
            listing = "\n".join(f"  - {violation}" for violation in violations)
>           raise AssertionError(
                f"Blueprint {blueprint_name!r} passes unknown module kwargs:\n{listing}\n\n"
                "Blueprint kwargs are forwarded into the module constructor. For modules "
                "with an explicit `config` annotation, use fields from that config model; "
                "for legacy modules with direct constructor parameters, use the declared "
                "`__init__` keyword names."
            )
E           AssertionError: Blueprint 'unitree-g1-sonic-webxr-teleop' passes unknown module kwargs:
E             - dimos.hardware.sensors.lidar.pointlio.module.PointLio: unknown kwargs ['lidar_ip']
E           
E           Blueprint kwargs are forwarded into the module constructor. For modules with an explicit `config` annotation, use fields from that config model; for legacy modules with direct constructor parameters, use the declared `__init__` keyword names.

atom       = BlueprintAtom(kwargs={}, module=<class 'dimos.teleop.webxr.sonic.SonicWebXRTeleopModule'>, streams=(StreamRef(name='le...e_cmd_vel', type=<class 'dimos.msgs.geometry_msgs.Twist.Twist'>, direction='out')), module_refs=(), instance_name=None)
blueprint  = Blueprint(blueprints=(BlueprintAtom(kwargs={'command_timeout_seconds': 0.1}, module=<class 'dimos.robot.unitree.g1.who...ts'}), requirement_checks=(<function _require_zenoh at 0xff6449ab1b20>,), configurator_checks=(), hosted_placements=())
blueprint_name = 'unitree-g1-sonic-webxr-teleop'
listing    = "  - dimos.hardware.sensors.lidar.pointlio.module.PointLio: unknown kwargs ['lidar_ip']"
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xff6449ac6780>
unknown_kwargs = []
violations = ["dimos.hardware.sensors.lidar.pointlio.module.PointLio: unknown kwargs ['lidar_ip']"]

dimos/codebase_checks/test_blueprint_kwargs.py:97: AssertionError

dimos.codebase_checks.test_docs_branding::test_docs_use_current_branding

Stack Traces | 0.014s run time

def test_docs_use_current_branding() -> None:
        """Fail if any file under docs/ spells the brand "DimOS" instead of "dimOS"."""
        hits = find_old_branding()
        if hits:
            listing = "\n".join(
                f"  - {p.relative_to(DIMOS_PROJECT_ROOT)}:{lineno}: {line.strip()}"
                for p, lineno, line in hits
            )
>           raise AssertionError(f'Found "DimOS" in docs/:\n{listing}\n\nThe brand is spelled "dimOS".')
E           AssertionError: Found "DimOS" in docs/:
E             - docs/usage/sonic-webxr.md:23: Use DimOS Desktop's Controller app for cameras and policy controls. This blueprint
E           
E           The brand is spelled "dimOS".

hits       = [(PosixPath('.../docs/usage/sonic-webxr.md'), 23, "Use DimOS Desktop's Controller app for cameras and policy controls. This blueprint")]
listing    = "  - docs/usage/sonic-webxr.md:23: Use DimOS Desktop's Controller app for cameras and policy controls. This blueprint"

dimos/codebase_checks/test_docs_branding.py:43: AssertionError

dimos.codebase_checks.test_no_init_files::test_no_init_files

Stack Traces | 0.023s run time

def test_no_init_files():
        dimos_dir = DIMOS_PROJECT_ROOT / "dimos"
        init_files = sorted(dimos_dir.rglob("__init__.py"))
        # The root dimos/__init__.py is allowed for the porcelain lazy import.
        init_files = [f for f in init_files if f != dimos_dir / "__init__.py"]
        if init_files:
            listing = "\n".join(f"  - {f.relative_to(dimos_dir)}" for f in init_files)
>           raise AssertionError(
                f"Found __init__.py files in dimos/:\n{listing}\n\n"
                "__init__.py files are not allowed because they lead to unnecessary "
                "extraneous imports. Everything should be imported straight from the "
                "source module."
            )
E           AssertionError: Found __init__.py files in dimos/:
E             - .../sonic/webxr/__init__.py
E           
E           __init__.py files are not allowed because they lead to unnecessary extraneous imports. Everything should be imported straight from the source module.

dimos_dir  = PosixPath('.../dimos/dimos/dimos')
init_files = [PosixPath('.../dimos/dimos/dimos/.../sonic/webxr/__init__.py')]
listing    = '  - .../sonic/webxr/__init__.py'

dimos/codebase_checks/test_no_init_files.py:25: AssertionError

experimental.gateway.tests.test_config::test_every_registered_blueprint_is_annotated

Stack Traces | 0.121s run time

def test_every_registered_blueprint_is_annotated() -> None:
        source = annotation_file.load()
        annotated = {name for robot in source["robots"].values() for name in robot["blueprints"]}
        missing = sorted(
            name
            for name in all_blueprints
            if name not in annotated and not annotation_file.HIDDEN_NAME.search(name)
        )
>       assert not missing, f"add these to experimental/gateway/annotations.yaml: {', '.join(missing)}"
E       AssertionError: add these to experimental/gateway/annotations.yaml: unitree-g1-sonic-wbc, unitree-g1-sonic-webxr-teleop
E       assert not ['unitree-g1-sonic-wbc', 'unitree-g1-sonic-webxr-teleop']

annotated  = {'a1z-planner-coordinator', 'alfred-keyboard-teleop', 'alfred-mls-nav', 'alfred-mls-nav-lidar', 'alfred-nav', 'coordinator-a1z', ...}
missing    = ['unitree-g1-sonic-wbc', 'unitree-g1-sonic-webxr-teleop']
source     = {'robots': {'a1z': {'blueprints': {'a1z-planner-coordinator': {'description': 'Motion planning and execution for a Gal...': {'label': 'Arms'}, 'dog': {'label': 'Dogs'}, 'drone': {'label': 'Drones'}, 'humanoid': {'label': 'Humanoids'}, ...}}

.../gateway/tests/test_config.py:255: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}