Skip to content

Preserve G1 motor-command freshness and MuJoCo timing - #4304

Open
Nabla7 wants to merge 9 commits into
mainfrom
pim/fix/g1-control-foundation
Open

Nabla7 wants to merge 9 commits into
mainfrom
pim/fix/g1-control-foundation

Conversation

@Nabla7

@Nabla7 Nabla7 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

What is this feature?

Keep G1 motor delivery independent of policy computation. An independent 500 Hz DDS publisher refreshes motor targets, checks feedback and configured command timeouts, and latches damping on control faults. Typed Zenoh subscriptions can retain only the latest frame to avoid accumulating stale motor commands.

MuJoCo waits for a complete PD command before advancing physics, can attach the G1 robot and its meshes to scene packages, and uses state-only native viewer synchronization to preserve physics timing with large scenes. Shared G1 motor fault checks live in dimos/robot/unitree/g1/control_safety.py. The driver accepts controller_fault: In[String], not a policy-specific fault port.

Layer 1 of the SONIC stack: #4304 -> #4305 -> #3557. This layer contains hardware, transport and simulation changes and their tests: 12 files, +782/-56. SONIC's policy lifecycle belongs to the standalone module in #3557; this layer does not change ControlCoordinator or the whole-body transport adapter.

Why do we need this?

SONIC needs fresh motor commands and simulation timing that keeps up with the policy. Keeping command delivery and simulation support separate makes their effect on existing robots independently reviewable.

How to Test

uv run --no-sync pytest \
  dimos/control/test_control.py::TestControlCoordinatorLifecycle \
  dimos/hardware/test_adapter_registries.py \
  dimos/control/tasks/test_registry.py \
  dimos/robot/unitree/g1/test_wholebody_connection.py \
  dimos/robot/test_all_blueprints_generation.py

At 5142a49ba, all 23 motor/fault tests pass, including controller-neutral fault reasons and latched damping. Targeted mypy and commit hooks pass. Earlier evidence at the preceding checkpoint comprised 60 focused checks. The hardware checks use fake IO; physical-robot watchdog timing has not been validated by these tests.

Which issue(s) does this PR close?

Part of #3557; no separate issue is closed.

Checklist

  • I have read and approved the CLA.

Stack update (2026-10-22)

Review and merge in order:

  1. Preserve G1 motor-command freshness and MuJoCo timing #4304: G1 motor delivery and simulation foundation.
  2. Add SONIC v1.1 inference and verified model setup #4305: SONIC inference and verified assets.
  3. Integrate standalone G1 SONIC policy and blueprint #3557: standalone policy module and blueprint.
  4. feat(g1): add SONIC Desktop activation and stop controls #4783 (draft): Desktop activation/stop APIs and run-bound gateway calls.
  5. feat(g1): add enterprise Pico WebXR full-body SONIC teleop #4784 (draft): enterprise Pico browser WebXR full-body teleop.

Controller app companion: jeff-hykin/dim-controller#1, separate repository.

Main at 42ca50dfef is merged through the original three branches without rewriting history. No sim2 or unmerged ControlCoordinator dependency was introduced. The native Pico/XRoboToolkit workaround is excluded.

Previous checkpoint, before the Mac CPU fix: 224 focused Python checks passed (SONIC/WebXR/movement, gateway/G1 safety/clocks/sim adapter, composition and registry); one hardware-specific check was skipped. Targeted mypy and commit hooks passed. The companion UI passed 290 tests and production-bundle desktop/mobile browser checks using mocked gateway replies.

Mac CPU simulation is now supported and verified with the actual SONIC models in MuJoCo (walking, stopping and staying upright). Hardware still requires CUDA. Enterprise-Pico tracking and physical-robot acceptance remain pending; no robot deployment or actuation was performed.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime

Details
Benchmark suite Current: b50b226 Previous: 576d547 Ratio
first frame wall 9.845 s 9.922 s 0.99
first frame cpu 20.474 s 20.847 s 0.98
run cpu 31.902 s 29.144 s 1.09
run cpu (user) 24.498 s 21.535 s 1.14
run cpu (system) 7.404 s 7.609 s 0.97
peak memory 2718.402 MB 2715.977 MB 1.00
peak threads 414 threads 409 threads 1.01
disk read 32.402 MB 29.492 MB 1.10
disk write 720.887 MB 753.289 MB 0.96

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'go2 replay realtime'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.08.

Benchmark suite Current: b50b226 Previous: 576d547 Ratio
run cpu 31.902 s 29.144 s 1.09
run cpu (user) 24.498 s 21.535 s 1.14
disk read 32.402 MB 29.492 MB 1.10

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.50289% with 96 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/robot/unitree/g1/wholebody_connection.py 75.25% 38 Missing and 11 partials ⚠️
dimos/simulation/engines/mujoco_sim_module.py 28.00% 18 Missing ⚠️
dimos/protocol/pubsub/impl/zenohpubsub.py 68.08% 8 Missing and 7 partials ⚠️
dimos/simulation/adapters/whole_body/g1.py 57.14% 5 Missing and 1 partial ⚠️
dimos/simulation/engines/mujoco_engine.py 28.57% 5 Missing ⚠️
dimos/robot/unitree/g1/control_safety.py 81.25% 2 Missing and 1 partial ⚠️
@@            Coverage Diff             @@
##             main    #4304      +/-   ##
==========================================
+ Coverage   82.28%   82.43%   +0.15%     
==========================================
  Files        1693     1694       +1     
  Lines      158389   158953     +564     
  Branches    13235    13316      +81     
==========================================
+ Hits       130326   131031     +705     
+ Misses      24428    24248     -180     
- Partials     3635     3674      +39     
Components Coverage Δ
Tests 96.86% <100.00%> (-0.03%) ⬇️
Flag Coverage Δ
OS-macos-latest 76.38% <81.50%> (+0.05%) ⬆️
OS-ubuntu-24.04-arm 76.21% <81.50%> (+0.05%) ⬆️
OS-ubuntu-latest 76.56% <81.50%> (+0.06%) ⬆️
Py-3.10 76.56% <81.50%> (+0.24%) ⬆️
Py-3.11 76.34% <81.50%> (-0.16%) ⬇️
Py-3.12 76.56% <81.50%> (+0.05%) ⬆️
SelfHosted-Large 32.10% <15.41%> (+0.23%) ⬆️
SelfHosted-Linux 40.45% <15.41%> (-0.09%) ⬇️
SelfHosted-macOS 40.77% <15.41%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/protocol/pubsub/impl/test_zenohpubsub.py 97.86% <100.00%> (+0.16%) ⬆️
...imos/robot/unitree/g1/test_wholebody_connection.py 100.00% <100.00%> (ø)
dimos/simulation/adapters/whole_body/test_g1.py 100.00% <100.00%> (ø)
dimos/simulation/engines/test_mujoco_sim_module.py 98.11% <100.00%> (+0.15%) ⬆️
dimos/robot/unitree/g1/control_safety.py 81.25% <81.25%> (ø)
dimos/simulation/engines/mujoco_engine.py 36.62% <28.57%> (+0.09%) ⬆️
dimos/simulation/adapters/whole_body/g1.py 70.32% <57.14%> (+11.35%) ⬆️
dimos/protocol/pubsub/impl/zenohpubsub.py 89.45% <68.08%> (-5.29%) ⬇️
dimos/simulation/engines/mujoco_sim_module.py 52.62% <28.00%> (+3.81%) ⬆️
dimos/robot/unitree/g1/wholebody_connection.py 67.48% <75.25%> (+21.55%) ⬆️

... and 22 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@greptile-apps

greptile-apps Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[High risk] Restructures hardware activation timing and adds safety checks to robot control.

Not ready to merge: shutdown can publish an actuator command before controller handoff. The missing-feedback diagnostic issue is non-blocking.

Findings

  1. P1 Shutdown commands before handoff ▶
  2. P2 Missing feedback stays silent ▶

Summary

The PR defers G1 controller handoff until a prepared command and adds command publication and fault handling. For an operator who starts and stops the connection without issuing a command, shutdown can still publish a motor-disable command before handoff. If the first feedback frame never arrives, the connection remains inactive without reporting a fault. The shutdown behavior must be fixed before merging.

Reviews (1) · Last reviewed commit: "fix(control): prepare G1 commands before..."

# Leave a faulted robot in damping; graceful cleanup must not replace
# a latched stop with a different actuator mode. Normal shutdown keeps
# its existing motor-disable behavior.
if self._publisher is not None and self._low_cmd is not None and self._crc is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Shutdown commands before handoff

If the connection stops before accepting a command, sport mode has not been released. stop() nevertheless publishes a motor-disable LowCmd on rt/lowcmd, sending an actuator command while the native controller is still in charge. Skip the final LowCmd when handoff has not occurred; this must be fixed before merging.

Artifacts

Mock DDS start-and-stop reproduction script

  • The executed script loads the parent or PR source, calls start and stop without supplying a motor command, and records DDS writes and handoff requests.

Parent behavior with no motor command

  • The parent run requested sport-mode release before publishing one 35-motor disable frame; assertions passed.

PR behavior with no motor command

  • The PR run published the same disable frame with no release request and handoff still false; assertions passed.

View artifacts

T-Rex Ran code and verified through T-Rex

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stop() now sends a final motor command only after controller handoff, while still closing the DDS connections in either case.

Comment on lines +526 to +527
if self._last_feedback_at is None:
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing feedback stays silent

If the first LowState never arrives, valid commands return without reporting a fault. The writer checks feedback freshness only after handoff, which cannot happen without an accepted command. The connection remains inactive with no fault reason, making a missing feedback stream harder to diagnose. This diagnostic issue is non-blocking.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Authored missing-feedback runtime test

  • The authored script drives real G1 handler and writer methods with mocked DDS feedback and time, and contains both comparison commands.

Run with no first LowState

  • The executed candidate handler received five commands over 0.8 seconds without feedback and reported no fault, confirming the silent stall.

In-memory guard comparison with no first LowState

  • The same executed sequence with an in-memory guard change reported robot feedback timeout, showing the fault can be surfaced before handoff.

View artifacts

T-Rex Ran code and verified through T-Rex

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commands received before the first feedback frame now produce a warning, and command_stream_status() exposes feedback_received, without taking control or latching a startup fault.

@greptile-apps

greptile-apps Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

  • P1 Stop publishes motor-disable LowCmd before sport-mode handoff ▶

    • Bug
      • With no accepted command, the PR leaves sport mode unreleased but stop() still publishes a LowCmd with all 35 motor modes set to 0. This can put a motor-disable request on rt/lowcmd while the stock controller remains active.
    • Cause
      • Handoff is deferred at line 243, while the stop() publish condition at line 280 checks only that DDS objects exist; Write() at line 294 does not require a completed handoff.
    • Fix
      • Skip the final LowCmd when sport mode has not been released, retaining the existing post-handoff safe-stop behavior.
  • P1 Missing first LowState leaves the connection silently stalled ▶

    • Bug
      • With no initial feedback, repeated motor commands and advancing time produce no handoff, no DDS writes, and no reported fault.
    • Cause
      • _on_motor_command returns when _last_feedback_at is None, while the writer checks feedback freshness only after sport-mode handoff; that handoff requires a command to pass the guard.
    • Fix
      • Track startup time and latch or report a feedback timeout if the first LowState has not arrived by the configured deadline, without requiring handoff.

@Nabla7 Nabla7 changed the title Prepare G1 control before activation and preserve command timing Preserve G1 motor-command freshness and MuJoCo timing Oct 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: f1c5558 Previous: 0ea05f6 Ratio
peak memory 1823.395 MB 1699.445 MB 1.07
peak threads 359 threads 357 threads 1.01
network (transport) 2731 MB 2725.914 MB 1.00
disk write 1.477 MB 1.469 MB 1.01
instructions 137.539 G 136.571 G 1.01

This comment was automatically generated by workflow using github-action-benchmark.

This branch was previously deployed

1 inactive deployment
cachix — f1c5558b Deployed Oct 22, 2026 by Nabla7 via cachix-build-macos #14050
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant