Browse Source

Phase 4: DDS bridge debugging, GR00T-WBC real robot prep, Vision Pro telepresence research

- Diagnosed DDS multicast blocked by UFW firewall on GB10 (root cause)
- Confirmed DDS active inside robot (342 pkts/4s from locomotion computer)
- Documented SSH credentials (unitree@192.168.123.164, pw: 123)
- Created real robot launch script and DDS test script
- Researched Vision Pro integration paths (xr_teleoperate WebXR, VisionProTeleop native)
- Documented GR00T-WBC decoupled architecture and CONTROL_GOAL_TOPIC integration point
- Added CURRENT-STATE.md handoff document for session continuity
- Updated networking-comms, open-questions, motion-retargeting, learning-and-ai context files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
master
Joe DiPrima 1 month ago
parent
commit
36a93e0e3a
  1. 3
      CLAUDE.md
  2. 145
      CURRENT-STATE.md
  3. 45
      context/gb10-offboard-compute.md
  4. 40
      context/learning-and-ai.md
  5. 28
      context/motion-retargeting.md
  6. 37
      context/networking-comms.md
  7. 99
      context/open-questions.md
  8. 32
      context/whole-body-control.md
  9. 108
      phases/phase-3-groot-wbc-deployment.md
  10. 71
      reference/glossary.yaml
  11. 98
      scripts/dds_test.py
  12. 34
      scripts/launch_real.sh

3
CLAUDE.md

@ -154,3 +154,6 @@ Operations
| 0 | 2026-02-13 | Context system scaffolding created — 15 topic files, glossary, templates | | 0 | 2026-02-13 | Context system scaffolding created — 15 topic files, glossary, templates |
| 1 | 2026-02-13 | Populated all context files from official docs, GitHub repos, 6 research papers, 5 community guides. ~30 source documents archived. Glossary expanded to 37 terms. 9 open questions resolved. | | 1 | 2026-02-13 | Populated all context files from official docs, GitHub repos, 6 research papers, 5 community guides. ~30 source documents archived. Glossary expanded to 37 terms. 9 open questions resolved. |
| 2 | 2026-02-13 | Expanded context for motion capture + robust balance. 3 new topic files (whole-body-control, motion-retargeting, push-recovery-balance). ~15 new source docs. Glossary expanded to ~57 terms. 6 new open questions. | | 2 | 2026-02-13 | Expanded context for motion capture + robust balance. 3 new topic files (whole-body-control, motion-retargeting, push-recovery-balance). ~15 new source docs. Glossary expanded to ~57 terms. 6 new open questions. |
| 3 | 2026-02-14 | GR00T-WBC deployed on GB10. 4 critical aarch64 bugs fixed (CycloneDDS buffer overflow, ROS2 path, shared libs, sync mode). Walking sim verified. NoMachine + web viewer for remote viz. 5 new open questions, 3 resolved. gb10-offboard-compute promoted to established. |
| 3.5 | 2026-02-15 | MuJoCo Playground training pipeline deployed on GB10. G1JoystickFlatTerrain verified (29-DOF, 103-dim obs). Brax PPO training at ~17K steps/sec on Blackwell. Locomotion-only baseline: 200M steps, reward +12.1, 3:08 training time. Researched unified WBC approach (ExBody2 paradigm) for AVP telepresence. Plan saved for Phase 4. |
| 4 | 2026-02-15 | DDS network bridge: GB10 ↔ G1 robot. Diagnosed DDS multicast blocked by UFW firewall (root cause). Confirmed DDS active inside robot (342 pkts/4s). SSH password documented (123). GR00T-WBC real robot config + launch script created. Researched Vision Pro telepresence: xr_teleoperate (WebXR, no app) and VisionProTeleop (native, open-source). GR00T-WBC architecture fully documented (decoupled upper/lower body, CONTROL_GOAL_TOPIC integration point). Vision Pro selected as primary telepresence device. |

145
CURRENT-STATE.md

@ -0,0 +1,145 @@
# Current State — 2026-02-15
Handoff document for continuing work from another machine.
---
## What's Working
### MuJoCo Playground RL Training (Complete)
- **Location:** GB10 (`~/mujoco_training/`)
- **Result:** 200M-step locomotion policy trained. Reward: -6.4 → +12.1
- **Time:** 3:08:37, ~17.9K steps/sec on Blackwell GPU
- **Checkpoint:** `~/mujoco_training/checkpoints/G1JoystickFlatTerrain-20260215-090148-full/params.pkl`
- **Scripts:** `train_g1.py`, `play_g1.py` (untested), `monitor.sh`
### GR00T-WBC Simulation (Complete)
- **Location:** GB10 (`~/GR00T-WholeBodyControl/`)
- **Status:** Walking robot in MuJoCo sim, keyboard controlled (w/s/a/d), 50 Hz, 3.5ms/iter
- **Launch:** `./launch_sim.sh` or `python3 run_g1_control_loop.py`
- **Viz:** NoMachine virtual desktop or web MJPEG viewer (`launch_with_web_viewer.py`)
### Network Bridge GB10 ↔ G1 (90% — needs final verification)
- **GB10 IP:** 10.0.0.68 (primary) + 192.168.123.100/24 (secondary on enP7s7)
- **Ping:** <1ms to 192.168.123.161 (locomotion) and .164 (Jetson)
- **DDS confirmed alive inside robot:** 342 multicast packets/4s from locomotion computer
- **Root cause found:** UFW firewall was blocking incoming DDS multicast
- **Fix applied:** User disabled UFW (`sudo ufw disable`)
- **NOT YET VERIFIED:** Robot was disconnected before we could confirm rt/lowstate data flows end-to-end
---
## What's Next (In Priority Order)
### 1. Verify DDS Connection (5 min)
When robot is reconnected to the network:
```bash
# On GB10:
sudo ip addr add 192.168.123.100/24 dev enP7s7 # if not persistent
sudo ufw status # confirm disabled or allows 192.168.123.0/24
cd ~/GR00T-WholeBodyControl && source .venv/bin/activate
python3 /tmp/dds_test.py
```
This tests ping → DDS multicast → SDK rt/lowstate in sequence.
### 2. Test GR00T-WBC on Real Robot (30 min)
Once DDS works:
1. Put robot on harness/stand
2. Enter debug mode: L2+R2 on wireless remote while robot is in damping state
3. Launch: `./launch_real.sh` (runs `--interface real`, auto-detects enP7s7)
4. Keyboard: `]` = enable walk, `w/s` = fwd/back, `a/d` = strafe, `q/e` = rotate
5. **WARNING:** No wireless remote control under GR00T-WBC — keyboard only
### 3. Install xr_teleoperate for Vision Pro (1-2 hrs)
Fastest path to telepresence — Vision Pro connects via Safari (WebXR), no app needed:
```bash
# On GB10:
cd ~ && git clone https://github.com/unitreerobotics/xr_teleoperate.git
# Follow their install instructions (needs Pinocchio, TeleVuer)
# Generate SSL certs for Vision Pro HTTPS connection
# Vision Pro opens: https://<gb10-ip>:8012
```
**Limitation:** xr_teleoperate bypasses GR00T-WBC and uses the stock controller for legs.
### 4. Build VisionProTeleop → GR00T-WBC Bridge (2-4 hrs)
Higher quality path — native visionOS app with RL-based balance:
1. Install VisionProTeleop's "Tracking Streamer" app on Vision Pro (App Store)
2. `pip install avp_stream` on GB10
3. Write bridge (~100-200 lines Python): `avp_stream` → wrist poses → Pinocchio IK → `ControlPolicy/upper_body_pose` ROS2 topic
4. Run alongside GR00T-WBC control loop
### 5. Unified WBC Training (Saved for Later)
Plan at `plans/eager-shimmying-raccoon.md`. Trains a policy that handles walking + arbitrary arm poses + balance simultaneously. ~400M steps, ~5.5 hrs on GB10.
---
## Key Architecture: GR00T-WBC
```
LOWER BODY (12 DOF legs) UPPER BODY (17 DOF waist+arms)
RL neural network (ONNX) Open-loop interpolation
Balance.onnx / Walk.onnx Targets from external source
Auto-switches based on Receives via ROS2 topic:
velocity command magnitude "ControlPolicy/upper_body_pose"
\ /
--> Merged 29-DOF joint positions --> DDS rt/lowcmd --> Motors
```
**Integration point for ANY telepresence/mocap system:** Publish to `ControlPolicy/upper_body_pose` ROS2 topic with:
- `target_upper_body_pose`: 17 joint angles (3 waist + 7 left arm + 7 right arm)
- `navigate_cmd`: `[vx, vy, wz]` velocity command (optional, overrides keyboard)
---
## Vision Pro Telepresence Options
| Path | What | Needs App? | Uses GR00T-WBC? | Status |
|------|------|:---:|:---:|--------|
| xr_teleoperate | WebXR via Safari | No | No (stock SDK) | Not yet installed |
| VisionProTeleop | Native visionOS app (MIT) | Yes (App Store) | Yes (via bridge) | Not yet installed |
| iPhone streamer | Socket.IO replication | Custom app | Yes (built-in) | Protocol documented |
**User has:** Apple Vision Pro + Meta Quest 3
**Decision:** Vision Pro selected as primary device
---
## Files on GB10
| File | Purpose |
|------|---------|
| `~/GR00T-WholeBodyControl/` | GR00T-WBC (patched for aarch64) |
| `~/GR00T-WholeBodyControl/launch_real.sh` | Real robot launcher (`--interface real`) |
| `~/GR00T-WholeBodyControl/launch_sim.sh` | Simulation launcher |
| `~/GR00T-WholeBodyControl/launch_with_web_viewer.py` | Sim + MJPEG web viewer |
| `~/GR00T-WholeBodyControl/gr00t_wbc/.../g1_29dof_gear_wbc_real.yaml` | Config copy with INTERFACE=enP7s7 |
| `~/mujoco_training/train_g1.py` | MuJoCo Playground RL training |
| `~/mujoco_training/play_g1.py` | Policy visualization (untested) |
| `~/mujoco_training/checkpoints/...full/params.pkl` | Trained locomotion policy |
| `/tmp/dds_test.py` | Quick DDS connectivity test |
| `/tmp/keysender.py` | ROS keyboard publisher for remote terminals |
---
## Credentials & Network
| Target | Address | User | Password |
|--------|---------|------|----------|
| GB10 | 10.0.0.68 | mitchaiet | (SSH key) |
| G1 Jetson | 192.168.123.164 | unitree | 123 |
| G1 Locomotion | 192.168.123.161 | — | Not SSH-accessible |
**GB10 interface:** enP7s7 (primary: 10.0.0.68 DHCP, secondary: 192.168.123.100/24 manual)
**DDS:** Domain 0, CycloneDDS 0.10.2, `ChannelFactoryInitialize(0, "enP7s7")`
**Firewall:** UFW was disabled. May need re-disabling after reboot.
---
## Context System Updates Made This Session
- `context/networking-comms.md` — Added §6: GB10 ↔ G1 DDS bridge (topology, setup, debugging findings, SSH creds)
- `context/open-questions.md` — Updated DDS relay question, updated Vision Pro mocap question, added "Vision Pro Telepresence Integration (Phase 5)" section, documented wireless remote answer
- `context/motion-retargeting.md` — Added §6: Apple Vision Pro telepresence paths (xr_teleoperate, VisionProTeleop, GR00T-WBC integration point)
- `context/learning-and-ai.md` — §8 MuJoCo Playground pipeline (added in prior session, still current)
- `CLAUDE.md` — Added Phase 4 to history
- `plans/eager-shimmying-raccoon.md` — Unified WBC training plan (from prior session, still current)

45
context/gb10-offboard-compute.md

@ -1,7 +1,7 @@
--- ---
id: gb10-offboard-compute id: gb10-offboard-compute
title: "Dell Pro Max GB10 — Offboard AI Compute" title: "Dell Pro Max GB10 — Offboard AI Compute"
status: proposed
status: established
source_sections: "Cross-referenced from git/spark context system" source_sections: "Cross-referenced from git/spark context system"
related_topics: [hardware-specs, networking-comms, learning-and-ai, sensors-perception, deployment-operations] related_topics: [hardware-specs, networking-comms, learning-and-ai, sensors-perception, deployment-operations]
key_equations: [] key_equations: []
@ -68,9 +68,52 @@ curl http://192.168.123.100:30000/v1/chat/completions \
Both systems are ARM64-native. Model files (.pt, .onnx, .gguf) trained on GB10 deploy directly to Orin NX without architecture conversion. Container images are interoperable (both aarch64). Both systems are ARM64-native. Model files (.pt, .onnx, .gguf) trained on GB10 deploy directly to Orin NX without architecture conversion. Container images are interoperable (both aarch64).
## 7. GR00T-WBC Deployment — Verified (2026-02-14) [T1]
GR00T-WBC runs successfully on the GB10. This is the primary use case: offboard whole-body control simulation and (eventually) real-time control relay.
**Network Configuration:**
- GB10 at `10.0.0.68` on LAN (not on robot's 192.168.123.0/24 subnet yet)
- SSH key auth configured for user `mitchaiet`
- Firewall (ufw) open for: SSH (22), VNC (5900), NoMachine (4000), Sunshine (47984-47990), web viewer (8080)
**Software Stack on GB10:**
- Ubuntu 24.04.3 LTS (Noble), kernel 6.14.0-1015-nvidia
- NVIDIA GB10 GPU, driver 580.95.05
- Python 3.12.3, ROS2 Jazzy
- GR00T-WBC cloned to `~/GR00T-WholeBodyControl` with Python venv
- CycloneDDS 0.10.4 (system) / 0.10.5 (ROS2) — ABI incompatible but works with FastRTPS RMW (default)
**Remote Access (headless — no monitor):**
- NoMachine (NX protocol) on port 4000 — best for interactive desktop
- Sunshine (NVENC game streaming) on ports 47984-47990 — installed but Moonlight client unstable on Win10
- x11vnc on port 5900 — works for basic desktop but cannot stream OpenGL content
- Xvfb virtual framebuffer on display :99 — used for headless rendering
**Critical Patches Applied:**
- Removed `<Tracing>` XML from `unitree_sdk2py/core/channel_config.py` (aarch64 buffer overflow fix)
- Created `ros2.pth` in venv for ROS2 package access
- Patched sync mode sim thread check in `run_g1_control_loop.py`
- Enabled auto-login in `/etc/gdm3/custom.conf`
**Launch Commands:**
```bash
# Simulation with viewer (from NoMachine terminal)
bash ~/GR00T-WholeBodyControl/launch_sim.sh --sync
# Headless simulation with web viewer
python ~/GR00T-WholeBodyControl/launch_with_web_viewer.py \
--interface sim --simulator mujoco --no-enable-onscreen \
--no-with-hands --sim-sync-mode --keyboard-dispatcher-type ros
# Keyboard sender (separate terminal)
python /tmp/keysender.py
```
## Key Relationships ## Key Relationships
- Computes for: [[learning-and-ai]] (training server) - Computes for: [[learning-and-ai]] (training server)
- Runs: [[whole-body-control]] (GR00T-WBC deployment verified)
- Connects via: [[networking-comms]] (Wi-Fi or Ethernet) - Connects via: [[networking-comms]] (Wi-Fi or Ethernet)
- Enhances: [[sensors-perception]] (large VLM inference) - Enhances: [[sensors-perception]] (large VLM inference)
- Deployed from: [[deployment-operations]] (trained models → real robot) - Deployed from: [[deployment-operations]] (trained models → real robot)

40
context/learning-and-ai.md

@ -167,6 +167,46 @@ Stage 5: Large pushes + concurrent upper-body task
This is the primary method for achieving the "always-on balance" goal. Papers arXiv:2505.20619 and arXiv:2511.07407 demonstrate this approach on real G1 hardware. See [[push-recovery-balance]] §3a for detailed parameters. This is the primary method for achieving the "always-on balance" goal. Papers arXiv:2505.20619 and arXiv:2511.07407 demonstrate this approach on real G1 hardware. See [[push-recovery-balance]] §3a for detailed parameters.
## 8. MuJoCo Playground Training Pipeline — Verified (2026-02-15) [T1]
GPU-parallelized RL training for G1 locomotion using MuJoCo Playground (Google DeepMind) on the Dell Pro Max GB10.
### Setup
- **Framework:** MuJoCo Playground (`playground` package from GitHub, not PyPI)
- **Environment:** `G1JoystickFlatTerrain` (29-DOF, 103-dim obs, velocity tracking with phase-based gait)
- **Training:** Brax PPO, JAX + CUDA 12 on Blackwell GPU, 8192 parallel MJX environments
- **Throughput:** ~17K steps/sec on GB10 Blackwell
### G1 Environment Details (from source inspection)
- **Observation (103 dims):** linvel(3) + gyro(3) + gravity(3) + command(3) + joint_pos-default(29) + joint_vel(29) + last_act(29) + phase(4)
- **Privileged state (165 dims):** state(103) + clean sensors + actuator force + contact + feet velocity
- **Actions:** 29 joint position targets (all DOF), residual from default pose, scaled by 0.25
- **Control rate:** 50 Hz (0.02s ctrl_dt), physics at 500 Hz (0.002s sim_dt)
- **Push perturbations:** Enabled by default (0.1-2.0 m/s velocity impulse, every 5-10s)
- **23 reward terms** including velocity tracking, gait phase, orientation, foot slip, joint deviation
- **Domain randomization:** Friction (0.4-1.0), mass (±10%), torso mass offset (±1kg), armature (1.0-1.05x)
### Also Available
- `G1JoystickRoughTerrain` — same env with procedural terrain
- H1 gait tracking environments — reference pattern for extending G1 with tracking rewards
- No existing whole-body tracking env for G1 (only H1 and Spot have gait tracking variants)
### Training Results (locomotion-only baseline)
- 5M steps (tiny): 6 min 41 sec, reward -6.4 → -2.8
- 200M steps (full): reward progression -6.4 → +8.8 at 117M steps (training in progress)
### Planned: Unified Whole-Body Control Training
Research direction: fork G1JoystickFlatTerrain to add upper body pose tracking for telepresence (Apple Vision Pro mocap + joystick locomotion). See `plans/eager-shimmying-raccoon.md` for full plan. Approach follows ExBody/ExBody2 paradigm: decouple velocity tracking (lower body) from keypoint tracking (upper body), 4-stage curriculum, ~400M steps.
### Key Open-Source Repos for G1 Whole-Body RL
| Repo | Approach | G1 Validated? |
|------|----------|---------------|
| MuJoCo Playground | GPU-parallelized MJX training, native G1 env | Yes [T1] |
| BFM-Zero (LeCAR-Lab) | Foundation model, motion tracking mode | Yes [T1] |
| BeyondMimic (HybridRobotics) | Whole-body tracking from LAFAN1 | Yes (claimed) |
| H2O / OmniH2O (LeCAR-Lab) | Real-time teleoperation | Humanoid (not G1-specific) |
| ExBody2 (UC San Diego) | Expressive whole-body with velocity decoupling | Humanoid |
## Key Relationships ## Key Relationships
- Trains in: [[simulation]] (MuJoCo, Isaac Lab, Isaac Gym) - Trains in: [[simulation]] (MuJoCo, Isaac Lab, Isaac Gym)
- Deploys via: [[sdk-programming]] (unitree_sdk2 DDS interface) - Deploys via: [[sdk-programming]] (unitree_sdk2 DDS interface)

28
context/motion-retargeting.md

@ -259,6 +259,34 @@ SMPL (Skinned Multi-Person Linear model) is the standard representation for huma
| rofunc | Robot learning from human demos + retargeting | Python | MIT | | rofunc | Robot learning from human demos + retargeting | Python | MIT |
| MuJoCo Menagerie | G1 model (g1.xml) for IK/simulation | MJCF | BSD-3 | | MuJoCo Menagerie | G1 model (g1.xml) for IK/simulation | MJCF | BSD-3 |
## 6. Apple Vision Pro Telepresence Paths (Researched 2026-02-15) [T1/T2]
### Available Integration Options
| Path | Approach | App Required? | GR00T-WBC Compatible? | Retargeting |
|------|----------|:---:|:---:|---|
| xr_teleoperate | WebXR via Safari | No (browser) | No (uses stock SDK) | Pinocchio IK |
| VisionProTeleop | Native visionOS app | Yes (App Store / open-source) | Yes (via bridge) | Custom (flexible) |
| iPhone streamer | Socket.IO protocol | Custom visionOS app | Yes (built-in) | Pinocchio IK in GR00T-WBC |
### xr_teleoperate (Unitree Official)
- Vision Pro connects via Safari to `https://<host>:8012` (WebXR)
- TeleVuer (Python, built on Vuer) serves the 3D interface
- WebSocket for tracking data, WebRTC for video feedback
- Pinocchio IK solves wrist poses → G1 arm joint angles
- Supports G1_29 and G1_23 variants
- **Limitation:** Bypasses GR00T-WBC — sends motor commands directly via DDS
### VisionProTeleop (MIT, Open-Source)
- Native visionOS app "Tracking Streamer" — on App Store + source on GitHub
- Python library `avp_stream` receives data via gRPC
- 25 finger joints/hand, head pose, wrist positions (native ARKit, better than WebXR)
- Robot-agnostic — needs a bridge to publish to GR00T-WBC's `ControlPolicy/upper_body_pose` ROS2 topic
- **Best path for GR00T-WBC integration with RL-based balance**
### GR00T-WBC Integration Point
The single integration point is the `ControlPolicy/upper_body_pose` ROS2 topic. Any source that publishes `target_upper_body_pose` (17 joint angles: 3 waist + 7 left arm + 7 right arm) and optionally `navigate_cmd` (velocity `[vx, vy, wz]`) can drive the robot. The `InterpolationPolicy` smooths targets before execution.
## Key Relationships ## Key Relationships
- Requires: [[joint-configuration]] (target skeleton — DOF, joint limits, link lengths) - Requires: [[joint-configuration]] (target skeleton — DOF, joint limits, link lengths)
- Executed via: [[whole-body-control]] (WBC provides balance during playback) - Executed via: [[whole-body-control]] (WBC provides balance during playback)

37
context/networking-comms.md

@ -111,8 +111,45 @@ Subscribers (user) ←── rt/dex3/*/state ←── Hand Controller (hand sta
**Critical:** For real-time control, use wired Ethernet or develop directly on the Jetson to minimize latency. WiFi adds variable latency that can affect control stability. [T2] **Critical:** For real-time control, use wired Ethernet or develop directly on the Jetson to minimize latency. WiFi adds variable latency that can affect control stability. [T2]
## 6. Offboard Compute — GB10 to G1 DDS Bridge (Verified 2026-02-15) [T1]
Connecting the Dell Pro Max GB10 to the G1's internal DDS network via Ethernet switch.
### Network Topology (Verified)
```
[G1 Robot Internal Switch]
├── Locomotion Computer: 192.168.123.161 (eth0)
├── Jetson Orin NX: 192.168.123.164 (eth0) + 10.0.0.64 (wlan0)
└── External Ethernet port ──► User's network switch
GB10: 10.0.0.68 (primary, DHCP)
+ 192.168.123.100/24 (secondary, manual)
Interface: enP7s7
```
### Setup Steps
1. Add secondary IP: `sudo ip addr add 192.168.123.100/24 dev enP7s7`
2. Verify ping: `ping 192.168.123.161` (should be <1ms)
3. **Critical:** Disable or allow DDS through firewall: `sudo ufw allow from 192.168.123.0/24`
### DDS Debugging Findings
- **DDS multicast (239.255.0.1:7400) IS active** on the robot's internal network — 342 packets/4s confirmed from Jetson
- Locomotion computer (192.168.123.161) publishes discovery continuously
- **UFW firewall was the root cause** of DDS failure — it blocks incoming UDP multicast by default
- Once firewall allows 192.168.123.0/24 traffic, DDS discovery and `rt/lowstate` subscription should work
- Use `ChannelFactoryInitialize(0, "enP7s7")` or `--interface real` (auto-detects 192.168.123.x interface)
### SSH Credentials (Verified)
- **Jetson:** `ssh unitree@192.168.123.164` — password: `123` (NOT "unitree") [T1]
- **Locomotion computer (192.168.123.161):** Not user-accessible via SSH
### Tools on GB10
- `/tmp/dds_test.py` — Quick DDS connectivity test (ping → multicast → SDK rt/lowstate)
- `~/GR00T-WholeBodyControl/launch_real.sh` — Launches GR00T-WBC with `--interface real`
## Key Relationships ## Key Relationships
- Transports: [[sdk-programming]] (SDK commands and data via DDS) - Transports: [[sdk-programming]] (SDK commands and data via DDS)
- Transports: [[ros2-integration]] (ROS2 topics via same DDS layer) - Transports: [[ros2-integration]] (ROS2 topics via same DDS layer)
- Affects: [[deployment-operations]] (remote operation reliability) - Affects: [[deployment-operations]] (remote operation reliability)
- Connected to: [[sensors-perception]] (LiDAR network address) - Connected to: [[sensors-perception]] (LiDAR network address)
- Connected to: [[gb10-offboard-compute]] (offboard AI compute via DDS bridge)

99
context/open-questions.md

@ -76,8 +76,9 @@ This file catalogs what we know, what we don't know, and what would resolve the
- _Would resolve:_ Testing or official confirmation. - _Would resolve:_ Testing or official confirmation.
- **Q:** What is the exact RL policy observation/action space? - **Q:** What is the exact RL policy observation/action space?
- _Partial:_ Inputs include IMU + joint encoders. Exact dimensions not documented.
- _Would resolve:_ Official policy architecture documentation or code inspection.
- _Partial:_ Stock `ai_sport` policy obs/action not documented. MuJoCo Playground G1JoystickFlatTerrain: 103-dim state obs, 165-dim privileged obs, 29-dim action. GR00T-WBC: 516-dim obs (86×6 history), 15-dim action (lower body only).
- _Update (2026-02-15):_ MuJoCo Playground env source inspected. Full obs breakdown documented in `context/learning-and-ai.md` §8.
- _Would resolve:_ Stock `ai_sport` binary analysis (still unknown).
- **Q:** Can a custom locomotion policy be deployed natively to the RK3588 locomotion computer? - **Q:** Can a custom locomotion policy be deployed natively to the RK3588 locomotion computer?
- _Partial:_ Root access achieved via BLE exploits (UniPwn, FreeBOT). The `ai_sport` binary is the stock policy. Nobody has publicly documented replacing it. Config files use FMX encryption (partially cracked). - _Partial:_ Root access achieved via BLE exploits (UniPwn, FreeBOT). The `ai_sport` binary is the stock policy. Nobody has publicly documented replacing it. Config files use FMX encryption (partially cracked).
@ -89,8 +90,9 @@ This file catalogs what we know, what we don't know, and what would resolve the
### Open ### Open
- **Q:** What are the optimal domain randomization parameters for G1 sim-to-real? - **Q:** What are the optimal domain randomization parameters for G1 sim-to-real?
- _Partial:_ Standard parameters (friction, mass, noise) used. Optimal ranges not published.
- _Would resolve:_ Research papers with ablation studies.
- _Partial:_ MuJoCo Playground defaults: friction U(0.4,1.0), body mass ±10%, torso mass offset ±1kg, DOF armature 1.0-1.05x, DOF frictionloss 0.5-2.0x. Whether these are optimal or need tuning for real G1 is unconfirmed.
- _Update (2026-02-15):_ Inspected MuJoCo Playground `randomize.py` source. Parameters documented in `context/learning-and-ai.md` §8.
- _Would resolve:_ Sim-to-real transfer testing on physical G1.
--- ---
@ -123,11 +125,13 @@ This file catalogs what we know, what we don't know, and what would resolve the
- _Would resolve:_ Physical push testing with force measurement, or Unitree internal documentation. - _Would resolve:_ Physical push testing with force measurement, or Unitree internal documentation.
- **Q:** Can GR00T-WBC run at 500 Hz on the Jetson Orin NX? - **Q:** Can GR00T-WBC run at 500 Hz on the Jetson Orin NX?
- _Partial:_ Orin NX has 100 TOPS. RL inference typically < 1ms. But GR00T-WBC may have WBC optimization overhead.
- _Would resolve:_ Benchmarking GR00T-WBC on Orin NX, or NVIDIA documentation on compute requirements.
- _Partial:_ On GB10 (much faster than Orin NX), loop runs at ~3.5ms/iteration at 50 Hz in sync mode. Orin NX benchmarking still needed.
- _Update (2026-02-14):_ GR00T-WBC default is 50 Hz control, not 500 Hz. At 50 Hz on GB10, it uses only 17.5% of time budget. Orin NX likely feasible at 50 Hz but 500 Hz unconfirmed.
- _Would resolve:_ Benchmarking GR00T-WBC on actual Orin NX hardware.
- **Q:** What is the end-to-end latency from mocap capture to robot execution? - **Q:** What is the end-to-end latency from mocap capture to robot execution?
- _Partial:_ DDS latency ~2ms. XR teleoperate latency not documented. Video-based pose estimation adds 30-100ms. - _Partial:_ DDS latency ~2ms. XR teleoperate latency not documented. Video-based pose estimation adds 30-100ms.
- _Update (2026-02-14):_ GR00T-WBC teleop accepts upper-body commands via `/ControlPolicy/upper_body_pose` ROS topic. Pico VR is the primary tested device. Camera-based mocap not yet integrated.
- _Would resolve:_ End-to-end timing measurement with each mocap source. - _Would resolve:_ End-to-end timing measurement with each mocap source.
- **Q:** Does residual policy overlay work with the proprietary locomotion computer, or does it require full replacement? - **Q:** Does residual policy overlay work with the proprietary locomotion computer, or does it require full replacement?
@ -144,8 +148,91 @@ This file catalogs what we know, what we don't know, and what would resolve the
--- ---
## GR00T-WBC Deployment (Phase 3)
### Open
- **Q:** How does GR00T-WBC Walk policy compare to stock G1 controller for push recovery?
- _Partial:_ Walk policy demonstrated working in sim (walking, turning, strafing). Push recovery not yet quantified.
- _Would resolve:_ Side-by-side push testing on real robot with force measurement.
- **Q:** What is the exact training recipe for the pre-trained ONNX policies (Balance, Walk)?
- _Partial:_ PPO via RSL-RL in Isaac Lab. MLP [512, 256, 128]. Domain randomization. Zero-shot transfer. Exact reward function and perturbation curriculum not published by NVIDIA.
- _Would resolve:_ NVIDIA publishing training code, or reverse-engineering from ONNX model + observation/action analysis.
- **Q:** Can GR00T-WBC relay real-time control from GB10 to G1 over the network?
- _Partial:_ Network bridge established (GB10 at 192.168.123.100, ping <1ms to .161/.164). DDS multicast confirmed active inside robot. UFW firewall was blocking now resolved. Full rt/lowstate subscription test pending (robot was disconnected before final verification).
- _Update (2026-02-15):_ UFW firewall identified as root cause. `sudo ufw allow from 192.168.123.0/24` fixes it. DDS test script ready at `/tmp/dds_test.py`. Launch script at `~/GR00T-WholeBodyControl/launch_real.sh`.
- _Would resolve:_ Final DDS data verification with robot reconnected, then `launch_real.sh` test.
- **Q:** What camera-based mocap solution integrates best with GR00T-WBC's upper body teleop?
- _Partial:_ GR00T-WBC supports Pico VR, LeapMotion, HTC Vive, iPhone natively. Camera-based (MediaPipe, OpenPose) not built-in but could publish to the same ROS topic.
- _Update (2026-02-15):_ **Apple Vision Pro selected as primary telepresence device.** Two integration paths identified:
- **Path 1 (fastest):** Unitree `xr_teleoperate` — Vision Pro connects via Safari WebXR, no app needed. Uses TeleVuer/Vuer web server. But bypasses GR00T-WBC (uses stock controller).
- **Path 2 (best quality):** VisionProTeleop (MIT, open-source native visionOS app "Tracking Streamer") → `avp_stream` Python lib (gRPC) → bridge to GR00T-WBC's `ControlPolicy/upper_body_pose` ROS2 topic. Enables RL-based balance via GR00T-WBC.
- _Would resolve:_ Implement one of the paths and measure tracking quality + latency.
- **Q:** Why does MuJoCo's GLFW passive viewer freeze on virtual/remote displays after a few seconds?
- _Partial:_ Observed on both Xvfb+VNC and NoMachine. GLFW event loop thread appears to stall. X11 framebuffer screenshots confirm rendering IS happening intermittently. ffmpeg x11grab also shows stalling.
- _Would resolve:_ GLFW debug logging, or switching to MuJoCo offscreen renderer + custom display pipeline.
## Unified WBC Training (Phase 4 Research Direction)
### Open
- **Q:** Can a unified policy (locomotion + upper body tracking) maintain balance when mocap drives arms to extreme positions?
- _Partial:_ ExBody/ExBody2 validate the approach on other humanoids. No published results on G1 specifically.
- _Plan:_ Fork G1JoystickFlatTerrain, add upper body tracking reward, 4-stage curriculum (400M steps, ~5.5 hrs on GB10). See `plans/eager-shimmying-raccoon.md`.
- _Would resolve:_ Training the unified policy and evaluating push survival with arbitrary upper body poses.
- **Q:** Are procedural upper body targets sufficient for training, or is AMASS motion data required?
- _Partial:_ MuJoCo Playground uses only parametric gait generation (no AMASS). ExBody2 uses motion capture. Procedural (uniform random + sinusoidal) may cover the config space but miss real-world motion correlations.
- _Would resolve:_ Compare tracking quality: procedural-trained vs. AMASS-trained on real AVP data.
- **Q:** What is the Apple Vision Pro → G1 retargeting latency and accuracy?
- _Partial:_ AVP provides full hand/body tracking. Multiple integration paths researched.
- _Update (2026-02-15):_ xr_teleoperate uses Pinocchio IK for retargeting (WebXR wrist poses → G1 arm joints). VisionProTeleop provides native ARKit with 25 finger joints/hand via gRPC. GR00T-WBC's `InterpolationPolicy` accepts 17-DOF upper body targets and interpolates smoothly.
- _Would resolve:_ Build AVP bridge, measure end-to-end latency and joint angle accuracy.
## Vision Pro Telepresence Integration (Phase 5)
### Open
- **Q:** Which Vision Pro integration path works best with GR00T-WBC?
- _Partial:_ Two paths identified. Path 1 (xr_teleoperate WebXR) is fastest but bypasses GR00T-WBC. Path 2 (VisionProTeleop native app → avp_stream → GR00T-WBC) is higher quality.
- _Would resolve:_ Prototype both, compare latency and tracking quality.
- **Q:** Can the GR00T-WBC Walk policy maintain balance with Vision Pro-driven arm poses?
- _Partial:_ The Walk ONNX policy receives upper body joint angles as observation input and can compensate. But it was NOT trained with arbitrary arm configurations — conservative motions likely fine, extreme poses may destabilize.
- _Related:_ Unified WBC training plan (plans/eager-shimmying-raccoon.md) would train specifically for this.
- _Would resolve:_ Test with real Vision Pro driving arms while walking. If unstable, proceed with unified training plan.
- **Q:** Does the Unitree wireless remote work under GR00T-WBC?
- **A:** No. GR00T-WBC takes over low-level motor control via rt/lowcmd, bypassing the stock controller that reads the remote. The rt/wirelesscontroller DDS topic is still published by the robot but nothing in GR00T-WBC subscribes to it on real hardware. Keyboard control (w/s/a/d/q/e) is the built-in alternative. [T1 — Source inspection, 2026-02-15]
---
## Resolved ## Resolved
### MuJoCo Playground Training (Resolved)
- **Q:** Can MuJoCo Playground train G1 policies on Blackwell (sm_121)?
- **A:** Yes. JAX 0.9.0.1 + CUDA 12 works. JIT compilation takes ~37s per operation first time, then cached. 8192 parallel envs, ~17K steps/sec. Full 200M-step training in ~3.5 hours. [T1 — Verified on GB10, 2026-02-15]
- **Q:** What is the reward function for G1JoystickFlatTerrain?
- **A:** 23 reward terms. Key ones: tracking_lin_vel(1.0), tracking_ang_vel(0.75), feet_phase(1.0), feet_air_time(2.0), orientation(-2.0), termination(-100.0). Push perturbations enabled (0.1-2.0 m/s, every 5-10s). Full breakdown in `context/learning-and-ai.md` §8. [T1 — Source inspection, 2026-02-15]
- **Q:** What training time estimates for different policy types on GB10?
- **A:** Locomotion-only: 5M steps in 7 min (sanity), 200M in ~3.5 hrs. Unified WBC (estimated): 400M in ~5.5 hrs. [T1 — Measured on GB10, 2026-02-15]
### GR00T-WBC Deployment (Resolved)
- **Q:** Does GR00T-WBC run on aarch64 (ARM64)?
- **A:** Yes, with patches. CycloneDDS `<Tracing>` XML causes buffer overflow on aarch64 — fix by removing the section. ROS2 Jazzy + Python 3.12 on Ubuntu 24.04 aarch64 works. [T1 — Verified on GB10, 2026-02-14]
- **Q:** Can ROS2 Jazzy and Unitree SDK2 coexist in the same process?
- **A:** Yes. CycloneDDS system lib (0.10.4) and ROS2 lib (0.10.5) are ABI-incompatible, but using the default FastRTPS RMW avoids the conflict. Second ChannelFactory init fails gracefully. [T1 — Verified on GB10, 2026-02-14]
- **Q:** What are the ONNX policy observation/action dimensions?
- **A:** Both Balance and Walk: 516-dim observation (proprioception + history), 15-dim action (lower body joint targets). [T1 — ONNX model inspection, 2026-02-14]
### Hardware & Mechanical (Resolved) ### Hardware & Mechanical (Resolved)
- **Q:** What are the exact DOF configurations for each G1 variant? - **Q:** What are the exact DOF configurations for each G1 variant?

32
context/whole-body-control.md

@ -114,6 +114,38 @@ The most G1-relevant WBC framework. Open-source, designed specifically for Unitr
### Why This Matters for Mocap + Balance ### Why This Matters for Mocap + Balance
GR00T-WBC is the most direct path to the user's goal: the locomotion policy maintains balance (including push recovery if trained with perturbations) while the upper body tracks mocap reference trajectories. The two are coordinated through the WBC layer. GR00T-WBC is the most direct path to the user's goal: the locomotion policy maintains balance (including push recovery if trained with perturbations) while the upper body tracks mocap reference trajectories. The two are coordinated through the WBC layer.
### Deployment on Dell Pro Max GB10 — Verified (2026-02-14) [T1]
GR00T-WBC has been **successfully deployed and tested** on the Dell Pro Max GB10 (NVIDIA Grace Blackwell, aarch64, Ubuntu 24.04). Key findings:
**Pre-trained ONNX Policies:**
- `GR00T-WholeBodyControl-Balance.onnx` — standing balance (15 lower-body joint targets)
- `GR00T-WholeBodyControl-Walk.onnx` — locomotion with velocity commands
- Both: 516-dim observation → 15-dim action. Pre-trained by NVIDIA (training code not open-sourced).
- Training: PPO via RSL-RL in Isaac Lab, domain randomization, zero-shot sim-to-real. Exact reward function and perturbation curriculum not published.
**Performance on GB10:**
- ~3.5 ms per control loop iteration at 50 Hz (sync mode) — only 17.5% of time budget
- 401% CPU usage (4 cores) — MuJoCo physics dominates
- Both Balance and Walk policies load and execute successfully
- Robot walks, turns, and strafes in simulation via keyboard velocity commands
**Critical Fixes Required for GB10 (aarch64):**
1. **CycloneDDS buffer overflow:** The `<Tracing>` XML section in `unitree_sdk2py/core/channel_config.py` triggers a glibc FORTIFY_SOURCE buffer overflow on aarch64. Fix: remove the `<Tracing>` section entirely. (See [[dev-environment]] for patch details.)
2. **ROS2 Python path:** venv needs `.pth` file pointing to `/opt/ros/jazzy/lib/python3.12/site-packages/`
3. **ROS2 shared libraries:** `export LD_LIBRARY_PATH=/opt/ros/jazzy/lib:$LD_LIBRARY_PATH`
4. **Sync mode bug:** `run_g1_control_loop.py` checks for sim thread in sync mode where none exists. Patch: add `not config.sim_sync_mode` guard.
**Keyboard Control:**
- `sshkeyboard` library fails in remote terminals (SSH, NoMachine). Workaround: use `--keyboard-dispatcher-type ros` and publish to `/keyboard_input` ROS topic from a separate process.
- Keys: `]`=enable Walk, `w/s`=fwd/back, `a/d`=strafe, `q/e`=rotate, `z`=stop, `backspace`=reset
**Visualization:**
- GLFW passive viewer freezes on virtual/remote displays (Xvfb, NoMachine) after a few seconds
- VNC (x11vnc) cannot capture OpenGL framebuffer updates
- Working solution: NoMachine virtual desktop (NX protocol) — viewer works initially but GLFW stalls
- Best solution: Web-based MJPEG streaming via MuJoCo offscreen renderer (bypasses all X11/GLFW issues)
## 4. Pinocchio + TSID (Model-Based WBC) ## 4. Pinocchio + TSID (Model-Based WBC)
An alternative to RL-based WBC using classical optimization. [T1 — Established framework] An alternative to RL-based WBC using classical optimization. [T1 — Established framework]

108
phases/phase-3-groot-wbc-deployment.md

@ -0,0 +1,108 @@
# Phase 3: GR00T-WBC Deployment on GB10
**Date:** 2026-02-14
**Goal:** Deploy and validate GR00T-WBC whole-body control framework on Dell Pro Max GB10 for G1 humanoid simulation
---
## Summary
Successfully deployed NVIDIA's GR00T-WholeBodyControl on the GB10 (aarch64, Ubuntu 24.04, Blackwell GPU). Identified and fixed 4 critical blockers. Achieved working simulation with walking robot controlled via keyboard. Established remote visualization via NoMachine and web-based MJPEG streaming.
## Key Achievements
1. **GR00T-WBC running in simulation** — Both Balance and Walk ONNX policies loaded and executing at 50 Hz
2. **Walking robot** — WASD keyboard control verified: forward, backward, strafe, rotate
3. **Performance validated** — 3.5 ms/iteration (17.5% of 20ms budget) in sync mode
4. **Remote visualization** — NoMachine virtual desktop + web MJPEG viewer
5. **4 critical bugs identified and fixed** on aarch64 platform
## Critical Fixes
### Fix 1: CycloneDDS aarch64 Buffer Overflow (ROOT CAUSE)
- **File:** `unitree_sdk2py/core/channel_config.py`
- **Problem:** `<Tracing>` XML section triggers glibc FORTIFY_SOURCE buffer overflow specifically on ARM64
- **Discovery:** Isolated via `Domain(0)` (works) vs `Domain(0, config_with_tracing)` (crashes) vs `Domain(0, config_without_tracing)` (works)
- **Fix:** Remove `<Tracing>` section from both `ChannelConfigHasInterface` and `ChannelConfigAutoDetermine`
- **Impact:** Without this fix, nothing in the Unitree SDK works on aarch64
### Fix 2: ROS2 Python Package Path
- **File:** `.venv/lib/python3.12/site-packages/ros2.pth` (created)
- **Problem:** ROS2 packages at `/opt/ros/jazzy/lib/python3.12/site-packages/` not visible in venv
- **Fix:** Create `.pth` file with that path
- **Impact:** sensor_msgs, std_msgs, rclpy all accessible
### Fix 3: ROS2 Shared Library Path
- **Problem:** rclpy native libraries (librcl_action.so etc.) not found
- **Fix:** `export LD_LIBRARY_PATH=/opt/ros/jazzy/lib:$LD_LIBRARY_PATH`
### Fix 4: Sync Mode Sim Thread Check
- **File:** `run_g1_control_loop.py` line 211
- **Problem:** Code checks `env.sim.sim_thread.is_alive()` but in sync mode no thread exists
- **Fix:** Add `not config.sim_sync_mode` guard to the check
## Architecture Findings
### Pre-trained ONNX Policies
- **Balance:** 516-dim obs → 15-dim action (lower body joint targets)
- **Walk:** Same dimensions, velocity-conditioned locomotion
- **Training:** PPO in Isaac Lab, domain randomization, zero-shot sim-to-real (exact recipe not published)
- **Not mocap-based:** Command-conditioned (velocity tracking), not motion imitation
### Keyboard Control Architecture
- `KeyboardDispatcher` (raw) uses `sshkeyboard` library — fails in remote terminals
- `ROSKeyboardDispatcher` subscribes to `/keyboard_input` topic — works via separate publisher
- Dispatchers call `handle_keyboard_button()` on registered listeners (policy, env, e-stop)
- Walk policy: `]`=enable, `w/s`=fwd/back (+/-0.2 m/s per press), `a/d`=strafe, `q/e`=rotate, `z`=zero
### Visualization Challenges (Headless GB10)
Attempted 5 approaches for remote visualization:
| Approach | Result | Issue |
|----------|--------|-------|
| x11vnc on physical display :0 | Black screen | No monitor connected, GPU output disabled |
| x11vnc on Xvfb :99 | Static frame | OpenGL renders to own framebuffer, x11vnc reads X11 pixmap |
| NoMachine physical shadow | Black screen | Shadowing black physical display |
| NoMachine virtual desktop | Works but GLFW freezes | GLFW passive viewer thread stalls after seconds |
| Web MJPEG streaming | Best solution | Offscreen renderer → JPEG → HTTP, no X11 dependency |
### DDS/ROS2 Coexistence
- CycloneDDS system (0.10.4) and ROS2 (0.10.5) are ABI-incompatible
- FastRTPS is the default RMW in ROS2 Jazzy — avoids CycloneDDS version conflict
- Second ChannelFactory init from BaseSimulator fails gracefully (try/except)
- `CYCLONEDDS_URI` environment variable controls Unitree SDK's DDS config
## Software Installed on GB10
| Package | Purpose |
|---------|---------|
| GR00T-WBC | Whole-body control framework |
| NoMachine 9.3.7 | Remote desktop (NX protocol) |
| Sunshine 2025.924 | Game streaming (NVENC) — installed but Moonlight client unstable |
| x11vnc | VNC server — limited use (can't capture OpenGL) |
| Xvfb | Virtual framebuffer for headless rendering |
| openbox | Lightweight window manager for Xvfb |
## Files Created/Modified on GB10
| File | Action | Purpose |
|------|--------|---------|
| `channel_config.py` | Modified | Removed Tracing XML (buffer overflow fix) |
| `.venv/.../ros2.pth` | Created | ROS2 Python path |
| `run_g1_control_loop.py` | Modified | Sync mode sim thread guard |
| `launch_sim.sh` | Created | Convenience launcher |
| `launch_with_web_viewer.py` | Created | Launcher with web MJPEG streaming |
| `web_viewer.py` | Created | MuJoCo offscreen → MJPEG HTTP server |
| `/tmp/keysender.py` | Created | ROS keyboard publisher for remote control |
| `/etc/gdm3/custom.conf` | Modified | Auto-login enabled |
| `/etc/x11vnc.pass` | Modified | VNC password set |
| Firewall (ufw) | Modified | Opened ports 5900, 4000, 47984-47990, 8080 |
## Next Steps
1. **Get web viewer working** — test `launch_with_web_viewer.py` for smooth browser-based visualization
2. **Connect mocap** — webcam → pose estimation → upper body teleop commands
3. **Network bridge to G1** — connect GB10 to robot's 192.168.123.0/24 subnet
4. **Deploy to real robot** — switch `--interface sim` to `--interface real`
5. **Push recovery testing** — benchmark stock vs. GR00T-WBC push robustness
6. **Custom RL training** — use Isaac Lab to train custom policies with perturbation curriculum

71
reference/glossary.yaml

@ -746,3 +746,74 @@ terms:
typical_range: null typical_range: null
related_terms: ["teleoperation", "lerobot", "dex3_1"] related_terms: ["teleoperation", "lerobot", "dex3_1"]
related_topics: ["manipulation", "learning-and-ai"] related_topics: ["manipulation", "learning-and-ai"]
# ============================================================
# GB10 & DEPLOYMENT (Phase 3)
# ============================================================
- term: "dell_pro_max_gb10"
full_name: "Dell Pro Max Desktop GB10"
definition: |
NVIDIA Grace Blackwell workstation (aarch64). 128 GB unified LPDDR5X,
1000 TFLOPS FP4, sm_121. Used as offboard AI brain for G1.
GR00T-WBC deployed and verified. Ubuntu 24.04, driver 580.95.05.
unit: null
typical_range: "1000 TFLOPS (FP4)"
related_terms: ["groot_wbc", "jetson_orin_nx"]
related_topics: ["gb10-offboard-compute", "whole-body-control"]
- term: "onnx_policy"
full_name: "ONNX Neural Network Policy"
definition: |
Pre-trained RL policy exported in ONNX format for deployment.
GR00T-WBC ships Balance and Walk policies: 516-dim observation
(proprioception + history) → 15-dim action (lower body joint targets).
Trained with PPO in Isaac Lab. Inference via onnxruntime.
unit: null
typical_range: "516-dim obs → 15-dim action, < 1ms inference"
related_terms: ["groot_wbc", "sim_to_real", "gait_conditioned_rl"]
related_topics: ["whole-body-control", "learning-and-ai"]
- term: "nomachine"
full_name: "NoMachine Remote Desktop"
definition: |
Remote desktop software using NX protocol. Used for GB10 headless
access. Creates virtual desktops on headless servers. Better than
VNC for GPU content but GLFW passive viewer still stalls.
unit: null
typical_range: "Port 4000 (NX protocol)"
related_terms: ["dell_pro_max_gb10"]
related_topics: ["gb10-offboard-compute"]
- term: "ppo"
full_name: "Proximal Policy Optimization"
definition: |
RL algorithm from OpenAI, standard for locomotion policy training.
Used by NVIDIA (via RSL-RL) to train GR00T-WBC Balance and Walk
policies. Clip-based surrogate objective for stable training.
unit: null
typical_range: null
related_terms: ["gait_conditioned_rl", "curriculum_learning", "sim_to_real"]
related_topics: ["learning-and-ai", "whole-body-control"]
- term: "rsl_rl"
full_name: "RSL-RL (Robotic Systems Lab RL)"
definition: |
GPU-optimized RL training library from ETH Zurich RSL. Used with
Isaac Lab for locomotion policy training. Implements PPO optimized
for parallel simulation. Standard for legged robot RL.
unit: null
typical_range: null
related_terms: ["ppo", "curriculum_learning"]
related_topics: ["learning-and-ai"]
- term: "isaac_lab"
full_name: "NVIDIA Isaac Lab"
definition: |
NVIDIA's robot learning framework built on Isaac Sim. GPU-parallelized
physics simulation for large-scale RL training. Used to train GR00T-WBC
policies. Supports domain randomization, terrain generation, and
whole-body control training workflows.
unit: null
typical_range: "4096+ parallel environments on single GPU"
related_terms: ["ppo", "rsl_rl", "groot_wbc", "sim_to_real"]
related_topics: ["simulation", "learning-and-ai", "whole-body-control"]

98
scripts/dds_test.py

@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""Quick DDS connectivity test for G1 robot.
Run this on the GB10 after connecting the robot to the network.
Usage: python3 dds_test.py [interface_name]
"""
import sys
import time
import os
import subprocess
import socket
import struct
def main():
# Auto-detect or use provided interface
iface = sys.argv[1] if len(sys.argv) > 1 else None
if iface is None:
result = subprocess.run(['ip', '-o', '-4', 'addr', 'show'], capture_output=True, text=True)
for line in result.stdout.splitlines():
if '192.168.123' in line:
iface = line.split()[1]
break
if iface is None:
print('ERROR: No interface with 192.168.123.x found. Add one with:')
print(' sudo ip addr add 192.168.123.100/24 dev <interface>')
sys.exit(1)
# Get our IP on that interface
result = subprocess.run(['ip', '-o', '-4', 'addr', 'show', iface], capture_output=True, text=True)
our_ip = None
for word in result.stdout.split():
if '192.168.123' in word:
our_ip = word.split('/')[0]
break
print(f'[1/4] Interface: {iface} ({our_ip})', flush=True)
# Step 1: Ping test
print('[2/4] Ping test...', flush=True)
for ip in ['192.168.123.161', '192.168.123.164']:
r = subprocess.run(['ping', '-c', '1', '-W', '1', ip], capture_output=True, text=True)
status = 'OK' if r.returncode == 0 else 'FAIL'
print(f' {ip}: {status}', flush=True)
# Step 2: Multicast receive test
print('[3/4] DDS multicast discovery (5s)...', flush=True)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', 7400))
mreq = struct.pack('4s4s', socket.inet_aton('239.255.0.1'), socket.inet_aton(our_ip))
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
sock.settimeout(1.0)
count = 0
sources = set()
start = time.time()
while time.time() - start < 5:
try:
data, addr = sock.recvfrom(4096)
count += 1
sources.add(addr[0])
except socket.timeout:
pass
sock.close()
robot_sources = {s for s in sources if s.startswith('192.168.123')}
print(f' {count} packets from {sources}', flush=True)
if robot_sources:
print(f' Robot DDS discovered: {robot_sources}', flush=True)
else:
print(' WARNING: No DDS multicast from robot!', flush=True)
print(' Check: sudo ufw disable OR sudo ufw allow from 192.168.123.0/24', flush=True)
sys.exit(1)
# Step 3: SDK subscriber test
print('[4/4] Unitree SDK rt/lowstate test (8s)...', flush=True)
try:
sys.path.insert(0, '/home/mitchaiet/GR00T-WholeBodyControl/.venv/lib/python3.12/site-packages')
from unitree_sdk2py.core.channel import ChannelFactoryInitialize, ChannelSubscriber
from unitree_sdk2py.idl.unitree_hg.msg.dds_ import LowState_
ChannelFactoryInitialize(0, iface)
sub = ChannelSubscriber('rt/lowstate', LowState_)
sub.Init()
for i in range(16):
msg = sub.Read()
if msg is not None:
gyro = msg.imu_state.gyroscope
print(f' GOT rt/lowstate! IMU gyro: [{gyro[0]:.4f}, {gyro[1]:.4f}, {gyro[2]:.4f}]', flush=True)
print(f' Motor states: {len(msg.motor_state)} joints', flush=True)
print('\n=== ALL TESTS PASSED ===', flush=True)
os._exit(0)
time.sleep(0.5)
print(' FAIL: No data on rt/lowstate after 8s', flush=True)
print(' DDS discovery works but topic data not received', flush=True)
except Exception as e:
print(f' SDK error: {e}', flush=True)
os._exit(1)
if __name__ == '__main__':
main()

34
scripts/launch_real.sh

@ -0,0 +1,34 @@
#!/bin/bash
# Launch GR00T-WBC on real G1 robot via GB10
# Prerequisites:
# 1. Robot powered on and connected via Ethernet (192.168.123.0/24 subnet)
# 2. GB10 has 192.168.123.100/24 on enP7s7: sudo ip addr add 192.168.123.100/24 dev enP7s7
# 3. UFW disabled or allows traffic from 192.168.123.0/24
# 4. Robot in debug mode (L2+R2 while in damping state on harness) for motor control
set -e
cd ~/GR00T-WholeBodyControl
source .venv/bin/activate
echo "=== GR00T-WBC Real Robot Launch ==="
echo ""
echo "[1/2] Checking network..."
if ping -c 1 -W 1 192.168.123.161 > /dev/null 2>&1; then
echo " Locomotion computer (192.168.123.161): OK"
else
echo " ERROR: Cannot reach 192.168.123.161"
echo " Fix: sudo ip addr add 192.168.123.100/24 dev enP7s7"
exit 1
fi
echo "[2/2] Launching GR00T-WBC (real mode)..."
echo " --interface real (auto-detects enP7s7 via 192.168.123.x)"
echo " Controls: ] = enable walk, w/s = fwd/back, a/d = strafe, q/e = rotate, z = zero"
echo " WARNING: Robot must be on harness + debug mode (L2+R2) before enabling walk!"
echo ""
export LD_LIBRARY_PATH=/opt/ros/jazzy/lib:$LD_LIBRARY_PATH
python3 gr00t_wbc/control/main/teleop/run_g1_control_loop.py \
--interface real \
"$@"
Loading…
Cancel
Save