# Current State — 2026-02-15

Handoff document for continuing work from another machine.

---

## What's Working

### MuJoCo Playground RL Training (Complete)
- **Location:** GB10 (`~/mujoco_training/`)
- **Result:** 200M-step locomotion policy trained. Reward: -6.4 → +12.1
- **Time:** 3:08:37, ~17.9K steps/sec on Blackwell GPU
- **Checkpoint:** `~/mujoco_training/checkpoints/G1JoystickFlatTerrain-20260215-090148-full/params.pkl`
- **Scripts:** `train_g1.py`, `play_g1.py` (untested), `monitor.sh`

### GR00T-WBC Simulation (Complete)
- **Location:** GB10 (`~/GR00T-WholeBodyControl/`)
- **Status:** Walking robot in MuJoCo sim, keyboard controlled (w/s/a/d), 50 Hz, 3.5ms/iter
- **Launch:** `./launch_sim.sh` or `python3 run_g1_control_loop.py`
- **Viz:** NoMachine virtual desktop or web MJPEG viewer (`launch_with_web_viewer.py`)

### Network Bridge GB10 ↔ G1 (90% — needs final verification)
- **GB10 IP:** 10.0.0.68 (primary) + 192.168.123.100/24 (secondary on enP7s7)
- **Ping:** <1ms to 192.168.123.161 (locomotion) and .164 (Jetson)
- **DDS confirmed alive inside robot:** 342 multicast packets/4s from locomotion computer
- **Root cause found:** UFW firewall was blocking incoming DDS multicast
- **Fix applied:** User disabled UFW (`sudo ufw disable`)
- **NOT YET VERIFIED:** Robot was disconnected before we could confirm rt/lowstate data flows end-to-end

---

## What's Next (In Priority Order)

### 1. Verify DDS Connection (5 min)
When robot is reconnected to the network:
```bash
# On GB10:
sudo ip addr add 192.168.123.100/24 dev enP7s7   # if not persistent
sudo ufw status   # confirm disabled or allows 192.168.123.0/24
cd ~/GR00T-WholeBodyControl && source .venv/bin/activate
python3 /tmp/dds_test.py
```
This tests ping → DDS multicast → SDK rt/lowstate in sequence.

### 2. Test GR00T-WBC on Real Robot (30 min)
Once DDS works:
1. Put robot on harness/stand
2. Enter debug mode: L2+R2 on wireless remote while robot is in damping state
3. Launch: `./launch_real.sh` (runs `--interface real`, auto-detects enP7s7)
4. Keyboard: `]` = enable walk, `w/s` = fwd/back, `a/d` = strafe, `q/e` = rotate
5. **WARNING:** No wireless remote control under GR00T-WBC — keyboard only

### 3. Install xr_teleoperate for Vision Pro (1-2 hrs)
Fastest path to telepresence — Vision Pro connects via Safari (WebXR), no app needed:
```bash
# On GB10:
cd ~ && git clone https://github.com/unitreerobotics/xr_teleoperate.git
# Follow their install instructions (needs Pinocchio, TeleVuer)
# Generate SSL certs for Vision Pro HTTPS connection
# Vision Pro opens: https://<gb10-ip>:8012
```
**Limitation:** xr_teleoperate bypasses GR00T-WBC and uses the stock controller for legs.

### 4. Build VisionProTeleop → GR00T-WBC Bridge (2-4 hrs)
Higher quality path — native visionOS app with RL-based balance:
1. Install VisionProTeleop's "Tracking Streamer" app on Vision Pro (App Store)
2. `pip install avp_stream` on GB10
3. Write bridge (~100-200 lines Python): `avp_stream` → wrist poses → Pinocchio IK → `ControlPolicy/upper_body_pose` ROS2 topic
4. Run alongside GR00T-WBC control loop

### 5. Unified WBC Training (Saved for Later)
Plan at `plans/eager-shimmying-raccoon.md`. Trains a policy that handles walking + arbitrary arm poses + balance simultaneously. ~400M steps, ~5.5 hrs on GB10.

---

## Key Architecture: GR00T-WBC

```
LOWER BODY (12 DOF legs)              UPPER BODY (17 DOF waist+arms)
  RL neural network (ONNX)               Open-loop interpolation
  Balance.onnx / Walk.onnx               Targets from external source
  Auto-switches based on                 Receives via ROS2 topic:
  velocity command magnitude             "ControlPolicy/upper_body_pose"
           \                                    /
            --> Merged 29-DOF joint positions --> DDS rt/lowcmd --> Motors
```

**Integration point for ANY telepresence/mocap system:** Publish to `ControlPolicy/upper_body_pose` ROS2 topic with:
- `target_upper_body_pose`: 17 joint angles (3 waist + 7 left arm + 7 right arm)
- `navigate_cmd`: `[vx, vy, wz]` velocity command (optional, overrides keyboard)

---

## Vision Pro Telepresence Options

| Path | What | Needs App? | Uses GR00T-WBC? | Status |
|------|------|:---:|:---:|--------|
| xr_teleoperate | WebXR via Safari | No | No (stock SDK) | Not yet installed |
| VisionProTeleop | Native visionOS app (MIT) | Yes (App Store) | Yes (via bridge) | Not yet installed |
| iPhone streamer | Socket.IO replication | Custom app | Yes (built-in) | Protocol documented |

**User has:** Apple Vision Pro + Meta Quest 3
**Decision:** Vision Pro selected as primary device

---

## Files on GB10

| File | Purpose |
|------|---------|
| `~/GR00T-WholeBodyControl/` | GR00T-WBC (patched for aarch64) |
| `~/GR00T-WholeBodyControl/launch_real.sh` | Real robot launcher (`--interface real`) |
| `~/GR00T-WholeBodyControl/launch_sim.sh` | Simulation launcher |
| `~/GR00T-WholeBodyControl/launch_with_web_viewer.py` | Sim + MJPEG web viewer |
| `~/GR00T-WholeBodyControl/gr00t_wbc/.../g1_29dof_gear_wbc_real.yaml` | Config copy with INTERFACE=enP7s7 |
| `~/mujoco_training/train_g1.py` | MuJoCo Playground RL training |
| `~/mujoco_training/play_g1.py` | Policy visualization (untested) |
| `~/mujoco_training/checkpoints/...full/params.pkl` | Trained locomotion policy |
| `/tmp/dds_test.py` | Quick DDS connectivity test |
| `/tmp/keysender.py` | ROS keyboard publisher for remote terminals |

---

## Credentials & Network

| Target | Address | User | Password |
|--------|---------|------|----------|
| GB10 | 10.0.0.68 | mitchaiet | (SSH key) |
| G1 Jetson | 192.168.123.164 | unitree | 123 |
| G1 Locomotion | 192.168.123.161 | — | Not SSH-accessible |

**GB10 interface:** enP7s7 (primary: 10.0.0.68 DHCP, secondary: 192.168.123.100/24 manual)
**DDS:** Domain 0, CycloneDDS 0.10.2, `ChannelFactoryInitialize(0, "enP7s7")`
**Firewall:** UFW was disabled. May need re-disabling after reboot.

---

## Context System Updates Made This Session

- `context/networking-comms.md` — Added §6: GB10 ↔ G1 DDS bridge (topology, setup, debugging findings, SSH creds)
- `context/open-questions.md` — Updated DDS relay question, updated Vision Pro mocap question, added "Vision Pro Telepresence Integration (Phase 5)" section, documented wireless remote answer
- `context/motion-retargeting.md` — Added §6: Apple Vision Pro telepresence paths (xr_teleoperate, VisionProTeleop, GR00T-WBC integration point)
- `context/learning-and-ai.md` — §8 MuJoCo Playground pipeline (added in prior session, still current)
- `CLAUDE.md` — Added Phase 4 to history
- `plans/eager-shimmying-raccoon.md` — Unified WBC training plan (from prior session, still current)