# Phase 2: Motion Capture + Robust Balance Context Expansion

**Date:** 2026-02-13
**Objective:** Expand the knowledge base to deeply support development of motion capture retargeting and robust push recovery on the G1, both independently and combined.

## User Goal

The user wants to:
1. Push the G1 and have it stay standing — robust push recovery during all activities
2. Replay motion capture data on the G1
3. Combine both — mocap playback with always-on balance

User has an EDU variant with Jetson Orin NX, dexterous hands (currently being repaired), running as 29-DOF body.

## Research Conducted

Three explore agents investigated:

### Agent 1: Robust Balance & Push Recovery
- Confirmed stock G1 controller has light push recovery (arXiv:2505.20619)
- Perturbation curriculum training is the primary method for enhanced push recovery
- Multiple papers validate push recovery on REAL G1 hardware
- Residual policy approach allows enhancing balance without replacing stock controller
- Control Barrier Functions provide formal safety guarantees (validated on G1)

### Agent 2: Motion Capture Retargeting
- Three approaches: IK-based (Pinocchio), optimization-based (CasADi), RL-based (BFM-Zero)
- AMASS dataset pre-retargeted for G1 exists on HuggingFace
- BFM-Zero has open-source motion tracking mode validated on G1
- XR teleoperation provides real-time mocap via Vision Pro / Quest 3
- SMPL body model is the standard format; mapping SMPL→G1 joints is well-defined

### Agent 3: Whole-Body Control Frameworks
- GR00T-WBC (NVIDIA) is G1-native, open-source, designed for exactly this use case
- Decouples locomotion (RL policy, lower body) from task (upper body) control
- H2O, OmniH2O, HumanPlus all prove combined mocap + balance works on humanoids
- SoFTA provides slow-fast decoupled control for loco-manipulation
- Pinocchio + TSID offers model-based alternative to RL-based WBC

## Key Architectural Decision: Three Approaches

```
Approach A: Overlay (Residual)
├── Keep stock locomotion controller
├── Control arms via rt/lowcmd
├── Optional: residual corrections on legs
├── Risk: Low | Authority: Limited
└── Best for: Initial experimentation

Approach B: GR00T-WBC (Recommended)
├── Replace stock with trained RL locomotion policy
├── WBC coordinates upper and lower body
├── Integrates with LeRobot for data collection
├── Risk: Medium | Authority: Full
└── Best for: Production mocap + balance system

Approach C: Full Custom RL Policy
├── Single policy handles everything
├── BFM-Zero approach — motion tracking + balance unified
├── Must handle all edge cases
├── Risk: High | Authority: Maximum
└── Best for: Advanced deployment after extensive sim validation
```

## What Changed

### New Context Files (3)
- `context/whole-body-control.md` — WBC paradigms, GR00T-WBC, Pinocchio/TSID, architectural approaches (overlay vs. replacement), H2O/OmniH2O/HumanPlus
- `context/motion-retargeting.md` — Retargeting problem, IK/optimization/RL approaches, motion sources (AMASS, CMU, XR, video), SMPL joint mapping, full pipeline
- `context/push-recovery-balance.md` — Ankle/hip/stepping strategies, perturbation curriculum, residual policies, CBFs, always-on architecture, fall recovery, metrics, development roadmap

### Updated Context Files (7)
- `context/locomotion-control.md` — Added §8 Custom Policy Replacement, cross-references to new topics
- `context/learning-and-ai.md` — Added §5 Motion Tracking Policies, §6 Residual Policy Learning, §7 Perturbation Curriculum
- `context/manipulation.md` — Expanded §4 with GR00T-WBC reference, added mocap-based manipulation note
- `context/simulation.md` — Added perturbation testing, motion retargeting validation, AMASS dataset note
- `context/safety-limits.md` — Added §8 CBFs for Balance, §9 Fall Risk Assessment for Custom Control
- `context/equations-and-bounds.md` — Added support polygon, CBF, kinematic scaling, residual policy equations
- `context/open-questions.md` — Added 6 new motion capture + balance open questions

### Glossary Expansion
- Added ~20 new terms across 4 new categories:
  - Whole-Body Control: whole_body_control, task_space_inverse_dynamics, operational_space_control, centroidal_dynamics, qp_solver, groot_wbc
  - Motion Retargeting: motion_retargeting, mocap, amass, smpl, kinematic_scaling, inverse_kinematics
  - Push Recovery: push_recovery, ankle_strategy, hip_strategy, stepping_strategy, residual_policy, control_barrier_function, support_polygon, perturbation_curriculum
  - Frameworks: pinocchio, h2o, omnih2o
- Total glossary: ~57 terms (was 37)

### Source Documents Archived (~15)
- Research papers: GR00T-WBC, H2O, OmniH2O, HumanPlus, TWIST, SoFTA, residual policy
- GitHub repos: gr00t-wbc, Pinocchio, LeRobot+GR00T integration
- Datasets: AMASS-G1, CMU Mocap
- Community: mocap retargeting tools survey

### CLAUDE.md Updates
- Added 3 rows to Quick Topic Lookup (total: 20 topics)
- Updated Key Concepts Quick Map with new topic hierarchy
- Added Phase 2 to History table

## Assessment: Achievability

**YES — this is an achievable goal.** Key evidence:
1. Multiple papers demonstrate push recovery on real G1 hardware
2. GR00T-WBC is designed specifically for G1 WBC
3. BFM-Zero demonstrates motion tracking on real G1
4. AMASS retarget for G1 already exists
5. The combined approach (balance + tracking) is validated by H2O/OmniH2O/HumanPlus on humanoids

## What's Next (Phase 3 candidates)

- Parse G1 URDF for exact link masses/inertias (needed for Pinocchio/TSID)
- Clone and document GR00T-WBC setup process
- Create worked examples in examples/ directory:
  - Example: IK-based retargeting of a simple arm gesture
  - Example: Perturbation testing in MuJoCo
  - Example: GR00T-WBC hello world
- Benchmark GR00T-WBC inference latency on Jetson Orin NX
- Test residual policy overlay on stock controller (is it possible?)