Browse Source
Add Unitree G1 robot integration document
Add Unitree G1 robot integration document
Detailed architecture for using GB10 as offboard AI brain for the G1 humanoid robot: connection options (Wi-Fi/10GbE), LLM task planning, VLM inference, RL training (Isaac Lab), imitation learning, latency budgets, communication protocols (REST/DDS/ROS2), and quick start guide. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>master
2 changed files with 306 additions and 4 deletions
@ -0,0 +1,299 @@ |
|||||
|
--- |
||||
|
id: g1-integration |
||||
|
title: "Unitree G1 Robot Integration" |
||||
|
status: proposed |
||||
|
source_sections: "Cross-referenced from git/unitree-g1 context system + Dell Pro Max GB10 context" |
||||
|
related_topics: [connectivity, ai-workloads, ai-frameworks, multi-unit-stacking] |
||||
|
key_equations: [] |
||||
|
key_terms: [unitree-g1, jetson-orin-nx, offboard-compute, cyclonedds, dds, teleoperation, isaac-lab, lerobot] |
||||
|
images: [] |
||||
|
examples: [] |
||||
|
open_questions: |
||||
|
- "DDS latency over Wi-Fi between GB10 and G1 under realistic conditions" |
||||
|
- "Optimal LLM size for real-time task planning (latency vs. capability tradeoff)" |
||||
|
- "Isaac Lab on GB10 vs. dedicated x86 workstation for RL training throughput" |
||||
|
- "Docker container availability for CycloneDDS 0.10.2 on ARM64 DGX OS" |
||||
|
- "Power-efficient always-on inference mode for persistent LLM service" |
||||
|
--- |
||||
|
|
||||
|
# Unitree G1 Robot Integration |
||||
|
|
||||
|
Architecture and use cases for pairing the Dell Pro Max GB10 as an offboard AI companion to the Unitree G1 humanoid robot. |
||||
|
|
||||
|
## 1. Why Integrate? |
||||
|
|
||||
|
The G1's development computer (Jetson Orin NX 16GB, 100 TOPS) is excellent for real-time control but severely limited for large-scale AI. The GB10 removes that ceiling. |
||||
|
|
||||
|
| Capability | G1 Orin NX | GB10 | Factor | |
||||
|
|---|---|---|---| |
||||
|
| AI compute | 100 TOPS | 1,000 TFLOPS (FP4) | ~10,000x | |
||||
|
| Memory for models | 16 GB | 128 GB unified | 8x | |
||||
|
| Max LLM (quantized) | ~7B | ~200B | ~30x | |
||||
|
| CUDA architecture | sm_87 (Orin) | sm_121 (Blackwell) | 2 gens newer | |
||||
|
| Storage | 32 GB eMMC (loco) | 2-4 TB NVMe | 60-125x | |
||||
|
|
||||
|
The GB10 acts as an **offboard AI brain** — handling reasoning, planning, training, and heavy inference that the Orin NX cannot. |
||||
|
|
||||
|
## 2. Connection Architecture |
||||
|
|
||||
|
``` |
||||
|
┌──────────────────────────┐ ┌───────────────────────────┐ |
||||
|
│ Unitree G1 │ │ Dell Pro Max GB10 │ |
||||
|
│ │ │ │ |
||||
|
│ ┌─────────────────┐ │ │ 128 GB unified memory │ |
||||
|
│ │ Locomotion CPU │ │ │ 1 PFLOP FP4 │ |
||||
|
│ │ RK3588 │ │ │ DGX OS (Ubuntu 24.04) │ |
||||
|
│ │ 192.168.123.161 │ │ │ │ |
||||
|
│ │ 500 Hz control │ │ │ ┌──────────────────┐ │ |
||||
|
│ └──────┬───────────┘ │ │ │ LLM Server │ │ |
||||
|
│ │ CycloneDDS │ │ │ (llama.cpp/ │ │ |
||||
|
│ ┌──────┴───────────┐ │ Wi-Fi 6/7 │ │ TensorRT-LLM/ │ │ |
||||
|
│ │ Development CPU │◄──┼──────────────┼──►│ Ollama) │ │ |
||||
|
│ │ Jetson Orin NX │ │ or 10GbE │ └──────────────────┘ │ |
||||
|
│ │ 192.168.123.164 │ │ │ │ |
||||
|
│ │ 100 TOPS │ │ │ ┌──────────────────┐ │ |
||||
|
│ └──────────────────┘ │ │ │ Training Server │ │ |
||||
|
│ │ │ │ (Isaac Lab / │ │ |
||||
|
│ Sensors: │ │ │ PyTorch / │ │ |
||||
|
│ - D435i (RGB-D) │ │ │ LeRobot) │ │ |
||||
|
│ - MID360 (LiDAR) │ │ └──────────────────┘ │ |
||||
|
│ - 6-axis IMU │ │ │ |
||||
|
│ - 4-mic array │ │ ┌──────────────────┐ │ |
||||
|
│ - Joint encoders (500Hz)│ │ │ VLM Server │ │ |
||||
|
│ │ │ │ (vision-language) │ │ |
||||
|
└──────────────────────────┘ └───────────────────────────┘ |
||||
|
``` |
||||
|
|
||||
|
### Connection Options |
||||
|
|
||||
|
| Method | Bandwidth | Latency | Best For | |
||||
|
|--------|-----------|---------|----------| |
||||
|
| Wi-Fi (G1 Wi-Fi 6 ↔ GB10 Wi-Fi 7) | ~1 Gbps theoretical | 5-50 ms variable | Untethered operation, API calls | |
||||
|
| 10GbE Ethernet (GB10 RJ45) | 10 Gbps | <1 ms | Lab/tethered, sensor streaming | |
||||
|
| QSFP (via switch) | 200 Gbps | <1 ms | Multi-robot + GB10 cluster (T4) | |
||||
|
|
||||
|
**Recommended:** Wi-Fi for mobile operation; 10GbE for lab/training workflows where the robot is stationary or tethered. |
||||
|
|
||||
|
### Network Configuration |
||||
|
|
||||
|
The G1 uses a fixed 192.168.123.0/24 subnet. The GB10 must join this subnet or use a router/bridge. |
||||
|
|
||||
|
**Option A — Direct subnet join (simplest):** |
||||
|
```bash |
||||
|
# On GB10, configure 10GbE or Wi-Fi interface |
||||
|
sudo nmcli con add type ethernet ifname eth0 con-name g1-net \ |
||||
|
ip4 192.168.123.100/24 |
||||
|
``` |
||||
|
|
||||
|
**Option B — Dual-network (GB10 keeps internet access):** |
||||
|
- GB10's 10GbE on 192.168.123.0/24 (to G1) |
||||
|
- GB10's Wi-Fi on home network (internet access for model downloads, updates) |
||||
|
|
||||
|
**Option C — G1 joins GB10's network:** |
||||
|
- Configure G1's Wi-Fi to connect to the same network as GB10 |
||||
|
- G1's internal subnet (192.168.123.0/24) remains separate for internal DDS |
||||
|
|
||||
|
## 3. Use Cases |
||||
|
|
||||
|
### 3a. LLM-Based Task Planning |
||||
|
|
||||
|
The G1's `learning-and-ai.md` lists "LLM-based task planning integration (firmware v3.2+)" as preliminary. The GB10 can serve as the LLM backend. |
||||
|
|
||||
|
**Architecture:** |
||||
|
``` |
||||
|
User (voice/text) → G1 mic array → STT on Orin NX |
||||
|
→ HTTP/gRPC to GB10 → LLM (Llama 70B+) generates plan |
||||
|
→ Plan sent back to G1 → G1 executes via locomotion policy + manipulation |
||||
|
``` |
||||
|
|
||||
|
**GB10 serves an OpenAI-compatible API** via llama.cpp or Ollama: |
||||
|
```bash |
||||
|
# On GB10: start LLM server |
||||
|
./llama-server --model ~/models/llama-70b-q4.gguf \ |
||||
|
--host 0.0.0.0 --port 30000 --n-gpu-layers 99 |
||||
|
|
||||
|
# On G1 Orin NX: call the API |
||||
|
curl http://192.168.123.100:30000/v1/chat/completions \ |
||||
|
-d '{"model":"llama","messages":[{"role":"user","content":"Walk to the kitchen table and pick up the red cup"}]}' |
||||
|
``` |
||||
|
|
||||
|
**Recommended models:** |
||||
|
| Model | Size | Speed (est.) | Task Planning Quality | |
||||
|
|-------|------|---------|----------------------| |
||||
|
| Llama 3.2 3B | 3B | ~100 tok/s | Basic commands | |
||||
|
| Nemotron-3-Nano 30B | 3B active | Fast | Good (built-in reasoning) | |
||||
|
| Llama 3.1 70B (Q4) | 70B | ~15-20 tok/s | Strong | |
||||
|
| Llama 3.1 70B (Q8) | 70B | ~10-15 tok/s | Strong, higher quality | |
||||
|
|
||||
|
For task planning, 1-5 second response time is acceptable — the robot doesn't need instant responses for high-level commands. |
||||
|
|
||||
|
### 3b. Vision-Language Models |
||||
|
|
||||
|
Stream camera frames from the G1's Intel RealSense D435i to the GB10 for analysis by large VLMs that can't run on the Orin NX. |
||||
|
|
||||
|
**Use cases:** |
||||
|
- Scene understanding ("describe what you see") |
||||
|
- Object identification ("find the red cup on the table") |
||||
|
- Spatial reasoning ("is the path ahead clear?") |
||||
|
- Anomaly detection ("does anything look unusual?") |
||||
|
|
||||
|
**Data path:** |
||||
|
``` |
||||
|
G1 D435i (1920x1080 @ 30fps) → RealSense SDK on Orin NX |
||||
|
→ JPEG compress → HTTP POST to GB10 |
||||
|
→ VLM inference → JSON response back to G1 |
||||
|
``` |
||||
|
|
||||
|
**Bandwidth:** A 1080p JPEG frame is ~100-500 KB. At 5 fps sampling: ~2.5 MB/s (~20 Mbps). Well within Wi-Fi capacity. |
||||
|
|
||||
|
### 3c. RL Policy Training |
||||
|
|
||||
|
Train locomotion and manipulation policies on the GB10 using Isaac Lab or MuJoCo, then deploy to the G1. |
||||
|
|
||||
|
**Why GB10 over the Orin NX:** |
||||
|
- Isaac Lab runs thousands of parallel environments — needs GPU memory |
||||
|
- GB10's 128 GB unified memory holds large batches + simulation state |
||||
|
- Blackwell GPU (6,144 CUDA cores, sm_121) accelerates physics simulation |
||||
|
- CUDA 13.0 + PyTorch container `nvcr.io/nvidia/pytorch:25.11-py3` |
||||
|
|
||||
|
**Workflow:** |
||||
|
``` |
||||
|
1. GB10: Train policy in Isaac Lab (unitree_rl_lab, G1-29dof config) |
||||
|
2. GB10: Validate in MuJoCo (sim2sim cross-validation) |
||||
|
3. GB10 → G1: Transfer trained model file (.pt / .onnx) |
||||
|
4. G1 Orin NX: Deploy via unitree_sdk2_python (low-gain start) |
||||
|
5. G1: Gradual gain ramp-up with tethered safety testing |
||||
|
``` |
||||
|
|
||||
|
**Key: Both systems are ARM64.** Model files trained on GB10 (ARM) deploy directly to Orin NX (ARM) without architecture-related issues. |
||||
|
|
||||
|
### 3d. Imitation Learning Data Processing |
||||
|
|
||||
|
Process teleoperation demonstration data collected on the G1 using larger models on the GB10. |
||||
|
|
||||
|
**Workflow:** |
||||
|
``` |
||||
|
1. G1: Collect demonstrations via XR teleoperation (Vision Pro / Quest 3) |
||||
|
2. Transfer data to GB10 (SCP over network) |
||||
|
3. GB10: Train behavior cloning / diffusion policy (LeRobot, 128GB memory) |
||||
|
4. GB10: Validate in simulation |
||||
|
5. Transfer model to G1 for deployment |
||||
|
``` |
||||
|
|
||||
|
The GB10's 128 GB memory enables training larger LeRobot policies and processing more demonstration episodes than the Orin NX's 16 GB allows. |
||||
|
|
||||
|
### 3e. Multi-Modal Interaction |
||||
|
|
||||
|
Combine the G1's sensor suite with GB10's compute for rich interaction: |
||||
|
|
||||
|
``` |
||||
|
G1 Microphones (4-mic array) → Speech-to-Text (Orin NX: Whisper small) |
||||
|
→ Text to GB10 → LLM reasoning → Response text |
||||
|
→ TTS on Orin NX → G1 Speaker (5W stereo) |
||||
|
|
||||
|
G1 Camera (D435i) → Frames to GB10 → VLM → Situational awareness |
||||
|
G1 LiDAR (MID360) → Point cloud to GB10 → Semantic mapping |
||||
|
``` |
||||
|
|
||||
|
### 3f. Simulation Environment |
||||
|
|
||||
|
Run full physics simulation of the G1 on the GB10: |
||||
|
|
||||
|
- **MuJoCo:** CPU-based simulation for policy validation |
||||
|
- **Isaac Lab:** GPU-accelerated parallel environments for training |
||||
|
- **Isaac Gym:** Legacy GPU simulation (unitree_rl_gym) |
||||
|
|
||||
|
The GB10 can run these natively (ARM64 NVIDIA stack) while the G1 operates in the real world. Sim2Real transfer between GB10 simulation and real G1 hardware. |
||||
|
|
||||
|
## 4. Latency Budget |
||||
|
|
||||
|
| Operation | Latency | Acceptable? | |
||||
|
|-----------|---------|-------------| |
||||
|
| G1 locomotion control loop | 2 ms (500 Hz) | N/A — stays on-robot | |
||||
|
| DDS internal (loco ↔ dev) | 2 ms | N/A — stays on-robot | |
||||
|
| Wi-Fi round-trip (G1 ↔ GB10) | 5-50 ms | Yes for planning, No for control | |
||||
|
| 10GbE round-trip (G1 ↔ GB10) | <1 ms | Yes for most tasks | |
||||
|
| LLM inference (70B Q4) | 1-5 seconds | Yes for task planning | |
||||
|
| VLM inference (per frame) | 0.5-2 seconds | Yes for scene understanding | |
||||
|
|
||||
|
**Critical rule:** Real-time control (500 Hz locomotion, balance, joint commands) MUST stay on the G1's locomotion computer. The GB10 handles only high-level reasoning with relaxed latency requirements. |
||||
|
|
||||
|
## 5. Communication Protocol Options |
||||
|
|
||||
|
### Option A: REST/gRPC API (Simplest) |
||||
|
|
||||
|
GB10 runs an HTTP API server (llama.cpp, Ollama, or custom FastAPI). G1's Orin NX sends requests and receives responses. |
||||
|
|
||||
|
- **Pros:** Simple, stateless, easy to debug, OpenAI-compatible |
||||
|
- **Cons:** Higher latency per request, no streaming sensor data |
||||
|
- **Best for:** Task planning, VLM queries, one-shot inference |
||||
|
|
||||
|
### Option B: CycloneDDS (Native G1 protocol) |
||||
|
|
||||
|
Install CycloneDDS 0.10.2 on the GB10, making it appear as another node on the G1's DDS network. |
||||
|
|
||||
|
- **Pros:** Native integration, low latency, pub/sub for streaming data |
||||
|
- **Cons:** More complex setup, must match exact DDS version (0.10.2) |
||||
|
- **Best for:** Streaming sensor data, continuous perception, tight integration |
||||
|
|
||||
|
```bash |
||||
|
# On GB10: Install CycloneDDS (must be 0.10.2 to match G1) |
||||
|
pip install cyclonedds==0.10.2 |
||||
|
|
||||
|
# Subscribe to G1 state |
||||
|
# (requires unitree_sdk2 IDL definitions) |
||||
|
``` |
||||
|
|
||||
|
### Option C: ROS 2 Bridge |
||||
|
|
||||
|
Both systems support ROS 2 (G1 via unitree_ros2, GB10 via standard Ubuntu packages). |
||||
|
|
||||
|
- **Pros:** Rich ecosystem, visualization (RViz), bag recording |
||||
|
- **Cons:** Additional overhead, ROS 2 setup complexity |
||||
|
- **Best for:** Research workflows, multi-sensor fusion, SLAM |
||||
|
|
||||
|
## 6. What NOT to Use the GB10 For |
||||
|
|
||||
|
- **Real-time joint control** — Too much latency over network. Locomotion stays on RK3588. |
||||
|
- **Balance/stability** — 500 Hz control loop cannot tolerate network jitter. |
||||
|
- **Safety-critical decisions** — Emergency stop must be on-robot, not dependent on network. |
||||
|
- **Field deployment brain** — GB10 requires wall power (280W). It's a lab/indoor companion only. |
||||
|
|
||||
|
## 7. Hardware Compatibility Notes |
||||
|
|
||||
|
| Aspect | G1 | GB10 | Compatible? | |
||||
|
|--------|-----|------|-------------| |
||||
|
| CPU architecture | ARM (A76/A55, Orin ARM) | ARM (Cortex-X925/A725) | Yes | |
||||
|
| CUDA | sm_87 (Orin) | sm_121 (Blackwell) | Model files portable | |
||||
|
| Linux | Custom (kernel 5.10 RT) | Ubuntu 24.04 (kernel 6.8) | Yes (userspace) | |
||||
|
| Python | 3.x on Orin NX | 3.x on DGX OS | Yes | |
||||
|
| DDS | CycloneDDS 0.10.2 | Not pre-installed (installable) | Yes (with setup) | |
||||
|
| Docker | Available on Orin NX | Pre-installed + NVIDIA Runtime | Yes | |
||||
|
| Network | Wi-Fi 6, Ethernet | Wi-Fi 7, 10GbE, QSFP | Yes | |
||||
|
|
||||
|
**ARM alignment is a significant advantage.** Both systems are ARM-native, eliminating cross-architecture issues with model files, shared libraries, and container images. |
||||
|
|
||||
|
## 8. Cost Context |
||||
|
|
||||
|
| Component | Price | Role | |
||||
|
|-----------|-------|------| |
||||
|
| G1 EDU Standard | $43,500 | Robot platform | |
||||
|
| G1 EDU Ultimate B | $73,900 | Robot + full DOF + tactile | |
||||
|
| Dell Pro Max GB10 (2TB) | $3,699 | Offboard AI brain | |
||||
|
| Dell Pro Max GB10 (4TB) | $3,999 | Offboard AI brain | |
||||
|
|
||||
|
The GB10 adds <10% to the cost of a G1 EDU while multiplying AI capability by orders of magnitude. Even at the $13,500 base G1 price, the GB10 is a ~27% add-on that transforms the AI capabilities. |
||||
|
|
||||
|
## 9. Quick Start (Proposed) |
||||
|
|
||||
|
1. **Network:** Connect GB10 to G1's subnet (Wi-Fi or Ethernet to 192.168.123.100) |
||||
|
2. **LLM server:** Start llama.cpp or Ollama on GB10 (port 30000 or 12000) |
||||
|
3. **Test connectivity:** `curl http://192.168.123.100:30000/v1/models` from G1 Orin NX |
||||
|
4. **First integration:** Simple Python script on Orin NX that sends voice-to-text to GB10 LLM, receives plan, prints it |
||||
|
5. **Iterate:** Add VLM, sensor streaming, RL training as needed |
||||
|
|
||||
|
## Key Relationships |
||||
|
|
||||
|
- GB10 compute: [[gb10-superchip]], [[ai-frameworks]], [[ai-workloads]] |
||||
|
- GB10 networking: [[connectivity]] |
||||
|
- G1 reference: `git/unitree-g1/context/` (separate knowledge base) |
||||
Write
Preview
Loading…
Cancel
Save
Reference in new issue