diff --git a/SETUP.md b/SETUP.md index 53589ce..96154db 100644 --- a/SETUP.md +++ b/SETUP.md @@ -1,63 +1,110 @@ # G1 Teleop — Quest 3 Native App Setup +## Overview + +Godot 4.6.1 Quest 3 VR app for teleoperating a Unitree G1 humanoid robot. Uses OpenXR with Meta Quest hand tracking, body tracking, and passthrough. + ## Prerequisites -- **Godot 4.3+** (download from https://godotengine.org) -- **Android SDK + NDK** (install via Android Studio or standalone) -- **Meta Quest Developer Hub** (for sideloading) or `adb` +- **Godot 4.6.1** — located at `Godot4_6_1/Godot_v4.6.1-stable_win64_console.exe` +- **Android SDK + NDK** — located at `android-sdk/` +- **JDK 17** — located at `jdk17/` +- **OpenXR Vendors plugin v4.3.0** — located at `addons/godotopenxrvendors/` +- **adb** — located at `C:\Users\John\Downloads\platform-tools\adb.exe` - Quest 3 in **developer mode** -## Step 1: Install Meta OpenXR Vendors Plugin +## Build Pipeline -1. Open project in Godot: `godot --editor --path C:\git\g1-teleop` -2. Go to **AssetLib** tab (top center) -3. Search for "Godot OpenXR Vendors" -4. Download and install (includes Meta Quest support) -5. Enable the plugin: **Project → Project Settings → Plugins → godotopenxrvendors** → Enable -6. Restart Godot when prompted - -## Step 2: Configure Android Export +```bash +# 1. Setup vendor plugin (copies AARs and .so files) +python build/setup_vendor_plugin.py -1. Go to **Editor → Editor Settings → Export → Android** -2. Set the path to your Android SDK -3. Set Java SDK path -4. Go to **Project → Export → Quest 3** -5. Download Android export templates if prompted (Editor → Manage Export Templates → Download) +# 2. Build APK (headless Godot export) +build\build_461.bat -## Step 3: Enable Body Tracking +# 3. Install to Quest 3 +adb install -r build/g1-teleop.apk +``` -Body tracking is configured via: -- `export_presets.cfg`: `xr_features/hand_tracking=2` (required) -- `android/AndroidManifest.xml`: body tracking features and metadata +The build script runs: `Godot_v4.6.1-stable_win64_console.exe --headless --quit --path . --export-debug "Quest 3" build/g1-teleop.apk` -The Meta OpenXR Vendors plugin v4.1.1+ exposes `XRBodyTracker` in GDScript. +## Architecture -## Step 4: Build and Sideload +### Two-Phase Startup +- **CONFIG phase**: Dark VR environment with start screen UI, G1 robot models spinning on each side, hand laser pointer with pinch-to-click for menu interaction +- **AR phase**: Meta Quest passthrough with body tracking, webcam quad, gaze-activated exit balls -1. Connect Quest 3 via USB (or Wi-Fi ADB) -2. In Godot: **Project → Export → Quest 3 → Export Project** (or one-click deploy) -3. APK will be at `build/g1-teleop.apk` -4. Sideload: `adb install build/g1-teleop.apk` +### Key Files -## Step 5: Robot Server +| File | Purpose | +|------|---------| +| `Main.gd` | Main controller — phase management, passthrough, gaze balls, G1 models, recenter | +| `Main.tscn` | Scene tree — XROrigin3D, camera, controllers, start screen, webcam quad | +| `scripts/vr_ui_pointer.gd` | Hand laser pointer, pinch-to-click, hand poke, controller ray interaction | +| `scripts/body_tracker.gd` | XRBodyTracker joint visualization and tracking data | +| `scripts/teleop_client.gd` | WebSocket client for robot communication | +| `scripts/start_screen.gd` | Config UI panel (connect, launch AR buttons) | +| `scripts/webcam_quad.gd` | Webcam video display | +| `project.godot` | Project settings including OpenXR extensions | +| `export_presets.cfg` | Android export config with Meta XR features | +| `models/g1_full.obj` | 3D model of G1 robot (76MB OBJ, Z-up) | -On the robot (or dev machine for testing): +### Critical Project Settings (`project.godot`) -```bash -cd server/ -pip install -r requirements.txt -python teleop_server.py --port 8765 +```ini +openxr/extensions/meta/passthrough=true # Required for Quest passthrough +openxr/extensions/meta/body_tracking=true # Required for body tracking +openxr/environment_blend_mode=0 # Must stay 0, passthrough enabled via code ``` -## Step 6: Configure App +### Critical Export Settings (`export_presets.cfg`) -Edit `scripts/teleop_client.gd` to set the robot's IP: -```gdscript -@export var server_host: String = "10.0.0.64" # Robot IP -@export var server_port: int = 8765 +```ini +meta_xr_features/hand_tracking=2 +meta_xr_features/passthrough=2 +meta_xr_features/body_tracking=2 ``` -Or configure at runtime via the Godot editor's export properties. +## Technical Notes + +### Meta Quest Passthrough +- Requires `openxr/extensions/meta/passthrough=true` in project.godot AND `meta_xr_features/passthrough=2` in export presets +- `get_supported_environment_blend_modes()` only returns `[0]` (opaque) on Quest — must force `set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND)` without checking support +- The Meta vendor plugin intercepts this call via `XR_FB_PASSTHROUGH` extension +- Also requires `transparent_bg = true` on the viewport and `clear_color = Color(0,0,0,0)` +- Do NOT add a WorldEnvironment with BG_COLOR black — it makes everything black in VR + +### Hand Tracking +- Palm joint's `-basis.z` points perpendicular to palm (upward when hand flat), NOT forward +- Ray direction must be computed geometrically: wrist to middle finger metacarpal +- XRHandTracker joints: PALM=0, WRIST=1, THUMB_TIP=5, INDEX_TIP=10, MIDDLE_META=11 +- Pinch detection: thumb tip to index tip distance with hysteresis (press <2.5cm, release >3.5cm) + +### Scene Parenting +- Nodes under `XRCamera3D` follow head tracking +- Nodes under `XROrigin3D` stay stationary in tracking space +- Webcam quad is under `XROrigin3D` (stationary), not `XRCamera3D` + +### OBJ Model Loading +- `load("res://models/g1_full.obj")` returns a `Mesh` resource directly (not PackedScene) +- OBJ files use Z-up; Godot uses Y-up — rotate -90 degrees on X axis to stand upright +- G1 model scale: `Vector3(0.0015, 0.0015, 0.0015)` +- Needs explicit material override and a DirectionalLight3D for shading + +### GDScript Gotchas +- Untyped Arrays return Variant — use explicit type: `var ball: MeshInstance3D = _gaze_balls[i]` +- Cannot infer types from Variant expressions — annotate variables explicitly + +### Gaze Ball System +- 4 balls above field of view: red (exit AR), yellow (reserved), green (reserved), blue (quit app) +- Head gaze detection via angle threshold (8 degrees) +- 5-second stare to activate with visual feedback (opacity, color, scale 1.0x to 1.8x) +- Laser beam from head to gazed ball + +### Recenter Support +- Connected to `XRInterface.pose_recentered` signal +- Must connect in both `is_initialized()` and `elif` branches of `_ready()` +- Repositions start screen + G1 models (CONFIG) or webcam + gaze balls (AR) ## Network @@ -77,3 +124,10 @@ Or configure at runtime via the Godot editor's export properties. - Verify robot IP is correct and reachable: `ping 10.0.0.64` - Check server is running: `ss -tlnp | grep 8765` - Check firewall: `sudo ufw allow 8765/tcp` + +### Passthrough shows black/gray +- Verify `openxr/extensions/meta/passthrough=true` in project.godot +- Verify `meta_xr_features/passthrough=2` in export_presets.cfg +- Do NOT set `openxr/environment_blend_mode=2` in project.godot (keep it 0) +- Force alpha blend mode via code, not project settings +- Remove any WorldEnvironment node with black background