# G1 Teleop — Quest 3 Native App Setup

## Overview

Godot 4.6.1 Quest 3 VR app for teleoperating a Unitree G1 humanoid robot. Uses OpenXR with Meta Quest hand tracking, body tracking, and passthrough.

## Prerequisites

- **Godot 4.6.1** — located at `Godot4_6_1/Godot_v4.6.1-stable_win64_console.exe`
- **Android SDK + NDK** — located at `android-sdk/`
- **JDK 17** — located at `jdk17/`
- **OpenXR Vendors plugin v4.3.0** — located at `addons/godotopenxrvendors/`
- **adb** — located at `C:\Users\John\Downloads\platform-tools\adb.exe`
- Quest 3 in **developer mode**

## Build Pipeline

```bash
# 1. Setup vendor plugin (copies AARs and .so files)
python build/setup_vendor_plugin.py

# 2. Build APK (headless Godot export)
build\build_461.bat

# 3. Install to Quest 3
adb install -r build/g1-teleop.apk
```

The build script runs: `Godot_v4.6.1-stable_win64_console.exe --headless --quit --path . --export-debug "Quest 3" build/g1-teleop.apk`

## Architecture

### Two-Phase Startup
- **CONFIG phase**: Dark VR environment with start screen UI, G1 robot models spinning on each side, hand laser pointer with pinch-to-click for menu interaction
- **AR phase**: Meta Quest passthrough with body tracking, webcam quad, gaze-activated exit balls

### Key Files

| File | Purpose |
|------|---------|
| `Main.gd` | Main controller — phase management, passthrough, gaze balls, G1 models, recenter |
| `Main.tscn` | Scene tree — XROrigin3D, camera, controllers, start screen, webcam quad |
| `scripts/vr_ui_pointer.gd` | Hand laser pointer, pinch-to-click, hand poke, controller ray interaction |
| `scripts/body_tracker.gd` | XRBodyTracker joint visualization and tracking data |
| `scripts/teleop_client.gd` | WebSocket client for robot communication |
| `scripts/start_screen.gd` | Config UI panel (connect, launch AR buttons) |
| `scripts/webcam_quad.gd` | Webcam video display |
| `project.godot` | Project settings including OpenXR extensions |
| `export_presets.cfg` | Android export config with Meta XR features |
| `models/g1_full.obj` | 3D model of G1 robot (76MB OBJ, Z-up) |

### Critical Project Settings (`project.godot`)

```ini
openxr/extensions/meta/passthrough=true    # Required for Quest passthrough
openxr/extensions/meta/body_tracking=true  # Required for body tracking
openxr/environment_blend_mode=0            # Must stay 0, passthrough enabled via code
```

### Critical Export Settings (`export_presets.cfg`)

```ini
meta_xr_features/hand_tracking=2
meta_xr_features/passthrough=2
meta_xr_features/body_tracking=2
```

## Technical Notes

### Meta Quest Passthrough
- Requires `openxr/extensions/meta/passthrough=true` in project.godot AND `meta_xr_features/passthrough=2` in export presets
- `get_supported_environment_blend_modes()` only returns `[0]` (opaque) on Quest — must force `set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND)` without checking support
- The Meta vendor plugin intercepts this call via `XR_FB_PASSTHROUGH` extension
- Also requires `transparent_bg = true` on the viewport and `clear_color = Color(0,0,0,0)`
- Do NOT add a WorldEnvironment with BG_COLOR black — it makes everything black in VR

### Hand Tracking
- Palm joint's `-basis.z` points perpendicular to palm (upward when hand flat), NOT forward
- Ray direction must be computed geometrically: wrist to middle finger metacarpal
- XRHandTracker joints: PALM=0, WRIST=1, THUMB_TIP=5, INDEX_TIP=10, MIDDLE_META=11
- Pinch detection: thumb tip to index tip distance with hysteresis (press <2.5cm, release >3.5cm)

### Scene Parenting
- Nodes under `XRCamera3D` follow head tracking
- Nodes under `XROrigin3D` stay stationary in tracking space
- Webcam quad is under `XROrigin3D` (stationary), not `XRCamera3D`

### OBJ Model Loading
- `load("res://models/g1_full.obj")` returns a `Mesh` resource directly (not PackedScene)
- OBJ files use Z-up; Godot uses Y-up — rotate -90 degrees on X axis to stand upright
- G1 model scale: `Vector3(0.0015, 0.0015, 0.0015)`
- Needs explicit material override and a DirectionalLight3D for shading

### GDScript Gotchas
- Untyped Arrays return Variant — use explicit type: `var ball: MeshInstance3D = _gaze_balls[i]`
- Cannot infer types from Variant expressions — annotate variables explicitly

### Gaze Ball System
- 4 balls above field of view: red (exit AR), yellow (reserved), green (reserved), blue (quit app)
- Head gaze detection via angle threshold (8 degrees)
- 5-second stare to activate with visual feedback (opacity, color, scale 1.0x to 1.8x)
- Laser beam from head to gazed ball

### Recenter Support
- Connected to `XRInterface.pose_recentered` signal
- Must connect in both `is_initialized()` and `elif` branches of `_ready()`
- Repositions start screen + G1 models (CONFIG) or webcam + gaze balls (AR)

## Network

- Quest 3 and robot must be on the same network
- No SSL required (raw WebSocket)
- Default port: 8765
- Firewall: allow TCP 8765 on the robot

## Troubleshooting

### Body tracking not activating
- Ensure Quest 3 system software is up to date
- Check Settings → Movement Tracking → Body Tracking is enabled
- Ensure the Meta OpenXR Vendors plugin version is ≥ 4.1.1

### WebSocket connection fails
- Verify robot IP is correct and reachable: `ping 10.0.0.64`
- Check server is running: `ss -tlnp | grep 8765`
- Check firewall: `sudo ufw allow 8765/tcp`

### Passthrough shows black/gray
- Verify `openxr/extensions/meta/passthrough=true` in project.godot
- Verify `meta_xr_features/passthrough=2` in export_presets.cfg
- Do NOT set `openxr/environment_blend_mode=2` in project.godot (keep it 0)
- Force alpha blend mode via code, not project settings
- Remove any WorldEnvironment node with black background