7.1 KiB

Raw Blame History

id	title	status	source_sections	related_topics	key_equations	key_terms	images	examples	open_questions
sensors-perception	Sensors & Perception	established	reference/sources/official-product-page.md, reference/sources/official-developer-guide.md	[hardware-specs locomotion-control sdk-programming safety-limits]	[]	[imu state_estimation realsense_d435i livox_mid360 tactile_sensor dual_encoder]	[]	[]	[IMU model and noise characteristics D435i resolution/FPS as configured on G1 specifically Force/torque sensor availability per variant]

Sensors & Perception

Sensor suite, specifications, and perception capabilities of the G1.

1. Sensor Suite Overview

Sensor Type	Model/Spec	Location	Data Type	Network/Interface	Tier
Depth Camera	Intel RealSense D435i	Head	RGB-D	USB, SDK access	T0
3D LiDAR	Livox MID360	Head	360° point cloud	Ethernet (192.168.123.20)	T0
IMU	Onboard IMU	Body (torso)	6-axis accel+gyro	Internal, via LowState	T0
Microphone Array	4-element array	Head	Audio (spatial)	Internal	T0
Speaker	5W stereo	Head	Audio output	Internal	T0
Joint Encoders	Dual encoders per joint	Each joint	Position + velocity	Internal, via LowState	T0
Temperature Sensors	Per-motor	Each actuator	Temperature	Internal, via LowState	T0
Tactile Sensors	33 per hand (Dex3-1)	Fingertips	Force/touch	Via Dex3-1 DDS topic	T0

[T0 — Official product page and developer guide]

2. Depth Camera — Intel RealSense D435i

The primary visual sensor for RGB-D perception, obstacle detection, and environment mapping.

Standard D435i specs: 1280x720 @ 30fps (depth), 1920x1080 @ 30fps (RGB), 87° x 58° FOV (depth)
Note: G1-specific configuration (resolution, FPS) may differ from standard D435i defaults — see open questions
Mounting: Head-mounted, forward-facing
Access: Via RealSense SDK on development computer, or via SDK2 camera API
Research: Used for end-to-end vision-based locomotion (arXiv:2602.06382) with depth augmentation [T1]

3. 3D LiDAR — Livox MID360

360-degree environment scanning for navigation, SLAM, and obstacle avoidance.

Horizontal FOV: 360°
Vertical angle: 59°
Network: Ethernet at 192.168.123.20
Access: Requires separate Livox driver setup (unilidar_sdk2)
Use cases: SLAM, navigation, environment mapping, point cloud generation

4. IMU (Inertial Measurement Unit)

Critical for balance and state estimation during locomotion.

Type: 6-axis (3-axis accelerometer + 3-axis gyroscope)
Location: Pelvis (body torso) — measures pelvis orientation only, NOT total body lean [T1]
Access: Available in rt/lowstate → imu_state field
Quaternion format: [w, x, y, z] — confirmed via code inspection and accelerometer cross-validation [T1]
Used by: Locomotion computer for real-time balance control, state estimation
User access: Via SDK2 LowState subscription

IMU Mounting Offset [T1 — Verified on real G1, 2026-02-15]

CRITICAL: The physical IMU in the G1's pelvis has a mounting offset of approximately 6° in pitch relative to what simulation assumes. This means:

In simulation, the IMU is perfectly aligned with the body frame (zero offset by definition)
On the real robot, the IMU's "zero" is ~6° off from true upright
The stock Unitree controller compensates for this offset internally (calibration in the locomotion MCU firmware)
Third-party controllers (e.g., GR00T-WBC) that bypass the stock controller and read raw IMU data via DDS will see a biased reference frame, causing persistent backward lean

Impact: Any RL policy trained in simulation that reads IMU orientation will have a systematically wrong reference for "upright" on the real robot. This cannot be fixed by PD gain tuning or pitch commands alone — the policy's balance loop actively fights corrections because it believes its (wrong) reference is correct.

Fix: Apply a quaternion pitch rotation to the IMU quaternion before computing the gravity vector:

# Rotate quaternion by IMU pitch offset (around Y axis)
half = imu_pitch_offset / 2.0  # ~np.deg2rad(-6.0) for forward correction
pitch_quat = np.array([np.cos(half), 0.0, np.sin(half), 0.0])  # w,x,y,z
corrected_quat = quat_multiply(pitch_quat, raw_imu_quat)
gravity = get_gravity_orientation(corrected_quat)

Note: The exact offset may vary per unit. Calibrate by adjusting until the robot stands upright. On our G1 EDU Ultimate E (U7), -6° was the calibrated value.

Total Body Lean vs IMU Measurement [T1]

The IMU only measures pelvis tilt. Total visible body lean compounds across the kinematic chain:

Source	Contribution
Pelvis IMU (measured)	~1-3°
Waist pitch sag	~1-2°
Knee compression	~1-2°
Ankle pitch error	~1-2°
Total visible lean	~5-10°

When the user reports "8-10° backward lean" but the IMU shows only 1.5-3°, this is NOT an error — it's the compounding effect across joints.

5. Joint Proprioception

Dual encoders per joint provide high-accuracy feedback:

Position feedback: Absolute joint angle (rad)
Velocity feedback: Joint angular velocity (rad/s)
Access: rt/lowstate → motor_state array
Rate: Updated at 500 Hz (control loop frequency)

6. Perception Pipeline

Sensors → Locomotion Computer → DDS → Development Computer → User Applications
   │              │
   │              └── IMU + encoders → state estimation → balance control (internal)
   │
   ├── D435i → RealSense SDK → RGBD frames → perception/planning
   ├── MID360 → Livox driver → point cloud → SLAM/navigation
   └── Microphones → audio processing → voice interaction

The locomotion computer handles proprioceptive sensing (IMU, encoders) internally for real-time control. Exteroceptive sensors (cameras, LiDAR) are processed on the development computer. [T1 — Inferred from architecture]

7. State Estimation

The locomotion computer fuses IMU and joint encoder data for real-time state estimation: [T2 — Observed from RL papers]

Body orientation (roll, pitch, yaw) from IMU
Angular velocity from IMU gyroscope
Joint positions and velocities from encoders
Contact state inference for foot contact detection

This state estimate feeds directly into the RL-based locomotion policy at 500 Hz.

Key Relationships

Mounted on: hardware-specs
Feeds into: locomotion-control (balance, terrain adaptation)
Accessed via: sdk-programming (sensor data API)
Network config: networking-comms

7.1 KiB Raw Blame History