- The **image server** has been changed to [teleimager](https://github.com/silencht/teleimager). Please refer to the repository README for details.
- The **image server** has been changed to [teleimager](https://github.com/silencht/teleimager). Please refer to the repository README for details.
- [televuer](https://github.com/silencht/televuer) has been upgraded. Please see the repository README for details.
- Upgraded [televuer](https://github.com/silencht/televuer). Please see the repository README for details.
> The new versions of [teleimager](https://github.com/silencht/teleimager/commit/ab5018691943433c24af4c9a7f3ea0c9a6fbaf3c) + [televuer](https://github.com/silencht/televuer/releases/tag/v3.0) support transmitting head camera images via **WebRTC**.
> The new versions of [teleimager](https://github.com/silencht/teleimager/commit/ab5018691943433c24af4c9a7f3ea0c9a6fbaf3c) + [televuer](https://github.com/silencht/televuer/releases/tag/v3.0) support transmitting **head camera images via WebRTC**.
> Supports **pass-through**, **ego**, and **immersive** modes.
- Improved the system’s **state machine** information and IPC mode.
- Improved the system’s **state machine** information and IPC mode.
- Added **pass-through mode**, allowing direct viewing of the external environment through a VR device camera (without using the robot’s head camera).
- ...
- Added support for **Inspire_FTP dexterous hand**.
- …
# 0. 📖 Introduction
# 0. 📖 Introduction
@ -128,19 +131,55 @@ For more information, you can refer to [Official Documentation ](https://support
# Generate the certificate files required for televuer submodule
# Configure SSL certificates for the televuer module so that XR devices (e.g., Pico / Quest / Apple Vision Pro) can securely connect via HTTPS / WebRTC
@ -167,10 +206,41 @@ For more information, you can refer to [Official Documentation ](https://support
>
>
> You can refer to [Harley Hahn's Guide to Unix and Linux](https://www.harley.com/unix-book/book/chapters/04.html#H) and [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to learn more.
> You can refer to [Harley Hahn's Guide to Unix and Linux](https://www.harley.com/unix-book/book/chapters/04.html#H) and [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to learn more.
| `--frequency` | Set the FPS for recording and control | Any reasonable float value | 30.0 |
| `--input-mode` | Choose XR input mode (how to control the robot) | `hand` (**hand tracking**)`controller` (**controller tracking**) | `hand` |
| `--display-mode` | Choose XR display mode (how to view the robot perspective) | `immersive` (immersive)`ego` (pass-through + small first-person window)`pass-through` (pass-through only) | `immersive` |
| `--arm` | Select the robot arm type (see 0. 📖 Introduction) | `G1_29``G1_23``H1_2``H1` | `G1_29` |
| `--ee` | Select the end-effector type of the arm (see 0. 📖 Introduction) | `dex1``dex3``inspire_ftp``inspire_dfx``brainco` | None |
| `--img-server-ip` | Set the image server IP address for receiving image streams and configuring WebRTC signaling | `IPv4` address | `192.168.123.164` |
| `--motion` | **Enable motion control mode**When enabled, the teleoperation program can run alongside the robot’s motion control program.In **hand tracking** mode, the [R3 controller](https://www.unitree.com/cn/R3) can be used to control normal robot walking; in **controller tracking** mode, joysticks can also control the robot’s movement. |
| `--headless` | **Enable headless mode**For running the program on devices without a display, e.g., the Development Computing Unit (PC2). |
| `--ipc` | **Inter-process communication mode**Allows controlling the xr_teleoperate program’s state via IPC. Suitable for interaction with agent programs. |
| `--affinity` | **CPU affinity mode**Set CPU core affinity. If you are unsure what this is, do not set it. |
| `--record` | **Enable data recording mode**Press **r** to start teleoperation, then **s** to start recording; press **s** again to stop and save the episode. Press **s** repeatedly to repeat the process. |
| `--task-*` | Configure the save path, target, description, and steps of the recorded task. |
# 2. 💻 Simulation Deployment
# 2. 💻 Simulation Deployment
## 2.1 📥 Environment Setup
## 2.1 📥 Environment Setup
> The v1.4 version’s simulation deployment is not yet available. Please use v1.3 for testing temporarily.
First, install [unitree_sim_isaaclab](https://github.com/unitreerobotics/unitree_sim_isaaclab). Follow that repo’s README.
First, install [unitree_sim_isaaclab](https://github.com/unitreerobotics/unitree_sim_isaaclab). Follow that repo’s README.
Then launch the simulation with a G1(29 DoF) and Dex3 hand configuration:
Then launch the simulation with a G1(29 DoF) and Dex3 hand configuration:
@ -195,23 +265,6 @@ Here is the simulation GUI:
This program supports XR control of a physical robot or in simulation. Choose modes with command-line arguments:
This program supports XR control of a physical robot or in simulation. Choose modes with command-line arguments:
| `--record` | Enable **data recording**<br/>After pressing **r** to start, press **s** to start/stop saving an episode. Can repeat. |
| `--motion` | Enable **motion mode**<br/>After enabling this mode, the teleoperation program can run alongside the robot's motion control.<br/>In **hand tracking** mode, you can use the [R3 Controller](https://www.unitree.com/cn/R3) to control the robot's walking behavior; <br/>in **controller tracking** mode, you can also use [controllers to control the robot’s movement](https://github.com/unitreerobotics/xr_teleoperate/blob/375cdc27605de377c698e2b89cad0e5885724ca6/teleop/teleop_hand_and_arm.py#L247-L257). |
| `--headless` | Run without GUI (for headless PC2 deployment) |
| `--sim` | Enable **simulation mode** |
Assuming hand tracking with G1(29 DoF) + Dex3 in simulation with recording:
Assuming hand tracking with G1(29 DoF) + Dex3 in simulation with recording:
```bash
```bash
@ -231,7 +284,7 @@ Next steps:
2. Connect to the corresponding Wi‑Fi
2. Connect to the corresponding Wi‑Fi
3. Open a browser (e.g. Safari or PICO Browser) and go to: `https://192.168.123.2:8012?ws=wss://192.168.123.2:8012`
3. Open a browser (e.g. Safari or PICO Browser) and go to: `https://192.168.123.2:8012/?ws=wss://192.168.123.2:8012`
> **Note 1**: This IP must match your **Host** IP (check with `ifconfig`).
> **Note 1**: This IP must match your **Host** IP (check with `ifconfig`).
>
>
@ -268,6 +321,8 @@ Next steps:
> **Note 1**: Recorded data is stored in `xr_teleoperate/teleop/utils/data` by default, with usage instructions at this repo: [unitree_IL_lerobot](https://github.com/unitreerobotics/unitree_IL_lerobot/tree/main?tab=readme-ov-file#data-collection-and-conversion).
> **Note 1**: Recorded data is stored in `xr_teleoperate/teleop/utils/data` by default, with usage instructions at this repo: [unitree_IL_lerobot](https://github.com/unitreerobotics/unitree_IL_lerobot/tree/main?tab=readme-ov-file#data-collection-and-conversion).
>
>
> **Note 2**: Please pay attention to your disk space size during data recording.
> **Note 2**: Please pay attention to your disk space size during data recording.
>
> **Note 3**: In v1.4 and above, the “record image” window has been removed.
## 2.3 🔚 Exit
## 2.3 🔚 Exit
@ -283,34 +338,50 @@ Physical deployment steps are similar to simulation, with these key differences:
In the simulation environment, the image service is automatically enabled. For physical deployment, you need to manually start the image service based on your specific camera hardware. The steps are as follows:
In the simulation environment, the image service is automatically enabled. For physical deployment, you need to manually start the image service based on your specific camera hardware. The steps are as follows:
Copy `image_server.py` in the `xr_teleoperate/teleop/image_server` directory to the **Development Computing Unit PC2** of Unitree Robot (G1/H1/H1_2/etc.),
1. Install the image service program on the **Development Computing Unit PC2** of the Unitree robot (G1/H1/H1_2, etc.)
```bash
# p.s. You can transfer image_server.py to PC2 via the scp command and then use ssh to remotely login to PC2 to execute it.
#Assumingthe IP address of thedevelopment computing unit PC2 is 192.168.123.164, the transmission process is as follows:
# log in to PC2 via SSH and create the folder for the image server
# Configure the environment according to the instructions in the teleimager repository README: https://github.com/silencht/teleimager/blob/main/README.md
```
and execute the following command **in the PC2**:
2. On the **local host**, execute the following commands:
```bash
# p.s. Currently, this image transmission program supports two methods for reading images: OpenCV and Realsense SDK. Please refer to the comments in the `ImageServer` class within `image_server.py` to configure your image transmission service according to your camera hardware.
# [Image Server] Head camera 0 resolution: 480.0 x 1280.0
# [Image Server] Image server has started, waiting for client connections...
```
```bash
# Copy the `key.pem` and `cert.pem` files configured in Section 1.1 from the **local host**`xr_teleoperate/teleop/televuer` directory to the corresponding path on PC2
# These two files are required by teleimager to start the WebRTC service
After image service is started, you can use `image_client.py`**in the Host** terminal to test whether the communication is successful:
3. On the **development computing unit PC2**, configure `cam_config_server.yaml` according to the teleimager documentation and start the image service.
# If the WebRTC image stream is set up, you can also open the URL [https://192.168.123.164:60001](https://192.168.123.164:60001) in a browser and click the Start button to test.
```
## 3.2 ✋ Inspire Hand Service (optional)
## 3.2 ✋ Inspire Hand Service (optional)
@ -318,7 +389,7 @@ After image service is started, you can use `image_client.py` **in the Host** te
>
>
> **Note 2**: For G1 robot with [Inspire DFX hand](https://support.unitree.com/home/zh/G1_developer/inspire_dfx_dexterous_hand), related issue [#46](https://github.com/unitreerobotics/xr_teleoperate/issues/46).
> **Note 2**: For G1 robot with [Inspire DFX hand](https://support.unitree.com/home/zh/G1_developer/inspire_dfx_dexterous_hand), related issue [#46](https://github.com/unitreerobotics/xr_teleoperate/issues/46).
>
>
> **Note 3**: For [Inspire FTP hand]((https://support.unitree.com/home/zh/G1_developer/inspire_ftp_dexterity_hand)), related issue [#48](https://github.com/unitreerobotics/xr_teleoperate/issues/48).
> **Note 3**: For [Inspire FTP hand]((https://support.unitree.com/home/zh/G1_developer/inspire_ftp_dexterity_hand)), related issue [#48](https://github.com/unitreerobotics/xr_teleoperate/issues/48). FTP dexterous hand is now supported. Please refer to the `--ee` parameter for configuration.
First, use [this URL: DFX_inspire_service](https://github.com/unitreerobotics/DFX_inspire_service) to clone the dexterous hand control interface program. And Copy it to **PC2** of Unitree robots.
First, use [this URL: DFX_inspire_service](https://github.com/unitreerobotics/DFX_inspire_service) to clone the dexterous hand control interface program. And Copy it to **PC2** of Unitree robots.
> 1. Everyone must keep a safe distance from the robot to prevent any potential danger!
> 2. Please make sure to read the [Official Documentation](https://support.unitree.com/home/zh/Teleoperation) at least once before running this program.
> 3. Without `--motion`, always make sure that the robot has entered [debug mode (L2+R2)](https://support.unitree.com/home/zh/H1_developer/Remote_control) to stop the motion control program, this will avoid potential command conflict problems.
> 4. To use motion mode (with `--motion`), ensure the robot is in control mode (via [R3 remote](https://www.unitree.com/R3)).
> 5. In motion mode:
> 1. Everyone must keep a safe distance from the robot to prevent any potential danger!
> 2. Please make sure to read the [Official Documentation](https://support.unitree.com/home/zh/Teleoperation) at least once before running this program.
> 3. To use motion mode (with `--motion`), ensure the robot is in control mode (via [R3 remote](https://www.unitree.com/R3)).
> 5. In motion mode:
> - Right controller **A** = Exit teleop
> - Right controller **A** = Exit teleop
> - Both joysticks pressed = soft emergency stop (switch to damping mode)
> - Both joysticks pressed = soft emergency stop (switch to damping mode)
> - Left joystick = drive directions;
> - Left joystick = drive directions;
@ -393,37 +465,37 @@ Same as simulation but follow the safety warnings above.
```
```
xr_teleoperate/
xr_teleoperate/
│
│
├── assets [Storage of robot URDF-related files]
│
├── hardware [3D‑printed hardware modules]
├── assets [Stores robot URDF-related files]
│
│
├── teleop
├── teleop
│ ├── image_server
│ │ ├── image_client.py [Used to receive image data from the robot image server]
│ │ ├── image_server.py [Capture images from cameras and send via network (Running on robot's Development Computing Unit PC2)]
│ ├── teleimager [New image service library, supporting multiple features]
> 您可以参考 [Harley Hahn's Guide to Unix and Linux](https://www.harley.com/unix-book/book/chapters/04.html#H) 和 [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) 来深入了解这些知识。
> 您可以参考 [Harley Hahn's Guide to Unix and Linux](https://www.harley.com/unix-book/book/chapters/04.html#H) 和 [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) 来深入了解这些知识。
silencht
4 months ago