epilectrik
/
vuer
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

[rename] mode fov to ego.

main
silencht 4 months ago
parent
562a9ca27d
commit
6b9aafd0b3
5 changed files with 25 additions and 25 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 4
      README.md
  2. 30
      src/televuer/televuer.py
  3. 10
      src/televuer/tv_wrapper.py
  4. 4
      test/test_televuer.py
  5. 2
      test/test_tv_wrapper.py

4
README.md
View File

@ -18,11 +18,11 @@ The image input of this library works in conjunction with the [teleimager](https
- pass-through: VR shows the real world through the VR headset cameras; no image from zmq or webrtc is displayed (even if enabled). - pass-through: VR shows the real world through the VR headset cameras; no image from zmq or webrtc is displayed (even if enabled).
- fov: a small window in the center shows the robot's first-person view, while the surrounding area shows the real world.
- ego: a small window in the center shows the robot's first-person view, while the surrounding area shows the real world.
2. Enhanced Immersion 2. Enhanced Immersion
Adjusted the image plane height for immersive and fov modes to provide a more natural and comfortable VR experience
Adjusted the image plane height for immersive and ego modes to provide a more natural and comfortable VR experience
### V3.0 🏷️ brings updates: ### V3.0 🏷️ brings updates:
1. Added `pass_through` interface to enable/disable the pass-through mode. 1. Added `pass_through` interface to enable/disable the pass-through mode.

30
src/televuer/televuer.py
View File

@ -12,7 +12,7 @@ from typing import Literal
class TeleVuer: class TeleVuer:
def __init__(self, use_hand_tracking: bool, binocular: bool=True, img_shape: tuple=None, display_fps: float=30.0, def __init__(self, use_hand_tracking: bool, binocular: bool=True, img_shape: tuple=None, display_fps: float=30.0,
display_mode: Literal["immersive", "pass-through", "fov"]="immersive", zmq: bool=False, webrtc: bool=False, webrtc_url: str=None,
display_mode: Literal["immersive", "pass-through", "ego"]="immersive", zmq: bool=False, webrtc: bool=False, webrtc_url: str=None,
cert_file: str=None, key_file: str=None): cert_file: str=None, key_file: str=None):
""" """
TeleVuer class for OpenXR-based XR teleoperate applications. TeleVuer class for OpenXR-based XR teleoperate applications.
@ -23,7 +23,7 @@ class TeleVuer:
:param img_shape: tuple, shape of the head image (height, width). :param img_shape: tuple, shape of the head image (height, width).
:param display_fps: float, target frames per second for display updates (default: 30.0). :param display_fps: float, target frames per second for display updates (default: 30.0).
:param display_mode: str, controls the VR viewing mode. Options are "immersive", "pass-through", and "fov".
:param display_mode: str, controls the VR viewing mode. Options are "immersive", "pass-through", and "ego".
:param zmq: bool, whether to use zmq for image transmission. :param zmq: bool, whether to use zmq for image transmission.
:param webrtc: bool, whether to use webrtc for real-time communication. :param webrtc: bool, whether to use webrtc for real-time communication.
:param webrtc_url: str, URL for the webrtc offer. must be provided if webrtc is True. :param webrtc_url: str, URL for the webrtc offer. must be provided if webrtc is True.
@ -35,10 +35,10 @@ class TeleVuer:
- display_mode controls what the VR headset displays: - display_mode controls what the VR headset displays:
* "immersive": fully immersive mode; VR shows the robot's first-person view (zmq or webrtc must be enabled). * "immersive": fully immersive mode; VR shows the robot's first-person view (zmq or webrtc must be enabled).
* "pass-through": VR shows the real world through the VR headset cameras; no image from zmq or webrtc is displayed (even if enabled). * "pass-through": VR shows the real world through the VR headset cameras; no image from zmq or webrtc is displayed (even if enabled).
* "fov": Field-of-View mode; a small window in the center shows the robot's first-person view, while the surrounding area shows the real world.
* "ego": a small window in the center shows the robot's first-person view, while the surrounding area shows the real world.
- Only one image mode is active at a time. - Only one image mode is active at a time.
- Image transmission to VR occurs only if display_mode is "immersive" or "fov" and the corresponding zmq or webrtc option is enabled.
- Image transmission to VR occurs only if display_mode is "immersive" or "ego" and the corresponding zmq or webrtc option is enabled.
- If zmq and webrtc simultaneously enabled, webrtc will be prioritized. - If zmq and webrtc simultaneously enabled, webrtc will be prioritized.
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
@ -48,7 +48,7 @@ class TeleVuer:
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
pass-through | Real world view (VR) | No | N/A | even if image source enabled, don't display pass-through | Real world view (VR) | No | N/A | even if image source enabled, don't display
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
fov | FOV view (robot + VR) | Yes (small) | zmq or webrtc | if both enabled, webrtc prioritized
ego | ego view (robot + VR) | Yes (small) | zmq or webrtc | if both enabled, webrtc prioritized
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
""" """
@ -114,9 +114,9 @@ class TeleVuer:
fn = self.main_image_binocular_zmq if self.binocular else self.main_image_monocular_zmq fn = self.main_image_binocular_zmq if self.binocular else self.main_image_monocular_zmq
else: else:
raise ValueError("[TeleVuer] immersive mode requires zmq=True or webrtc=True.") raise ValueError("[TeleVuer] immersive mode requires zmq=True or webrtc=True.")
elif self.display_mode == "fov":
elif self.display_mode == "ego":
if self.webrtc: if self.webrtc:
fn = self.main_image_binocular_webrtc_fov if self.binocular else self.main_image_monocular_webrtc_fov
fn = self.main_image_binocular_webrtc_ego if self.binocular else self.main_image_monocular_webrtc_ego
elif self.zmq: elif self.zmq:
self.img2display_shm = shared_memory.SharedMemory(create=True, size=np.prod(self.img_shape) * np.uint8().itemsize) self.img2display_shm = shared_memory.SharedMemory(create=True, size=np.prod(self.img_shape) * np.uint8().itemsize)
self.img2display = np.ndarray(self.img_shape, dtype=np.uint8, buffer=self.img2display_shm.buf) self.img2display = np.ndarray(self.img_shape, dtype=np.uint8, buffer=self.img2display_shm.buf)
@ -125,9 +125,9 @@ class TeleVuer:
self.stop_writer_event = threading.Event() self.stop_writer_event = threading.Event()
self.writer_thread = threading.Thread(target=self._xr_render_loop, daemon=True) self.writer_thread = threading.Thread(target=self._xr_render_loop, daemon=True)
self.writer_thread.start() self.writer_thread.start()
fn = self.main_image_binocular_zmq_fov if self.binocular else self.main_image_monocular_zmq_fov
fn = self.main_image_binocular_zmq_ego if self.binocular else self.main_image_monocular_zmq_ego
else: else:
raise ValueError("[TeleVuer] fov mode requires zmq=True or webrtc=True.")
raise ValueError("[TeleVuer] ego mode requires zmq=True or webrtc=True.")
elif self.display_mode == "pass-through": elif self.display_mode == "pass-through":
fn = self.main_pass_through fn = self.main_pass_through
else: else:
@ -208,7 +208,7 @@ class TeleVuer:
def close(self): def close(self):
self.process.terminate() self.process.terminate()
self.process.join(timeout=0.5) self.process.join(timeout=0.5)
if self.display_mode in ("immersive", "fov") and not self.webrtc:
if self.display_mode in ("immersive", "ego") and not self.webrtc:
self.stop_writer_event.set() self.stop_writer_event.set()
self.new_frame_event.set() self.new_frame_event.set()
self.writer_thread.join(timeout=0.5) self.writer_thread.join(timeout=0.5)
@ -480,8 +480,8 @@ class TeleVuer:
) )
await asyncio.sleep(1.0 / self.display_fps) await asyncio.sleep(1.0 / self.display_fps)
## FOV MODE
async def main_image_binocular_zmq_fov(self, session):
## ego MODE
async def main_image_binocular_zmq_ego(self, session):
if self.use_hand_tracking: if self.use_hand_tracking:
session.upsert( session.upsert(
Hands( Hands(
@ -536,7 +536,7 @@ class TeleVuer:
# 'jpeg' encoding should give you about 30fps with a 16ms wait in-between. # 'jpeg' encoding should give you about 30fps with a 16ms wait in-between.
await asyncio.sleep(1.0 / self.display_fps) await asyncio.sleep(1.0 / self.display_fps)
async def main_image_monocular_zmq_fov(self, session):
async def main_image_monocular_zmq_ego(self, session):
if self.use_hand_tracking: if self.use_hand_tracking:
session.upsert( session.upsert(
Hands( Hands(
@ -576,7 +576,7 @@ class TeleVuer:
) )
await asyncio.sleep(1.0 / self.display_fps) await asyncio.sleep(1.0 / self.display_fps)
async def main_image_binocular_webrtc_fov(self, session):
async def main_image_binocular_webrtc_ego(self, session):
if self.use_hand_tracking: if self.use_hand_tracking:
session.upsert( session.upsert(
Hands( Hands(
@ -613,7 +613,7 @@ class TeleVuer:
) )
await asyncio.sleep(1.0 / self.display_fps) await asyncio.sleep(1.0 / self.display_fps)
async def main_image_monocular_webrtc_fov(self, session):
async def main_image_monocular_webrtc_ego(self, session):
if self.use_hand_tracking: if self.use_hand_tracking:
session.upsert( session.upsert(
Hands( Hands(

10
src/televuer/tv_wrapper.py
View File

@ -194,7 +194,7 @@ class TeleData:
class TeleVuerWrapper: class TeleVuerWrapper:
def __init__(self, use_hand_tracking: bool, binocular: bool=True, img_shape: tuple=(480, 1280), display_fps: float=30.0, def __init__(self, use_hand_tracking: bool, binocular: bool=True, img_shape: tuple=(480, 1280), display_fps: float=30.0,
display_mode: Literal["immersive", "pass-through", "fov"]="immersive", zmq: bool=False, webrtc: bool=False, webrtc_url: str=None,
display_mode: Literal["immersive", "pass-through", "ego"]="immersive", zmq: bool=False, webrtc: bool=False, webrtc_url: str=None,
cert_file: str=None, key_file: str=None, return_hand_rot_data: bool=False): cert_file: str=None, key_file: str=None, return_hand_rot_data: bool=False):
""" """
TeleVuerWrapper is a wrapper for the TeleVuer class, which handles XR device's data suit for robot control. TeleVuerWrapper is a wrapper for the TeleVuer class, which handles XR device's data suit for robot control.
@ -205,7 +205,7 @@ class TeleVuerWrapper:
:param img_shape: tuple, shape of the head image (height, width). :param img_shape: tuple, shape of the head image (height, width).
:param display_fps: float, target frames per second for display updates (default: 30.0). :param display_fps: float, target frames per second for display updates (default: 30.0).
:param display_mode: str, controls the VR viewing mode. Options are "immersive", "pass-through", and "fov".
:param display_mode: str, controls the VR viewing mode. Options are "immersive", "pass-through", and "ego".
:param zmq: bool, whether to use ZMQ for image transmission. :param zmq: bool, whether to use ZMQ for image transmission.
:param webrtc: bool, whether to use webrtc for real-time communication. :param webrtc: bool, whether to use webrtc for real-time communication.
:param webrtc_url: str, URL for the webrtc offer. must be provided if webrtc is True. :param webrtc_url: str, URL for the webrtc offer. must be provided if webrtc is True.
@ -217,10 +217,10 @@ class TeleVuerWrapper:
- display_mode controls what the VR headset displays: - display_mode controls what the VR headset displays:
* "immersive": fully immersive mode; VR shows the robot's first-person view (zmq or webrtc must be enabled). * "immersive": fully immersive mode; VR shows the robot's first-person view (zmq or webrtc must be enabled).
* "pass-through": VR shows the real world through the VR headset cameras; no image from zmq or webrtc is displayed (even if enabled). * "pass-through": VR shows the real world through the VR headset cameras; no image from zmq or webrtc is displayed (even if enabled).
* "fov": Field-of-View mode; a small window in the center shows the robot's first-person view, while the surrounding area shows the real world.
* "ego": a small window in the center shows the robot's first-person view, while the surrounding area shows the real world.
- Only one image mode is active at a time. - Only one image mode is active at a time.
- Image transmission to VR occurs only if display_mode is "immersive" or "fov" and the corresponding zmq or webrtc option is enabled.
- Image transmission to VR occurs only if display_mode is "immersive" or "ego" and the corresponding zmq or webrtc option is enabled.
- If zmq and webrtc simultaneously enabled, webrtc will be prioritized. - If zmq and webrtc simultaneously enabled, webrtc will be prioritized.
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
@ -230,7 +230,7 @@ class TeleVuerWrapper:
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
pass-through | Real world view (VR) | No | N/A | even if image source enabled, don't display pass-through | Real world view (VR) | No | N/A | even if image source enabled, don't display
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
fov | FOV view (robot + VR) | Yes (small) | zmq or webrtc | if both enabled, webrtc prioritized
ego | ego view (robot + VR) | Yes (small) | zmq or webrtc | if both enabled, webrtc prioritized
-------------- ------------------- -------------- ----------------- ------- -------------- ------------------- -------------- ----------------- -------
""" """
self.use_hand_tracking = use_hand_tracking self.use_hand_tracking = use_hand_tracking

4
test/test_televuer.py
View File

@ -20,7 +20,7 @@ def run_test_TeleVuer():
binocular=camera_config['head_camera']['binocular'], binocular=camera_config['head_camera']['binocular'],
img_shape=camera_config['head_camera']['image_shape'], img_shape=camera_config['head_camera']['image_shape'],
display_fps=camera_config['head_camera']['fps'], display_fps=camera_config['head_camera']['fps'],
display_mode="immersive", # "fov" or "immersive" or "pass-through"
display_mode="immersive", # "ego" or "immersive" or "pass-through"
zmq=camera_config['head_camera']['enable_zmq'], zmq=camera_config['head_camera']['enable_zmq'],
webrtc=camera_config['head_camera']['enable_webrtc'], webrtc=camera_config['head_camera']['enable_webrtc'],
webrtc_url=f"https://192.168.123.164:{camera_config['head_camera']['webrtc_port']}/offer" webrtc_url=f"https://192.168.123.164:{camera_config['head_camera']['webrtc_port']}/offer"
@ -30,7 +30,7 @@ def run_test_TeleVuer():
# binocular=True, # binocular=True,
# img_shape=(480, 1280), # img_shape=(480, 1280),
# display_fps=30.0, # display_fps=30.0,
# display_mode="fov", # "fov" or "immersive" or "pass-through"
# display_mode="ego", # "ego" or "immersive" or "pass-through"
# zmq=False, # zmq=False,
# webrtc=True, # webrtc=True,
# webrtc_url="https://192.168.123.164:60001/offer" # webrtc_url="https://192.168.123.164:60001/offer"

2
test/test_tv_wrapper.py
View File

@ -32,7 +32,7 @@ def run_test_tv_wrapper():
# binocular=True, # binocular=True,
# img_shape=(480, 1280), # img_shape=(480, 1280),
# display_fps=30.0, # display_fps=30.0,
# display_mode="fov",
# display_mode="ego",
# zmq=True, # zmq=True,
# webrtc=True, # webrtc=True,
# webrtc_url="https://192.168.123.164:60001/offer" # webrtc_url="https://192.168.123.164:60001/offer"

Write Preview
Loading…
Cancel
Save