ArXiv

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

2026-05-12 · 15:51 UTC ·Luming Wang, Hao Shi, Jiajun Zhai... ·1 min read
Authors
Luming Wang, Hao Shi, Jiajun Zhai...
Categories
cs.CV, cs.RO, eess.IV
arXiv
https://arxiv.org/abs/2605.12297v1
PDF
https://arxiv.org/pdf/2605.12297v1

Brief

EgoEV-HandPose tackles egocentric 3D bimanual hand-pose estimation and gesture recognition from stereo event cameras by introducing KeypointBEV, which lifts stereo features into a bird's-eye-view and iteratively reprojection-refines depth and kinematic estimates. Trained and evaluated on the new EgoEVHands dataset (5,419 sequences, 38 gestures), it reports MPJPE 30.54 mm and 86.87% Top-1 accuracy, outperforming RGB-stereo and prior event-based methods, notably under low-light and occlusion.

Why it matters

EgoEV-HandPose introduces KeypointBEV, a stereo fusion module that lifts features into a canonical bird's-eye-view and uses an iterative reprojection-guided refinement loop to resolve depth uncertainty and enforce kinematic consistency for egocentric bimanual 3D hand pose and gesture estimation.

Key details

  • The authors collected EgoEVHands, the first large-scale real-world stereo event-camera egocentric hand dataset: 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination, to be released with code.
  • EgoEV-HandPose achieves state-of-the-art results: MPJPE = 30.54 mm and Top-1 gesture accuracy = 86.87%, significantly outperforming RGB stereo and prior event-camera methods, especially in low-light and bimanual occlusion scenarios.
Source evidence

Abstract

Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

Comment: Extended version of SMC 2025 paper arXiv:2503.12419. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose