Maniformer, in collaboration with researchers from Northwestern Polytechnical University and Xi’an Jiaotong University, has introduced MEgoVista, an offline system that reconstructs hand and head motion from video captured with the MEgo View headset.
The system is designed to convert everyday first-person footage into metric-scale 4D motion data: movement tracked over time and measured in real-world units. This could help make egocentric video more useful for training embodied AI systems, which need to learn how people move and interact with objects.
Paper: https://arxiv.org/abs/2609.16684
Recording an activity is only the first step. For video to support model training, a system must identify the wearer’s hands, estimate finger poses, and determine where the hands are in three-dimensional space. It must also account for camera movement and distinguish the wearer’s hands from those of other people.
Why Egocentric Motion Is Difficult to Measure
Existing motion-capture approaches often involve a trade-off between accuracy and flexibility. Multi-camera systems can produce precise measurements, but they typically rely on equipment installed in a dedicated studio. Large datasets of everyday video capture more varied activities, but often lack metric-scale 3D annotations for hand position and movement.
Single-camera methods have another limitation: they infer depth from patterns learned during training rather than measuring it directly. As a result, a hand reconstruction may look convincing in the image while being misplaced in 3D space.
Real-world scenes can make the task harder. During an activity such as a manicure, the wearer’s hands may touch or overlap with a bystander’s. Tools and forearms can obscure fingers, while head turns move the camera and complicate efforts to separate camera motion from hand motion.
How MEgoVista Reconstructs Motion
MEgoVista uses a three-stage pipeline to identify hands, estimate their distance, and refine motion across a complete sequence.
First, it identifies the wearer’s hands. The system detects hands, determines whether each is a left or right hand, and classifies whether it belongs to the wearer. This helps it avoid confusing the wearer’s hands with a bystander’s, including during close interactions.
Next, it estimates distance using multiple cameras. The MEgo View headset records video at 60 frames per second with three fisheye cameras. The center camera is paired with each side camera to create two stereo pairs. Using the headset’s calibrated camera baselines, MEgoVista rectifies the images and estimates depth with FoundationStereo. It then combines the two depth results, giving greater weight to estimates with higher confidence.
Finally, it refines motion across the sequence. Rather than relying on isolated frame-by-frame estimates, MEgoVista selects reliable observations and optimizes the sequence as a whole. This process is intended to reduce jitter and produce smoother motion and anatomically plausible finger poses.
The system also reconstructs head movement from the multi-camera footage. It establishes metric scale at initialization and uses inertial data to determine the vertical direction. The resulting hand and head motion share a metric-scale coordinate system.
Evaluation Shows Lower Errors and Fewer Missed Hands
The researchers evaluated MEgoVista against reference measurements from an independent optical motion-capture system with 26 cameras. The evaluation included missed hand detections, preventing a method from appearing more accurate simply by skipping difficult frames.
MEgoVista recorded a finger-pose error, measured as PA-MPJPE-p, of 4.29 mm. The comparison results were 16.30 mm for HaWoR and 49.18 mm for Dyn-HaMR. For absolute wrist position in the camera frame, measured as CT-p, MEgoVista’s error was 14.30 mm, compared with 46.70 mm for HaWoR and 184.38 mm for Dyn-HaMR.
On the test set, MEgoVista had no missed or false hand detections. It also produced lower motion jitter than either baseline, at 1.11 mm/frame². In two head-trajectory tests, its average position errors across the complete sequences were 2.87 mm and 0.75 mm. An offline-refined ORB-SLAM3 baseline recorded errors of 3.27 mm and 1.00 mm, respectively.
The team also compared the methods in three everyday scenarios:
- Milk-tea preparation: MEgoVista and Dyn-HaMR produced skeletons that closely followed both hands during rapid movement. HaWoR showed a noticeable misalignment in the left-hand skeleton.
- Manicure: When the wearer’s and a bystander’s hands came into contact, HaWoR and Dyn-HaMR sometimes assigned the bystander’s hand to the wearer. MEgoVista distinguished between the two people.
- Animal washing: MEgoVista and Dyn-HaMR detected hands that were moving quickly and covered in dirt. HaWoR showed displacement around the left hand and wrist.
These comparisons suggest that MEgoVista can track hands more consistently in fast-moving activities and interactions involving multiple people. Its reconstructions also more closely followed hand positions in the evaluated scenes.
By combining hand identification, stereo depth estimation, and sequence-level refinement, MEgoVista aims to turn first-person video into measured hand and head motion that can support robot learning.
About Maniformer
Maniformer is a Physical AI data platform focused on infrastructure for embodied intelligence. Its work spans data-collection hardware, data processing, validation, and related services, with the goal of turning real-world physical interactions into data for developing and deploying Physical AI systems.
Find out more about Maniformer here: https://maniformer.ai/en
