EngramEgo

Generative 4D
hand–object reconstruction

From first-person recordings, we reconstruct how hands and objects move together in 3D over time. Explore a grasp, follow the object’s motion, and compare the reconstruction with the recorded action.

Explore 4D reconstruction Explore our data samples

Full-body reconstruction example. Reference, headset input and EgoAllo output are shown together.

Generative 4D reconstruction

Reconstruct hand–object interaction over time

Reconstruct hand geometry, object motion and their spatial relationship from first-person recordings. Compare the footage with the 3D result to follow how the interaction changes over time.

Picking up and moving a pump bottle, with synchronized hand–object reconstruction.

Handling a carton and a can at the checkout, with synchronized hand–object reconstruction.

Tilting a bottle over a bowl, shown alongside the 3D reconstruction

Look closer at the hand geometry

In these HOT3D examples, the estimated hand mesh follows the fingers as they grip a utensil or lift a bowl. The views below show the grasp in finer detail.

10.61 mmWrist-aligned joint error (MPJPE-p)

2.09×Speedup over our previous version

Benchmark and timing details

Wrist-aligned joint error (MPJPE-p) is 10.61 mm for our model and 11.38 mm for ACE-Ego-Hand. The corresponding errors after Procrustes alignment (PA-p) are 6.44 mm and 6.75 mm. In a separate test on one RTX 5090, median processing time per 81-frame batch at 672 × 480 fell from 2.383 s in our previous version to 1.140 s.

Moving toy food with a spatula

Moving toy food between a plate and a bowl

Lifting a wooden bowl with both hands

Watch the complete hand-tracking demonstration (34 s)

Hand meshes overlaid on HOT3D footage. The chart shows the larger of the two hands’ errors at each frame.

EngramEgo data samples

Explore a recorded episode

This bowl-and-lid recording is one of our data samples. Open EngramEgo to inspect the camera views, depth, head pose and inertial readings on the same timeline.

Browse the data samples on EngramEgo

Data sample · Handling a bowl and lid · 8.4 sInspect the synchronized sample on EngramEgo ↗

Task coverage

People at work, at home and on the move

These 21 first-person clips show the tasks being recorded, from tightening a screw at a workbench to choosing groceries in a shop.

Office Open a jar

Office Close a laptop

Office Fill a cup at the dispenser

Office Wash and dry hands

Convenience store Choose snacks from a shelf

Convenience store Inspect a packet of chilled noodles

Convenience store Pick out vegetables

Convenience store Turn a bottle to read the label

Convenience store Scan a bottle at the self-checkout

Restaurant Pick up noodles with a fork

Travel Use the steering-wheel controls

Travel Open a parcel locker

Travel Unscrew a bottle cap on the metro

Clothing store Check a garment tag

Factory Tighten a screw on a bracket

Factory Sort tools and parts

Home Take a drink from the fridge

Home Wash hands at home

Mall Read an information leaflet

Mall Try a keyboard and a phone

Mall Connect a phone to a shared power bank

Hardware

Capture hardware

The headband carries four cameras and an inertial sensor. A portable processing unit synchronizes the signals and displays depth and hand keypoints during recording.

Front view of the headband’s camera array — Four-camera headband

EngramEgo headband connected to the processing unit — Headband and processing unit

EngramEgo processing unit — Portable processing unit

Capture and recording controls

Cameras	Two wide-angle cameras and two fisheye cameras
Depth	Depth is estimated on the device in real time and synchronized with the RGB video.
Muscle activity	Multichannel EMG from a forearm band
Voice controls	Voice commands start or stop recording. An audio prompt alerts the wearer when a hand moves out of view.
Motion	6DoF head pose, plus accelerometer and gyroscope readings at 400 Hz
Sample playback	Seven synchronized streams in the sample, with camera video at about 30 fps

Device specifications

Headband	350 g · 8 h of continuous capture
Camera array	4 global-shutter cameras · 300° × 137° combined field of view
Stereo pair	1920 × 1200 per camera · 110° × 80° field of view
Fisheye cameras	1920 × 1200 per camera · 204° × 137° field of view
On-device processing	34 TOPS of AI compute
Outputs	Multiview video, with 3D body, hand and object motion available after reconstruction

View the capture interface

Live capture interface with four camera views, a depth map and estimated keypoints for both hands

Generative 4Dhand–object reconstruction