Objects and parts
Color-coded views show individual objects and parts within each reconstructed scene.
World perception models for embodied AI
From a photograph to a world you can edit, interact with and manipulate.
Understanding object structure and physical properties
Reconstruct editable, interactive 3D environments from photographs, with spatial relationships, internal structures and moving parts that support manipulation in simulation.
Choose a scene to explore its structure and motion.
Kitchen
Cabinet doors swing open, drawers slide out, and the microwave and oven doors reveal their interiors.
Watch the scenes side by side
From object structure to physical interaction
Doors rotate about hinges. Drawers follow a sliding direction. Objects fall, collide and come to rest. Structure and physical properties shape how a scene responds to interaction.
Joints and motion
Hinges and sliding joints define how a part moves and its range of motion.
Physical properties
Mass, friction and collision geometry provide the simulation with the properties needed for contact and motion.
Edit objects within the scene
Change a wooden tabletop to marble or a pendant lamp to brass with a text instruction. Objects can be edited individually within the reconstructed scene.
Build a world model for your task
Tell us about your target environment and the interactions you want to simulate.
Technical foundations
From observations to actionable environments
Object reconstruction and part generation provide a structural foundation for actionable 3D environments. Scale calibration, physical properties and simulation checks prepare scenes for specific interaction tasks.
Reconstruct objects and where they belong
CUPID jointly estimates object geometry and camera pose, using image-aligned features to preserve observed detail. SPOON uses multi-view geometry to coordinate object scale, position and orientation in a coherent scene. Together, these research directions address fidelity to observations and the spatial arrangement of objects.
Re-rendering from the recovered camera pose checks fidelity to visible evidence. Regions outside the photograph are completed by the model and can be checked against additional views.
Explore the scene-layout method
Preserve part detail and whole-object structure
STRUCTPARTS alternates between part-local and object-global coordinates to preserve fine geometry and the arrangement of parts. Changing the segmentation groups allows parts to be split or merged. Explicit parts provide a basis for assigning joints and motion limits.
Test structure and motion in simulation
A scene intended for robot manipulation needs to be tested in simulation. The team’s workflow brings scale calibration, part modeling, physical properties and simulation checks together, using the results to validate structure and motion.
Calibrate camera and scale
Use structural lines and objects with known dimensions as references, then cross-check the scale against other standard objects.
Align objects and parts
Align objects to shared planes such as walls, counters and cabinet fronts, while keeping doors and drawers as separate parts.
Assign motion and physical properties
Assign hinges, sliding joints and motion limits to moving parts, and estimate object mass from material.
Check rendering and simulation
Compare rendered views with the input photograph. Check stability, full part motion and interpenetration in MuJoCo, followed by an independent validation pass.