Inhabit

Phone video → metric 3D reconstruction → a room a robot can explore.

The full pipeline in one shot: phone capture → metric 3D reconstruction → a Go2 robot exploring the scanned room at metric scale.

Fidelity check on a benchmarked scene (Replica room0): original video (left) and our reconstruction (right), along the same path. Sub-centimetre accuracy against the ground-truth mesh.

Interactive viewers (live)

Quality vs ground truth (Replica, visibility-culled · cm / F-score@5cm)

Method	Accuracy ↓	Completion ↓	Chamfer-L1 ↓	F-score ↑
PGSR	1.13	7.60	4.37	0.898
DN-Splatter	0.57	6.14	3.36	0.936
Consensus (ours)	1.07	6.47	3.77	0.913

5-scene average. Sub-centimetre accuracy. Full protocol + per-scene table in the repo's docs/BENCHMARK.md.

Chamfer-L1 (lower is better) and F-score@5cm (higher is better) across the three methods, on the Replica average and on a real iPhone capture.

All five benchmark scenes, input video (left) vs reconstruction (right) along the same path. Top: room0, room1, room2; bottom: office0, office1.

And on a real iPhone capture

A real handheld iPhone scan (MuSHRoom coffee_room), reconstructed and validated at roughly 2 cm against a Faro laser ground-truth mesh. Rougher than the rendered scenes, as expected for real capture.

A robot moves through it, at metric scale

A Go2 quadruped traverses the phone-scanned room in Genesis at metric scale (fixed camera, planned path with a trot gait; a kinematic traversal, since physics-driven locomotion needs a trained policy). Rigid-body physics is validated separately: a dropped object rests on the floor within 2 cm. See scripts/genesis/ and docs/PHASE2_GENESIS.md.

Why this matters

Reconstruction is the front half of the loop; the back half is what the metric mesh is for. Because the output is metric and loads as a rigid-body collider, every capture becomes an environment an agent can be trained and tested in. Paired with a physics engine like Genesis, a phone becomes an environment generator.

• Training data at scale. Each scan is a sim-ready environment. Domain-randomise it and render egocentric trajectories to produce labelled embodied data, with no capture rig and no manual scene authoring.
• Evaluation in diverse, real environments. A new test scene is just a phone recording: kitchens, stairwells, cluttered offices, the long tail of real rooms instead of a few hand-built ones.
• Real-to-sim from data you already have. Egocentric and teleoperation footage becomes simulatable RL environments, so policies can be re-tested against the exact out-of-distribution situations the fleet met in the real world.

Code

Repo & full write-up on GitHub — see the repo, docs/ARCHITECTURE.md, and docs/BENCHMARK.md.