LiteReality-Agent turns LiDAR scans into editable 3D scene code

LiteReality-Agent is an Apache-2.0 toolkit that converts a LiDAR room capture into an articulated 3D scene represented by Python code. An LLM agent edits Room.py, renders each revision, compares it with the source imagery, and repeats the process before exporting the result to Blender, Unity, Unreal, web viewers, or physics simulators.

Researchers from Cambridge, Imperial College, and independent teams built the project around a free scanner app for LiDAR-equipped iPhones and iPads. The app records RGB images, depth data, and Apple RoomPlan’s room.usdz description during a walkthrough.

A five-step room build

1. Capture: Walk through the room once with the LiteReality Scanner app.
2. Transfer: Move the RGB, depth, and RoomPlan files into a capture directory.
3. Initialize: Run a deterministic pass that creates the seed scene.
4. Author: Let the agent compare renders with the capture and revise Room.py.
5. Validate and export: Run automated checks, then write the scene to Blender and glTF formats.

Apple’s RoomPlan export supplies a structured estimate of walls, openings, and detected room elements. LiteReality uses that data as the geometric foundation for a scene that can contain separate objects, materials, and articulated parts such as doors or drawers.

The deterministic first pass makes repeated initialization predictable. The agentic second pass handles visual authoring, including material selection, object placement, asset generation, and revisions driven by rendered comparisons. An automated quality-control gate checks the finished result before export.

Room.py becomes the scene graph

The generated Room.py file defines geometry, materials, transforms, and articulation in one program. Developers can inspect changes as source diffs, replay them, modify them manually, and regenerate exports without editing an opaque combined mesh.