Synthetic ground truth for 3D reconstruction: 3,879 Unreal Engine frames, 35.7 million points, and the error I found in my own depth
retained text
Every frame my Unreal Engine capture plugin renders comes with a depth map in metres and the camera's pose and intrinsics. I wanted to see how far that gets you with no reconstruction algorithm at all, so I fused 13 captures straight from those files: lift every pixel with its depth, place it with the recorded pose, drop anything labelled (the people are re-placed every frame), average into 10 cm cubes. 3,879 frames, 35.7 million points, 45 to 112 seconds per map. No feature matching, no pose estimation.
Then I checked whether the frames agree with each other. Take a frame, project its depth into the nearest frame that sees the same place using only the recorded poses and intrinsics, and compare with the depth that frame recorded. The median difference is 5 to 15 mm at median distances of 25 to 60 m.
The part I didn't expect: that median is almost exactly what rounding produces. My depth.npy files are float32, but they hold half-float values, because the render target was 16-bit. There are never more than 1,024 distinct values between one power of two and the next, so the steps are 2 cm between 20 and 41 m. Simulating rounding alone at 2 cm gives a median of 5.2 mm. The Paris flyover measures 5.1. So the poses are more precise than the depth can show. The fix is a render target change that's in the plugin source now; every capture in the video predates it.
Where frames disagree most: vegetation (wind or LOD, I haven't separated the two), the sea on the beach (the surface moves, and that pair is fog at noon against clear at 7 am), and tower faces seen almost edge-on. One I can't explain: a flat quarry floor where two frames disagree by more than 5 cm across a whole patch. If you've seen that before, I'd like to know what it was.
A gotcha I only noticed while writing it up: a flipped axis passes this check. Every frame is mirrored the same way, so they still agree with each other. You only catch it by comparing the map with a rendered frame.
The viewer is plain WebGL2 with eye-dome lighting. The points are stored in the order of the first frame that saw them, so the replay in the video is just drawing more of them each frame.
If you work on MVS, NeRF or SLAM with synthetic data: would a per-scene consistency number like this change which dataset you pick? And what would you want exported besides the per-frame camera JSON? COLMAP and transforms.json are the obvious ones.document fetched 2026-09-27 00:23 · canonical https://reddit.com/r/computervision/comments/1wqxpc7/synthetic_ground_truth_for_3d_reconstruction_3879