Turns a short video (a slow orbit of a subject) into a single .ply Gaussian splat.

Note

This cog sometimes fails on Replicate, it appears that long running jobs on Replicate can get pre-empted. If your prediction is failing, try lower values for splats / iterations / frames to increase processing speed and therefore the chance of the job succeeding.

This is a Cog packaging of vid2scene by Sam / SamuSynth. All the real work is theirs; vid2scene was open sourced under Apache 2.0 [1] and I have adapted this for running on Replicate model even though it’s not a traditional AI model with weights. This just makes the reconstruction pipeline callable in the same fashion as an AI model and therefore easily meter-able as a per-job variable cost.

Source

Original source: https://github.com/samuelm2/vid2scene

Cog build/deploy (modified) source and instructions: https://github.com/3DStreet/vid2scene-cog

Usage

Upload a video and (optionally) tune:

target_framecount: frames sampled from the video training_num_steps: gsplat steps (quality vs. speed) training_max_num_gaussians: caps Gaussian count / ply size & detail reconstruction_method, equirectangular, use_background_sphere, remove_background, apriltag_size_meters (real-world scale)

Output is one .ply that you can drop in to any splat viewer.

Runs on L40S.

About

Built for use in 3DStreet.app (https://3dstreet.app) to mix output splats with 3D maps and custom geometry. Provided as-is under Apache 2.0, with no warranty or guarantee of updates; fork and self-host if you’d rather run your own.