System Design

One of the biggest challenges in real-time Gaussian splat rendering is sorting the splats so that they can be drawn and blended in back-to-front order, known as the Painter's algorithm.

Rendering data flow cycle

ForgeRenderer is a key component in Forge that manages this process. It traverses the visible THREE.js scene graph and compiles a complete list of all splats across the scene, generated by instances of SplatMesh in the scene hierarchy.

Each ForgeRenderer has a default ForgeViewpoint that reads back a list of all splat viewpoint distances from the GPU, then determines the splat draw order using an efficient bucket sort algorithm, run in a background worker thread via SplatWorker. You can spawn additional ForgeViewpoints to create multiple simultaneous render viewpoints.

Finally, on the next THREE.js render() call, ForgeRenderer invokes a single instanced geometry draw call to draw all the scene's splats in the correct back-to-front order, merging with other opaque THREE.js geometry using the Z buffer. The sort order lags the render by at least one frame, but possibly more on older devices, but is not usually perceptible.

// Optionally add a ForgeRenderer to the scene to manage SplatMesh rendering.
// If none is created, Forge will create one for you automatically.
const forge = new ForgeRenderer({ renderer: webGlRenderer });
scene.add(forge);

This design allows splats from distinct objects/scenes to coexist in space and sort correctly w.r.t. each other's splats. Splats from independent SplatMeshes are aggregated using a SplatAccumulator, which produces a PackedSplats, a collection splats stored in a cache-efficient 16-byte/splat format.

"Programmable Splats"

Forge also uses this opportunity to run a user-programmable data pipeline on each splat on the GPU. The standard pipeline provides high-level functionality, such as applying rigid transforms, adjusting RGB / opacity, and spherical harmonics, but also color editing and perturbations (via SplatEdit) and a dual-quaternion skeletal animation system (SplatSkinning). The standard pipeline also allows injecting arbitrary code to modify each splat via dyno shader graph system.

SplatMesh derives from a more general base class SplatGenerator, which itself derives from THREE.Object3D. As such, it can be placed anywhere in the scene hierarchy and obeys expected local and global coordinate transforms. A SplatGenerator is the most general form of a "splat object", whose splats are produced programmatically via a dyno shader graph function that maps { index: "int" } to { gsplat: "Gsplat" }. A SplatMesh is a higher-level object that implements such a mapping, reading source splats from a template (loaded via a url constructor parameter or otherwise) at the given index, then applying functions such as transforming to world space.

In contrast, implementing a SplatGenerator gives you full control to write any function that programmatically computes a splat's attributes (center, scales, quaternion, rgba). These could be stateless (relying only on index, random-number generators, etc), or could rely on a complex combination of splat files, textures, and other global parameters for real-time procedural generation, and can vary with time to produce real-time animations.

The dyno shader graph system allows you to create these programmatic pipelines with Javascript code, which is synthesized into GLSL code and compiled and run on the GPU. This dyno system powers other components of Forge as well, such as Readback (which can perform any computation and read back the resulting value), used to compute the sort distance metric for pairs of splats and read them back for CPU sorting.