Gsplat SOG v2 file format proposal

[slimbuck]

Introduction

PlayCanvas added support for SOG gaussian splat scenes thanks to the work of @vincentwoo and @w-m (see https://github.com/playcanvas/sogs). The SOG format is an image-based encoding of gaussian splat data.

We are now updating this format to better match our runtime requirements:

• rather than using PLAS to order data, we order by morton code instead
  • this results in roughly similar compression by webp
  • it means we don't have to calculate morton code and reorder data at load time
• improve data robustness to cover a wider spectrum of scenes (see SOGS spherical harmonics #20)
  • introduce a codebook for scales, sh0 and shN
  • simplify meta.json layout
  • introduce a single-file, bundled binary format which is more convenient to use compared to separate files

Since this is an evolution on the SOG format, we keep the name, but consider them Spatially Ordered Gaussians.

A SOG scene comprises multiple files:

• meta.json: contains the metadata and the names of the linked webp images
• images: images containing the packed gaussian data, by default in webp format
  • means_u, means_l: upper and lower bits of position
  • scales: sizes
  • quats: orientation/quaternions
  • sh0: color and opacity
  • shN_centroids, shN_labels: (optional) palette of spherical harmonics and their per-gaussian labels

All images have the same dimension apart from shN_centroids.

Metadata v2 format

interface SogMeta {
    version: 2;            // file format version
    count: number;         // number of splats (pixels encoded)
    antialias: boolean,

    means: {
        // Per-axis affine ranges used to (de)normalize log-transformed positions.
        // Values are AFTER logTransform(value) = sign(v)*log(|v|+1).
        mins: [number, number, number];   // [xmin', ymin', zmin']
        maxs: [number, number, number];   // [xmax', ymax', zmax']
        files: ["means_l.webp", "means_u.webp"];
    };

    scales: {
        // Scale codebook for k=256 k-means over [scale_0, scale_1, scale_2]
        codebook: number[];  // length 256
        files: ["scales.webp"]; // per-splat byte labels in RGB
    };

    quats: {
        // Orientation texture only; ranges are implicit in encoding.
        files: ["quats.webp"];
    };

    sh0: {
        // DC color codebook for k=256 k-means over [f_dc_0, f_dc_1, f_dc_2].
        codebook: number[];  // length 256
        files: ["sh0.webp"]; // per-splat byte labels in RGB; A = opacity (sigmoid*255)
    };

    // Present only if higher-order SH (f_rest_*) exists.
    shN?: {
        // A 256-entry 1D codebook built over the SH centroids.
        codebook: number[];  // length 256
        files: ["shN_centroids.webp", "shN_labels.webp"];
        // - shN_centroids.webp packs centroid coefficient triplets per coeff in RGBA.
        // - shN_labels.webp stores per-splat palette indices (uint16 little-endian in RG).
    };
}

Data layout

A summary of the image data is as follows:

 ┌───────────────────┐
 │ means_l.webp      │   Position (log-space, normalized, 16-bit split)
 │───────────────────│
 │ R = x low byte    │
 │ G = y low byte    │
 │ B = z low byte    │
 │ A = 255           │
 └───────────────────┘

 ┌───────────────────┐
 │ means_u.webp      │
 │───────────────────│
 │ R = x high byte   │
 │ G = y high byte   │
 │ B = z high byte   │
 │ A = 255           │
 └───────────────────┘

 ┌───────────────────┐
 │ quats.webp        │   Orientation (compressed quaternion)
 │───────────────────│
 │ R = comp0         │
 │ G = comp1         │
 │ B = comp2         │
 │ A = 252 + index   │   (which component was dropped)
 └───────────────────┘

 ┌───────────────────┐
 │ scales.webp       │   Log-scales
 │───────────────────│
 │ R = scale_x       │   (cluster index, 0–255)
 │ G = scale_y       │
 │ B = scale_z       │
 │ A = 255           │
 └───────────────────┘

 ┌───────────────────┐
 │ sh0.webp          │   Color + opacity
 │───────────────────│
 │ R = f_dc_0 label  │   (cluster index, 0–255)
 │ G = f_dc_1 label  │
 │ B = f_dc_2 label  │
 │ A = opacity (0–255, sigmoid mapped)
 └───────────────────┘

 ┌─────────────────────────┐
 │ shN_centroids.webp      │   (only if higher-order SH present)
 │─────────────────────────│
 │ R,G,B = centroid coeffs │
 │ A = 255                 │
 └─────────────────────────┘

 ┌───────────────────┐
 │ shN_labels.webp   │   Per-splat SH palette index
 │───────────────────│
 │ R = low byte      │
 │ G = high byte     │
 │ B = 0             │
 │ A = 255           │
 └───────────────────┘

Bundled format

It can be inconvenient working with multiple files so we support a bundled version of SOG data.

A bundled scene has a .sog extension and is a simple zip archive (i.e. uncompressed zip) of the constituent files.

The bundle loader uses the following logic when resolving filenames appearing in meta.json:

• if the file is present in the bundle, use it
• otherwise consider the filename a URL and resolve it relative to current href

This will allow bundles to reference resources external to the bundle.

Many thanks to @nobbis for suggesting we use a zip archive instead of custom binary format (see discussion below).

[slimbuck]

For the bundled .sog format we also considered zipping the unbundled files.

However, in terms of file size there would be no benefit, because webp is already compressed.

And the advantage of not zipping is that we can perform range requests directly to load the webp files. This way we get most of the same download benefits of the unbundled version.

[nobbis]

You could use an uncompressed zip file (like USDZ does) and still allow range requests. A stored zip file has several advantages: forwards compatibility, no custom parser, standard tooling, >4GB file sizes, CRC checking.

[lyehe]

Maybe this is a dumb question. If no zipping, does it make sense to stack multiple webps side-by-side then? I don't think that will increase file size by too much.

[lyehe]

Or use multi frames (add black borders to smaller frames). And keep the metadata in XMP. Then you only need a single webp file.

[slimbuck]

You could use an uncompressed zip file (like USDZ does) and still allow range requests. A stored zip file has several advantages: forwards compatibility, no custom parser, standard tooling, >4GB file sizes, CRC checking.

Loading an uncompressed zip over a network would require an extra request:

• request zip header and file size
• request zip file offsets (from end of file)
• request files

In the case of a binary file you can save one round trip:

• read header (includes file offsets)
• request files

[slimbuck]

Maybe this is a dumb question. If no zipping, does it make sense to stack multiple webps side-by-side then? I don't think that will increase file size by too much.

The webp images are loaded into webgl/WebGPU texture memory (by the browser) and having them be a consistent size is desirable. Combing them could result in a too-large texture not supported by gpu.

Also, loading multiple images in parallel results in much faster loads on fast connections than a single load request.

[nobbis]

In the case of a binary file you can save one round trip:

• read header (includes file offsets)
• request files

You don't need to know file size to make an end of file request, e.g. Range: bytes=-65536, so you'd multiplex with the header. Same number of round trips.

[slimbuck]

You don't need to know file size to make an end of file request, e.g. Range: bytes=-65536, so you'd multiplex with the header. Same number of round trips.

Yes I guess you're right. Loading n bytes from the end of the file seems messy (and then extracting the last meta.json), but perhaps worth it given the advantages of using a standard zip format.

I actually wrote a simple uncompressed zip writer for supersplat's .ssproj format, so I could test this quite easily.

Thanks for your input @nobbis (and @lyehe) it's much appreciated!

[Maksims]

Is it possible to use an existing bundles system, which is simply a tar? It is pretty flexible, allows to bundle as many things together. The bundling process is very simple: just tar'ing files together.

It is not a proprietary bundle format, so people will have existing tools to bundle and unbundle if needed.

I believe with current bundles it is already possibile, just not "convenientised" enough.

[slimbuck]

Is it possible to use an existing bundles system, which is simply a tar? It is pretty flexible, allows to bundle as many things together. The bundling process is very simple: just tar'ing files together.

It is not a proprietary bundle format, so people will have existing tools to bundle and unbundle if needed.

I believe with current bundles it is already possibile, just not "convenientised" enough.

I investigated tar and a few other archiving options, but they don't have a table of contents easily accessible (files must be stepped through).

It looks like a zip archive will support this though, as @nobbis has suggested.

[jo-chemla]

Thanks for sharing Playcanvas workflow regarding enhancing the SOG format, really amazed to see you building out in the open!

Sharing a few thoughts/feedbacks here (ended up being a bit ore than a few, beware long read):

• First, indeed using an existing uncompressed zip for container will probably facilitate implementation for other renderers - either web or native apps. A successful similar take is the SZI format for large image pyramids see spec or below screen, which is basically an uncompressed DZI directory very much used for microscopy/astronomy images. Produced with libvips and js reader implementation highlighting how they fetch the zip central-directory at the end-of-file - probably variable length because szi can have a large directory structure. Other single-file tiled formats discussed in the OpenSeaDragon repo here, mentions also some hurdles you might face at the end - [1] gzip transport has some browser inconsistencies esp with firefox, [2] don't use head requests to get actual filesize etc

Image format:

• format: it's great you specify what is the content of your images (+ metadata) and how to decode that into splats attributes, but is it necessary to also mention how these images are encoded? You decided to use webp after probably some time tinkering with different options, but should not this be left as the responsibility of other data-producers or renderers implementation? They might decide otherwise for various reasons - some we might even discover in the future - so I'm questioning whether this should be part of the format specification. In the end, since you are the initial implementor, it is also probable the choices you do will influence upcoming implementations but some room might be left here.
• KTX2: Regarding alternatives to webp for encoding SOG images, it will be a choice of file-size/compression ratio which impacts bandwidth necessary for the scene to reach the client, vs total time it takes for the image to reach the gpu memory from CPU thread as well as image size in gpu memory. On our end, when encoding massive meshes as 3D-tiles tilesets (HLOD hierarchies of glTF tile meshes with textures), we took the decision to use KTX2 to have a 3x increase in file-size but VRAM usage reduced 10x (see screen in details), and the texture became GPU ready in shader faster. It is probably a pretty different use-case because our textures look more natural and compression probably works better on these than on SOG noisy textures
• AVIF/JXL: Comparing "non gpu-ready" image formats, WebP vs eg AVIF or JPEGXL, comments in this thread by J. Cupitt from libvips roughly stated AVIF/AV1 being more future-proof (vs WebP) for compressed images for the web, at least until JPEGXL has better browser support (JXL has a lot of the old webp devs behind it).
• TIFF: also wanted to mention TIFF which could support a handful of your requirements - apart maybe from the GPU-ready data I introduced. Great GDAL doc on the GTiff format. This is what ended-up being used as a basis for COG Cloud Optimized Geotiff (for massive raster datasets), for its extensibility. Such images can:
  • have arbitrary number of bands, so you could encode them all into a single image file - maybe one image for means_u, means_l, scales, quats, sh0, and another image for higher-level spherical harmonics
  • be read/write by all imagery softwares (probably not very useful) or CLI js/py/whatever tiff RW libraries and SDKs
  • be stored with contiguous memory blocks BLOCKXSIZE=YES, BLOCKXSIZE, BLOCKYSIZE so you could stream splats (with all their attributes) before loading the whole with INTERLEAVE=[BAND​/​PIXEL]
  • Compression can take values among (shortlist) COMPRESS=[JPEG​/​/​DEFLATE​//​WEBP​/​JXL​/​NONE]
  • Natively supports internal overviews, basically subsampled versions of the image raster. You could even use subdatasets in case you want multiple SOG splats scenes shared within the same tiff file. In the COG case tiff pages are used to store downsampled versions of the same scene, with the same tile block structure, to allow readers to efficiently fetch the data best fit to the user viewport.

Other possible container: glTF?
Combining all these thoughts together also yielded a possible nice container alternative (compared to uncompressed zip), also since I saw you comment on the PR for glTF extension KHR_gaussian_splatting - joining discussions for a possible KHR_gaussian_splatting_SOGv2_compression incoming 👀. There is already a pretty rich ecosystem for glTF, with the aim at storing/transferring as much gpu-ready assets as possible. glTF can be exploded as multiple files (which for meshes are usually json metadata, bin geometry, and image texture files), but can also be stored in a glb binary container. For initial testing, you could even just have a single tri/quad geometry with references to all the existing webp texture files each in its slot, and even tinker with glTF-transform to parse through that splat scene and convert all textures at once to ktx2 or bundle to binary glb container. You probably have in mind gathering community feedback to finalize a solid possible file format that could be encapsulated as a glTF extension, just wanted to mention it.

Hope this helps, and sorry for the lengthy read!

SZI spec p.5

KTX vs WebP vs AVIF memory footprint