Allow archetypes to clear and/or update existing components

[emilk]

Backstory

• provide empty() method on Rust archetype implementations to log empty #3279
• Decide if archetypes logging APIs should log non-provided components as empty-lists  #3341

What should the following code do:

log("points", Points3D::new(positions).with_colors(…));
…
log("points", Points3D::new(new_positions));

Should the new positions inherit the colors of the previous log call?

That would be quite non-intuitive.

Related to this: how does one update just the colors of a point cloud?

Design

We've all converged on the following API:

// We do NOT implement `Default` for archetypes.
// users need to explcitly use `clear` or `update`
struct Points3D {
   positions: Optional<Vec<Vec3>>,
   color: Optional<Vec<Color>>,
   radii: Optional<Vec<f32>>,
}

impl Points3D {
    /// Clear all components of this archetype to the empty list.
    fn clear() -> Self {
        Self {
            positions: Some(vec![]),
            color: Some(vec![]),
            radii: Some(vec![]),
        }
    }
    
    /// Create a new `Point3d` point cloud with some positions,
    /// and with emtpy lists for all other archetypes.
    fn new(positions: Vec<Vec3>) -> Self {
        Self {
            positions: Some(positions),
            ..Self::clear()
        }
    }
    
    /// An empty `Point3d` point cloud, with `None` for each component.
    fn update() -> Self {
        Self {
            positions: None,
            color: None,
            radii: None,
        }
    }
}

fn main() {
    // Log a point cloud:
    log("points", Points3D::new(…).with_colors(…));
    
    log("points", Clear{}); // will clear all components of all archetypes
    log("points", Points3D::clear()); // will clear this archetype of the entity
    log("points", Points3D::new(positions).with_colors(…)); // overwrites everything
    log("points", Points3D::update().with_colors(…)); // overwrites only colors

    // This is also possible, though a somewhat weird way to write it:
    log("points", Points3D::clear().with_colors(…)); // overwrites everything
}

Python:

log("points", Clear()); # will clear all components of all archetypes
log("points", Points3D.clear()); # will clear this archetype of the entity
log("points", Points3D(positions, colors=[…], radii=None); # overwrites everything, even radii
log("points", Points3D.update(colors=[…], radii=None)); # overwrites only colors and radii

Necessary changes

• All components are Options, even the required ones
• When querying, an empty lists should be treated the same as a missing components
• In the UI we should probably hide empty components in the streams view

Component lists implications

A component list (e.g. radii) can have one of the four special values:

• null/missing
• 0 length ("empty")
• 1 length ("splat")
• N length (1<N)

null components aren't logged, but empty components are.
When querying the store, missing components are treated the same as empty components.

Related issues:

• Visualize density graph of empty batches differently #7117

[emilk]

If this proves to be more complex than anticipated, then we'll push it to 0.10

[emilk]

Needs a follow-up where we hide empty components from the streams view

[emilk]

We need to think through how num_instances would work for an update. We could take the max-length of all its components, but that won't work if you're only logging splats. Perhaps we don't need num_instances for updates; it's really only used for the primary (e.g. the indicator).

[jleibs]

This API should also handle specifying the InstanceKeys for the collection that's being updated.

[teh-cmc]

Instance keys and num_instances are not a problem anymore (or won't be, soon).

[jleibs]

@teh-cmc we were talking through this with respect to Transforms I realized this approach has the potential to cause some annoying performance issues for chunks.

In particular, every unused optional component will still require growing the offset array for each of the optional components by one element (in many cases, just a repeated 0). This is similar to the overhead of growing sparse unions (growing each child array), and strictly worse than the overhead of sparse unions (just growing one child array).

This suggests to me we're going to need some kind of chunk-specific optimization.

This also has an annoying ripple-effect on datafusion / dataframe UI of showing a bunch of empty / cleared (not nulled) columns all over the place.