re_renderer: cross-platform shader imports & build-system

[teh-cmc]

Alright, I'll just quote the module docs directly.

---

Main module

This module implements one half of our cross-platform #import system.

The other half is provided as an extension to the build system, see the build.rs file
at the root of this crate.

While it is agnostic to the type of files being imported, in practice this is only used
for shaders, thus this is what this documentation will linger on.
In particular, integration with our hot-reloading capabilities can get tricky depending
on the platform/target.

Usage

#import <x/y/z/my_file.wgsl>

Syntax

Import clauses follow the general form of #import <x/y/z/my_file.wgsl>.
The path to be imported can be either absolute or relative to the path of the importer,
or relative to any of the paths set in the search path (RERUN_SHADER_PATH).

The actual parsing rules themselves are very barebones:

• An import clause can only span one line.
• An import clause line must start with #import (exl. whitespaces).
• Everything between the first < and the last > is interpreted as the import
  path, as-is. We do so because, between the 4 major platforms (Linux, macOS, Window, Web),
  basically any string is a valid path.

Everything is trim()ed at every step, you do not need to worry about whitespaces.

Resolution

Resolution is done in three steps:

1. First, we try to interpret the imported path as absolute.
   1.1. If this is possible and leads to an existing file, we're done.
   1.2. Otherwise, we go to 2.

2. Second, we try to interpret the imported path as relative to the importer's.
   2.1. If this leads to an existing file, we're done.
   2.2. Otherwise, we go to 3.

3. Finally, we try to interpret the imported path as relative to all the directories
   present in the search path, in their prefined priority order, similar to e.g. how
   the standard $PATH environment variable behaves.
   3.1. If this leads to an existing file, we're done.
   3.2. Otherwise, resolution failed: throw an error.

Interpolation

Interpolation is done in the simplest way possible: the entire line containing the import
clause is overwritten with the contents of the imported file.
This is of course a recursive process.

A word about #pragma semantics

Imports can behave in two different ways: #pragma once and #pragma many.

#pragma once means that each unique #import clause is only be resolved once even if it
used several times, e.g. assuming that a.txt contains the string "xyz" then:

#import <a.txt>
#import <a.txt>

becomes

xyz

#pragma many on the other hand will resolve the clause as many times as it is used:

#import <a.txt>
#import <a.txt>

becomes

xyz
xyz

Both have their use cases and drawbacks, there's no "right solution".

At the moment, our import system only provides support for #pragma many semantics.
We will most likely add support for #pragma once at some point.

Hot-reloading: platform specifics

This import system transparently integrates with the renderer's hot-reloading capabilities.
What that actually means in practice depends on the platform/target.

A general over-simplification of what we're aiming for can be expressed as:

Be lazy in debug, be eager in release.

When targeting native debug builds, we want everything to be as lazy as possible, everything
to happen just-in-time, e.g.:

• We always talk directly with the filesystem and check for missing files at the last moment.
• We do resolution & interpolation just-in-time, e.g. just before calling
  create_shader_module.
• Etc.

On the web, we don't even have an actual filesystem to access at runtime, so not only we'd
like to be as eager can be, we don't have much of a choice to begin with.
That said, we don't want to be too eager either: while we do have to make sure that every
single shader that we're gonna use (whether directly or indirectly via an import) ends up
in the final artifact one way or another, we still want to delay interpolation as much as
we can, otherwise we'd be bloating the binary artifact with N copies of the exact same
shader code.

Still, we'd like to limit the number of differences between targets/platforms.
And indeed, the current implementation uses a virtual filesystem approach to effectively
remove any difference between how the different platforms behave at run-time.

Debug builds (excl. web)

Native debug builds are straightforward:

• We handle resolution & interpolation just-in-time (i.e. when fetching file contents).
• We always talk directly to the filesystem.

No surprises there.

Release builds (incl. web)

Things are very different for release artifacts, as 1) we disable hot-reloading there and
2) we never interact with the OS filesystem at run-time.
Still, in practice, we handle release builds just the same as debug ones.

What happens there is we have a virtual, hermetic, in-memory filesystem that gets pre-loaded
with all the shaders defined within the Cargo workspace.
This happens in part through a build script that you can find at the root of this crate.

From there, everything behaves exactly the same as usual. In fact, there is only one code
path for all platforms at run-time.

There are many issues to deal with along the way though: paths comparisons across
environments and build-time/run-time, hermeticism, etc...
We won't cover those here: please refer to the code if you're curious.

For developers

Canonicalization vs. Normalization

Comparing paths can get tricky, especially when juggling target environments and
run-time vs. compile-time constraints.
For this reason you'll see plenty mentions of canonicalization and normalization all over
the code: better make sure there's no confusion here.

Canonicalization (i.e. std::fs::canonicalize) relies on syscalls to both normalize a path
(including following symlinks!) and make sure the file it references actually exist.

It's the strictest form of path normalization you can get (and therefore ideal), but
requires 1) to have access to an actual filesystem at run-time and 2) that the file
being referenced already exists.

Normalization (not available in std) on the other hand is purely lexicographical: it
normalizes paths as best as it can without ever touching the filesystem.

See also "Getting Dot-Dot Right".

Hermeticism

When shipping release artifacts (whether web or otherwise), we want to avoid leaking state
from the original build environments into the final binary (think: paths, timestamps, etc).
We need to the build to be hermetic.

Rust's toolchain already takes care of that to some extent, and we need to match that
behaviour on our side (e.g. by not leaking local paths), otherwise we won't be able to
compare paths at runtime.

Think of it as chrooting into our Cargo workspace :)

In our case, there's an extra invariant on top on that: we must never embed shaders from
outside the workspace into our release artifacts!

Things we don't support

• Async: everything in this module is done using standard synchronous APIs.
• Compression, minification, etc: everything we embed is embedded as-is.
• Importing via network requests: only the (virtual) filesystem is supported for now.
• Implicit file suffixes: e.g. #import <myshader> for myshader.wglsl.
• Embedding raw Naga modules: not yet, though we have everything in place for it.

Build extension

This build script implements the second half of our cross-platform shader #import system.
The first half can be found in src/file_resolver.rs.

It finds all WGSL shaders defined anywhere within our Cargo workspace, and embeds them
directly into the released artifact for our re_renderer library.

At run-time, for release builds only, those shaders will be available through an hermetic
virtual filesystem.
To the user, it will look like business as usual.

See re_renderer/src/workspace_shaders.rs for the end result.

[Wumpf]

amazing work, love it!

The only thing that tripped me off was the env var query in the release version of the file macro, that seems wrong to me and I believe that corner could be simplified with a more pragmatic view on path normalization (aka never do canonicalize in memfs world ever, also do canonicalize/normalize late), but I might be missing details.

[teh-cmc]

The only thing that tripped me off was the env var query in the release version of the file macro, that seems wrong to me and I believe that corner could be simplified with a more pragmatic view on path normalization (aka never do canonicalize in memfs world ever, also do canonicalize/normalize late), but I might be missing details.

I think a lot of the confusion comes from the fact that this is a compile-time env!() query, not a run-time one (std::env::var)? Maybe I'm misunderstanding your misunderstanding though, let's huddle 😄

[Wumpf]

niiiiice (:

[Wumpf]

🚢!!