Feature: Upstream `bevy_web_asset`, allowing assets loaded via `http`

[mrchantey]

What problem does this solve or what need does it fill?

Being able to load assets from http and https urls.
This is particulary relevant for examples, files that are too big for source control are often stored in external zip files requiring awkward download & extract workflows for people just getting started with a crate.
It also makes both wasm and native binaries more portable, as the additional 'assets' folder isnt required to always be placed alongside them, a big win for game jams etc.

What solution would you like?

For url assets to 'just work', meaning if a path starts with http/https the asset will be loaded via the http client:

# intercepted by http client
asset_server.load("http://example.com/fox.glb");
# https also accepted
asset_server.load("https://example.com/fox.glb");
# load normally
asset_server.load("/fox.glb");

What alternative(s) have you considered?

Keep this as a third party crate

Pros:

• No changes/maintenance needed for bevy core

Cons:

• Bevy examples face same pain point mentioned above
• No blessed solution for this issue in ecosystem crates

Additional context

Considerations

• Disk Caching: Browsers usually quietly (and unpredictably) cache files on disk so that they arent fetched on every reload, we will need something similar for native builds, and possibly a more predictable implementation on wasm builds, for example using IndexedDB.
• Choice of HTTP client: It makes sense to go with the same client used elsewhere in Bevy, ie bevy_remote, unless it is missing some critical feature.
• Feature Flags: Because this adds a http client to bevy_asset it should be behind a feature flag, we will need to decide whether this is to be included by default, in my opinion it should be.
• CORS: files fetched by native builds dont need cors enabled but it will be required for wasm builds to work, this creates an inconsistency that we may want to flag on native builds if the headers are not present.
• Work: @johanhelsing has mentioned that they dont have the capacity to work on this at the moment. I'm happy to get the ball rolling when I get some time, though this will be my most contraversial PR so I may need some mentorship.

bevy_web_asset is a tiny crate, with <250 lines of code. I guess the most work required here will be a caching solution.

[alice-i-cecile]

This would be useful for #13875, to the point where I think something shaped like this is probably a prerequisite.

[villor]

On caching: I think the simplest solution here is to rely on HTTP-caching. Maybe add an option to put a hash of each asset as a suffix on the file name, along with generating a manifest (never cached) containing each up-to-date path. This way we can fully rely on the browsers cache while ensuring changed assets are reloaded. (Similar to what web tools like Vite do) Maybe this would be possible with asset preprocessing? 🤔

The above could solve caching on WASM builds. For native builds we might be able to use the same approach with a client library that supports caching.

Choice of HTTP client: It makes sense to go with the same client used elsewhere in Bevy, ie bevy_remote, unless it is missing some critical feature.

bevy_remote does not use any HTTP client, just the server implementation. The client example uses ureq directly.

ureq is not async though. reqwest or surf might be better alternatives. Both has cache middleware implementations through the http-cache crate.

[mrchantey]

Yes its totally reasonable for us to use default http caching, but if we go that way we should make the limitations clear to users and ensure the api is extensible so an ecosystem crate can implement a more formal mechanisim.

Limitations

These are the particular issues i have with the http cache spec:

1. It is largely opaque with very few guarantees
2. Each browser implements it differently
3. Even chrome doesnt cache large files approx > 25Mb

For me the third issue is the dealbreaker. Afik the only solution for large files like video, complex 3d models or client-side AI models is IndexedDb. Browsers can still evict that content if they need to but theres a high chance of it being available across page reloads, which is particularly important during development and unit tests.

For reference:

• MDN Storage Docs outlines some browser differences
• Chrome Storage Docs goes deeper on chrome policies
• http cache spec describes what is and isnt guaranteed

Next steps

I think there's enough for a first draft:

1. Use built-in browser caches and the http-cache middleware for surf
2. Ensure the api is extensible for third parties to implement bevy_big_web_asset or similar

Maybe once thats in and we've all had a chance to play with the behavior we can revisit manifests, large files etc in a second iteration.