WASM support

[Byron]

The ability to compile to --target wasm32-unknown-unknown and wasm32-wasi. Here we track what's needed to make this happen as well as the progress thus far.

Note that this doesn't necessarily mean everything will work there - that's an entirely separate task

Tasks

• CI setup for compiling crates to WASM, to avoid regressions (--target wasm32-unknown-unknown seems like a good baseline)
• preliminary wasm32 support for git-pack #735
• Analysis - know what works and what doesn't

Crates that need the wasm feature toggle to compile

• git-diff
• git-pack

Crates to be fixed

Fixing them might move them up to the list of crates with the wasm feature toggle. Major crates to fix are git-sec and git-tempfile - with these more would probably run right away.

• git-config
• git-config-value
• git-credentials
• git-discover
• git-index
• git-lock
• git-prompt
• git-protocol (due to git-sec)
• git-ref
• git-sec
• git-tempfile
• git-transport (due to `git-sec
• git-url - home crate should be optional - it doesn't compile
• gix

Features of git-features that won't compile

And that's OK as wasm32 doesn't need the fastest implementation just yet.

• zlib-ng
• zlib-ng-compat
• zlib-stock

What to target

• look at this insightful comment which provides an idea on how WASM support could look like.

[dzmitry-lahoda]

you may consider support of no_std(it is the proper way to do wasm rather than just compile to target). afaik embedded and blockchain targets are no_std and these (wasm stems from being no_std

[dzmitry-lahoda]

on top of allowing in-browser ide to work better, this feature will allow more secure software release and dependency verifications. (there is usefull stuff to git in blockchain to verify gpg signatures. code oracle later when nix will be on chain, can verify whole build, for now just ci oracle would be good. ) just couple of words about future application.

[dzmitry-lahoda]

after no_std is done for core libs, can support wasi (os/fs interface of some no_std hosts). that is separate task.

[uberroot4]

Hi @Byron, as there was not much effort/interest in this issue the last two years I want to ask if there are any plans currently?
Looking at the list of crates to be fixed, are they still up-to-date? As there was much change since opening the issue, I assume it is not exhaustive anymore?

I executed cargo tree in the gix package and tried to come up with a bottom-up list to identify an order of the crates to be fixed:

• gix-hash
• gix-validate
• gix-trace
• gix-tempfile
• gix-utils
• gix-actor
• gix-date
• gix-chunk
• gix-config-value
• gix-quote
• gix-features
• gix-path
• gix-sec
• gix-fs
• gix-traverse
• gix-object
• gix-attributes
• gix-index
• gix-commitgraph
• gix-ref
• gix-discover
• gix-revwalk
• gix-credentials
• gix-archive
• gix-diff
• gix-hashtable
• gix-ignore
• gix-config
• gix-filter
• gix-odb
• gix-pack
• gix-negotiate
• gix-worktree
• gix-pathspec
• gix-mailmap
• gix-command
• gix-revision
• gix-status
• gix-url
• gix-refspec
• gix-dir
• gix-merge
• gix-submodule
• gix-worktree-state
• gix-worktree-stream
• gix

There might be better tools to invert me a whole cargo tree, however this was the "simplest" variant for me.

As @dzmitry-lahoda correctly pointed out that the no_std feature would be the cleanest version of bringing gix into the browser (at least for a readonly version). Maybe you could give me an idea what crates you consider "core" (i.e. a no_std would be good)? Maybe no_std is not the best option at all imo as git cannot work without an std env and having a no_std feature means rewriting probably most of the crates Cargo.toml.

I already started working on porting some crates, however in wasm32-unknown-unknown (i.e. wasm-in-the-browser) the biggest challenge is the need for async reading/writing files/directories. For now I simply use an patched version of the web_fs crate. I am, however, not familiar if async is also required for e.g. wasm-wasi.

Another thing I would like to know, as of right now I messed up the use of #[cfg(feature = "wasm")] and #[cfg(target_arch = wasm32)]. Are there any preferences/project-guidelines when to use what?

Thanks in advance.

[Byron]

Hi @Byron, as there was not much effort/interest in this issue the last two years I want to ask if there are any plans currently?

It's correct that this is currently a community-contributed feature, I am not working on it.

Looking at the list of crates to be fixed, are they still up-to-date? As there was much change since opening the issue, I assume it is not exhaustive anymore?

The list could be re-evaluated for sure - the ones which failed will still fail, and there may be some new ones to check. It's probably easiest to check Cargo.toml/workspace.members for an exhaustive and flat list.

As @dzmitry-lahoda correctly pointed out that the no_std feature would be the cleanest version of bringing gix into the browser (at least for a readonly version). Maybe you could give me an idea what crates you consider "core" (i.e. a no_std would be good)? Maybe no_std is not the best option at all imo as git cannot work without an std env and having a no_std feature means rewriting probably most of the crates Cargo.toml.

Doing so might be harder than just getting it to compile for the browser, even though I wouldn't argue that it's the most flexible option. I'd approach it from "what do I want to do in the browser" - then usually clone comes up which definitely exercises quite a bunch of crates.

I already started working on porting some crates, however in wasm32-unknown-unknown (i.e. wasm-in-the-browser) the biggest challenge is the need for async reading/writing files/directories. For now I simply use an patched version of the web_fs crate. I am, however, not familiar if async is also required for e.g. wasm-wasi.

That's exciting, thank you! It's good that there isn't too much file-IO going on, and but when it is it's definitely all blocking. I don't know how to best solve this except for assuming that a 'git server' is going to be run from a web-worker and the API is completely message-based. Then it's natively asynchronous, while the core can still be blocking in nature.
That seems cleanest to me, and like it's a better fit for how Gitoxide works.
I see it like git::clone_or_open(<url>, <identifier/handle>) to get a handle to a possibly freshly cloned repo, and that can then be used in a completely asynchronous fashion. The fact that it is a webworker behind the scenes could be hidden completely.

Another thing I would like to know, as of right now I messed up the use of #[cfg(feature = "wasm")] and #[cfg(target_arch = wasm32)]. Are there any preferences/project-guidelines when to use what?

It seems like the target_arch variant is preferable as it works out of the box. That assumes it will indeed catch all the WASM backends, and my feeling is that it's more difficult to be compatible to everything. In any case, I am open to change it to whatever is (now) best.

I hope that helps.

[uberroot4]

I already started working on porting some crates, however in wasm32-unknown-unknown (i.e. wasm-in-the-browser) the biggest challenge is the need for async reading/writing files/directories. For now I simply use an patched version of the web_fs crate. I am, however, not familiar if async is also required for e.g. wasm-wasi.

That's exciting, thank you! It's good that there isn't too much file-IO going on, and but when it is it's definitely all blocking. I don't know how to best solve this except for assuming that a 'git server' is going to be run from a web-worker and the API is completely message-based. Then it's natively asynchronous, while the core can still be blocking in nature. That seems cleanest to me, and like it's a better fit for how Gitoxide works. I see it like git::clone_or_open(<url>, <identifier/handle>) to get a handle to a possibly freshly cloned repo, and that can then be used in a completely asynchronous fashion. The fact that it is a webworker behind the scenes could be hidden completely.

I'd say yes and no. There is indeed some I/O ongoing, at least when I call gix::discover or any other kind of "getting a repo to travers" (which was my very first intention, just iterate the tree). As I reseach some more and thought about your comment, it's not quite clear to me right now.
Maybe some more thought from my idea(s):

• I intended to use the FileSystem API which builds upon the Origin private file system (OPFS) concept. It is still relatively new, but the browser compatiblity seems to be ok and gaining momentum. And overall it relatively easy to use from Typescript and also Rust.
• Data access (except from stuff like LocalStorage) is per design async (see web.dev)
• This brings up the major problem i am facing right now. Rust std::fs stuff is just partially implemented for the target; e.g. Path or PathBuf works out-of-the box, but Path::new(...).exists() returns false as WASM cannot access the OPFS per default. I am not sure it ever will, I did not find anything. That's why i patched the gix crates with:

#[cfg(not(all(target_family = "wasm", target_os = "unknown")))]
use {
    std::fs
};

#[cfg(all(target_family = "wasm", target_os = "unknown"))]
use {
    web_fs as fs,
};

I do not think it is the cleanest version but it works already for many crates, e.g. gix-discover, gix-path, gix-worktree and gix-fs. But the biggest point keeps to be the async stuff.. as for now I simply duplicated the code, one async and one sync (with cfg toggle), but well, that's the worst solution as we would have duplicate stuff. Do you have any better ideas for this point?

• The sync webworker idea seems not bad, but we again face the problem that std is not fully implemented for the target and just raises Errors where it would not to due to the OPFS. Also I did not find any really good solution for the server idea. As far as I research today is a webworker just a JS snippet which is executed in the background and ends when its done (i.e. it is at the end). I did not find anything useful about a "WebWorker-Server".

[Byron]

I do not think it is the cleanest version but it works already for many crates, e.g. gix-discover, gix-path, gix-worktree and gix-fs. But the biggest point keeps to be the async stuff.. as for now I simply duplicated the code, one async and one sync (with cfg toggle), but well, that's the worst solution as we would have duplicate stuff. Do you have any better ideas for this point?

I think doing so serves as proof-of-concept, but couldn't be merged as 'unmaintainable'. However, my hope rests on the next Rust edition which is supposed to come with QoL improvements related to async - maybe that will make it possible to use the same implementation for both sync and async variants. Alternatively, there is the maybe_async macro that I have already been using to avoid such duplication, which might also work here. It's more of a crutch though and I truly believe that async needs better Rust support to be viable in codebases like this one.

• Also I did not find any really good solution for the server idea. As far as I research today is a webworker just a JS snippet which is executed in the background and ends when its done (i.e. it is at the end). I did not find anything useful about a "WebWorker-Server".

I don't think this will be a problem, and doing so would seemingly solve the 'everything should be async' requirement. It will just be a lot of work to do this right, and the work would at least be en par with implementing the backend of a GitForge :D. It's a mini-forge in your browser, basically.

[dzmitry-lahoda]

There is a lot of boring work around these things.

I know nothing about async.
But may hope that so it may appear that wasm/no_std support of core git algorithms and data structures may be useful in many application in browser or in clouds to handle and use git data.
And ultimately open way for multi signature audit/release/vetting flows within some verifiable executor (SGX/TEE/blockchain).

Usually deps can be hidden in some mymodule. That module hides from consumers difference of wasm/no_std/code/alloc things. So most of gitoxide code will not have cfg attrs in cases when code paths are supported in several targets.

Unlike async, there is no need to wait for rust 2024/effects.

jj prolonged git to be default tool for devs, so it may be safe to expect git is reusable longer than less in more areas and not replaced by pijul/darcs, so may be invested by startups doing wasm/no_std git.

[Byron]

Thanks for sharing.

I forgot to mention that in those rare cases where I had to duplicate code, I managed to not duplicate the tests to assure the copies don't diverge (beyond what the tests can capture, at least). It's the exception though.

[uberroot4]

Hi @Byron I again managed to get my hands back on that topic. I threw everything away I tried with target wasm32-unknown-unknown and decided to give wasm32-waspi2 a chance. I successfully (and with just a few changes) managed to get the latest main branch compiling with target wasm-wasip2 and running in the browser, e.g. I can discover a repo and partly traverse the commit graph.
Currently I use the following feature flags: features = ["revision", "basic","comfort", "attributes", "dirwalk", "excludes", "mailmap", "revparse-regex"], the default ones did not work at first so I decided to just use it like this.

What I currently cannot figure out is the following: When I call repo.find_commt(oid) it works perfectly fine if I execute it on my machine. When compiled to target wasm-wasip2 then I can find just some commits and some return an error (Err(Find(NotFound { oid: Sha1(<oid>) }))).

For example I can find the child of a certain Merge-Commit but not its parents, there it fails. I think I managed to find the source which should be down in the store crate. Since I cannot truly debug the program and need to work with println statements its quite tricky to find the correct root cause. Do you maybe have an idea what the cause could be or where I need to dig into? One thought I had was that pack/idx files might be a problem or the upload process via File System API, but I am pretty sure now that this is not the case (but not 100% sure tbh). I also tried to upload a zipped .git folder an defalting, but it did not change the outcome. Hence I do not think this is the cause.

[uberroot4]

I think I found the root cause of the problem above. The gix-pack crate uses the memmap library which has

impl MmapInner {
    fn new() -> io::Result<MmapInner> {
        Err(io::Error::new(
            io::ErrorKind::Other,
            "platform not supported",
        ))
    }
...
}

The gix-pack crate (gix-pack/src/index/init.rs#33) returns an error then, which I think -- as of now -- should be the reason.

Edit: With some more minor changes and a simple implementation for replacing memmap for wasm32-wasip2 I got everything building and running. So I can truly traverse the commit graph within the browser environment. Please note that building it with this target the nightly channel is required.