DataFusion does not support wasm32-unknown-unknown target

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11615

The Arrow crate successfully compiles to WebAssembly (e.g. https://github.com/domoritz/arrow-wasm) but the DataFusion crate currently does not support thewasm32-unknown-unknown target.

Try out the repository at https://github.com/domoritz/datafusion-wasm/tree/73105fd1b2e3ca6c32ec4652c271fb741bda419a.

{code}
error[E0433]: failed to resolve: could not find unix in os
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/dirs-1.0.5/src/lin.rs:41:18
|
41 | use std::os::unix::ffi::OsStringExt;
| ^^^^ could not find unix in os

error[E0432]: unresolved import unix
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/dirs-1.0.5/src/lin.rs:6:5
|
6 | use unix;
| ^^^^ no unix in the root

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:98:9
|
98 | sys::duplicate(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:101:9
|
101 | sys::allocated_size(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:104:9
|
104 | sys::allocate(self, len)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:107:9
|
107 | sys::lock_shared(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:110:9
|
110 | sys::lock_exclusive(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:113:9
|
113 | sys::try_lock_shared(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:116:9
|
116 | sys::try_lock_exclusive(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:119:9
|
119 | sys::unlock(self)
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:126:5
|
126 | sys::lock_error()
| ^^^ use of undeclared crate or module sys

error[E0433]: failed to resolve: use of undeclared crate or module sys
--> /Users/dominik/.cargo/registry/src/github.com-1ecc6299db9ec823/fs2-0.4.3/src/lib.rs:169:5
|
169 | sys::statvfs(path.as_ref())
| ^^^ use of undeclared crate or module sys

Compiling num-rational v0.3.2
error: aborting due to 10 previous errors
{code}

[alippai]

Polars proof of concept (shows that arrow-rs and datafusion like API can work): https://github.com/ritchie46/polars/blob/master/js-polars/app.js

[alippai]

#218 is a great step helping this

[seddonm1]

@alamb @Dandandan @jorgecarleitao
I have done some digging into this. There are two blockers:

1. dirs-rs
   problem dirs-rs is pulled in due to prettyprint-r pulled in due to pretty printing dataframes https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/dataframe_impl.rs#L34. dirs does not have a wasm32-unknown implementation and the owner seems reluctant to add it: wasm-pack support missing.  dirs-dev/dirs-rs#17.
   solution The arrow crate has a feature flag which enables pretty printing. If we agree we can add this to the datafusion (default enabled) to prevent this first problem.

2. lz4
   problem the parquet crate depends on lz4 which is generated by lz4-sys (not native rust) and does not compile to wasm32-unknown-unknown. I have done a test of ripping parquet out of datafusion and everything compiles.
   solution we could either put anything parquet behind another feature flag or swap out the lz4 for a pure rust implementation like the redox: https://lib.rs/crates/lz4-compress.

It would be relatively easy for me to do the two feature flags unless someone has an objection?

[alamb]

dirs-rs

Note that apache/arrow-rs#656 from @PsiACE has removed the pretty-table dependency in arrow-rs upstream. This will be included in the 6.0 arrow release (in 2ish months); I am not sure if/how this affects your decision

lz4:

I think lz4 is an optional dependency of parquet: https://github.com/apache/arrow-rs/blob/master/parquet/Cargo.toml#L40 thus perhaps we could just have a lz4 feature flag for datafusion?

[seddonm1]

Thanks @alamb .
I will do some experiments but seems like a good solution.

[ivanceras]

I think the biggest hurdles is using tokio specific types in the underlying implementation of datafusion, as I've found in this code
I understand this is for performance reason, but also there is no easy way to abstract this instead of using executor specific types. In wasm, we can use wasm-bindgen-futures as a way to run async functions.

[alamb]

@ivanceras I am not sure if we are using tokio specific futures stuff due to performance or just convenience

It might be worth trying to replace tokio specific structs with things from the futures crate (if that works on wasm32) and see what it looks like

[seddonm1]

@ivanceras what are you experiencing? I have managed to compile to WASM with very slight code modifications.

[roee88]

@seddonm1 compile and run?

I have experimented with that yesterday. I tried wasm32-wasi first and a simple sample works in single threaded mode after disabling some parquet features. See this gist for the example: https://gist.github.com/roee88/91f2b67c3e180fa0dfb688ba8d923dae

For wasm32-unknown-unknown adding getrandom with js as a dependency of the sample makes it compile IIRC, but actually running it is a different story. I tried to get a sample working with wasm-pack and it stops execution on the datafusion context creation, I suspect that it uses some sync primitives that are unsupported in wasm32-unknown-unknown but I didn't investigate further.

I didn't try wasm32-unknown-emscripten yet since my local rust version is incompatible with my installed emcc version (both latest at the time of this writing).

Edit: re tokio, the sample above worked on wasm32-wasi with other executors in single threaded mode including futures 0.3, https://github.com/richardanaya/executor, and async-global-executor.
As long as you don't hit code paths that use things like tokio::spawn (used in hash aggregate) it might be fine to use another executor. I'm not sure what's the best approach for library code that needs to spawn tasks. I have seen opinions for 1) a library should never spawn, 2) futures should be universally supported, 3) a library should accept an executor trait (as implemented in https://github.com/najamelan/async_executors). I didn't check the state of futures and WebAssembly recently.
I didn't try wasmbindgen-futures because it's officially no longer compatible with wasi and emscripten and as I said I couldn't get anything running with wasm32-unknown-unknown.

[roee88]

I got the basic sample from the gist in the previous message working with wasm-pack (wasm32-unknown-unknown) on single threaded tokio after:

1. Removing the "lz4" and "zstd" features from parquet dependency. This is the same change I had to do for running in wasm32-wasi.
2. Disabling the default time feature from tokio-stream by changing the dependency to: tokio-stream = { version = "0.1", default-features = false }
3. Removing the use of Instant::now() which is unsupported in wasm32-unknown-unknown and panics. For the basic sample that I'm running I just removed some lines from ScopedTimerGuard, but it's used in other places too if you run more complex stuff. Maybe https://github.com/sebcrozet/instant can be used instead.
4. Enabling js feature of getrandom: getrandom = { version = "0.2", features = ["js"] }. I did this in the sample application itself and not in datafusion (as it breaks other targets like wasm32-wasi).
5. Enabling wasmbind feature of chrono: chrono = {version = "0.4", features = ["wasmbind"]}. I did this in the sample application itself and not in datafusion (as it breaks other targets like wasm32-wasi).

Tested in chrome and seems to work. Again, there are definitely some code paths that lead to panic as not all of std is supported in wasm32 targets and I only tested something basic. Also, for multi-threading to work in the browser some parts of tokio can't be used directly from the datafusion codebase (this is a more complex topic).

[milesrichardson]

Good news, fellow WebAssembly enthusiasts! It looks like the stars are finally aligning, and with relatively minimal patching, I successfully compiled the code from the gist (create, insert and query a MemTable) to wasm32-wasi and wasm32-unknown-unknown, and ran it in wasmedge and the browser (via wasmpack):

❯ docker run --rm -it -v $(pwd)/target/wasm32-wasi/debug:/app wasmedge/slim:0.11.2-rc.1 wasmedge --reactor dfwasm.wasm _start
+---+----+
| a | b  |
+---+----+
| b | 10 |
| c | 10 |
+---+----+
0

I pushed the proof-of-concept to a public repository at splitgraph/experimental-datafusion-webassembly. There are two branches:

• wasm32-wasi
  • This is the target I got working first. The readme on this branch contains all the details and you should be able to reproduce it yourself.
• wasm32-unknown-unknown
  • This is branched from wasm32-wasi and the diff of wasm32-wasi..wasm32-unknown-unknown shows the changes
  • The top of the readme includes instructions for running this in the browser, but the patch is still very messy and might not be easily reproducible. Make sure you check Cargo.toml for any patched crates that you need to have checked out at a local path.

In the near future, I intend to cleanup these changes and submit a PR to DataFusion feature-flagging WebAssembly support.

In general, the summary of requirements for wasm-wasi:

• Patching arrow to use its upstream Git repository fixed some initial compilation issues, but otherwise no changes were required. It seems like the latest release on master of arrow-rs (v26.0.0) can compile to both webassembly targets, and so DataFusion just needs to upgrade to that. (There was also one minor change to datafusion/physical-expr required to support the upgraded Arrow package).
• Compiling with RUSTFLAGS="--cfg tokio_unstable" is necessary, and will benefit from recently stabilized wasm support in Tokio
• Removing cloud providers for object_store, as they are not compatible with the wasm32-unknown-unknown target (they may be with wasm32-wasi in some runtimes, but I disabled them).
• Removing local file system calls/writes (at least where necessary)
• Removing bzip2
• Using a forked reqwest to add some wasm compatibility (note: I'm not sure how much of this was/is necessary, and/or if it forces tokio to resolve to a version that other packages incidentally benefit from)
• Replacing two calls to spawn_blocking in sort with spawn, making the compiler happy but possibly causing runtime/logic errors
• ...and some other stuff in the patch

for wasm32-unknown-unknown, in addition to all those requirements, it was also necessary to:

• Replace usage of std::time with Instant, in both datafusion and arrow
• Make sure every library that calls getrandom is also passing it the js feature flag, which I did by just patching getrandom and making that the default

To get it to run (without a runtime error related to std::time being unreachable), a few more changes were made:

• Don't run the demo code in a Tokio main runtime, even with flavor = current-thread. Instead, use wasm-bindgen-futures to await a future that performs the asynchronous task that calls datafusion

This is all very messy. I will clean it up and submit a PR to DataFusion once I have a better sense of the most minimal changes required and the proper way to feature flag them. Also, general disclaimer that I'm new to Rust and YMMV, especially on the wasm-unknown-unknown patch - after all, I barely got it to run. But it does compile and create and query a small in-memory table, which is pretty good!

[alamb]

This sounds very cool @milesrichardson - DataFusion should be upgraded to arrow 26.0.0 shortly: #4039. I think @jimexist is in the process of making bzip support optional #3993

In terms of being messy / submitting a PR -- if it is possible I suggest trying to do it incrementally -- like for example we can probably sort out the calls to spawn_blocking in a separate PR

But all in all this is pretty exciting

[REASY]

Hello, folks.

I'm trying to add WASM support to DataFusion's dependencies. Started with bzip2-rs trifectatechfoundation/bzip2-rs#93