Skip to content

Commit

Permalink
WebAssembly build
Browse files Browse the repository at this point in the history
This is a WebAssembly build of sharp usable in Node.js and Stackblitz environments.

There is quite a lot going on here to make it work, so I won't list all the implementation details I had to make it in the PR description, but instead would ask you to refer to the comments which hopefully explains the individual hacks, and feel free to ask any questions in the review.

The build script itself piggy-backs on the excellent work done by @kleisauke in [wasm-vips](https://github.com/kleisauke/wasm-vips) with some modifications to use it only for building libvips rather than custom bindings.

One thing I'll emphasise is that this is a build made specifically for Node.js / Stackblitz with the intention of being 1:1 API-compatible - among other things, it means that Wasm instantiation is intentionally synchronous, and that it will use the native filesystem via Node.js raw filesystem. I had / have a
separate branch with an almost working browser build of sharp, but that requires some API changes to be usable in a browser without locking the main thread and with accepting e.g. `File` object instead of using virtual filesystem, so for now keeping it out of scope.

The concurrency is currently limited to a fixed-size threadpool. While I made it possible to create threads on-demand in recent versions of Emscripten, there are still some issues and bugs when trying to use it with internal libvips threadpool, so for now keeping a fixed-size threadpool is a safer and time-tested option. Among other things, this will run only one concurrent sharp operation at a time. Also, in Stackblitz environment libvips will be limited to only 2 threads to keep memory usage under control - this is done because libvips already always needs +3 extra threads, and async emnapi operation will need yet another +1 thread per async operation, so the number of Workers quickly adds up.

Additionally, some formats and operations - namely, SVG, DZI and text operations - are currently unsupported just like they're in wasm-vips. Text (both on its own and in SVG) is notoriously difficult due to lots of questions around font loading (Local Font Access API, remote fonts, etc.), but other formats might come in
time. For now, though, this Wasm build should already cover most common use-cases.

Finally, for now I committed the build script together with prebuilt binaries as part of the PR, as it made testing easiest, but I'd ask the maintainers to integrate it properly into their "prebuilt addon" system somehow - I can't do that from my end, as I neither know how it works nor have access to the storage.
  • Loading branch information
RReverser committed Jan 9, 2023
1 parent 844deaf commit 65e504a
Show file tree
Hide file tree
Showing 34 changed files with 10,851 additions and 639 deletions.
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
src/libvips/* linguist-vendored
build/** linguist-generated
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
build
build/*
!build/Release
build/Release/*
!build/Release/sharp-emscripten-wasm32.node.*
node_modules
/coverage
test/bench/node_modules
Expand All @@ -16,3 +19,4 @@ vendor
package-lock.json
.idea
.firebase
/wasm-scripts/wasm-vips
55 changes: 50 additions & 5 deletions binding.gyp
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,15 @@
'NAPI_VERSION=7'
],
'dependencies': [
'<!(node -p "require(\'node-addon-api\').gyp")',
'libvips-cpp'
],
'variables': {
'runtime_link%': 'shared',
'conditions': [
['OS != "emscripten"', {
'runtime_link%': 'shared',
}, {
'runtime_link%': 'static',
}],
['OS != "win"', {
'pkg_config_path': '<!(node -p "require(\'./lib/libvips\').pkgConfigPath()")',
'use_global_libvips': '<!(node -p "Boolean(require(\'./lib/libvips\').useGlobalLibvips()).toString()")'
Expand All @@ -97,10 +100,52 @@
'src/utilities.cc',
'src/sharp.cc'
],
'include_dirs': [
'<!(node -p "require(\'node-addon-api\').include_dir")',
],
'conditions': [
['OS == "emscripten"', {
# .js because that's how Emscripten knows what to build, and .node
# in front so that `require('[...].node')` would just work.
'product_extension': 'node.js',
'defines': [
# Limit to 1 async task to avoid stealing threads from the pool
# that are intended for libvips.
'EMNAPI_WORKER_POOL_SIZE=1',
],
'cflags': ['-g2'],
'ldflags': [
'-g2',
'-fexceptions',
# We don't know in advance how large images will be, best to allow growth
# rather than overallocate in resource-constrained environments.
'-sALLOW_MEMORY_GROWTH',
# Building for Node.js, we want sync instantiation for compat with native
'-sWASM_ASYNC_COMPILATION=0',
# Re-export emnapi bindings as the main exports
'--pre-js=<!(node -p "require.resolve(\'./wasm-scripts/pre.js\')")',
# Using JS implementation of text decoder is actually faster when using pthreads.
'-sTEXTDECODER=0',
# Support 64-bit integers.
'-sWASM_BIGINT',
# CPU cores + 3 extra threads for libvips + 1 extra thread for an emnapi async task.
'-sPTHREAD_POOL_SIZE="vipsConcurrency+3+1"',
'-sPTHREAD_POOL_SIZE_STRICT=2',
# Don't wait for the pthread pool to keep initialization synchronous.
'-sPTHREAD_POOL_DELAY_LOAD',

# We're building only for Node.js for now
'-sENVIRONMENT=node',
# Propagate filesystem to Node.js.
'-sNODERAWFS',
# Exit helper
'-sEXPORTED_FUNCTIONS=["_vips_shutdown"]',
],
}, {
'dependencies': [
'<!(node -p "require(\'node-addon-api\').gyp")',
],
'include_dirs': [
'<!(node -p "require(\'node-addon-api\').include_dir")',
],
}],
['use_global_libvips == "true"', {
# Use pkg-config for include and lib
'include_dirs': ['<!@(PKG_CONFIG_PATH="<(pkg_config_path)" pkg-config --cflags-only-I vips-cpp vips glib-2.0 | sed s\/-I//g)'],
Expand Down
Loading

0 comments on commit 65e504a

Please sign in to comment.