Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack doesn’t work well with wasm modules created with Emscripten #7352

Open
surma opened this issue May 20, 2018 · 13 comments
Open

Webpack doesn’t work well with wasm modules created with Emscripten #7352

surma opened this issue May 20, 2018 · 13 comments
Labels
wasm wasm related issues

Comments

@surma
Copy link

surma commented May 20, 2018

Feature request

What is the current behavior?
The modularized JS emitted by Emscripten registers a global with a given name that loads the wasm file on invocation, initializes the wasm runtime and returns a Module.

Making it work with Webpack is quite hard as there seems to be interference with Webpack 4 defaults.
This is the webpack.config.js that I came up with:

const webpack = require("webpack");
const path = require("path");

module.exports = {
  mode: "development",
  entry: "./index.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "bundle.js"
  },
  module: {
    defaultRules: [
      {
        type: "javascript/auto",
        resolve: {}
      }
    ],
    rules: [
      {
        test: /fibonacci\.js$/,
        loader: "exports-loader"
      },
      {
        test: /fibonacci\.wasm$/,
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
  // This is necessary due to the fact that emscripten puts both Node and
  // web code into one file. The node part uses Node’s `fs` module to load
  // the wasm file.
  // Issue: https://github.com/kripken/emscripten/issues/6542.
  plugins: [new webpack.IgnorePlugin(/(fs)/)]
};

(Here is a minimal test project in a gist that you can clone and build with npm start. Docker required!)

edit:
In the meantime, @sokra informed that that I can simplify the configuration a bit (and make it less like a sledgehammer):

module.exports = {
  /* ... */
  browser: { 
    "fs": false // ← !!
  },
  module: {
    rules: [
      /* ... */
      {
        test: /fibonacci\.wasm$/,
        type: "javascript/auto", // ← !!
        loader: "file-loader",
        options: {
          publicPath: "dist/"
        }
      }
    ]
  },
};

Unexpected things I had to do

  • I needed to overwrite defaultRules as otherwise some sort of default rule will run in addition to the ones I specified and making webpack error “Module parse failed: magic header not detected” (try it!)
  • I needed to specify file-loader for the wasm file as otherwise webpack tries to resolve the names of the wasm module’s import object like env, which are provided by the JS file.
  • I needed to set a locateFile() function as webpack changes the file (and potentially path) of the wasm file and Emscripten hardcodes that name (not visible here but in the gist)

I am not sure what the right course of action here is, but considering that most wasm projects are going to be built with Emscripten, I feel like it’s worth making it easier.

Happy to answer Qs to give y’all a clearer picture.

What is the expected behavior?

Ideally, Webpack would recognize the typical Emscripten JS files and automatically bundle the accomodating wasm module and make paths work.

Other relevant information:
webpack version: 4.8.3
Node.js version: 10
Operating System: Mac OS 10.13.4
Additional tools:

@xtuc
Copy link
Member

xtuc commented May 20, 2018

Here's my suggestion: #7264 (comment)

We don't need to explicitly support emscripten, that would be generic enough.

Note that wasn-bindgen already does that.

@sokra
Copy link
Member

sokra commented May 22, 2018

It would be great to see a ESM target for Emscripten. This is how webpack's wasm support works. It expects WASM to be imported via import { ... } from "./something.wasm". WASM can get access to JS functions via (import "./something.js" "exportName") (import section).

wasm-bindgen creates two files abc.wasm and abc.js. The abc.js exports all exposed functions (public api) and all internal functions. It also imports the abc.wasm file. The abc.wasm imports all internal functions from abc.js.

That works great but exposes all internal functions too. I think a 3 file output would be better:

  • abc.js exports all public functions, imports abc_internal.js and abc_internal.js
  • abc_internal.js exports all internal functions
  • abc_internal.wasm imports abc_internal.js

Even better if repeated runtime code could be moved into a npm package: This way the wasm could import it directly from this package. This could get handy if you got many wasm modules in an application.

@xtuc
Copy link
Member

xtuc commented May 22, 2018

For ref emscripten-core/emscripten#6483

Also note that the same works for a module on npm. I think it would be more practical.

@gogins gogins mentioned this issue May 28, 2018
janhohenheim added a commit to myelin-ai/myelin that referenced this issue Sep 13, 2018
The workaround is described in webpack/webpack#7352
Note that now the webserver cannot load our wasm:
"TypeError: _myelin_visualization_bg__WEBPACK_IMPORTED_MODULE_0__.init is not a function"
This might be caused by a transcient dependency on stdweb:
https://github.com/sebcrozet/nphysics/blob/master/build/nphysics2d/Cargo.toml#L31
Which is incompatible with wasm-bindgen:
koute/stdweb#263
agilgur5 added a commit to agilgur5/physijs-webpack that referenced this issue Sep 16, 2018
- this is what I was having trouble with when I first created this repo
  - c.f. kripken/ammo.js#109 (comment) ,
    webpack/webpack#7352 ,
    and emscripten-core/emscripten#6542
- can't just upgrade to a new ammo as that might very well break
  physijs, so just make the Node environment check false
  - in this case I changed `var qa="object"===typeof process,` to
    `var qa=false,` in the minified code
    - could also remove the `if(qa){`... parts now, though that'll
      probably get re-minified too and nbd
agilgur5 added a commit to agilgur5/physijs-webpack that referenced this issue Sep 16, 2018
- this is what I was having trouble with when I first created this repo
  - c.f. kripken/ammo.js#109 (comment) ,
    webpack/webpack#7352 ,
    and emscripten-core/emscripten#6542
- can't just upgrade to a new ammo as that might very well break
  physijs, so just make the Node environment check false
  - in this case I changed `var qa="object"===typeof process,` to
    `var qa=false,` in the minified code
  - and also remove thed `if(qa){`...`}` parts
    - this was the part that had `require` statements, so now webpack
      etc should be able to parse it without a problem
      - there was a `require("fs")` and a `require("path")` in there
  - alternatively, could build and replace stuff with webpack, but I'd
    need to provide that anyway for auto-config
    - downside is it might no longer work on Node, but not the target
      audience so w/e
agilgur5 added a commit to VerifiableRobotics/LTLMoPWeb3D that referenced this issue Oct 3, 2018
- use physijs-webpack, a fork I created a few years ago and finally
  took some time to debug and get working
  - use via github since someone took the NPM name a few years ago and
    referenced my repo in it even though it never worked until today...
    - maybe will update once I get that resolved
  - now we don't have to vendor in all those scripts anymore and don't
    need to manually configure Physijs either!
    - remove the vendored physijs and three scripts
  - had to debug and modify ammo to get that version working with
    webpack / bundlers in general
    - newer versions of Emscripten can target specific envs
    - c.f. kripken/ammo.js#109 (comment) ,
      webpack/webpack#7352 ,
      and emscripten-core/emscripten#6542
  - and then made the worker config more flexible
  - add worker-loader as a devDep per physijs-webpack instructions
    - fix publicPath location as this is now actually used
      - the worker is loaded based on the publicPath

- add three-window-resize and three-trackballcontrols as deps as well
  - since three's examples/js/ folder doesn't quite work with bundlers
  - c.f. mrdoob/three.js#9562
  - maybe if I made some modifications, updated to newer Three
    revision, and used imports-loader it might work 🤷 TBD

- upgrade physijs to latest and Three from r60 to r73
  - latest physijs uses r73, so remain consistent
  - also physijs-webpack has a peerDep for that specific version
  - three-trackballcontrols@0.0.3 also requires r73 as dep
- had to refactor a bit due to upgrade
  - WebGLRenderer() -> WebGLRenderer({alpha: true})
    - the canvas now defaults to black without this, which was extremely
      disorienting
  - shadowMapEnabled -> shadowMap.enabled
  - CubeGeometry -> BoxGeometry
  - quaternion._euler -> rotation
    - I probably could've just used this before, couldn't have I...?

.
.

- TODO: improve webpack perf (CPU + Memory) and build speed
- this change slows down initial build times quite a bit (~20s), since
  Three, Physijs, and Ammo are all parsed by Webpack now
  - will want to update webpack to get a dev-server (wepback-serve)
    running
    - webpack itself is faster in later versions as well
    - and perhaps add HardSource for caching otherwise or split out
      vendored stuff into a DLL
    - probably can't update until decaffeinate'd since I believe the
      loaders used here are no longer maintained :/
      - and then would need to figure out the literate programming
        and its sourcemaps
  - will want to output a separate vendor bundle per best practice
    - may also want to output HTML via webpack too while at it
      - the templates don't do any templating so would be nbd
      - and would allow for hashing and therefore cache-busting
        - no need to manually clean cache then
  - and prod build / uglification is even _slower_ (~80s)
    - may want to exclude some files from Uglify
      - i.e. Ammo and use three.min so it can be excluded too
cbreak-black added a commit to disneyresearch/jeri that referenced this issue Dec 14, 2018
Trying to get webpack 4 to work by hacking on the code until it stops
complaining. Done with input from:
webpack/webpack#8412
webpack/webpack#7352
@mikeheddes
Copy link

mikeheddes commented Mar 18, 2019

I also came across this problem when using a JS library I made, that uses WebAssembly, in a React web project.

I solved it by adding some hacky pre.js code to basically hijack the wasm module instantiation from the Emscripten created glue code. This way I have all the control over initialising the WASM module. The pre.js file overrides the createWasm function and adds a custom module initialisation function which it exports using ES6. Note that I compile with the default JS glue code settings (see cpp build script) so the EXPORT_ES6 and modularize options are not set but because the pre.js code has an ES6 export statement the generated glue code is an ES6 module.

After that I combine the generated JS glue code with the JS API interface code bundled with Rollup. The entry of this can be found in index.js.

Then finally when using the library I need to provide it an arrayBuffer (node.js) or a fetch instance (browsers) of the generated dcgp.wasm file. With webpack that means I needed the following rule:

{
  test: /\.wasm$/,
  type: 'javascript/auto',
  loader: 'file-loader',
}

Some implementation references:

So my conclusion being...
I agree with @surma that Webpack and Emscripten don't work well together but I think the main solution should be found in the way Emscripten generates the JS glue code and not how Webpack should handle them. I'm also not sure if @surma's suggestion in #6542 to make separate platform outputs will fix the root cause of this problem. For me the following implementation would make more sense in the JS ecosystem:

// node.js
import { initializer } from './wasm-glue-code'
import { fileToArrayBuffer } from 'emscripten-helpers'

const myLibrary = initializer(fileToArrayBuffer('./wasm-file.wasm'))

// web
import { initializer } from './wasm-glue-code'

const myLibrary = initializer(fetch('./wasm-file.wasm'))

Hope this was helpful in some way.

@VirtualTim
Copy link

VirtualTim commented Apr 7, 2020

Hi @surma, @sokra,
I know this is fairly old, but it seems like:

browser: { 
  "fs": false // ← !!
},

no longer works, and I can't see any reference to browser in the webpack documentation.
Any ideas what we should be using instead?

Edit: figured it out. Use this instead:

node: { 
  fs: "empty"
},

@vankop vankop added wasm wasm related issues and removed Web Assembly labels Nov 18, 2021
@mmarczell-graphisoft
Copy link

You can specify -sENVIRONMENT=web to stop Emscripten from including node-targeted JavaScript.

Conzel added a commit to Conzel/zipnet that referenced this issue Mar 24, 2022
The default webpack options don't play with wasm32-emscripten.
Refer to the open issue in: webpack/webpack#7352
@dhdaines
Copy link

dhdaines commented May 6, 2022

For Webpack 5, disabling the node-specific stuff is relatively easy:

resolve: {
    fallback: {
        crypto: false,
        fs: false,
        path: false
    },
},

To include the wasm files, one solution is just to use copy-webpack-plugin, which is annoying since it makes it non-transparent for people to use your module, but then again, they're already adding the magic resolve incantation...

const CopyPlugin = require("copy-webpack-plugin");
module.exports = {
    plugins: [
        new CopyPlugin({
          patterns: [
              { from: "node_modules/MYPACKAGE/MYPACKAGE.wasm*",
                to: "[name][ext]" },
          ],
      }),
    ],
};

It would sure be swell if someone could write a webpack plugin or loader that would automate this so people could, like, just use Emscripten-compiled modules transparently. It seems like it should be relatively easy to recognize an Emscripten .js/.wasm file pair?

@marczellm
Copy link

@dhdaines
I once managed to get a working solution with setting the Emscripten build to EXPORT_ES6, and then simply importing the .js file in my Webpack based (actually Angular) project. Webpack intelligently detected that the .js module further depends on the .wasm file (and maybe even the .data file for preloaded files?)

Though this seemed to work, it printed some - to me - unintelligible, and unsilenceable warning on every build, so ultimately I used Angular's configuration options to bring in the JS file as a regular script and the WASM as a resource/asset.

@9oelM
Copy link

9oelM commented May 10, 2022

Hi guys, I felt extremely lost while I was digging into this issue, so after I got everything working, I made a example repo containing detailed walkthroughs:

Hope it helps someone.

@dhdaines
Copy link

Note also that you still need to disable Webpack's "mocking" of __filename, at the very least, or you will not be able to load your WASM unless it is in the root directory (Webpack sets __filename = "/index.js" and then Emscripten' s preamble takes this as the path of the current script in order to find the WASM, which is, obviously, wrong):

    node: {
        global: false,
        __filename: false,             
        __dirname: false,
    },

@dhdaines
Copy link

dhdaines commented Jan 26, 2023

@dhdaines I once managed to get a working solution with setting the Emscripten build to EXPORT_ES6, and then simply importing the .js file in my Webpack based (actually Angular) project. Webpack intelligently detected that the .js module further depends on the .wasm file (and maybe even the .data file for preloaded files?)

Can confirm that this works now, and it produces something like this: https://github.com/dhdaines/soundswallower-demo/tree/main/docs

...as long as you don't use Angular, because Angular doesn't include whatever magical Webpack configuration makes Webpack use the appropriate loader for the WASM file.

If you try to fix this by treating the WASM as a normal asset, Angular will use Webpack's default broken behaviour of munging the import.meta.url used in the JavaScript code to refer to the WASM to a file:// URL on your local file system. (see angular/angular-cli#22388)

So, you can use a custom Webpack config to disable this behaviour, but this breaks testing among other things, and probably means you can't target non-ES6 browsers. (see https://stackoverflow.com/questions/74038161/angular-cli-and-loading-assets-with-import-meta-url-causing-not-allowed-to-load)

Use ES6 modules, they said. It'll be great, they said.

@dhdaines
Copy link

dhdaines commented Jan 27, 2023

Just to follow up on this. The problem is actually Angular and its mysterious built-in webpack configuration, which disables module.parser.javascript.url for some unknown reason! If you re-enable it in a custom webpack configuration, then EXPORT_ES6=1 creates modules that work just fine, as noted above - the call to new URL("foo.wasm", import.meta.url).href in the JavaScript code is parsed by webpack, which duly adds your WASM as an asset and rewrites the URL properly.

The correct (as far as I can tell) workaround is to enable module.parser.javascript.url in a custom webpack config: angular/angular-cli#24617

@mmarczell-graphisoft
Copy link

@dhdaines Thank you for the deep dive! I recently tried to reproduce my own previously reported success and failed. Now I can try again...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wasm wasm related issues
Projects
None yet
Development

No branches or pull requests