Rust WebAssembly module in an ES module wrapper from wasm-pack fails to load in Next.js

[gthb]

What version of Next.js are you using?

11.1.2

What version of Node.js are you using?

14.17.6

What browser are you using?

Firefox, Chrome

What operating system are you using?

macOS

How are you deploying your application?

Not yet deployed

Describe the Bug

A simple Rust WebAssembly module packaged with its glue code into an ES module with wasm-pack (patched as in rustwasm/wasm-pack#1061) loads and works just fine under webpack, as illustrated in webpack/webpack#14313, but fails to import under Next.js. This is apparently because Next.js generates the .wasm generated at one path but then tries to load it from a different path.

Expected Behavior

I expected the npm run dev to successfully run the application and render a page with the greeting from inside the WebAssembly module.

I expected npm run build to successfully build a production distribution that would do the same.

To Reproduce

This is demonstrated in https://github.com/gthb/try-to-use-wasm-in-next.js/ — the README there recounts my circuitous path of trying to get this to work, but the current state serves to illustrate the problem I'm reporting here.

In that repo, first setup:

yarn install
yarn run build-wasm # or just copy the `pkg` from https://github.com/webpack/webpack/pull/14313 into `hi-wasm/pkg`
yarn run link-wasm

Then try yarn run dev — it fails like this:

yarn run v1.22.11
$ next dev
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
info  - Using webpack 5. Reason: Enabled by default https://nextjs.org/docs/messages/webpack5
warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

event - compiled successfully
event - build page: /next/dist/pages/_error
wait  - compiling...
event - compiled successfully
event - build page: /
wait  - compiling...
event - compiled successfully
error - pages/index.js (22:51) @ Home
TypeError: (0 , hi_wasm__WEBPACK_IMPORTED_MODULE_4__.greeting) is not a function
  20 |         <p className={styles.description}>
  21 |           Greeting:
> 22 |           <code className={styles.code}>{ greeting("Bob") }</code>
     |                                                   ^
  23 |         </p>
  24 |
  25 |         <div className={styles.grid}>

Then try yarn run build — it fails like this:

yarn run v1.22.11
$ next build
info  - Using webpack 5. Reason: Enabled by default https://nextjs.org/docs/messages/webpack5
warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

info  - Checking validity of types
info  - Creating an optimized production build
info  - Compiled successfully

> Build error occurred
[Error: ENOENT: no such file or directory, open '/Users/gthb/git/try-to-use-wasm-in-next.js/.next/server/static/wasm/1905d306caca373cb9a6.wasm'] {
  type: 'Error',
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/Users/gthb/git/try-to-use-wasm-in-next.js/.next/server/static/wasm/1905d306caca373cb9a6.wasm'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Note the path to the .wasm chunk. The actual generated .wasm chunks are:

$ find .next -iname '*.wasm'
.next/server/chunks/static/wasm/1905d306caca373cb9a6.wasm
.next/static/wasm/b9a4f4672eb576798896.wasm

So apparently the failure is that Next.js generates the .wasm chunk on the server side with an extra chunks/ path element (which presumably should not be there) and then looks for it at a path without that path element and fails to find it.

I'm guessing that the same problem is the cause of the npm run dev failure (i.e. the module imports as empty).

[gthb]

The yarn run dev problem is fixed in v11.1.3-canary.13, was still broken in v11.1.3-canary.7, so something inbetween there fixed it (maybe the webpack upgrade).

But yarn run build still fails in the same way, in that canary.13 build. So I guess it's not actually the (exact) same problem.

In the latest canary build, v11.1.3-canary.32, yarn run build now fails on something else, presumably unrelated:

yarn run v1.22.11
$ next build
warn  - You have enabled experimental feature(s).
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

info  - Checking validity of types

> Build error occurred
Error: 'entryOptions.layer' is only allowed when 'experiments.layers' is enabled
    at Function.entryDescriptionToOptions (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:36878:10)
    at Function.applyEntryOption (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:36846:39)
    at /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:36827:22
    at Hook.eval [as call] (eval at create (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:139224:10), <anonymous>:7:16)
    at Hook.CALL_DELEGATE [as _call] (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:139036:14)
    at WebpackOptionsApply.process (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:60381:30)
    at createCompiler (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:130634:28)
    at create (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:130680:16)
    at webpack (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:130704:32)
    at Object.f [as webpack] (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/compiled/webpack/bundle5.js:86594:16)
    at /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/compiler.js:31:40
    at new Promise (<anonymous>)
    at Object.runCompiler (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/compiler.js:30:12)
    at /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/index.js:358:59
    at async Span.traceAsyncFn (/Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/trace/trace.js:73:20)
    at async /Users/gthb/git/try-to-use-wasm-in-next.js/node_modules/next/dist/build/index.js:343:9
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

But enabling that layers experiment makes this go away so we can test our problem ... and yep, yarn run build still fails with the same Error: ENOENT: no such file or directory problem as described above, in the latest canary build, canary.32.

[brhx]

As per #29485, I'm actually not sure this is wasm-pack's (or wasm-bindgen's fault). I'm seeing this behaviour with just importing the vanilla .wasm file when building the Next.js application for production. Out of interest, are you able to replicate the problem in my issue?

[brhx]

I believe it is #22697 that's causing this conflict.

On a production build with webpack 5 the output configuration is as follows:

{
  publicPath: '/_next/',
  path: '/Users/sam/Desktop/with-webassembly-app/.next/server/chunks',
  filename: '../[name].js',
  library: undefined,
  libraryTarget: 'commonjs2',
  hotUpdateChunkFilename: 'static/webpack/[id].[fullhash].hot-update.js',
  hotUpdateMainFilename: 'static/webpack/[fullhash].[runtime].hot-update.json',
  chunkFilename: '[name].js',
  strictModuleExceptionHandling: true,
  crossOriginLoading: undefined,
  webassemblyModuleFilename: 'static/wasm/[modulehash].wasm'
}

It seems as though webassemblyModuleFilename interprets this to output to .next/server/chunks/static/wasm/[modulehash].wasm, or alternatively {output.path}/{webassemblyModuleFilename}. Chunks are outputted to .next/server/chunks but ordinary files are outputted to the parent directory with ../[name].js. webassemblyModuleFilename doesn't handle this parent recursion correctly, and so ends up in chunks. When importing, it doesn't appear to look for chunks because it's looking for webassemblyModuleFilename at the root of the webpack config, or .next/server

[brhx]

@gthb Lord forgive me, but I've created a workaround:

// next.config.js
module.exports = {
  webpack(config, { isServer, dev }) {
    // Enable webassembly
    config.experiments = { asyncWebAssembly: true };

    // In prod mode and in the server bundle (the place where this "chunks" bug
    // appears), use the client static directory for the same .wasm bundle
    config.output.webassemblyModuleFilename =
      isServer && !dev ? "../static/wasm/[id].wasm" : "static/wasm/[id].wasm";

    // Ensure the filename for the .wasm bundle is the same on both the client
    // and the server (as in any other mode the ID's won't match)
    config.optimization.moduleIds = "named";

    return config;
  },
};

Then to ensure the .wasm file is always included in the client bundle (you don't need to do this if you're using it ONLY on the client, or in both the server and the client), place this somewhere in the root of your _app.tsx:

(function () {
  import("wasm/add.wasm");
  // Import which .wasm files you need here
});

This won't download the file onto the client, just ensure that it's in the bundle

This is obviously far from an ideal solution, because it still generates the server bundle at .next/server/chunks/../static/wasm/[id].wasm (i.e. .next/server/static/wasm/[id].wasm) that webpack is trying to import for the server, but instead gets the static one because it's looking for .next/server/../static/wasm/[id].wasm (i.e. .next/static/wasm/[id].wasm)

[brhx]

The above workaround breaks when using experiments.layers = true because even though the module ID is named, a chunk hash is still appended to the output assets. The only way the above fix can work is if the client and server output the same filename for both the client and the sever.

I've created a more reliable workaround in the form of an embedded webpack plugin. This behaviour is pretty close to how Next.js internally handles the chunks directory and then walking back up with ../

// next.config.js

module.exports = {
  webpack(config, { isServer, dev }) {
    config.experiments = {
      asyncWebAssembly: true,
      layers: true,
    };

    if (!dev && isServer) {
      config.output.webassemblyModuleFilename = "chunks/[id].wasm";
      config.plugins.push(new WasmChunksFixPlugin());
    }

    return config;
  },
};

class WasmChunksFixPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap("WasmChunksFixPlugin", (compilation) => {
      compilation.hooks.processAssets.tap(
        { name: "WasmChunksFixPlugin" },
        (assets) =>
          Object.entries(assets).forEach(([pathname, source]) => {
            if (!pathname.match(/\.wasm$/)) return;
            compilation.deleteAsset(pathname);

            const name = pathname.split("/")[1];
            const info = compilation.assetsInfo.get(pathname);
            compilation.emitAsset(name, source, info);
          })
      );
    });
  }
}

[icyJoseph]

The above workaround breaks when using experiments.layers = true because even though the module ID is named, a chunk hash is still appended to the output assets. The only way the above fix can work is if the client and server output the same filename for both the client and the sever.

I've created a more reliable workaround in the form of an embedded webpack plugin. This behaviour is pretty close to how Next.js internally handles the chunks directory and then walking back up with ../

// next.config.js

module.exports = {
  webpack(config, { isServer, dev }) {
    config.experiments = {
      asyncWebAssembly: true,
      layers: true,
    };

    if (!dev && isServer) {
      config.output.webassemblyModuleFilename = "chunks/[id].wasm";
      config.plugins.push(new WasmChunksFixPlugin());
    }

    config.module.rules.push({
      test: /\.svg$/,
      use: ["@svgr/webpack"],
    });

    return config;
  },
};

class WasmChunksFixPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap("WasmChunksFixPlugin", (compilation) => {
      compilation.hooks.processAssets.tap(
        { name: "WasmChunksFixPlugin" },
        (assets) =>
          Object.entries(assets).forEach(([pathname, source]) => {
            if (!pathname.match(/\.wasm$/)) return;
            compilation.deleteAsset(pathname);

            const name = pathname.split("/")[1];
            const info = compilation.assetsInfo.get(pathname);
            compilation.emitAsset(name, source, info);
          })
      );
    });
  }
}

Oh man! 4 hours later I bump into this thread.

I had packed a battle snake solver with wasm-pack and had placed it on my pages/api routes, everything worked fine locally, but then the madness began when trying to publish it.

However this seems to work only with version 11, on Next 12, it breaks during import, I guess I gotta read further.

Thanks for the snippet! @sam3d !

[brhx]

@icyJoseph is it breaking when you try and deploy it on Vercel? The snippet is working for me in Next 12 for both dev and prod build locally. However on Vercel it seems like output file tracing doesn't detect the .wasm file. I believe the place this is happening in @vercel/nft is here:

https://github.com/vercel/nft/blob/e89f3ca8cfa41838d0c4d7c16255eba7f561609f/src/node-file-trace.ts#L40-L42

[icyJoseph]

@icyJoseph is it breaking when you try and deploy it on Vercel? The snippet is working for me in Next 12 for both dev and prod build locally. However on Vercel it seems like output file tracing doesn't detect the .wasm file. I believe the place this is happening in @vercel/nft is here:

https://github.com/vercel/nft/blob/e89f3ca8cfa41838d0c4d7c16255eba7f561609f/src/node-file-trace.ts#L40-L42

Ah I spoke too soon, when I deployed it on vercel it fails on, with Next 11.

ERROR	TypeError: wasm.get_info is not a function

And locally with next 12, it all falls apart.

Here's how I import the wasm file, on pages/piage/index.js

export default async function handler(req, res) {
  if (req.method !== "GET") return res.status(404).json({ error: "No" });

  try {
    const wasm = await import("../../../snake/pkg/logic");

    return res.status(200).json(wasm.get_info());
  } catch (e) {
    console.error(e);
    return res.status(500).json({ message: "Internal Server Error" });
  }
}

Here's a link to my config https://github.com/icyJoseph/vercel-battle-snake-rust/blob/fix-runtime/next.config.js, on the branch where I am trying to get this sorted out.

I think for now I'll just solve these with TypeScript.

[brhx]

@icyJoseph What happens if you remove the WasmPackPlugin? I don't think they're interoperable

[icyJoseph]

@sam3d I had to take a moment to re-do the entire thing, my code had changed quite a lot from the fork. Now I have it all running on TypeScript, and a sub route /api/wasm where I am trying to get one function to work correctly.

https://github.com/icyJoseph/next-battlesnake/tree/with-wasm -> set at the correct branch

I did as you suggested, and now things work fine locally, with Next 12, but still, when deploying to Vercel, upon reaching for the /api/wasm endpoint I get:

2021-11-23T12:54:00.429Z	acc7efcd-6066-45df-8452-37d1feb32982	ERROR	[Error: ENOENT: no such file or directory, open '/var/task/.next/server/chunks/817.wasm'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/var/task/.next/server/chunks/817.wasm'
}

Edit

Looking at https://github.com/vercel/nft/blob/e89f3ca8cfa41838d0c4d7c16255eba7f561609f/src/node-file-trace.ts#L40-L42, as you suggest, seems very telling, have you had a chance to try and debug it or make PR?

 if (path.endsWith('.js') || path.endsWith('.cjs') || path.endsWith('.mjs') || path.endsWith('.node') || job.ts && (path.endsWith('.ts') || path.endsWith('.tsx'))) {
      return job.emitDependency(path);
    }

[brhx]

@icyJoseph I filed this issue about a week ago (https://github.com/vercel/nft/issues/247) but I'm afraid I haven't yet the chance to work on a reproduction or PR for it (as I don't have the Next.js dev stack configured locally). My assumption is that a simple addition of path.endsWith('.wasm') would fix the issue, but I don't know that for sure

[brhx]

(p.s. you don't need the following by the way, I posted my patch hastily and forgot to remove this part from my own Next.js config:)

config.module.rules.push({
  test: /\.svg$/,
  use: ["@svgr/webpack"],
});