Modules not resolved with adapter-cloudflare (buffer, stream, util, crypto)

[simcolin]

Describe the bug

The sveltekit build crash with current latest next versions of cloudflare adapter and sveltekit

Reproduction

use versions provided in system info
try to build

Logs

11:42:54.036	Run npm run preview to preview your production build locally.
11:42:54.048	
11:42:54.048	> Using @sveltejs/adapter-cloudflare
11:42:54.207	 > node_modules/jws/lib/verify-stream.js:5:21: error: Could not resolve "stream" (use "platform: 'node'" when building for node)
11:42:54.207	    5 │ var Stream = require('stream');
11:42:54.208	      ╵                      ~~~~~~~~
11:42:54.208	
11:42:54.208	 > node_modules/jws/lib/verify-stream.js:7:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.208	    7 │ var util = require('util');
11:42:54.208	      ╵                    ~~~~~~
11:42:54.208	
11:42:54.209	 > node_modules/jws/lib/tostring.js:2:21: error: Could not resolve "buffer" (use "platform: 'node'" when building for node)
11:42:54.209	    2 │ var Buffer = require('buffer').Buffer;
11:42:54.209	      ╵                      ~~~~~~~~
11:42:54.209	
11:42:54.209	 > node_modules/jws/lib/sign-stream.js:5:21: error: Could not resolve "stream" (use "platform: 'node'" when building for node)
11:42:54.209	    5 │ var Stream = require('stream');
11:42:54.209	      ╵                      ~~~~~~~~
11:42:54.209	
11:42:54.209	 > node_modules/jws/lib/sign-stream.js:7:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.209	    7 │ var util = require('util');
11:42:54.209	      ╵                    ~~~~~~
11:42:54.209	
11:42:54.210	 > node_modules/safe-buffer/index.js:2:21: error: Could not resolve "buffer" (use "platform: 'node'" when building for node)
11:42:54.210	    2 │ var buffer = require('buffer')
11:42:54.210	      ╵                      ~~~~~~~~
11:42:54.210	
11:42:54.210	 > node_modules/jws/lib/data-stream.js:3:21: error: Could not resolve "stream" (use "platform: 'node'" when building for node)
11:42:54.210	    3 │ var Stream = require('stream');
11:42:54.210	      ╵                      ~~~~~~~~
11:42:54.210	
11:42:54.210	 > node_modules/jws/lib/data-stream.js:4:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.210	    4 │ var util = require('util');
11:42:54.210	      ╵                    ~~~~~~
11:42:54.210	
11:42:54.211	 > node_modules/jwa/index.js:3:21: error: Could not resolve "crypto" (use "platform: 'node'" when building for node)
11:42:54.211	    3 │ var crypto = require('crypto');
11:42:54.211	      ╵                      ~~~~~~~~
11:42:54.211	
11:42:54.211	 > node_modules/jwa/index.js:5:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.211	    5 │ var util = require('util');
11:42:54.211	      ╵                    ~~~~~~
11:42:54.211	
11:42:54.214	 > node_modules/buffer-equal-constant-time/index.js:3:21: error: Could not resolve "buffer" (use "platform: 'node'" when building for node)
11:42:54.214	    3 │ var Buffer = require('buffer').Buffer; // browserify
11:42:54.214	      ╵                      ~~~~~~~~
11:42:54.214	
11:42:54.237	> Build failed with 11 errors:
11:42:54.237	node_modules/buffer-equal-constant-time/index.js:3:21: error: Could not resolve "buffer" (use "platform: 'node'" when building for node)
11:42:54.237	node_modules/jwa/index.js:3:21: error: Could not resolve "crypto" (use "platform: 'node'" when building for node)
11:42:54.237	node_modules/jwa/index.js:5:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.237	node_modules/jws/lib/data-stream.js:3:21: error: Could not resolve "stream" (use "platform: 'node'" when building for node)
11:42:54.237	node_modules/jws/lib/data-stream.js:4:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.237	...
11:42:54.238	node_modules/buffer-equal-constant-time/index.js:3:21: error: Could not resolve "buffer" (use "platform: 'node'" when building for node)
11:42:54.238	node_modules/jwa/index.js:3:21: error: Could not resolve "crypto" (use "platform: 'node'" when building for node)
11:42:54.238	node_modules/jwa/index.js:5:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.238	node_modules/jws/lib/data-stream.js:3:21: error: Could not resolve "stream" (use "platform: 'node'" when building for node)
11:42:54.238	node_modules/jws/lib/data-stream.js:4:19: error: Could not resolve "util" (use "platform: 'node'" when building for node)
11:42:54.238	...
11:42:54.238	    at failureErrorWithLog (/opt/buildhome/repo/node_modules/esbuild/lib/main.js:1493:15)
11:42:54.238	    at /opt/buildhome/repo/node_modules/esbuild/lib/main.js:1151:28
11:42:54.238	    at runOnEndCallbacks (/opt/buildhome/repo/node_modules/esbuild/lib/main.js:1069:65)
11:42:54.238	    at buildResponseToResult (/opt/buildhome/repo/node_modules/esbuild/lib/main.js:1149:7)
11:42:54.238	    at /opt/buildhome/repo/node_modules/esbuild/lib/main.js:1258:14
11:42:54.238	    at /opt/buildhome/repo/node_modules/esbuild/lib/main.js:629:9
11:42:54.238	    at handleIncomingPacket (/opt/buildhome/repo/node_modules/esbuild/lib/main.js:726:9)
11:42:54.238	    at Socket.readFromStdout (/opt/buildhome/repo/node_modules/esbuild/lib/main.js:596:7)
11:42:54.238	    at Socket.emit (events.js:400:28)
11:42:54.238	    at addChunk (internal/streams/readable.js:293:12)
11:42:54.258	npm ERR! code ELIFECYCLE
11:42:54.259	npm ERR! errno 1
11:42:54.262	npm ERR! vadrouilleurs-front@0.0.1 build: `svelte-kit build`
11:42:54.262	npm ERR! Exit status 1
11:42:54.262	npm ERR! 
11:42:54.262	npm ERR! Failed at the vadrouilleurs-front@0.0.1 build script.
11:42:54.262	npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
11:42:54.277	
11:42:54.277	npm ERR! A complete log of this run can be found in:
11:42:54.277	npm ERR!     /opt/buildhome/.npm/_logs/2022-01-27T10_42_54_262Z-debug.log
11:42:54.282	Failed: build command exited with code: 1

System Info

System:
    OS: Windows 10 10.0.19041
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 8.79 GB / 15.92 GB
  Binaries:
    Node: 16.3.0 - F:\Program Files\nodejs\node.EXE
    npm: 7.16.0 - F:\Program Files\nodejs\npm.CMD
  Browsers:
    Chrome: 97.0.4692.71
    Edge: Spartan (44.19041.423.0), Chromium (97.0.1072.69)
    Internet Explorer: 11.0.19041.1
  npmPackages:
    @sveltejs/adapter-cloudflare: 1.0.0-next.9 => 1.0.0-next.9
    @sveltejs/adapter-node: ^1.0.0-next.67 => 1.0.0-next.67
    @sveltejs/adapter-static: ^1.0.0-next.26 => 1.0.0-next.26
    @sveltejs/kit: next => 1.0.0-next.245
    svelte: ^3.46.3 => 3.46.3

Severity

blocking an upgrade

Additional Information

standard node adapter compiles without any problem

using { platform: 'node' } as build parameters for the cloudflare adapter fixes the build crash but break the cloudflare deployment

[Karlinator]

I am unable to replicate. Tested in a brand new Svelte demo project.

System info:

System:
    OS: Windows 10 10.0.19044
    CPU: (16) x64 AMD Ryzen 7 3700X 8-Core Processor
    Memory: 15.02 GB / 31.90 GB
  Binaries:
    Node: 17.4.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.10 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 8.3.1 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Chrome: 97.0.4692.99
    Edge: Spartan (44.19041.1266.0), Chromium (97.0.1072.69)
    Internet Explorer: 11.0.19041.1202
  npmPackages:
    @sveltejs/adapter-cloudflare: ^1.0.0-next.9 => 1.0.0-next.9
    @sveltejs/adapter-node: ^1.0.0-next.67 => 1.0.0-next.67
    @sveltejs/adapter-static: ^1.0.0-next.26 => 1.0.0-next.26
    @sveltejs/kit: next => 1.0.0-next.245
    svelte: ^3.44.0 => 3.46.3

I also tried with node 16.3.0 in WSL without any errors.

What are the contents of the project? Anything unusual in the config? Dependencies?

[simcolin]

Here is my svelte config, nothing fancy :

import preprocess from 'svelte-preprocess';
import { resolve } from 'path';
import adapter from '@sveltejs/adapter-cloudflare';

/** @type {import('@sveltejs/kit').Config} */
const config = {
    // Consult https://github.com/sveltejs/svelte-preprocess
    // for more information about preprocessors
    preprocess: preprocess(),

    kit: {
        target: '#svelte',
        vite: {
            resolve: {
                alias: {
                    $src: resolve('./src')
                }
            }
        },
        adapter: adapter()
    }
};

export default config;

Maybe the problem is linked to one of my dependencies ?

  "devDependencies": {
    "@sveltejs/adapter-cloudflare": "^1.0.0-next.8",
    "@sveltejs/adapter-node": "^1.0.0-next.66",
    "@sveltejs/adapter-static": "^1.0.0-next.26",
    "@sveltejs/kit": "next",
    "@types/cookie": "^0.4.1",
    "@types/google.maps": "^3.47.2",
    "@types/jsonwebtoken": "^8.5.8",
    "@types/luxon": "^2.0.9",
    "node-sass": "^7.0.1",
    "svelte": "^3.46.2",
    "svelte-check": "^2.3.0",
    "svelte-preprocess": "^4.10.2",
    "tslib": "^2.3.1",
    "typescript": "^4.5.5"
  },
  "type": "module",
  "dependencies": {
    "cookie": "^0.4.1",
    "firebase": "^9.6.3",
    "jsonwebtoken": "^8.5.1",
    "luxon": "^2.3.0",
    "quill": "^1.3.7",
    "sass": "^1.49.0"
  }

Honestly no idea how to reproduce the bug other than that. The logs are not helping me to pinpoint the bug on any library with my current knowledge.

The only things that I can think of is maybe the fact that the cloudflare adapter doesn't handle hooks.js or API routes (I have one POST request handler). But the package says that it supports all sveltekit features and those seems too basic to not be included in that phrasing.

[Conduitry]

From the error message, it would certainly sound like the issue is that https://www.npmjs.com/package/jws uses Node APIs which are not available on Cloudflare. I imagine you would need to find another library that doesn't use these APIs.

[simcolin]

Do you have a recommendation on how can I find out which one(s) of my dependencies depends on jws ? (CTRL + F on my vscode doesn't find anything)

[Karlinator]

That'd be jsonwebtoken: https://github.com/auth0/node-jsonwebtoken/blob/master/package.json#L39

EDIT: there are JWT libraries that use webcrypto, like this one: https://github.com/tsndr/cloudflare-worker-jwt

[simcolin]

EDIT: there are JWT libraries that use webcrypto, like this one: https://github.com/tsndr/cloudflare-worker-jwt

I just tested this library, it breaks as well with my cloudflare pages builds (and my local builds) :

16:07:46.423 | > crypto is not defined
16:07:46.424 | at new JWT (/opt/buildhome/repo/node_modules/@tsndr/cloudflare-worker-jwt/index.js:12:9)
16:07:46.424 | at Object.<anonymous> (/opt/buildhome/repo/node_modules/@tsndr/cloudflare-worker-jwt/index.js:115:18)
16:07:46.424 | at Module._compile (internal/modules/cjs/loader.js:1085:14)
16:07:46.424 | at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
16:07:46.424 | at Module.load (internal/modules/cjs/loader.js:950:32)
16:07:46.424 | at Function.Module._load (internal/modules/cjs/loader.js:790:12)
16:07:46.424 | at ModuleWrap.<anonymous> (internal/modules/esm/translators.js:199:29)
16:07:46.424 | at ModuleJob.run (internal/modules/esm/module_job.js:183:25)
16:07:46.424 | at async Loader.import (internal/modules/esm/loader.js:178:24)
16:07:46.424 | at async prerender (file:///opt/buildhome/repo/node_modules/@sveltejs/kit/dist/chunks/index5.js:352:28)

I am going to find another way to solve my needs without using JWT.

[Karlinator]

I think you could solve that by disabling prerendering? Looks like it tried to use the WebCrypto API while prerendering, which wouldn't work (and presumably you wouldn't want to do that anyway).

I am going to find another way to solve my needs without using JWT.

Hah, you very well might be happier for it.

[simcolin]

Disabling prerendering in svelte.config.js solves the issue.

But it's probably smarter to let it activated because some pages might need it, and deal with the error in the onError callback of the config, to ignore that exact error that souldn't be a problem on cloudflare servers I suppose.

I guess problem solved, Should I close the issue ?

[Karlinator]

But it's probably smarter to let it activated because some pages might need it, and deal with the error in the onError callback of the config, to ignore that exact error that souldn't be a problem on cloudflare servers I suppose.

Yeah, sounds reasonable. Though if you're using the package in a hook then that will effectively disable prerendering. If you're using it in an endpoint or something then you should be able to work around it.

I guess problem solved, Should I close the issue ?

That's usually what happens when an issue is resolved, yes 🙂

[hyunbinseo]

@simcolin You can use the following libraries to create your own JWS functions.

They are tested in SvelteKit projects that are deployed on Cloudflare Workers.

• base64url
• crypto-es
• js-sha256

The following code example generates parts of JWT using the HS256 algorithm.

import base64url from 'base64url';
import { sha256 } from 'js-sha256';

const encodedHeader = base64url.encode(JSON.stringify(header));
const encodedPayload = base64url.encode(JSON.stringify(payload));
const encodedVerifySignature = base64url.encode(
  sha256.hmac(secret, `${encodedHeader}.${encodedPayload}`),
  'hex',
);

---

Custom functions can be written using the Web Crypto API and the TextEncoder.

[silverwolfdoc]

stripe uses these libraries, so can't use stripe on cloudfare? getting the same issues.

[colecrouter]

stripe uses these libraries, so can't use stripe on cloudfare? getting the same issues.

I was able to get it to compile, using esbuild-plugin-node-polyfills out of the box. However, when I went to import Stripe, I get "process" is not defined. Aside from that, crypto, http, https, and event seemed to resolve fine. I ended up just giving up, using Stripe as a type import, then just fetch()-ing by hand.