webpack bundled node versions fails with `ENOENT: no such file or directory, open 'iam_service.json'`

[elyobo]

1. Is this a client library issue or a product issue?

This is occurring when bundling @google-cloud/pubsub, however the underlying issue is with this library.

2. Did someone already solve this?

Not that I can see.

3. Do you have a support contract?

No.

Environment details

• OS: Linux
• Node.js version: v12.14.1
• npm version: v6.14.8
• google-gax version: google-gax@2.10.0

Steps to reproduce

Edit: A minimal reproduction has been provided at https://github.com/elyobo/gax-node-948-repo, original notes below.

1. Use webpack to bundle an application using @google-cloud/pubsub
2. Run the application, such that it attempts to publish to a PubSub topic
3. The attempt to publish fails with an error as follows

Error: ENOENT: no such file or directory, open 'iam_service.json'
  at Object.openSync (fs.js:462:3)
  at Object.readFileSync (fs.js:364:35)
  at f (/srv/www/app/webpack:/node_modules/protobufjs/src/root.js:172:34)
  at p.e [as load] (/srv/www/app/webpack:/node_modules/protobufjs/src/root.js:206:13)
  at p.loadSync (/srv/www/app/webpack:/node_modules/protobufjs/src/root.js:247:17)
  at Object.t.loadSync (/srv/www/app/webpack:/node_modules/@grpc/proto-loader/build/src/index.js:223:29)
  at t.GrpcClient.loadFromProto (/srv/www/app/webpack:/src/grpc.ts:145:40)
  at t.GrpcClient.loadProto (/srv/www/app/webpack:/src/grpc.ts:179:17)
  at new t.IamClient (/srv/www/app/webpack:/src/iamService.ts:101:34)
  at new t.PublisherClient (/srv/www/app/webpack:/src/v1/publisher_client.ts:137:22)
  at m.getClientAsync_ (/srv/www/app/webpack:/src/pubsub.ts:1068:19)
  at m.getClient_ (/srv/www/app/webpack:/src/pubsub.ts:1031:10)
  at m.request (/srv/www/app/webpack:/src/pubsub.ts:1117:10)
  at t.Queue._publish (/srv/www/app/webpack:/src/publisher/message-queues.ts:78:11)
  at t.Queue.publish (/srv/www/app/webpack:/src/publisher/message-queues.ts:144:10)
  at Timeout.<anonymous> (/srv/www/app/webpack:/src/publisher/message-queues.ts:128:44) 

The underlying issue appears to be in iamService.ts, where a path to the iam_service.json is passed down that the application tries to load later. The path is correct in un-bundled code, but is incorrect in bundled code and so loading fails at runtime.

Is this meant to work? Is there a known workaround for the problem?

Thanks for any assistance.

CC: @alexander-fenster @xiaozhenliu-gg5 as authors of the iamService.ts.

[alexander-fenster]

Thank you for the report. I'll hope to take a look at this next week. A few questions for you:

• are you bundling it with target: 'node'? This is not for any kind of browser usage, right?
• do you use any of the IAM service calls in your code? (if not, you might be able to use null-loader to ignore everything related to IAM as a quick fix)

[elyobo]

Thanks @alexander-fenster, appreciate any assistance you can give here.

Yes, it's being bundled with target: 'node'. The usage in this case is via @google-cloud/pubsub with a service account identified by JSON credentials.

I don't directly call any IAM stuff, but I did notice that the PR that added this stuff was explicitly noting that PubSub was one of the libraries that used it and so I'm not sure what might be happening indirectly - I'm not clear what I'd be looking at using null-loader to suppress, exactly, the iam_service.json isn't being require()-d - otherwise it'd be bundled and there'd be no problem!

Interestingly it does look like it is actually in the bundle as well, probably because webpack picks up the conditional require() call for the opt.fallback case and bundles it. I could null load that, but that wouldn't resolve the problem, because in the node case opt.fallback is false and it's only a path that is passed through, and that path is then loaded at runtime, causing the observed file not found behaviour.

I can look at providing a minimal reproduction if that would help to look at the issue? Edit: Added reproduction at https://github.com/elyobo/gax-node-948-repo, hope that helps!

[alexander-fenster]

Thank you, I reproduced the problem. The bad thing here is that it's not just about the IAM JSON - even if you comment out all references to this.iamClient from node_modules/\@google-cloud/pubsub/build/src/v1/subscriber_client.js and node_modules/\@google-cloud/pubsub/build/src/v1/publisher_client.js, it will fail in pretty much the same way being unable to load PubSub's own proto JSON file here since the logic there is exactly the same.

I will need to think for a while how to make this work with the bundler: I don't see an obvious solution. If I have one, I'll let you know :)

One thing you could try (with no guarantee that it will work!) is to pass {fallback: true}: the fallback not only works in browser, it works in Node too (it will use node-fetch over HTTP/1 instead of gRPC). It should work unless you use Pub/Sub streaming methods, but I must say we never actually tested Pub/Sub with fallback (the fallback mode is introduced for different purposes).

There is another way to make it work by copying JSON files to some (seemingly random) locations near your script. Locations are weird because the way __dirname works in a bundle is weird too.

cp node_modules/google-gax/build/protos/*.json .
mkdir ../protos  ## note: one level up
cp node_modules/\@google-cloud/pubsub/build/protos/protos.json ../protos/

The bundled code will happily find all the required files if you do this. Ugly, I know.

[alexander-fenster]

(marking as "feature request" because we don't officially support this use case, but we definitely need to have this sorted out)

[elyobo]

Thanks, I tried hard coding the fallback option in earlier and ran into other issues, but I have another workaround in mind: I’ll try bundling my app as a library and then calling it from an unbundled launcher which will pass in the unbundled version of the library. Should be straightforward as the app already passes though external dependencies like this from the top to simplify mocking during testing. Hopefully that will sidestep the issue for me, at least.

…

On Fri, 29 Jan 2021 at 11:21, Alexander Fenster ***@***.***> wrote: Thank you, I reproduced the problem. The bad thing here is that it's not just about the IAM JSON - even if you comment out all references to this.iamClient from ***@***.***/pubsub/build/src/v1/subscriber_client.js and ***@***.***/pubsub/build/src/v1/publisher_client.js, it will fail in pretty much the same way being unable to load PubSub's own proto JSON file here <https://github.com/googleapis/nodejs-pubsub/blob/master/src/v1/publisher_client.ts#L166> since the logic there is exactly the same. I will need to think for a while how to make this work with the bundler: I don't see an obvious solution. If I have one, I'll let you know :) One thing you could try (with no guarantee that it will work!) is to pass {fallback: true}: the fallback not only works in browser, it works in Node too (it will use node-fetch over HTTP/1 instead of gRPC). It should work unless you use Pub/Sub streaming methods, but I must say we never actually tested Pub/Sub with fallback (the fallback mode is introduced for different purposes). There is another way to make it work by copying JSON files to some (seemingly random) locations near your script. Locations are weird because the way __dirname works in a bundle is weird too. cp node_modules/google-gax/build/protos/*.json . mkdir ../protos ## note: one level up cp ***@***.***/pubsub/build/protos/protos.json ../protos/ The bundled code will happily find all the required file if you do this. Ugly, I know. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#379>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADDR7VQ6SZ35RHQBQIW473S4IEQTANCNFSM4WMWHJUA> .

[elyobo]

If it doesn’t I’ll investigate the other options you’ve suggested - thanks!

…

On Fri, 29 Jan 2021 at 11:30, Liam O'Boyle ***@***.***> wrote: Thanks, I tried hard coding the fallback option in earlier and ran into other issues, but I have another workaround in mind: I’ll try bundling my app as a library and then calling it from an unbundled launcher which will pass in the unbundled version of the library. Should be straightforward as the app already passes though external dependencies like this from the top to simplify mocking during testing. Hopefully that will sidestep the issue for me, at least. On Fri, 29 Jan 2021 at 11:21, Alexander Fenster ***@***.***> wrote: > Thank you, I reproduced the problem. The bad thing here is that it's not > just about the IAM JSON - even if you comment out all references to > this.iamClient from > ***@***.***/pubsub/build/src/v1/subscriber_client.js and > ***@***.***/pubsub/build/src/v1/publisher_client.js, it > will fail in pretty much the same way being unable to load PubSub's own > proto JSON file here > <https://github.com/googleapis/nodejs-pubsub/blob/master/src/v1/publisher_client.ts#L166> > since the logic there is exactly the same. > > I will need to think for a while how to make this work with the bundler: > I don't see an obvious solution. If I have one, I'll let you know :) > > One thing you could try (with no guarantee that it will work!) is to pass {fallback: > true}: the fallback not only works in browser, it works in Node too (it > will use node-fetch over HTTP/1 instead of gRPC). It should work unless > you use Pub/Sub streaming methods, but I must say we never actually tested > Pub/Sub with fallback (the fallback mode is introduced for different > purposes). > > There is another way to make it work by copying JSON files to some > (seemingly random) locations near your script. Locations are weird because > the way __dirname works in a bundle is weird too. > > cp node_modules/google-gax/build/protos/*.json . > mkdir ../protos ## note: one level up > cp ***@***.***/pubsub/build/protos/protos.json ../protos/ > > The bundled code will happily find all the required file if you do this. > Ugly, I know. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#379>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADDR7VQ6SZ35RHQBQIW473S4IEQTANCNFSM4WMWHJUA> > . >

[elyobo]

And looking at that option, I realised that there is a much more simple solution 🤦

Webpack allows marking dependencies as external, so that they're not bundled at all. This can be easily be achieved en masse with the webpack-node-externals package.

An example is included as a PR in the reproduction issue for reference if anyone else comes across this problem. With that change node build fails with an expected error (the topic doesn't exist) rather than a failure to load files.

Thanks for looking into this and your suggestions, it seems like there is a reasonable solution that doesn't require any changes here.

[alexander-fenster]

Thanks @elyobo! It's something I never knew about webpack, and it makes perfect sense for bundling a backend application.