Unable to run with webpack

[ipryshlyak]

I'm trying to develop an aws lambda function which calls a language api.
The code is the bootstrap example.
Lambda Error: ENOENT: no such file or directory, open 'node_modules/protobufjs/google/protobuf/api.proto' ; stack: Error: ENOENT: no such file or directory, open 'node_modules/protobufjs/google/protobuf/api.proto' at Object.openSync (fs.js:440:3) at Object.readFileSync (fs.js:342:35) at fetch (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/protobufjs/src/root.js:160:1) at Root.load (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/protobufjs/src/root.js:194:1) at Root.loadSync (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/protobufjs/src/root.js:235:1) at Object.loadSync (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/protobufjs/src/index-light.js:69:1) at Object../node_modules/@grpc/proto-loader/build/src/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@grpc/proto-loader/build/src/index.js:235:1) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object.<anonymous> (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/google-gax/build/src/grpc.js:34:25) at Object../node_modules/google-gax/build/src/grpc.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/analyze.js:55452:30) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object../node_modules/google-gax/build/src/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/google-gax/build/src/index.js:33:16) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object.<anonymous> (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@google-cloud/language/src/v1/language_service_client.js:18:13) at Object../node_modules/@google-cloud/language/src/v1/language_service_client.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/analyze.js:34096:30) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object../node_modules/@google-cloud/language/src/v1/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@google-cloud/language/src/v1/index.js:17:31) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object../node_modules/@google-cloud/language/src/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@google-cloud/language/src/index.js:35:7) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at main (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/analyzer/analyze.ts:11:22) at /Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:710:30 at Promise._execute (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/debuggability.js:384:9) at Promise._resolveFromExecutor (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:518:18) at new Promise (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:103:10) at AwsInvokeLocal.invokeLocalNodeJs (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:663:12) at AwsInvokeLocal.invokeLocal (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:169:19) at AwsInvokeLocal.tryCatcher (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromiseCtx (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:641:10) at _drainQueueStep (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:97:12) at _drainQueue (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues [as _onImmediate] (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:15:14) at processImmediate (internal/timers.js:439:21) at process.topLevelDomainCallback (domain.js:130:23)

It looks like webpack doesn't know how to deal with protos.
Maybe, there is already a solution for this? Is it even possible to use google/language api with aws lambda via serverless framework?

[ipryshlyak]

if I copy a protobuf folder like this:
new CopyPlugin([{ from: 'node_modules/protobufjs/google/protobuf', to: 'node_modules/protobufjs/google/protobuf', context: __dirname }, ]), then I get this: Error: ENOENT: no such file or directory, lstat '/protos' at EventEmitter.<anonymous> (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/walkdir/walkdir.js:265:1) at Object.onceWrapper (events.js:300:26) at EventEmitter.emit (events.js:210:5) at EventEmitter.emit (domain.js:475:20) at fn (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/walkdir/walkdir.js:100:1) at statter (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/walkdir/walkdir.js:137:1) at walkdir (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/walkdir/walkdir.js:269:1) at Function../node_modules/walkdir/walkdir.js.walkdir.sync (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/walkdir/walkdir.js:15:1) at Object.<anonymous> (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/google-gax/build/src/grpc.js:50:1) at Object../node_modules/google-gax/build/src/grpc.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/analyze.js:55452:30) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object../node_modules/google-gax/build/src/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/google-gax/build/src/index.js:33:16) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object.<anonymous> (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@google-cloud/language/src/v1/language_service_client.js:18:13) at Object../node_modules/@google-cloud/language/src/v1/language_service_client.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/analyze.js:34096:30) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object../node_modules/@google-cloud/language/src/v1/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@google-cloud/language/src/v1/index.js:17:31) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at Object../node_modules/@google-cloud/language/src/index.js (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/node_modules/@google-cloud/language/src/index.js:35:7) at __webpack_require__ (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/webpack/bootstrap:19:1) at main (/Users/ihorpryshliak/Desktop/Workspace/rForgeProject/rforge/packages/analyzerService/.webpack/service/analyzer/webpack:/analyzer/analyze.ts:11:22) at /Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:710:30 at Promise._execute (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/debuggability.js:384:9) at Promise._resolveFromExecutor (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:518:18) at new Promise (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:103:10) at AwsInvokeLocal.invokeLocalNodeJs (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:663:12) at AwsInvokeLocal.invokeLocal (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:169:19) at AwsInvokeLocal.tryCatcher (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:547:31) at Promise._settlePromise (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:604:18) at Promise._settlePromiseCtx (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:641:10) at _drainQueueStep (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:97:12) at _drainQueue (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:86:9) at Async._drainQueues (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:102:5) at Immediate.Async.drainQueues [as _onImmediate] (/Users/ihorpryshliak/.nvm/versions/node/v12.13.1/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:15:14) at processImmediate (internal/timers.js:439:21) at process.topLevelDomainCallback (domain.js:130:23)

The path is unresolvable even if I copy a 'protos' folder.
Any suggestions?

[bcoe]

Hey @ipryshlyak, with a few tweaks, it should be possible to run any of our gRPC libraries with WebPack, @alexander-fenster has provided the steps here.

Let me know if you bump into any issues.

[ipryshlyak]

@bcoe the problem is that it fails on 'require'/'import' statement before I can call a constructor with a fallback option.
I see that language_service_client.js during init calls this
const gax = require('google-gax');
which requires
const grpc_1 = require("./grpc");
which requires
const grpcProtoLoader = require("@grpc/proto-loader");
which does this:

`// Load Google's well-known proto files that aren't exposed by Protobuf.js.
{
// Protobuf.js exposes: any, duration, empty, field_mask, struct, timestamp,
// and wrappers. compiler/plugin is excluded in Protobuf.js and here.

var wellKnownProtos = ['api', 'descriptor', 'source_context', 'type'];
var sourceDir = path.join(path.dirname(require.resolve('protobufjs')), 'google', 'protobuf');
for (var _i = 0, wellKnownProtos_1 = wellKnownProtos; _i < wellKnownProtos_1.length; _i++) {
    var proto = wellKnownProtos_1[_i];
    var file = path.join(sourceDir, proto + ".proto");
    var descriptor_1 = Protobuf.loadSync(file).toJSON();
    // @ts-ignore
    Protobuf.common(proto, descriptor_1.nested.google.nested);
}

}`
and fails, cause webpack doesn't know about .protos.

Here's my code:
`const language = require('@google-cloud/language');

const data = JSON.parse(event.body);

const client = new language.LanguageServiceClient({
  fallback: true // I see that there is opt arg in client's constructor, right?
});
const [result] = await client.analyzeSentiment({
  document: {
    content: data.text,
    type: 'PLAIN_TEXT'
  }
});
const sentiment = result.documentSentiment;`

[alexander-fenster]

@ipryshlyak Unfortunately, our dependencies (gRPC and protobuf modules) are not webpackable (they use fs to do stuff), so you'll need to make webpack ignore them. See our webpack.config.js for an example:

https://github.com/googleapis/nodejs-language/blob/master/webpack.config.js#L41-L60

In a fallback mode, those dependencies are not used so it will work.

You have two options:

1. Add the same ignore rules to your webpack.config.js (make sure to npm install --save-dev null-loader);

2. Make a bundle out of our library and use it directly from your code:

$ git clone https://github.com/googleapis/nodejs-language.git
$ cd nodejs-language
$ npm install
$ npx webpack  # generates dist/language-service.js

and then in your HTML code:

<script src="/path/to/dist/language-service.js"></script>
<script>
const client = new LanguageService.LanguageServiceClient({fallback: true, auth: ......});
</script>

In both cases, make sure you pass an auth instance to the constructor as described here.

It works, but since we haven't properly documented this yet, we don't officially support it - you can try it and let us know if it helped!

[telpirion]

Thank you for filing this issue. We proposed a workaround for you a while ago but haven't heard back. Please feel free to re-open this issue with more information.