When I add an external library (as a directory), there is an error installing dependencies of the package.

[angelgarrido]

This is a (Bug Report)

Description

For bug reports:

• What went wrong?
  I'm using serverless-webpack with a project that uses a lambda layer made with typescript for common modules and code.
  We are deploying the layer in remote and in local in "/opt" directory so i want to exclude this directories on the Webpack bundle.

For doing so, I've added our directory as a regular expression to the externals declaration of Webpack config.
This works fine, as the dependency is treated as external, but when is trying to install dependencies it generates a 'empty' package.json package name & package version and the npm install to for creating dependencies fails.

On line 128 of packExternalModules.js in serverless-webpack package is where it tries to push the module without name or version.

      if (!packageJson.devDependencies || !packageJson.devDependencies[module.external]) {
        // Add transient dependencies if they appear not in the service's dev dependencies
        const originInfo = _.get(dependencyGraph, 'dependencies', {})[module.origin] || {};
        moduleVersion = _.get(_.get(originInfo, 'dependencies', {})[module.external], 'version');
        if (!moduleVersion) {
          this.serverless.cli.log(`WARNING: Could not determine version of module ${module.external}`);
        }
        prodModules.push(moduleVersion ? `${module.external}@${moduleVersion}` : module.external);

I've used a workaround with forceExclude in serverless.yml

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules:
      forceExclude:
        - aws-sdk
        ## Fix to not try to deploy dependencies using webpack.config.js ^\/opt\/layer-utils\// regex
        - ''

• What did you expect should have happened?
  As it is an external directories with my own packages, I expect to be identified as that and don't try to install them. Also detect the name and version correctly.
• What was the config you used?
  This is my Webpack.config.js file

const path = require('path')
const webpack = require('webpack')
const slsw = require('serverless-webpack')
const nodeExternals = require('webpack-node-externals')
const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin')

module.exports = {
    context: __dirname,
    mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
    entry: slsw.lib.entries,
    devtool: slsw.lib.webpack.isLocal
        ? 'cheap-module-eval-source-map'
        : 'source-map',
    resolve: {
        extensions: ['.mjs', '.json', '.ts','.js', '.jsx'],
        symlinks: false,
        cacheWithContext: false
    },
    output: {
        libraryTarget: 'commonjs',
        path: path.join(__dirname, 'dist'),
        filename: '[name].js'
    },
    target: 'node',
    externals: [nodeExternals(),
                function(context, request, callback) {
                    if (/^\/opt\/layer-utils\//.test(request)){
                    return callback(null, 'commonjs ' + request);
                    }
                    callback();
                }      
            ],
    module: {
        rules: [
            // all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`
            {
                test: /\.(tsx?)$/,
                loader: 'ts-loader',
                exclude: [
                    [
                        path.resolve(__dirname, 'node_modules'),
                        path.resolve(__dirname, '.serverless'),
                        path.resolve(__dirname, '.webpack'),
                        path.resolve('/opt')
                    ]
                ],
                options: {
                    transpileOnly: true,
                    experimentalWatchApi: true
                }
            }
        ]
    },
    plugins: [

    ]
}

• What stacktrace or error message from your provider did you see?

Serverless: Fetch dependency graph from /app/package.json
Serverless: WARNING: Could not determine version of module 
Serverless: WARNING: Could not determine version of module 
Serverless: Excluding external modules: aws-sdk@^2.664.0
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: 
 
  Error --------------------------------------------------
 
  Error: npm install failed with code 1
      at ChildProcess.<anonymous> (/app/node_modules/serverless-webpack/lib/utils.js:91:16)
      at ChildProcess.emit (events.js:310:20)
      at ChildProcess.EventEmitter.emit (domain.js:482:12)
      at maybeClose (internal/child_process.js:1021:16)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

And it has created the file ./dist/dependencies/package.json with this content:

{
  "name": "MyName",
  "version": "1.0.0",
  "description": "MyDescription",
  "private": true,
  "scripts": {},
  "dependencies": {
    "": ""
  }

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
• If there is additional config how would it look

It should be an easier way to exclude directories without try to install it as a dependency.
Or add an optional boolean parameter to externals to forceExclude on dependenciesPackage

Similar or dependent issue(s):

Additional Data

• Serverless-Webpack Version you're using:
  "^5.3.4"
• Webpack version you're using:
  "^4.43.0",
• Serverless Framework Version you're using:
  Framework Core: 1.77.0
  Plugin: 3.6.18
  SDK: 2.3.1
  Components: 2.33.0
• Operating System:
  Dockerized linux alpine
• Stack Trace (if available):

[ssanri]

same issue.

Error: npm.cmd ls -prod -json -depth=1 failed with code 1
      at ChildProcess.child.on.exitCode (C:\dev\serverless\services-legacy\assesments-service\node_modules\serverless-webpack\lib\utils.js:91:16)
      at ChildProcess.emit (events.js:189:13)
      at ChildProcess.EventEmitter.emit (domain.js:441:20)
      at maybeClose (internal/child_process.js:970:16)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)

also in my case the issue is caused by a private repository in package.json with git+ssh://git@gitlab.com... target.

[lenamikaze]

Facing this as well. Any updates?

[vicary]

@angelgarrido @lenamikaze serverless-webpack will try to install dependencies in an internal directory to create a "clean" node_modules copy, this error usually means that you don't have the required packages installed to the dependencies section of your package.json.

Try running npm install at project root and check if it gives more verbose errors.

[vicary]

@ssanri serverless-webpack will not try to copy your .npmrc or anything alike when performing the internal installation, so for authentications to private repo you may have to use npm login to hoist the config to user level instead of project level. Please share more information and what you have tried if it doesn't work.

[luiznazari]

I'm having a similar error (maybe is the same related by the author).

We're build a lambda layer to provide shared node_modules packages and a common Database connection logic shared across all lambdas that are exported as a module and required as /opt/nodejs/external-lib-provided-by-lambda-layer on the lambdas' code.

When running sls package it gives the following error:

$ sls package
Running "serverless" from node_modules

Packaging packagename for stage dev (region)
Bundling with Webpack...
asset src/handler.js 4.04 KiB [emitted] [minimized] (name: src/handler) 1 related asset
./src/handler.ts 7.57 KiB [built] [code generated]
external "child_process" 42 bytes [built] [code generated]
external "mongodb" 42 bytes [built] [code generated]
external "@internal/somepackage1" 42 bytes [built] [code generated]
external "@middy/secrets-manager" 42 bytes [built] [code generated]
external "/opt/nodejs/external-lib-provided-by-lambda-layer" 42 bytes [built] [code generated]
webpack compiled successfully in 513 ms
WARNING: Could not determine version of module 
Excluding external modules: mongodb@4.3.1, @internal/somepackage1@0.0.18-beta
Package lock found - Using locked versions
Packing external modules: @middy/secrets-manager@^2.5.7, 
WARNING: Could not determine version of module 
Environment: darwin, node 14.18.1, framework 3.7.5 (local) 3.7.3v (global), plugin 6.1.5, SDK 4.3.2
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Error: npm install failed with code 1
    at ChildProcess.<anonymous> (/<project_root>/node_modules/serverless-webpack/lib/utils.js:91:16)
    at ChildProcess.emit (events.js:400:28)
    at ChildProcess.emit (domain.js:475:12)
    at maybeClose (internal/child_process.js:1058:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:293:5)
Waiting for the debugger to disconnect...

Solved it by addind '' to the custom.webpack.includeModules.forceExclude configuration in serverless.ts:

    webpack: {
      includeModules: {
        // Modules provided by lambda layer
        forceExclude: [
          '@internal/somepackage1',
          '@internal/somepackage2',
          'mongodb',
          'mongodb-client-encryption',
          'mongoose',
          '', // <--------------------------
        ],
      },

---

Project configuration files:

serverless.ts (irrelevant lines ommited)

const serverlessConfiguration: AWS = {
  frameworkVersion: '3',
  plugins: [
    'serverless-webpack',
    'serverless-offline',
  ],
  provider: {
    name: 'aws',
    runtime: 'nodejs14.x',
  },
  package: { individually: true },
  custom: {
    webpack: {
      includeModules: {
        // Modules provided by lambda layer
        forceExclude: [
          '@internal/somepackage1',
          '@internal/somepackage2',
          'mongodb',
          'mongodb-client-encryption',
          'mongoose',
        ],
      },
    },
  },
}

module.exports = serverlessConfiguration

webpack.config.js

const path = require('path')
const slsw = require('serverless-webpack')
const webpack = require('webpack')
const nodeExternals = require('webpack-node-externals')

module.exports = {
  context: __dirname,
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  entry: slsw.lib.entries,
  devtool: slsw.lib.webpack.isLocal ? 'eval-cheap-module-source-map' : 'source-map',
  resolve: {
    extensions: ['.js', '.mjs', '.json', '.ts'],
    symlinks: false,
    cacheWithContext: false,
  },
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js',
    sourceMapFilename: '[file].map',
  },
  optimization: {
    concatenateModules: false,
  },
  target: 'node',
  externals: [
    nodeExternals(),
    { '/opt/nodejs/external-lib-provided-by-lambda-layer': 'commonjs2 /opt/nodejs/external-lib-provided-by-lambda-layer' },
  ],
  module: {
    rules: [
      // all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`
      {
        test: /\.(tsx?)$/,
        loader: 'ts-loader',
        exclude: [
          [
            path.resolve(__dirname, 'node_modules'),
            path.resolve(__dirname, '.serverless'),
            path.resolve(__dirname, '.webpack'),
          ],
        ],
        options: {
          transpileOnly: true,
          experimentalWatchApi: true,
        },
      },
    ],
  },
}

Project's package.json

{
  "name": "somepackage",
  "version": "1.0.0",
  "main": "serverless.ts",
  "scripts": {
    "start": "cross-env NODE_ENV=test serverless offline start --stage dev",
    "deploy": "npx serverless deploy"
  },
  "dependencies": {
    "@middy/secrets-manager": "^2.5.7"
  },
  "optionalDependencies": { // Modules provided by lambda layer
    "@internal/somepackage1": "0.0.18-beta",
    "@internal/somepackage2": "0.0.3-beta",
    "mongodb": "4.3.1",
    "mongodb-client-encryption": "^2.0.0",
    "mongoose": "^6.2.3"
  },
  "devDependencies": {
    "@serverless/typescript": "^3.3.0",
    "@types/aws-lambda": "^8.10.92",
    "@types/jest": "^27.4.1",
    "@typescript-eslint/eslint-plugin": "^5.12.1",
    "@typescript-eslint/parser": "^5.12.1",
    "aws-sdk": "^2.1081.0",
    "cross-env": "^7.0.3",
    "eslint": "^8.10.0",
    "eslint-config-airbnb-base": "^15.0.0",
    "eslint-config-prettier": "^8.4.0",
    "eslint-import-resolver-typescript": "^2.5.0",
    "eslint-plugin-import": "^2.25.4",
    "eslint-plugin-prettier": "^4.0.0",
    "jest": "^27.5.1",
    "jest-mock": "^27.5.1",
    "prettier": "^2.5.1",
    "serverless": "^3.7.1",
    "serverless-offline": "^8.5.0",
    "serverless-plugin-typescript": "^2.1.1",
    "serverless-vpc-plugin": "^1.0.4",
    "serverless-webpack": "^5.6.1",
    "ts-jest": "^27.1.3",
    "ts-loader": "^9.2.6",
    "ts-node": "^10.5.0",
    "typescript": "4.5.5",
    "webpack": "^5.70.0",
    "webpack-node-externals": "^3.0.0"
  }
}

.webpack/dependencies/package.json (obtained while debugging sls package command).

{
  "name": "packagename",
  "version": "1.0.0",
  "description": "Packaged externals for somepackage",
  "private": true,
  "scripts": {},
  "dependencies": {
    "@middy/secrets-manager": "^2.5.7",
    "": ""
  }
}

[Hideman85]

Having the same troubles right now.
I'm having a module targeting a git repo and all the deps of that module are found without a version resulting in the console warn and a package.json with lot of "module": "" dependencies 🙁

Anyone found a workaround I would love to finish my feature 🙂

Edit: Could we have a fallback like require('aws-sdk/package.json').version?
Edit2: There is my custom fallbacks:

if (!moduleVersion) {
  try {
    moduleVersion = require(`${module.external}/package.json`).version;
  } catch {}
}
if (!moduleVersion) {
  try {
    //  Need to remove potential nested subfolders and file from the path
    const modulePath = require.resolve(module.external).replace(new RegExp(`${module.external}/.*$`), module.external);
    moduleVersion = require(`${modulePath}/package.json`).version;
  } catch {}
}

[rdsedmundo]

Also facing this, it's adding several { origin: undefined, external: '' } entries of external modules because the logic it's using to get the origin/external data is off. It generated empty values for this module:

<ref *1> ExternalModule {
  dependencies: [
    StaticExportsDependency {
      _parentModule: [Circular *1],
      _parentDependenciesBlock: [Circular *1],
      _parentDependenciesBlockIndex: 0,
      weak: false,
      optional: false,
      _locSL: 0,
      _locSC: 0,
      _locEL: 0,
      _locEC: 0,
      _locI: undefined,
      _locN: undefined,
      _loc: undefined,
      exports: true,
      canMangle: false
    }
  ],
  blocks: [],
  parent: undefined,
  type: 'javascript/dynamic',
  context: null,
  layer: null,
  needId: true,
  debugId: 2589,
  resolveOptions: {},
  factoryMeta: undefined,
  useSourceMap: true,
  useSimpleSourceMap: false,
  _warnings: undefined,
  _errors: undefined,
  buildMeta: { async: false, exportsType: 'dynamic' },
  buildInfo: { strict: true, topLevelDeclarations: Set(0) {}, module: false },
  presentationalDependencies: undefined,
  codeGenerationDependencies: undefined,
  request: '/Users/work/redacted/redacted/node_modules/.pnpm/generate-password@1.7.0/node_modules/generate-password/main.js',
  externalType: 'commonjs',
  userRequest: 'generate-password'
}

[rdsedmundo]

In my case I think it's related to passing a custom node-externals function that it's not recognizing, which I did as a workaround for liady/webpack-node-externals#95.

This messes up with the logic totally:

module.exports.nodeExternalsWithResolve = function nodeExternalsWithResolve(options) {
  let currentContext;
  const externals = nodeExternals({
    ...options,
    importType(request) {
      const resolved = require.resolve(request, {
        paths: [currentContext],
      });

      return `commonjs ${resolved}`;
    },
  });

  return ({ context, request }, callback) => {
    currentContext = context;
    return externals(context, request, callback);
  };
};

...
  externals: [nodeExternalsWithResolve()]

This works:

...
  externals: [nodeExternals()]

[james-hu]

@angelgarrido @lenamikaze serverless-webpack will try to install dependencies in an internal directory to create a "clean" node_modules copy, this error usually means that you don't have the required packages installed to the dependencies section of your package.json.

Try running npm install at project root and check if it gives more verbose errors.

Is there a reason why .npmrc is not copied together with package.json? I mean is this a design decision or just a missing feature or a bug? I feel it should be quite simple to add copying.

[vicary]

@james-hu AFAIK it was not skipped on purpose, accepting PRs.