Multi-app Monorepo?

[mysterycommand]

Hello, I have use case that I think is common, but has been described from various angles and maybe not given a name. I'd like to propose Multi-app Monorepo to describe my use case, and maybe even be so bold as to suggest a solution in the form of a flag to pass to npm install and/or npm ci (I would also love to find out that the functionality I'm about to describe is already possible and I just missed some key documentation). Here's my basic setup:

A Monorepo with Two Apps

My monorepo has two apps, "a" and "b". They're in an apps workspace and they both depend on a package "c" that's in a packages workspace, so the top-level package.json looks like this:

{
  "name": "@my/monorepo",
  "private": "true",
  "workspaces": [
    "apps/*",
    "packages/*"
  ],
  // etc.

... and the tree looks (roughly) like this:

$ tree . --dirsfirst
.
├── apps
│   ├── a
│   │   ├── src
│   │   └── package.json
│   └── b
│       ├── src
│       └── package.json
├── packages
│   └── c
│       ├── src
│       └── package.json
├── package-lock.json
└── package.json

Where Things Get Dicey

For any number of reasons apps "a" and "b" are deployed to two different locations. They could be built as separate Docker containers, or two AWS Lambdas, or anything else. Also for reasons package "c" is not (maybe cannot be) published to a registry, either private or public.

In the interest of following best practices, and in the pursuit of an absolute minimum of build tooling, I'd like to deploy an unbundled Node app to my targets. My goal here is an app structure that looks something like this (whether there's a dist folder or src just gets built into the package root is kinda trivial to me):

$ tree apps/a --dirsfirst
apps/a
├── dist
├── node_modules
└── package.json

This structure might be COPY'd into a Docker image, or zipped up and pushed to AWS, or whatever else. Importantly, that node_modules directory is created with npm install --only=prod (or npm ci --only=prod) so that it's the minimum, production-only set of dependencies. There could also be an npm-shrinkwrap.json involved if that makes anything here easier.

The problem is really the fact that my package "c" is not published to a registry. So why not just publish to a private registry? I think even if package "c" were published, I'd still be in a weird situation when I want to create a pull request in my monorepo that affects one or both apps and package "c". In this case I'd need to be sure that as part of my CI/CD setup I publish a pre-release of package "c", then installed that into my app(s), and then deployed staging/integration versions of the app(s), and then if all that works and tests pass etc. publish a "real" release of package "c" (only to be consumed by my own apps "a" and "b") and then I guess re-release the final versions of my apps? Seems like a lot of extra work.

Maybe there are other, simpler ways around this, but I think if installing from a local/sibling workspace could be supported "natively" by npm it'd make the workflow a lot clearer. I also have an inkling that there are other use cases that could take advantage of this functionality.

A Flag to the Rescue?

Okay, so what'd be really great (imo) is if I could prompt npm install (or npm ci) to check my workspace root before any registries in npm's config. When npm install/npm ci are so prompted, I'd like for npm to resolve the dependency to my local filesystem. I think the root-level package-lock.json could still look something like this (for my above-mentioned scenario):

{
  "name": "@my/monorepo",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "@my/monorepo",
      "workspaces": [
        "apps/*",
        "packages/*"
      ]
    },
    "apps/a": {
      "name": "@my/a",
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "@my/c": "^1.0.0"
      }
    },
    "apps/b": {
      "name": "@my/b",
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "@my/c": "^1.0.0"
      }
    },
    "node_modules/@my/a": {
      "resolved": "apps/a",
      "link": true
    },
    "node_modules/@my/b": {
      "resolved": "apps/b",
      "link": true
    },
    "node_modules/@my/c": {
      "resolved": "packages/c",
      "link": true
    },
    "packages/c": {
      "name": "@my/c",
      "version": "1.0.0",
      "license": "ISC"
    }
  },
  "dependencies": {
    "@my/a": {
      "version": "file:apps/a",
      "requires": {
        "@my/c": "^1.0.0"
      }
    },
    "@my/b": {
      "version": "file:apps/b",
      "requires": {
        "@my/c": "^1.0.0"
      }
    },
    "@my/c": {
      "version": "file:packages/c"
    }
  }
}

What'd be different is that when npm finds a local match, instead of reaching out to a registry it'd "serve" the result of calling npm pack on the requested dependency.

For example, if I were to call npm i -w @my/a --prefer-workspaces from the workspace-root of the above-mentioned example project, I'd like npm's handling of the @my/c dependency to behave as if I'd called npm pack -w @my/c --pack-destination apps/a && npm i my-c-1.0.0.tgz -w @my/a && rm apps/a/my-c-1.0.0.tgz but without updating the dependency in app "a"'s package.json (e.g. from "@my/c": "^1.0.0" to "@my/c": "file:my-c-1.0.0.tgz" in this case).

I guess there'd be some potentially exotic configurations of transitive local dependencies, but in that case I think a kind of ephemeral local npm registry'd be able to handle serving the results of npm pack for any number of sibling packages. Anything more than one dependency-level deep would have to reach out to the npm config(ure)d registry (kind of like how the alias rule works for dependency renaming).

I've offered --prefer-workspaces above as an optional name for this flag, and I think --prefer-local has similarly intuitive (to me anyway) semantics.

A thought I just had while writing this is that maybe this functionality could piggyback on npm config's registry option. This'd mean no new config, and maybe add the option to specify it in a monorepo-root-level .npmrc file, but would also depend on something like --registry=. (or --registry=file:///.) being able to identify that the specified directory is in fact a workspace root. A benefit of this approach is that it could maybe be run as a CLI flag from app workspaces directly (e.g. a build script that includes something like npm ci --only=prod --registry=file:///../.. could be pretty useful for some scenarios).

In Conclusion

I am sleepy. It's late where I am and I took a melatonin, so I'm going to finish this up tomorrow. I've tried a bunch of other ways of approaching this use case and will try to document several of them and call out their pitfalls.

I hope this message finds you well, and that I haven't missed anything glaringly obvious (though if I have, thank goodness! I can't wait to hear about how my use case is already well solved). Thank you and good night.

Edit

Fixed a couple silly typos before shutting my eyes.

[bcbrian]

If the shared package is not published and will always be there, why not a local dependency?

{
  "name": "a",
  "dependencies": {
    "c": "file:../packages/c"
  }
}

https://stackoverflow.com/questions/15806241/how-to-specify-local-modules-as-npm-package-dependencies

[mysterycommand]

Unfortunately I think there are at least two things that make this not the thing I'm looking for:

1. Depending on "@my/c": "file:../../packages/c" will not let me hook into any of npm's lifecycle scripts for preparing package "c". It'll just link the entire folder (including "raw" src folders and maybe other assets or dev-time stuff that you'd normally ignore via an .npmignore file).
2. Related to the "links the entire folder" issue, just running npm i -w @my/a (when it depends on package "c" as above) doesn't actually copy packages/c into apps/a/node_modules
   • to get it to copy I've got to cd apps/a && npm i --only=prod
   • maybe then I run npm shrinkwrap to convert apps/a/package-lock.json into a (meant for publishing) apps/a/npm-shrinkwrap.json and add a build script like npm ci --only=prod which I can call from the root like npm run build -w @my/a
   • this works, but again appears to copy the entire folder, and I believe I've run into issues where it doesn't copy the entire folder, but appears to move some parts of it? That could've been a bug that's been fixed, it's been a while since I tried this

[mysterycommand]

Other Things I Have Tried: Lerna

Lerna has a concept of Git Hosted Dependencies that sort of does what I'm proposing, but it requires additional setup that I kind of hate (even the Git docs on using filter-branch start with a warning like "here be dragons").

Lerna's --force-local is kinda related maybe? It ignores semver ranges, and always resolves to the workspace when linking during lerna bootstrap but I think it only applies to bootstrap which means it only really affects a monorepo-root-level node_modules ... unless I'm missing something?

[MylesBorins]

Hey all,

since this discussion looks a bit more like a scoped feature request I've transferred this discussion to the RFC repo. As a next step you might want to open an issue to gather feedback.

[mysterycommand]

Thanks @MylesBorins! I wasn't quite sure where to put it 😅

[mysterycommand]

Created #463