Improve monorepo tooling with Rushjs or Turborepo

[humphd]

We are evolving our app to use microservices, and thinking about having a collection of smaller apps (front-end, services) vs. a monolithic, single application. At the same time, we're also trying to embrace the idea of having a single monorepo (all code in one place), so that we don't have to duplicate setup for things like prettier, eslint, CI, etc. for each new repo.

Until now we've been using a series of npm scripts to manage having multiple package.json files, and it's worked OK, but the number of these is about to grow as we split the back-end into services. Our app is eventually going to look like this:

/
    src/
        /web
            package.json
        /api
            /image
                package.json
            /user
                package.json
            ...
package.json

Here we have a single package.json at the root, and then sub-dirs for each major "app" within the project (/web is the next.js front-end and all the microservices live in /api).

There are a number of ways to manage these separate apps-with-an-app from the stand-point of the package manager:

• yarn workspaces
• pnpm
• Lerna
• npm7 workspaces

We need to pick one of these and migrate our repo to use it.

Here are some considerations:

• Telescope is developed by students, who are new to most of the tech we use. When we pick something, let's make sure we can easily teach it to new devs.
• It should work for our workflows. If we have to bend our workflows a lot to accommodate the tool, maybe it's not a good fit?
• It needs to work on every OS

The first part of this issue is to research the various solutions for doing a monorepo with JS, and then we need to pick one and implement it.

[HyperTHD]

Reading the docs for all 4, I'm going to rule out yarn and lerna. Lerna imo fails the first two considerations since it's a new technology that has to be introduced with it's own unique setup that would take alot of work to migrate over. Yarn is a bit easier to teach to new devs than lerna, but has it's own disadvantages, primarily the package layout being different between devs and consumers and having packages being installed from npm instead of from the linked package.json of another workspace if the dependency versions are even slightly different, possibly breaking all microservices that use such dependency.

That leaves us with npm7 and pnpm. The latter seems to be built on top of npm while npm7 is strictly npm that we're familiar with. I'll continue doing research on both, but early reviews is making me lean towards npm7. One advantage for npm7 is that future students of the course should already have it if they got it from installing the latest LTS version of Node.js.

[humphd]

@HyperTHD thanks for your useful comments here, you helped me a lot with my thinking. I agree that using Lerna or Yarn is going to be yet another stumbling block for new devs.

Another detail we need to consider is how we'll Dockerize these workspaces. When we move to some kind of "workspace" approach, we'll have our dependencies hoisted into the root node_modules folder. This is fine when you build in the native filesystem, but problematic for Docker, which is building what you copy into the container.

I found a few approaches doing this (mostly with Yarn, but npm workspaces are similar in how they work):

• https://stackoverflow.com/questions/50215553/how-to-build-a-docker-image-from-a-nodejs-project-in-a-monorepo-with-yarn-worksp/52805882#52805882
• https://xfor.medium.com/yarn-workspaces-and-docker-39e30402b69b

The idea seems to be to define and copy your shared "package" into the Dockerfile as well as the current project, then do multi-stage docker builds.

I'll have to get @raygervais to weigh in on this too, since the choice we make has to work well with Docker too.

[humphd]

Another thing I see people saying is to run the docker command from the root directory of the shared package code and your app's code, so that you can pull in both directories. https://lucas.love/sharing-code-in-a-node-monorepo-with-docker is an example of this.

For us, this might mean:

src/
  /api
    shared/
    service-1/
    service-2/

We'd need to run our various docker commands from src/api which has access to both shared/ and service-*/ so that we can copy things into the container either via multi-stage builds or using npm tricks.

[humphd]

Reading https://orta.io/notes/js/yarn-vs-npm and https://medium.com/edgybees-blog/how-to-move-from-an-existing-repository-to-a-monorepo-using-npm-7-workspaces-27012a100269, I think we should hold-off on this until npm adds more to their workspaces feature, which they are slated to do this year.

Let's revisit this in a few months.

[humphd]

I'd like to experiment with this, and use pnpm. I'm reading more and more about it lately, especially for projects doing monorepos. We need a better solution than the one we have now, since we have so many sub-packages in Telescope. Also, I want to be able to do internal workspace references with Satellite in the Telescope tree vs. as a separate repo. One deal-breaker will be getting the Docker stuff to work, but there seems to be a solution.

With the upcoming changing of the guard after we ship 2.0, it's a good time to do major breaking changes like this, since it won't disrupt too many devs.

[humphd]

I think Rush might be the way to go here. We should do an experiment that uses it to define our entire tree of projects. We can probably use it to pull Satellite in too.

[DukeManh]

@humphd, it looks like we have to use Rush on top of pnpm because Rush doesn't work well with with newer versions of npm https://rushjs.io/pages/maintainer/package_managers/.
So if anyone takes on this, can they use our initial pnpm PR as a starting point?

[humphd]

pnpm is amazing. The only down side of using it is that dependabot doesn't support it (yet). To me, this isn't a deal breaker.

Also, Node 16.9 includes support for corepack to let a project define its package manager: https://nodejs.org/dist/latest-v16.x/docs/api/corepack.html#configuring-a-package, and it will download it automatically. And, it supports pnpm.

We could update to 16.9.0 for our base engine, since LTS is currently at 16.13.0.

I'm happy to try pnpm + rushjs, yes. It will make installation so, so much faster, easier, and smaller!

[humphd]

@menghif is going to give this a try.

[humphd]

All the cool kids are talking about https://turborepo.org/. We should look at both and see if either is a fit.

The main idea here is to define build steps that take into account dependencies between different parts of our monorepo.

[menghif]

I heard about this tool a few weeks ago but didn't really understand it. I would like to look into it more to see if we should use it vs going with Rushjs.