mismatch BUILD_ID in multi server

[troywith77]

I have two server running a next program for load balancing, when I run npm run build in this two server, they generate two different BUILD_ID in their own .next dir. Then when I visit my site, the server would throw an error Build id mismatch! Seems like the server and the client version of files are not the same. do you know how to solve this problem?

[arunoda]

In this case, you should simply run the same build as multiple instances.
Build once, then start the server multiple times.

If not, try to use sticky session, but I think above one is simple.

[nkzawa]

Or I wonder if we can create BUILD_ID from a directory (maybe .next) then BUILD_ID can be a same in this situation if you have a lockfile of dependencies.

[arunoda]

@n8agrin What do you mean by the lockfile of deps? yarn.lock?
If we need to create consistent build ids, we need to hash all the code, static content, lockfiles and other stuff.
That's doable, but I don't think that's worthy.
Build once, run many instances would be the best approach.

As an easy fix, we could provide a way to override the build id.

[nkzawa]

You meant to mention me ? :D

yep, I meant yarn.lock and npm-shrinkwrap.json.
I think we can just create hash from .next dir (except .next/dist) since we serve only results of webpack compilations using buildId, as far as I read #745.
Maybe it's not so difficult ?

Another problem of the current approach is that build artifacts would be invalidated even if you just updated custom server and served content was not changed at all, for example.

[nkzawa]

As another approach to generate buildId, we might be able to reuse hash or chunkhash of webpack. https://webpack.github.io/docs/long-term-caching.html

[arunoda]

@nkzawa That's a pretty good one.

[iaincollins]

Okay I was about to raise an issue for this and so wrote something up. Rather than creating a new ticket, I'll just paste it here (apologies for labouring over the same thing as above, but have some additional info and comments that might be relevant/helpful).

BACKGROUND

I am using Next.js on a couple of new projects (some with public source code, some private). On one of the projects I've run into an issue deploying on a scaling cluster on AWS Elastic Beanstalk, althought the problem is not AWS or EB specific. I've got suggestion for a fix but wanted to run it by folks first.

SCENARIO

With the sort of deployment system with EB, I'm deploying a git repo and it runs npm install then I just have it run npm build; npm start to start the service. Not every deployment system uses the same approach of course, but it's pretty generic mechanism.

PROBLEM

When each server in the cluster (in AWS terminology "Scaling Group") runs it's own build step it generates it's own unqiue hashes for files in .next, these hashes are not the same as on other servers, even though the code is identical.

So, when a user loads the site, it will request resources dynamically (like the JS files that are automatically generated by Next.js) BUT because they happen over seperate HTTP requests they may be served by different servers.

This means that you get HTTP 5000 "Internal Server" errors when the client tries to load a page, because it references resources with server specific hashes in them like:

http://my-server-name.us-east-1.elasticbeanstalk.com/_next/4ada604c-2219-4821-8a59-fb2a12d7a1f9/commons.js

ABSOLUTE PATHS IN .next/dist/pages/index.js

I thought I could address the issue by bundling a pre-built .next assets folder with each deploy, but that doesn't work well as '.next/dist/pages/index.js' includes absolute (not relative) paths so then running 'npm start' on the server fails because it can't find the files it's referencing because the absolute path to the modules it's referencing are not the same between the computer the assets are pre-built on and the server.

SUGGESTED FIXES

1. Use only relative paths in pages like .next/dist/pages/index.js

For example, references like this:

var _regenerator = require('/Users/iain/Development/my-project/node_modules/next/node_modules/babel-runtime/regenerator');

Would then look something like this:

var _regenerator = require('../../../node_modules/next/node_modules/babel-runtime/regenerator');

This seems to work fine and be low impact, so if people think that seems sane I'm happy to submit a pull request that does that.

[NB: So far I have only changed these manually with a regex and not programmatically.]

2. Use a predictable hash (such as based on file contents) that always returns to the same value each time when run.

Personally I'd prefer this (either instead or as well - in some environments option 1 would be preferable, but in my case option 2 is even better), but I don't know the codebase and so am unsure of the ramifications. This is what is suggested above.

3. Use sticky load balancing as a workaround

I see this has already been suggested. This is a decent workaround and is what I've done for now - but it's not ideal for building a scaleable service and it won't be an option open to everyone (as some switches don't support it and/or enabling it can come with limitations).

I agree the idea of build once, run many instances is nice, but (sadly) lots of deployment systems don't work that way out of the box and can be a pain to achieve in production.

I am relieved to find out I'm not the only one having this as an issue at least. :-)

[arunoda]

@iaincollins This is a good research. Thanks.
For the absolute path issue, I suggest to refer this tread: #198

For the buildId, sticky session is a good solution, but it's not for scaling. So, we need to go for something else.
We can go for a hash based on webpack or file content. But we need to do some good testing and research on that. So, I'd like to defer that.

But for the short term, I'd like to show you a way to override the BUILD_ID. Add the following into your num script.

"postbuild": "echo MY_NEW_ID > .next/BUILD_ID",

And you can also get the BUILD_ID via env variables like this too:

"postbuild": "echo $BUILD_ID > .next/BUILD_ID",

[iaincollins]

Thank you, I was just thinking about how being able to set the ID (and maybe via an env var) would be a great option that might be pragmatic! I look forward to trying that out in production tomorrow.

Many thanks also for the link to #198

[sedubois]

Random thought but how does Now detect when the project changes? Can a similar mechanism be reused in Next.js? Or something similar to the way Git computes its commit hash.

[arunoda]

@iaincollins may be there's already something similar to BUILD_ID in the beanstalk env. So, you could use that too.

[iaincollins]

@arunoda Thanks again, I've updated to beta 18 in production (with a green/blue deployment method) and using this approach is working well.

Details for the record, in case it helps others:

Annoyingly, it turns out the auto-generated app version is not available as an environment variable (though you can get it by looking for and unzipping a file at a specific location, or calling the AWS API from the host) so I opted instead to just grab the app version from package.json and echo that to .next/BUILD_ID.

Due to AWS Elastic Beanstalk only supporting one of a range of specific commands start the server (i.e. you can tell it to do 'npm start' but not 'npm run build; npm start') my package.json contains a script that now looks like this:

"start": "next build; node -e \"process.stdout.write(require('./package.json').version)\" > .next/BUILD_ID; node server.js"

It seems that being able to just pass a Build ID as an option if invoking next() from a JS file might be nice way to let people in similar situation set the ID if they want to in a simple way (using whatever logic they want without bloating next). Of course, they could equally just write their own logic to write to the BUILD_ID file themselves before calling next(), but the option to pass it explicitly would make for a simple self-documenting mechanism.

Of course EBS is slow for deployment and something like Zeit's now platform is both faster much less hassle, but for anyone building front ends in enterprise environments (in my case, because I want to leverage AWS specific features on the back end) I'm happy to say the provided solution works just fine - and it's now in production and I've been able to turn sticky load balancing back off.

[arunoda]

@iaincollins Thanks for the feedback and I glad my suggestion helped you.
Anyway, I'd like to have a way to expose a user defined BUILD_ID via an option in the CLI.

If you have some free time, feel free to send a PR.

[gcpantazis]

Anyway, I'd like to have a way to expose a user defined BUILD_ID via an option in the CLI. If you have some free time, feel free to send a PR.

I can look into this tomorrow if nobody else is on it, also critical for us.

[rickysahu]

riffing off of @arunoda heres what we are using to get around this

"build": "next build && echo $(git rev-parse HEAD) > .next/BUILD_ID && echo {\\\"app.js\\\":{\\\"hash\\\":\\\"$(git rev-parse HEAD)\\\"}} > .next/build-stats.json",

[flippidippi]

I got this working with AWS EBS without Docker using the git hash. I had an issue with something like @rickysahu's solution because eb deploy uses git archive which doesn't copy over the .git/ folder.

You can get around this by using .gitattributes that runs on git archive to replace build file contents with the git hash.

My working solution

• Add a .gitattributes file:

build/BUILD_ID export-subst
build/build-stats.json export-subst

• Create a folder build/ in the root of your project
• Create build/BUILD_ID file:

$Format:%H$

• Create build/build-stats.json file:

{"app.js":{"hash":"$Format:%H$"}}

• And lastly, update your package.json scripts to copy over the build files after a build

"prestart": "NODE_ENV=production next build && cp -a build/. .next/",
"start": "NODE_ENV=production node server.js"

[nickdandakis]

Came across this issue as well, and just went with a Docker setup.

Wrote an article that describes how to deploy to Elastic Beanstalk with Docker that avoids mismatched BUILD_IDs, and absolute path issues.

[craigmcnamara]

I made a patch that should allow a build id to be passed in through the environment.

#3873

[nexdrew]

Thanks to #3873, you can now use the next-build-id package within your next.config.js module to easily set your BUILD_ID to the latest git commit hash (instead of using the default uuid). See an example here.

[flippidippi]

Update to get this working with AWS EBS without Docker using the git hash and the new generateBuildId.

Use .gitattributes that runs on git archive to replace generateBuildId with the git hash.

Updated working solution

• Add a .gitattributes file:

next.config.js export-subst

• In next.config.js file set:

generateBuildId: () => '$Format:%H$'

• Update your package.json to run build and start the server

"prestart": "NODE_ENV=production next build",
"start": "NODE_ENV=production node server.js"