Jest memory problems in node environment

[mrrinot]

💬 Questions and Help

Stack:

  System:
    OS: macOS High Sierra 10.13.4
    CPU: x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
  Binaries:
    Node: 8.1.4 - ~/.nvm/versions/node/v8.1.4/bin/node
    npm: 6.0.0 - ~/.nvm/versions/node/v8.1.4/bin/npm
  npmPackages:
    jest: ^23.0.1 => 23.0.1
    sequelize: ^4.37.10
    pg: ^7.4.3
    koa: ^2.3.0
    babel-jest: ^22.4.3

1. Preface

We recently switched all our API's test from Mocha to Jest.
We have around 90 tests, half of them require to run synchronously due to them using our testing database (running seeds between them), so we have to use --runInBand.

I unfortunately can't share my code as it is private.

2. The problem

Running tests one by one was fine, I then tried to run all of them at once and things went bad.
With the --logHeapUsage, it seems context memory isn't GC'd resulting in a Javascript heap out of memory.

I tried using the new option --detectOpenHandles to see what could prevent the GC to work but this is what came out:

  ●  PROMISE
          at Promise.catch (<anonymous>)
      at node_modules/core-js/library/modules/es6.promise.js:244:30
      at Object.<anonymous>.module.exports (node_modules/core-js/library/modules/_iter-detect.js:19:5)
      at Object.<anonymous> (node_modules/core-js/library/modules/es6.promise.js:243:74)
      at Object.<anonymous> (node_modules/core-js/library/fn/promise.js:4:1)

I have around 6-8 of these, and no clue whatsoever of where to look.

I searched around and found out it was most likely the database connection so I added these as a global teardown:

afterAll(async () => {
    await db.close(); // Sequelize instance
    server.close(); // Koa server instance used with supertest (when needed)
});

This didn't change much, memory still goes up very quickly (30-40 MB per test).
In the end, I wrote a small file launching jest multiple times to avoid memory problems and stitching coverage report together but this isn't ideal.

[thymikee]

Maybe unrelated, but please make sure you have jest and babel-jest versions matching up

[SimenB]

It would be super helpful if you were able to create a reproduction showing a leak (even if it doesn't leak enough to crash)

[mrrinot]

Just tried with babel-jest 23 and that doesn't seem to fix it.
Also my code cannot be shared, I know that sucks :/

[SimenB]

Sure, but a reduced case would help. As is, this report isn't actionable for us

[mrrinot]

I'm working on a reduced case of the problem, will get back to you when/if needed

[mrrinot]

Ok, turns out the problem comes from us importing fs-extra in one of our model. We use the latest version (6.0.1).

Just commenting the import fixes the memory going up between test suites.
I don't really know if it's a problem with fs-extra or jest.

[mrrinot]

Here is a quick repro if you want: https://github.com/mrrinot/fs-extra-jest-repro
After a more thorough test, it seems it is graceful-fs instead.
Issues that seem related:

• Memory leak when required repeatedly isaacs/node-graceful-fs#102
• Updating to Jest 22 causes coverage to run out of memory #5239

[rickhanlonii]

Can you run with --detectLeaks? If that fails then your environment is leaking so it's not Jest

[rickhanlonii]

All 27 tests leak:

Unfortunately this isn't Jest related

[SimenB]

Might be worth opening up an issue with graceful-fs.

(note that we use it internally, so maybe it's installed multiple times or something?)

[mrrinot]

I updated the one I referenced above but the project doesn't seem to get a lot of attention.
I noticed that some jest packages use graceful-js, so you might want to get rid of it.

"_requiredBy": [
    "/jest-haste-map",
    "/jest-runner",
    "/jest-runtime",
    "/jest-util",
    "/jest/jest-cli"
  ],```

[SimenB]

It's pretty much required for windows, unfortunately.

[lev-kazakov]

@rickhanlonii, I'm also experiencing this bug, and I do suspect it's a bug with jest's module registry.

Jest does not cache modules as node's require() does and probably does not release properly require()'d modules between test suites.

Created a repo to illustrate this issue.

[SimenB]

Interesting find!

It's something that Webpack does though - removing that import (but keeping the 2 others) keeps the memory usage stable. So I don't think it's an issue with Jest in general, but Webpack (or one of its dependents) is doing something that leaks.

(note that --detectLeaks also complains until require('webpack') is removed).