[Bug]: Memory consumption issues on Node JS 16.11.0+

[EternallLight]

🚨 As of Node 21.1.0, this has been fixed upstream. Hopefully the fixes will be backported to v18 and v20 as well (as of writing (Oct. 26 2023) they have not. Node 18.20.0 and Node 20.10.0 have the backports), but that is up to the Node.js project and nothing we control from here. Note that (native) ESM still has memory leaks - that can be tracked here: #14605. If you're unable to upgrade your version of Node, you can use --workerIdleMemoryLimit in Jest 29 and later. See https://jestjs.io/docs/configuration/#workeridlememorylimit-numberstring 🚨

Version

27.0.6

Steps to reproduce

1. Install the latest Node JS (16.11.0 or later) or use the appropriate Docker image
2. Set up a project with a multiplicity Jest tests
3. Run node --expose-gc node_modules/.bin/jest --logHeapUsage and see how the memory consumption starts increasing.

Expected behavior

Since Jest calls global.gc() when Garbage Collector is exposed and --logHeapUsage flag is present, the memory usage should be stable.

Actual behavior

The memory usage increases with every new test

Additional context

We had some issues with Jest workers consuming all available RAM both on CI machine and locally.
After doing some research, we found that if we run Jest like the following node --expose-gc node_modules/.bin/jest --logHeapUsage, the heap size remains stable. After upgrading to Node JS v16.11.0, the issue was back. Node v16.10.0 works fine. I believe it was something accidentally introduced in the new Node, but it might be useful to take a look at this from Jest perspective in search of possible workarounds.
I'm also having the same behavior on my working machine, environment of which I'm pasting below 👇🏻

Environment

System:
    OS: macOS 11.6
    CPU: (8) x64 Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
  Binaries:
    Node: 16.11.0 - ~/.nvm/versions/node/v16.11.0/bin/node
    Yarn: 1.22.0 - ~/SomeFancyDir/webapp/node_modules/.bin/yarn
    npm: 8.0.0 - ~/.nvm/versions/node/v16.11.0/bin/npm
  npmPackages:
    jest: 27.0.6 => 27.0.6

[rthreei]

We're also experiencing this issue. Node 16.11+ and Jest v27 consumes significantly more memory. Node 16.10 and Jest v27 seems OK.

[blimmer]

We're also unable to update to the LTS version of Node 16 (at the time of writing 16.13.0) because of this issue. We bisected the changes and identified that the upgrade from Node 16.10 to 16.11 caused our large Jest suite to hang indefinitely.

I took a look at the Node 16.11 changelog and I think the most likely culprit for this issue comes from the V8 update to 9.4 (PR). In V8 9.4, the new Sparkplug compiler is enabled by default (see also this Node issue).

I was hoping I could try disabling sparkplug to see verify that this is the issue. node exposes a V8 option to disable (--no-sparkplug), but I don't think it's passing through to the Jest workers when I call it like this:

node --no-sparkplug node_modules/.bin/jest

I also tried setting the V8 option in jest-environment-node here: https://github.com/facebook/jest/blob/42b020f2931ac04820521cc8037b7c430eb2fa2f/packages/jest-environment-node/src/index.ts#L109 via

setFlagsFromString('--no-sparkplug');

but I didn't see any change. I'm not sure if that means Sparkplug isn't causing the problem or if I'm not setting the V8 flag properly in the jest workers.

@SimenB - I see you've committed a good deal to jest-environment-node - any tips for how I might pass that V8 flag down through all the workers? If it's possible (even via patch-pacakge or something) I'd be happy to give it a shot on our test suite that's exhibiting the problem.

So I'm not exactly positive that this is the cause of the issue, but it seems like a potentially promising place to start looking.

[SimenB]

any tips for how I might pass that V8 flag down through all the workers?

I would have thought https://github.com/facebook/jest/blob/e0b33b74b5afd738edc183858b5c34053cfc26dd/packages/jest-worker/src/workers/ChildProcessWorker.ts#L93-L94 made it so it was passed down...

[blimmer]

It's possible the sparkplug thing is a red herring. Nothing else jumped out at me from that changelog.

I captured some heap size metrics to show the scale of the difference.

Node 16.10.0 (baseline)

> node --expose-gc ./node_modules/.bin/jest --logHeapUsage --maxWorkers ${TEST_PARALLELISM:-8} "--reporters=default" "--reporters=jest-junit"
(14.297 s, 211 MB heap size)
(16.811 s, 211 MB heap size)
(17.913 s, 211 MB heap size)
(18.382 s, 213 MB heap size)
(18.38 s, 214 MB heap size)
(19.122 s, 238 MB heap size)
(19.241 s, 211 MB heap size)
(21.969 s, 222 MB heap size)
(10.783 s, 268 MB heap size)
(6.974 s, 268 MB heap size)
(7.575 s, 267 MB heap size)
(7.404 s, 269 MB heap size)
(6.788 s, 288 MB heap size)
(6.907 s, 267 MB heap size)
(10.324 s, 268 MB heap size)
(6.796 s, 276 MB heap size)
(5.177 s, 282 MB heap size)
(5.638 s, 269 MB heap size)
(5.702 s, 269 MB heap size)
(268 MB heap size)
(6.432 s, 271 MB heap size)
(7.227 s, 268 MB heap size)
(7.276 s, 272 MB heap size)
(5.192 s, 278 MB heap size)
(271 MB heap size)
(5.053 s, 286 MB heap size)
(269 MB heap size)
(5.3 s, 272 MB heap size)
(281 MB heap size)
(274 MB heap size)
(281 MB heap size)
(5.018 s, 272 MB heap size)
(274 MB heap size)
(288 MB heap size)
(275 MB heap size)
(272 MB heap size)
(276 MB heap size)
(278 MB heap size)
(285 MB heap size)
(274 MB heap size)
(290 MB heap size)
(277 MB heap size)
(275 MB heap size)
(278 MB heap size)
(278 MB heap size)
(280 MB heap size)
(287 MB heap size)
(277 MB heap size)
(293 MB heap size)
(280 MB heap size)
(277 MB heap size)
(281 MB heap size)
(281 MB heap size)
(280 MB heap size)
(283 MB heap size)
(289 MB heap size)
(296 MB heap size)
(283 MB heap size)
(280 MB heap size)
(284 MB heap size)
(283 MB heap size)
(282 MB heap size)
(285 MB heap size)
(292 MB heap size)
(298 MB heap size)
(282 MB heap size)
(285 MB heap size)
(286 MB heap size)
(286 MB heap size)
(285 MB heap size)
(287 MB heap size)
(295 MB heap size)
(301 MB heap size)
(285 MB heap size)
(288 MB heap size)
(289 MB heap size)
(289 MB heap size)
(288 MB heap size)
(290 MB heap size)
(298 MB heap size)
(216 MB heap size)
(304 MB heap size)
(288 MB heap size)
(290 MB heap size)
(292 MB heap size)
(292 MB heap size)
(296 MB heap size)
(294 MB heap size)
(295 MB heap size)
(293 MB heap size)
(6.766 s, 283 MB heap size)
(7.887 s, 273 MB heap size)
(7.654 s, 308 MB heap size)
(8.406 s, 272 MB heap size)
(298 MB heap size)
(298 MB heap size)
(297 MB heap size)
(295 MB heap size)
(297 MB heap size)
(300 MB heap size)
(289 MB heap size)
(305 MB heap size)
(301 MB heap size)
(300 MB heap size)
(291 MB heap size)
(297 MB heap size)
(300 MB heap size)
(303 MB heap size)
(303 MB heap size)
(292 MB heap size)
(303 MB heap size)
(308 MB heap size)
(311 MB heap size)
(300 MB heap size)
(303 MB heap size)
(306 MB heap size)
(295 MB heap size)
(306 MB heap size)
(310 MB heap size)
(305 MB heap size)
(297 MB heap size)
(303 MB heap size)
(306 MB heap size)
(309 MB heap size)
(297 MB heap size)
(313 MB heap size)
(310 MB heap size)
(307 MB heap size)
(305 MB heap size)
(299 MB heap size)
(308 MB heap size)
(311 MB heap size)
(315 MB heap size)
(300 MB heap size)
(311 MB heap size)
(220 MB heap size)
(208 MB heap size)
(310 MB heap size)
(302 MB heap size)
(309 MB heap size)
(242 MB heap size)
(311 MB heap size)
(284 MB heap size)
(322 MB heap size)
(296 MB heap size)
(312 MB heap size)
(313 MB heap size)
(315 MB heap size)
(317 MB heap size)
(316 MB heap size)
(304 MB heap size)
(315 MB heap size)
(308 MB heap size)
(315 MB heap size)
(317 MB heap size)
(318 MB heap size)
(317 MB heap size)
(320 MB heap size)
(306 MB heap size)
(311 MB heap size)
(317 MB heap size)
(320 MB heap size)
(318 MB heap size)
(320 MB heap size)
(323 MB heap size)
(320 MB heap size)
(310 MB heap size)
(313 MB heap size)
(323 MB heap size)
(320 MB heap size)
(321 MB heap size)
(340 MB heap size)
(326 MB heap size)
(323 MB heap size)
(312 MB heap size)
(326 MB heap size)
(316 MB heap size)
(322 MB heap size)
(328 MB heap size)
(323 MB heap size)
(326 MB heap size)
(343 MB heap size)
(315 MB heap size)
(329 MB heap size)
(325 MB heap size)
(319 MB heap size)
(328 MB heap size)
(326 MB heap size)
(331 MB heap size)
(345 MB heap size)
(318 MB heap size)
(331 MB heap size)
(328 MB heap size)
(321 MB heap size)
(329 MB heap size)
(332 MB heap size)
(334 MB heap size)
(348 MB heap size)
(320 MB heap size)
(334 MB heap size)
(331 MB heap size)
(324 MB heap size)
(335 MB heap size)
(332 MB heap size)
(337 MB heap size)
(350 MB heap size)
(323 MB heap size)
(265 MB heap size)
(333 MB heap size)
(337 MB heap size)
(302 MB heap size)
(240 MB heap size)
(327 MB heap size)
(257 MB heap size)
(266 MB heap size)
(334 MB heap size)
(337 MB heap size)
(9.083 s, 342 MB heap size)
(9.235 s, 319 MB heap size)
(9.967 s, 321 MB heap size)
(9.455 s, 345 MB heap size)
(9.686 s, 359 MB heap size)
(12.938 s, 313 MB heap size)
(13.969 s, 340 MB heap size)
(14.6 s, 333 MB heap size)
(6.149 s, 340 MB heap size)
(8.209 s, 347 MB heap size)
(9.345 s, 337 MB heap size)
(10 s, 344 MB heap size)
(8.202 s, 360 MB heap size)
(7.368 s, 333 MB heap size)
(6.233 s, 342 MB heap size)
(6.243 s, 359 MB heap size)
(6.108 s, 342 MB heap size)
(363 MB heap size)
(5.324 s, 315 MB heap size)
(332 MB heap size)
(6.147 s, 338 MB heap size)
(345 MB heap size)
(5.909 s, 341 MB heap size)
(7.36 s, 325 MB heap size)
(5.155 s, 342 MB heap size)
(324 MB heap size)
(5.058 s, 364 MB heap size)
(313 MB heap size)
(5.34 s, 276 MB heap size)
(273 MB heap size)
(342 MB heap size)
(281 MB heap size)
(271 MB heap size)
(327 MB heap size)
(273 MB heap size)
(344 MB heap size)
(276 MB heap size)
(296 MB heap size)
(273 MB heap size)
(281 MB heap size)
(271 MB heap size)
(276 MB heap size)
(289 MB heap size)
(299 MB heap size)
(347 MB heap size)
(277 MB heap size)
(275 MB heap size)
(283 MB heap size)
(274 MB heap size)
(279 MB heap size)
(291 MB heap size)
(350 MB heap size)
(302 MB heap size)
(279 MB heap size)
(278 MB heap size)
(286 MB heap size)
(277 MB heap size)
(281 MB heap size)
(294 MB heap size)
(353 MB heap size)
(283 MB heap size)
(284 MB heap size)
(289 MB heap size)
(281 MB heap size)
(281 MB heap size)
(285 MB heap size)
(297 MB heap size)
(285 MB heap size)
(355 MB heap size)
(287 MB heap size)
(283 MB heap size)
(292 MB heap size)
(282 MB heap size)
(287 MB heap size)
(300 MB heap size)
(286 MB heap size)
(358 MB heap size)
(287 MB heap size)
(294 MB heap size)
(290 MB heap size)
(285 MB heap size)
(290 MB heap size)
(302 MB heap size)
(289 MB heap size)
(297 MB heap size)
(290 MB heap size)
(292 MB heap size)
(361 MB heap size)
(365 MB heap size)
(5.125 s, 298 MB heap size)
(6.13 s, 295 MB heap size)
(6.526 s, 304 MB heap size)
(6.928 s, 301 MB heap size)
(7.982 s, 281 MB heap size)
(8.516 s, 317 MB heap size)
(9.841 s, 313 MB heap size)
(5.477 s, 295 MB heap size)
(5.506 s, 312 MB heap size)
(296 MB heap size)
(294 MB heap size)
(301 MB heap size)
(298 MB heap size)
(326 MB heap size)
(301 MB heap size)
(297 MB heap size)
(294 MB heap size)
(294 MB heap size)
(330 MB heap size)
(295 MB heap size)
(301 MB heap size)
(330 MB heap size)
(296 MB heap size)
(323 MB heap size)
(297 MB heap size)
(297 MB heap size)
(316 MB heap size)
(297 MB heap size)
(304 MB heap size)
(332 MB heap size)
(298 MB heap size)
(308 MB heap size)
(319 MB heap size)
(300 MB heap size)
(300 MB heap size)
(300 MB heap size)
(334 MB heap size)
(306 MB heap size)
(301 MB heap size)
(310 MB heap size)
(303 MB heap size)
(322 MB heap size)
(302 MB heap size)
(302 MB heap size)
(337 MB heap size)
(310 MB heap size)
(212 MB heap size)
(209 MB heap size)
(304 MB heap size)
(253 MB heap size)
(230 MB heap size)
(265 MB heap size)
(286 MB heap size)
(312 MB heap size)
(5.316 s, 317 MB heap size)
(5.007 s, 307 MB heap size)
(8.481 s, 309 MB heap size)
(8.669 s, 289 MB heap size)
(10.421 s, 313 MB heap size)
(283 MB heap size)
(5.801 s, 316 MB heap size)
(336 MB heap size)
(13.269 s, 344 MB heap size)
(288 MB heap size)
(14.837 s, 338 MB heap size)
(288 MB heap size)
(331 MB heap size)
(285 MB heap size)
(302 MB heap size)
(287 MB heap size)
(340 MB heap size)
(297 MB heap size)
(297 MB heap size)
(287 MB heap size)
(313 MB heap size)
(291 MB heap size)
(343 MB heap size)
(300 MB heap size)
(279 MB heap size)
(298 MB heap size)
(290 MB heap size)
(293 MB heap size)
(314 MB heap size)
(303 MB heap size)
(345 MB heap size)
(282 MB heap size)
(301 MB heap size)
(25.518 s, 290 MB heap size)
(293 MB heap size)
(295 MB heap size)
(317 MB heap size)
(305 MB heap size)
(284 MB heap size)
(348 MB heap size)
(303 MB heap size)
(295 MB heap size)
(298 MB heap size)
(320 MB heap size)
(306 MB heap size)
(308 MB heap size)
(287 MB heap size)
(351 MB heap size)
(306 MB heap size)
(298 MB heap size)
(301 MB heap size)
(322 MB heap size)
(310 MB heap size)
(309 MB heap size)
(290 MB heap size)
(353 MB heap size)
(308 MB heap size)
(300 MB heap size)
(264 MB heap size)
(303 MB heap size)
(312 MB heap size)
(326 MB heap size)
(292 MB heap size)
(313 MB heap size)
(268 MB heap size)
(356 MB heap size)
(233 MB heap size)
(220 MB heap size)
(254 MB heap size)
(252 MB heap size)
(232 MB heap size)
(255 MB heap size)
(232 MB heap size)

Node 16.11.0 (problem introduced)

> node --expose-gc --no-sparkplug ./node_modules/.bin/jest --logHeapUsage --maxWorkers ${TEST_PARALLELISM:-8} "--reporters=default" "--reporters=jest-junit"
(41.727 s, 217 MB heap size)
(42.441 s, 217 MB heap size)
(42.452 s, 216 MB heap size)
(45.134 s, 217 MB heap size)
(45.868 s, 220 MB heap size)
(46.713 s, 245 MB heap size)
(49.371 s, 247 MB heap size)
(9.725 s, 328 MB heap size)
(12.574 s, 329 MB heap size)
(54.825 s, 218 MB heap size)
(8.787 s, 340 MB heap size)
(13.189 s, 329 MB heap size)
(13.471 s, 328 MB heap size)
(10.219 s, 341 MB heap size)
(14.752 s, 333 MB heap size)
(8.109 s, 397 MB heap size)
(10.887 s, 386 MB heap size)
(13.786 s, 387 MB heap size)
(8.03 s, 385 MB heap size)
(13.016 s, 388 MB heap size)
(9.036 s, 419 MB heap size)
(332 MB heap size)
(16.237 s, 328 MB heap size)
(10.056 s, 390 MB heap size)
(7.183 s, 458 MB heap size)
(6.975 s, 446 MB heap size)
(9.844 s, 411 MB heap size)
(7.222 s, 423 MB heap size)
(7.276 s, 426 MB heap size)
(6.49 s, 520 MB heap size)
(7.583 s, 444 MB heap size)
(14.041 s, 416 MB heap size)
(10.527 s, 386 MB heap size)
(6.191 s, 415 MB heap size)
(8.422 s, 510 MB heap size)
(6.528 s, 581 MB heap size)
(7.28 s, 486 MB heap size)
(9.558 s, 523 MB heap size)
(7.271 s, 423 MB heap size)
(6.011 s, 545 MB heap size)
(7.171 s, 477 MB heap size)
(9.289 s, 414 MB heap size)
(16.893 s, 496 MB heap size)
(9.017 s, 642 MB heap size)
(7.173 s, 521 MB heap size)
(9.823 s, 548 MB heap size)
(6.973 s, 606 MB heap size)
(7.009 s, 565 MB heap size)
(6.953 s, 416 MB heap size)
(8.185 s, 483 MB heap size)
(581 MB heap size)
(8.146 s, 489 MB heap size)
(5.061 s, 609 MB heap size)
(5.397 s, 666 MB heap size)
(6.634 s, 541 MB heap size)
(5.866 s, 476 MB heap size)
(8.275 s, 670 MB heap size)
(529 MB heap size)
(9.576 s, 478 MB heap size)
(6.206 s, 643 MB heap size)
(7.119 s, 670 MB heap size)
(6.208 s, 538 MB heap size)
(10.041 s, 550 MB heap size)
(6.383 s, 623 MB heap size)
(8.896 s, 727 MB heap size)
(6.355 s, 477 MB heap size)
(6.294 s, 704 MB heap size)
(5.112 s, 599 MB heap size)
(12.077 s, 748 MB heap size)
(5.699 s, 731 MB heap size)
(730 MB heap size)
(6.233 s, 722 MB heap size)
(7.113 s, 549 MB heap size)
(8.579 s, 684 MB heap size)
(6.936 s, 776 MB heap size)
(6.933 s, 659 MB heap size)
(6.843 s, 794 MB heap size)
(10.345 s, 539 MB heap size)
(5.048 s, 842 MB heap size)
(609 MB heap size)
(5.361 s, 783 MB heap size)
(6.001 s, 682 MB heap size)
(6.456 s, 828 MB heap size)
(5.785 s, 854 MB heap size)
(5.447 s, 451 MB heap size)
(5.668 s, 670 MB heap size)
(5.305 s, 843 MB heap size)
(7.549 s, 720 MB heap size)
(6.11 s, 903 MB heap size)
(5.761 s, 743 MB heap size)
(731 MB heap size)
(6.164 s, 889 MB heap size)
(5.513 s, 914 MB heap size)
(5.31 s, 904 MB heap size)
(5.111 s, 963 MB heap size)
(6.897 s, 512 MB heap size)
(6.214 s, 782 MB heap size)
(5.568 s, 805 MB heap size)
(573 MB heap size)
(5.445 s, 949 MB heap size)
(1024 MB heap size)
(6.722 s, 793 MB heap size)
(842 MB heap size)
(6.405 s, 964 MB heap size)
(8.361 s, 975 MB heap size)
(5.613 s, 865 MB heap size)
(902 MB heap size)
(5.719 s, 1009 MB heap size)
(5.861 s, 1084 MB heap size)
(5.734 s, 854 MB heap size)
(5.225 s, 1026 MB heap size)
(7.716 s, 633 MB heap size)
(5.191 s, 926 MB heap size)
(6.251 s, 974 MB heap size)
(1069 MB heap size)
(1085 MB heap size)
(1145 MB heap size)
(6.744 s, 962 MB heap size)
(5.617 s, 915 MB heap size)
(6.088 s, 653 MB heap size)
(1034 MB heap size)
(5.324 s, 987 MB heap size)
(5.146 s, 1130 MB heap size)
(1146 MB heap size)
(713 MB heap size)
(5.612 s, 1223 MB heap size)
(5.552 s, 975 MB heap size)
(1094 MB heap size)
(6.886 s, 963 MB heap size)
(6.49 s, 1048 MB heap size)
(1205 MB heap size)
(5.253 s, 1192 MB heap size)
(5.241 s, 775 MB heap size)
(1265 MB heap size)
(1156 MB heap size)
(1022 MB heap size)
(6.167 s, 1035 MB heap size)
(5.235 s, 1266 MB heap size)
(5.017 s, 1273 MB heap size)
(7.344 s, 1107 MB heap size)
(5.872 s, 1329 MB heap size)
(5.267 s, 1084 MB heap size)
(6.312 s, 1216 MB heap size)
(9.271 s, 837 MB heap size)
(1329 MB heap size)
(1315 MB heap size)
(10.02 s, 1107 MB heap size)
(1389 MB heap size)
(6.675 s, 983 MB heap size)
(1144 MB heap size)
(1276 MB heap size)
(5.482 s, 775 MB heap size)
(1375 MB heap size)
(5.25 s, 1390 MB heap size)
(5.264 s, 1204 MB heap size)
(5.69 s, 1449 MB heap size)
(5.953 s, 1042 MB heap size)
(6.997 s, 1031 MB heap size)
(1339 MB heap size)
(6.208 s, 845 MB heap size)
(5.602 s, 1435 MB heap size)
(1265 MB heap size)
(1092 MB heap size)
(6.002 s, 1103 MB heap size)
(898 MB heap size)
(9.446 s, 1462 MB heap size)
(6.544 s, 1510 MB heap size)
(6.159 s, 1400 MB heap size)
(1495 MB heap size)
(1328 MB heap size)
(5.141 s, 1152 MB heap size)
(1513 MB heap size)
(5.133 s, 957 MB heap size)
(5.965 s, 1164 MB heap size)
(1571 MB heap size)
(6.002 s, 1397 MB heap size)
(5.063 s, 1556 MB heap size)
(5.68 s, 1388 MB heap size)
(1018 MB heap size)
(6.518 s, 1212 MB heap size)
(5.141 s, 1631 MB heap size)
(6.407 s, 1574 MB heap size)
(6.51 s, 1224 MB heap size)
(5.082 s, 1466 MB heap size)
(1616 MB heap size)
(1078 MB heap size)
(1448 MB heap size)
(6.306 s, 1273 MB heap size)
(5.142 s, 1691 MB heap size)
(1677 MB heap size)
(5.811 s, 1077 MB heap size)
(1139 MB heap size)
(6.226 s, 1435 MB heap size)
(1508 MB heap size)
(1138 MB heap size)
(6.122 s, 1213 MB heap size)
(1736 MB heap size)
(1199 MB heap size)
(1496 MB heap size)
(15.376 s, 1528 MB heap size)
(1272 MB heap size)
(5.026 s, 1198 MB heap size)
(10.881 s, 1763 MB heap size)
(1258 MB heap size)
(8.77 s, 1580 MB heap size)
(1556 MB heap size)
(5.052 s, 1797 MB heap size)
(1586 MB heap size)
(1333 MB heap size)
(1322 MB heap size)
(1631 MB heap size)
(5.567 s, 1259 MB heap size)
(1616 MB heap size)
(5.411 s, 1857 MB heap size)
(6.565 s, 1755 MB heap size)
(1646 MB heap size)
(1691 MB heap size)
(1322 MB heap size)
(5.522 s, 1382 MB heap size)
(5.783 s, 1393 MB heap size)
(1549 MB heap size)
(5.275 s, 1676 MB heap size)
(1918 MB heap size)
(1442 MB heap size)
(1751 MB heap size)
(5.368 s, 1382 MB heap size)
(5.263 s, 1454 MB heap size)
(1661 MB heap size)
(5.433 s, 1736 MB heap size)
(1978 MB heap size)
(1502 MB heap size)
(1442 MB heap size)
(1812 MB heap size)
(1514 MB heap size)
(1737 MB heap size)
(1796 MB heap size)
(2038 MB heap size)
(17.555 s, 1800 MB heap size)
(1563 MB heap size)
(1502 MB heap size)
(1872 MB heap size)
(1574 MB heap size)
(1797 MB heap size)
(1428 MB heap size)
(1857 MB heap size)
(2099 MB heap size)
(1750 MB heap size)
(1562 MB heap size)
(6.206 s, 1569 MB heap size)
(1635 MB heap size)
(1933 MB heap size)
(6.031 s, 1560 MB heap size)
(1917 MB heap size)
(1862 MB heap size)
(2159 MB heap size)
(1623 MB heap size)
(1629 MB heap size)
(1694 MB heap size)
(5.348 s, 1637 MB heap size)
(1938 MB heap size)
(2218 MB heap size)
(6.073 s, 1978 MB heap size)
(1689 MB heap size)
(1683 MB heap size)
(1755 MB heap size)
(1543 MB heap size)
(1697 MB heap size)
(1998 MB heap size)
(2038 MB heap size)
(5.306 s, 2285 MB heap size)
(1749 MB heap size)
(1815 MB heap size)
(1677 MB heap size)
(2072 MB heap size)
(1757 MB heap size)
(2059 MB heap size)
(2098 MB heap size)
(2345 MB heap size)
(1875 MB heap size)
(1753 MB heap size)
(1817 MB heap size)
(2115 MB heap size)
(8.092 s, 1810 MB heap size)
(2118 MB heap size)
(2158 MB heap size)
(2405 MB heap size)
(1936 MB heap size)
(1814 MB heap size)
(5.142 s, 1877 MB heap size)
(1869 MB heap size)
(2138 MB heap size)
(5.164 s, 2192 MB heap size)
(2180 MB heap size)
(1874 MB heap size)
(2465 MB heap size)
(1996 MB heap size)
(1931 MB heap size)
(5.237 s, 1938 MB heap size)
(2252 MB heap size)
(5.079 s, 2247 MB heap size)
(2239 MB heap size)
(2057 MB heap size)
(1934 MB heap size)
(2525 MB heap size)
(1997 MB heap size)
(1990 MB heap size)
(2312 MB heap size)
(2307 MB heap size)
(2305 MB heap size)
(2116 MB heap size)
(2258 MB heap size)
(1995 MB heap size)
(2586 MB heap size)
(5.015 s, 2058 MB heap size)
(2050 MB heap size)
(2365 MB heap size)
(2373 MB heap size)
(2176 MB heap size)
(2390 MB heap size)
(2055 MB heap size)
(2337 MB heap size)
(2645 MB heap size)
(2118 MB heap size)
(2110 MB heap size)
(2433 MB heap size)
(2053 MB heap size)
(2237 MB heap size)
(2114 MB heap size)
(2450 MB heap size)
(2447 MB heap size)
(2178 MB heap size)
(2706 MB heap size)
(2180 MB heap size)
(2296 MB heap size)
(2175 MB heap size)
(2511 MB heap size)
(2507 MB heap size)
(2239 MB heap size)
(2767 MB heap size)
(2553 MB heap size)
(2241 MB heap size)
(2163 MB heap size)
(2235 MB heap size)
(2362 MB heap size)
(2567 MB heap size)
(5.929 s, 2598 MB heap size)
(2826 MB heap size)
(2613 MB heap size)
(2301 MB heap size)
(2293 MB heap size)
(2422 MB heap size)
(2301 MB heap size)
(2627 MB heap size)
(2284 MB heap size)
(2623 MB heap size)
(2887 MB heap size)
(2673 MB heap size)
(2286 MB heap size)
(2367 MB heap size)
(2369 MB heap size)
(2361 MB heap size)
(2687 MB heap size)
(2684 MB heap size)
(2419 MB heap size)
(2733 MB heap size)
(2963 MB heap size)
(2429 MB heap size)
(2427 MB heap size)
(2421 MB heap size)
(2747 MB heap size)
(2495 MB heap size)
(2489 MB heap size)
(2744 MB heap size)
(2793 MB heap size)
(2488 MB heap size)
(3007 MB heap size)
(2481 MB heap size)
(2807 MB heap size)
(2549 MB heap size)
(2804 MB heap size)
(2555 MB heap size)
(2854 MB heap size)
(2547 MB heap size)
(3067 MB heap size)
(2541 MB heap size)
(2867 MB heap size)
(2863 MB heap size)
(2615 MB heap size)
(2609 MB heap size)
(2607 MB heap size)
(2914 MB heap size)
(3127 MB heap size)
(2601 MB heap size)
(2927 MB heap size)
(2669 MB heap size)
(2667 MB heap size)
(2974 MB heap size)
(3187 MB heap size)
(2662 MB heap size)
(2681 MB heap size)
(2988 MB heap size)
(2729 MB heap size)
(2984 MB heap size)
(2727 MB heap size)
(2723 MB heap size)
(3247 MB heap size)
(2757 MB heap size)
(3048 MB heap size)
(2789 MB heap size)
(5.435 s, 3044 MB heap size)
(11.665 s, 2788 MB heap size)
(12.754 s, 2859 MB heap size)

I'll log out the code you posted to make sure it's being passed down.

[blimmer]

Yep, @SimenB you were correct - the --no-sparkplug flag does appear to be making it down to the workers. Thanks for pointing me to that code.

@EternallLight and @rthreei - is there anything about your codebase that's notable to potentially cause this issue? The only thing I can think of on my end is that we make heavy use of async_hooks and have historically had issues upgrading in the 16.x series related to them.

I'm sure the jest team would love to have some kind of reproducibility to try to look more closely at this, but I'm struggling to develop a small reproducible case.

[rthreei]

@blimmer nothing in particular is notable about our codebase. It's not a small codebase, but not very big either. Prior to 16.11, there were known memory leaks when running test suite (potentially around lodash); maybe made worse by 16.11.

[SimenB]

If it's specifically in 16.11, you can probably try to build node yourself and bisect https://github.com/nodejs/node/compare/v16.10.0..v16.11.0. Figuring out which commit that introduced it might help understand when one (or more) of your code, Node and Jest does wrong 🙂

[pustovalov]

Looks like in new version of node, garbage collector will no longer clear memory until it reaches the limit.

RAM: 4GB

Node v14.17.3

node --expose-gc --trace_gc ./node_modules/.bin/jest app/javascript --forceExit --ci -w=2 --coverage --logHeapUsage > log/jest_memory_usage.log

Max memory usage:

[552:0x49fcc00]   116300 ms: Scavenge 700.3 (726.1) -> 692.0 (729.6) MB, 4.4 / 0.0 ms  (average mu = 0.983, current mu = 0.990) allocation failure 

full log:
https://gist.github.com/pustovalov/ad8bcd84b0b6bd6abec301982e03ed55

Node v16.13.0

node --expose-gc --trace_gc ./node_modules/.bin/jest app/javascript --forceExit --ci -w=2 --coverage --logHeapUsage > log/current-node-16.log

Max memory usage:

[10321:0x7fdd67100000]   865787 ms: Scavenge 2939.4 (3351.0) -> 2924.0 (3351.0) MB, 3.1 / 0.0 ms  (average mu = 0.346, current mu = 0.315) allocation failure 

full log:
https://gist.github.com/pustovalov/01cc1f484566f4b692fb405335b5f78d

Node v16.13.0 with max-old-space-size

based on how much memory the process was using when it used node v14, set a limit in 700mb

node --expose-gc --trace_gc --max-old-space-size=700 ./node_modules/.bin/jest app/javascript --forceExit --ci -w=2 --coverage --logHeapUsage > log/jest_memory_usage.log

Max memory usage:

[1040:0x66e7250]   193826 ms: Scavenge 603.6 (640.0) -> 592.4 (643.3) MB, 9.2 / 0.0 ms  (average mu = 0.823, current mu = 0.844) allocation failure 

full log:
https://gist.github.com/pustovalov/a86545aa38708d6dda479ee9a1bc4e1d

If you set a limit that will be equal to the maximum amount of RAM, then the process will crash because the cleaning will begin only when this limit is reached

  System:
    OS: macOS 11.6
    CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  Binaries:
    Node: 16.13.0 - /usr/local/Cellar/node@16/16.13.0/bin/node
    Yarn: 1.22.17 - /usr/local/bin/yarn
    npm: 8.1.0 - /usr/local/Cellar/node@16/16.13.0/bin/npm
  npmPackages:
    jest: ^27.3.1 => 27.3.1

[luiz290788]

If it's specifically in 16.11, you can probably try to build node yourself and bisect https://github.com/nodejs/node/compare/v16.10.0..v16.11.0. Figuring out which commit that introduced it might help understand when one (or more) of your code, Node and Jest does wrong 🙂

I've tried that. I was able to reproduce the issue on the last commit on v8 upgrade. It looks like the problem is related to v8 upgrade inside of node.

[B4nan]

Looks like I am hitting this as well. It happens only when doing coverage (as it eats more memory), and it ends up with hard failure. Locking node version to 16.10 helps, node 17 fails the same. Fails locally too.

Maybe one interesting note, it started to happen after I upgraded node-mongodb to v4, failed pipeline here.