EFAULT, EPIPE errors on Big Sur on Apple Silicon (M1)

[andreialecu]

• Version: 15.5.0
• Platform: Darwin Kernel Version 20.2.0: Wed Dec 2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 arm64
• Subsystem: streams(?)

What steps will reproduce the bug?

Using node 15.5.0 compiled for arm64.

➜ node -p process.arch
arm64

I'm baffled that I couldn't find anything related to this yet, and I'm not sure whether it's a nodejs bug or something else.

While running nodejs apps that use network or file streams it seems that they very frequently break with errors like EPIPE or EFAULT.

For example, a recent crash:

Error: read EFAULT
    at Pipe.onStreamRead (node:internal/stream_base_commons:213:20)
    at Pipe.callbackTrampoline (node:internal/async_hooks:131:14)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:80:21) {
  errno: -14,
  code: 'EFAULT',
  syscall: 'read'
}

This also happens quite a lot in the nodejs subsystem used in VSCode: microsoft/vscode#113410 (see various screenshots attached there)

I haven't verified if it happens in Rosetta emulation.

How often does it reproduce? Is there a required condition?

Very often, at least once every 10-60 minutes.

I don't have a specific reproduction, but I've seen it happen when something like tsc --watch reinitiates a compilation based on file system changes. It also happens randomly in simple expressjs servers that don't watch the file system, but just keep a connection to a mongodb server.

What is the expected behavior?

More stability.

What do you see instead?

Frequent errors.

Additional information

If I can run with some sort of flag that produces an useful crash dump, let me know of the instructions and I'll do it asap.

[andreialecu]

@karlhorky you're going to have a bad time with 11.2: #37061

[karlhorky]

Ooh looks fancy, thanks for the link! Hopefully don't run into this! In any case, worth it to have Bluetooth working reliably again.

[maximilianschmid]

Same here, long running node script produces this error on M1 Mac mini, Big Sur 11.1 and 11.2 and NodeJS 15.x.
Same script on x86 and same NodeJS version does not produce this error.

Recent crash:

Tue Feb 02 2021 08:51:37 GMT+0100 (Mitteleuropäische Normalzeit): no updates detected.
node:events:356
throw er; // Unhandled 'error' event
^

Error: read EFAULT
    at Pipe.onStreamRead (node:internal/stream_base_commons:211:20)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:81:21) {
  errno: -14,
  code: 'EFAULT',
  syscall: 'read'
}
npm ERR! code 1
npm ERR! path /Users/cypress1/planfred3/testapp
npm ERR! command failed
npm ERR! command sh -c node run.js

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/cypress1/.npm/_logs/2021-02-02T07_51_44_152Z-debug.log

[andreialecu]

I haven't recently seen this error at all, but I upgraded my M1 from an initial 8GB of RAM one, to 16.

I have also been using early 11.2 betas, back before nodejs got broken completely in recent RCs (#37061)

There was a bug reported in macOS 11.1 on M1 chips that occurs when memory pressure is high, which is much more likely on the 8GB versions. See discussion in: libuv/libuv#3095 (comment)

So I think it is relevant to mention which version of the M1 everyone is using:

8GB or 16GB?

[karlhorky]

16GB. Updated my post above: #36826 (comment)

[Biboswan]

We are facing the same issue. A meteor application after some time is giving the same EFAULT error... being discussed here.
Device is Macbook Pro M1 8gb RAM

[maximilianschmid]

8GB mac mini which produces this error.

[zvictor]

We are facing the same issue. A meteor application after some time is giving the same EFAULT error... being discussed here.
Device is Macbook Pro M1 8gb RAM

Increasing the memory limit has helped us immensely. Could it be that Rosetta uses much more memory to do its work?

If so, this should help you all for now:

#!/bin/bash

if [ "$(sysctl -in sysctl.proc_translated)" = "1" ]; then
    echo "Meteor: Rosetta2: Limitting memory to 2gb"
    export NODE_OPTIONS="--max_old_space_size=2048"
else
    echo "Meteor: Native-intel: Limitting memory to 512mb"
    export NODE_OPTIONS="--max_old_space_size=512"
fi

your-cmd-here

[JvManuel]

Is there someone who was able to fix this issue?

[timmywheels]

Is there someone who was able to fix this issue?

I was able to get around this issue by adding --max_old_space_size=2048 flag to my npm run start script per @zvictor's comment

[AltamimiYasser]

Having the same issue, Macbook Air, m1 ship, 16gp ram.

[GautierT]

Hi.

--max_old_space_size=2048 doesn't change anything for me.

When i have a process auto-restarting with nodemon or PM2 sometimes it crashed with this error :

node:events:346
      throw er; // Unhandled 'error' event
      ^

Error: read EFAULT
    at Pipe.onStreamRead (node:internal/stream_base_commons:211:20)
    at Pipe.callbackTrampoline (node:internal/async_hooks:132:14)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:81:21) {
  errno: -14,
  code: 'EFAULT',
  syscall: 'read'
}

node -v : v15.14.0
macos : 11.2 (20D64)
Apple M1 - 8g ram

So i use until nodemon src/; do sleep 1; done to auto restart when it crash but it's an ugly fix and i think it eats my CPU (iterm2 is 100% cpu after a few hours.)

Any idead other than changing for a 16gb mac ? @aduh95 @andreialecu

Thanks a lot.

[karlhorky]

Switching to a 16GB machine is not a solution - this also happens for me on a 16GB M1 MacBook Pro: #36826 (comment)

[andreialecu]

Indeed, this does happen on the 16GB M1 - but extremely rarely, at least based on my own experience. I have only seen it once or twice over 2 months or so.

It was however extremely frequent on the 8GB M1 (several times per hour). As per my brief research in previous comments I strongly believe this is memory/page fault related. So until it is acknowledged and a fix is found, the only solution seems to be upgrading to 16GB.

[karlhorky]

I see it multiple times per day on the 16GB M1 MacBook Pro.

[andreialecu]

@karlhorky when it happens, is the system under high memory pressure?

[karlhorky]

I don't know if there's a better way of testing this, but in Activity Monitor, here's a snapshot from now, which should be pretty representative of the normal work I do (12.8GB / 16GB used, 7GB Swap, 3GB Cached Files):

[andreialecu]

@karlhorky that looks almost identical to mine. Try observing the Memory Pressure graph and see if you can correlate EFAULT crashes with spikes on the graph.

I've been able to reproduce the EFAULT error in VSCode by adding memory pressure. More details about that here:
microsoft/vscode#113410 (comment)

[andreialecu]

Pinging @AshCripps, hopefully this can be prioritized/acknowledged. 🤞

[dannyking]

Experiencing this issue on M1 8gb mac mini

[quentin-aslan]

Same, macbook air M1 8gb

[bl-ue]

Unfortunately, this is hard to troubleshoot for two reasons:

• Not everyone who has a Mac or even Big Sur has an M1 (this is the case with me)
• If it's intermittent, it's always harder to deal with

If someone has the time, an M1, and the error frequently enough, (@ashveen, @tobspr?) I'll try to work with you.

[GautierT]

@bl-ue : I have the error very frequently !
I'm on a Macbook Pro M1 8gb Big sur 11.2.

What can i do to help you fix this ?

I run all my node process with until nodemon src/; do sleep 1; done now to enable auto restart when it crash but it's not very convenient.

[bl-ue]

Hi @GautierT — great. First of all, could you consider updating to Big Sur 11.4? I wonder if that would resolve the issue 🤔
If it doesn't, at least we know it still happens on the latest version.

[GautierT]

@bl-ue
I'm on the beta channel. I can update to 11.5 beta 2. I will launch the update tonight.

[bl-ue]

Also, could someone try it out on macOS Monterey 12?

[ChALkeR]

I obtained a lowest-config M1 Mac Mini (8GB ram, 256GB disk) running Big Sur and can't reproduce this yet.
Does anyone have a way to reproduce?

I tried vs code, most I got from it was SIGSEGV for Extension Host, but I had to put the system on high memory pressure (i.e. constant reallocating random-filled buffers into a separate process to avoid them going into swap) to get that, and the whole system was acting laggy/slow at that point.

I can't reproduce EFAULT/EPIPE yet.

Upd: by increasing the memory pressure further, I managed to freeze the whole system (it freezed for ~20 seconds, then blinked and went into a reboot), but I still didn't get EFAULT/EPIPE in vscode or node processes.

I need some way to reproduce this.

[GautierT]

i'm sorry @bl-ue, I still haven't had time to update.
I will try to do it this week.

[andreialecu]

@GautierT do you have a way to reproduce it as per @ChALkeR's request above? Since I upgraded to the 16GB version I'm not able to reproduce this any more.

[GautierT]

@ChALkeR : for me it's happening a lot when I run nodemon index.js on a codebase of 100 files (build with feathersjs). It's a private repo so i can't give you access.

but when I run ungit on this repo, ungit crash after some time (2-4 minutes) when I change file on this codebase.

I tried running ungit on a small repo and it eventually crash after some change on the codebase and some time.

~/Documents/Projets/feathers-basics master ?5                                                                                                                                                                                      11s
❯ ungit
## Ungit started ##

Took 273ms to start server.
Navigate to http://localhost:8448/#/repository?path=/Users/gautier/Documents/Projets/feathers-basics
2021-06-28T14:56:48.268Z - error: Error: read EFAULT
    at Pipe.onStreamRead (node:internal/stream_base_commons:211:20)
Stopped keeping ungit alive

[ChALkeR]

@GautierT Thanks!
Managed to reproduce EFAULT (at Pipe.onStreamRead) on ungit, investigating.
First, I'll check if it reproduces reliably and then will try to create a reproducible testcase.

[ChALkeR]

Status update: the reproduction setup is reliable, though the exact moment when it fails fluctuates a lot and seems to depend on the system memory pressure. Though even with low pressure it seems to fail eventually.
So for this problem to occur, high memory pressure is not required (though that seems to increase the frequency of failures significantly).

[ChALkeR]

Also reproduced Illegal hardware instruction, same setup, slightly different parameters.

[ChALkeR]

Both Illegal hardware instruction and EFAULT at Pipe.onStreamRead seem to be reproducible with this:

const { spawn } = require('child_process')
const tick = () => {
  const proc = spawn('echo', ['1'])
  proc.stdout.on('data', () => {})
  proc.stderr.on('data', () => {})
  proc.on('close', () => {
    setTimeout(tick, 0)
  })
}
for (let i = 0; i < 8; i++) tick()

With very low memory pressure it might effectively never occur.

Increasing memory pressure (by spawning something memory-consuming) makes failures much more common.

It doesn't have to consume all the available memory to fail, e.g. 60-70% makes failures common enough to occur in several seconds on my setup.

Could someone confirm this on their setup, please?

---

Memory pressure could be increased e.g. with this script:

const { randomBytes } = require('crypto')
let y
for (let i = 0; i < 30; i++) {
  console.log(i)
  y = Array(1).fill().map(() => randomBytes(2e9))
}

Parameters (1, 2e9) might need to be tweaked depending on the setup.

Or just with opening some pages in separate Safari windows.

[ChALkeR]

cc @cjihrig and @deepak1556 perhaps, libuv/libuv#3050 might be somewhat related. Or not.

Also, is it correct that VS Code Extension Host spawns child processes (e.g. git)?
Or has this occured without child_process usage?

I'll look deeper though.

[ChALkeR]

Upon updating macOS from stock 11.2.3 to 11.4, I can't reproduce this anymore.
Will try again tomorrow.
q: does anyone observe this on 11.4?

[deepak1556]

Hi @ChALkeR ,

Also, is it correct that VS Code Extension Host spawns child processes (e.g. git)?

Yup that is correct.

the reports we got so far indicated the SIGILL, EFAULT crashes only came from extensions that used child_process module in the extension host.

It is also to note that VS Code stable carries the patch for libuv/libuv#3050, if you see the crash even in stable then the libuv issue might be unrelated. Also, users who had reported the issue on 16GB devices recently mentioned that they no longer see the issue. So it might be related to 11.4 update.

[ChALkeR]

I can't reproduce this on macOS 11.4 anymore on 8GB/256GB with the isolated testcase that I got above in #36826 (comment), which reproduced this reliably on macOS 11.2.3.
Also, I can't reproduce the original issue with ungit from #36826 (comment) anymore after the update.

Unless there are or will be reports with this issue occuring on 11.4 setups, I think it's safe to assume that this was fixed in macOS.

[ChALkeR]

I'm going to close this shortly unless anyone objects.

Will reopen if there would be any new cases reported with macOS 11.4 or newer.

[GautierT]

Hi @ChALkeR, I spent the last 2 days with 11.4 and no EFAULT anymore ! 🥳 🥳

[ianmartorell]

I've been on 11.4 for a couple weeks and haven't had the issue 😄

[andreialecu]

Awesome! I've been on 11.4 for a while also.

I'll close this myself. Thanks everyone!

[vashisth00]

I am still getting the issue on MacOS 11.4 ...

[vashisth00]

events.js:377js
throw er; // Unhandled 'error' event
^

Error: read EFAULT
at Pipe.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on Socket instance at:
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:82:21) {
errno: -14,

[maximelafarie]

Same issue with Node:current (16.x.x) on similar Mac configuration. Several memory pressure tests on a route using createReadStream seem to make it crash:

Error: read EFAULT
    at Pipe.onStreamRead (node:internal/stream_base_commons:211:20)
    at Pipe.callbackTrampoline (node:internal/async_hooks:130:17)
[ERROR] 17:45:21 Error: read EFAULT

Additional note: it seems it crashes only few minutes after the pressure test passed (without errors during the test).

[ChALkeR]

@vashisth00 @maximelafarie What do you use to reproduce this?
The script from #36826 (comment) or something else?

[maximelafarie]

@vashisth00 @maximelafarie What do you use to reproduce this? The script from #36826 (comment) or something else?

I'm creating files massively with fs.writeStream, but I will run your scripts anyway to be sure.

[marcotas]

I'm having this issue with M1 Pro Chip since I got the new MacBook Pro 16". Anyone else having the same issue? Did anyone solve it for M1 Pro?

Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
Exited with code: 1

Node version: 14.18.1 (but happens with any version).

[ChALkeR]

@marcotas

1. What's the macOS version?
2. Could you reproduce using the testcase above? EFAULT, EPIPE errors on Big Sur on Apple Silicon (M1) #36826 (comment)
3. If not, could you share a testcase?

[andreialecu]

@ChALkeR I'm not sure if this is directly related, but just going to add it here.

I was recently troubleshooting a node crash inside docker builds on my M1 machine.
There is a repro here:
https://github.com/andreialecu/yarn-docker-crash-repro

Run it with: docker build --tag repro --no-cache --progress=plain --file packages/backend/Dockerfile .

Basically when Docker is configured to the default on Mac (which is maximum 2GB of memory), the yarn install step crashes a majority of the time with exit code 1 but no error output at all. Sometimes it goes through, but most times it crashes on the Fetch step.

If I increase Docker memory to 4GB it builds properly every time.

I believe this may be caused by memory pressure. I wasn't able to reproduce it with --max-old-space-size at all however.

It could also be a docker bug of some sort, I'm not sure. I tried node 16 and 17 and it occurs on both.

There was more discussion on this topic on the Yarn discord here: https://discord.com/channels/226791405589233664/910944274953175082/911221057732636672

[micaiah-effiong]

I'm having this issue with MacBook Pro M1, 2020, 8Gb ram, 2020.

events.js:377
      throw er; // Unhandled 'error' event
      ^

Error: read EFAULT
    at Pipe.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  errno: -14,
  code: 'EFAULT',
  syscall: 'read'
}
error Command failed with exit code 1.

Node version v14.18.0

[ChALkeR]

@micaiah-effiong could you please share a testcase?
And macOS version.

Also see #36826 (comment)

[marcotas]

@ChALkeR I'm using Monterrey 12.0.1. Tried node v17.1.0, 16.13.0 and 14.18.1 with no success. The application I'm running is meteor framework and they launched a recent version 2.5.1 compatible with M1 chips, but didn't work anyways too. It doesn't seem to be a framework problem (I suppose).

I have no idea how to solve this and it seems to be related to memory usage? Anyways I'm following up with this issue to see if someone shows up with the same issue. I also tried to use nvm and didn't solve my problem too.

[ChALkeR]

@marcotas Thanks for the OS info! Could you try to reproduce the issue on #36826 (comment)?
That's a stand-alone script with no deps.

If it doesn't reproduce, please try adding memory pressure (e.g. via the bottom part of that comment).

Also: does the setup with Meteor spawn child processes at all or not?

[micaiah-effiong]

@micaiah-effiong could you please share a testcase? And macOS version.

Also see #36826 (comment)

I using a macOS Big Sur version 11.1.
I am running an express server in a dev environment am I use nodemon to restart the server. After a couple of restarts, it crashes with the error

events.js:377
      throw er; // Unhandled 'error' event
      ^

Error: read EFAULT
    at Pipe.onStreamRead (internal/stream_base_commons.js:209:20)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  errno: -14,
  code: 'EFAULT',
  syscall: 'read'
}
error Command failed with exit code 1.

[ChALkeR]

@micaiah-effiong This was known on 11.1 due to a macOS issue and it was previously thought that it was resolved in macOS 11.4.

But there are new reports on 12 above, though I wasn't yet able to reproduce.
It would be great if someone would be able to either reproduce with the testcase from #36826 (comment) on any macOS version >= 11.4 or share a testcase that reproduces this on a macOS version >= 11.4.

[marcotas]

@ChALkeR I wasn't able to reproduce the error with your comment you suggested. Nothing happened at all. It seems to me it's something related to websockets though. Meteor framework uses websockets a LOT so I think this might be the case. I also tried this solution in Safari but it still happening, also tried to use only chrome but didn't change anything. Unfortunately it's not something I'm able to reproduce, because I'm just facing this in a meteor framework application. I'll give any more updates if I figure out something more.

[ChALkeR]

@marcotas
Thanks! If you could provide at least some way to reproduce (e.g. a repo/setup) I could try to work from there to build a minimal testcase.

A repo which has some meteor setup triggering this might also help.

Preferably without docker or any other extra parts.

[marcotas]

@ChALkeR after updating my MacOS to 12.1 it just stop with this error. I'm not facing this error anymore. It seems it was something on Apple's end.

[IliaEremin]

@marcotas to me upgrade to 12.1 fixed damn error for a while and then I started to get it again. Problem goes away after I restart my mac. So probably upgrade to 12.1 has nothing to do with problem. It's rather reboot made a difference for you and me.

[ChALkeR]

Hmm. I still couldn't reproduce this, but i didn't try keeping mac active for a while.

@IlyaEremin A few qustions that might help:

1. Is it the reboot or does logout/relogin help?
2. Does just closing all other apps help?
3. Is this perhaps memory load related?
4. Is there a way to trigger this quicker after a reboot?

[marcotas]

@marcotas to me upgrade to 12.1 fixed damn error for a while and then I started to get it again. Problem goes away after I restart my mac. So probably upgrade to 12.1 has nothing to do with problem. It's rather reboot made a difference for you and me.

@IlyaEremin I was facing the same issue sometimes until I figured out that the problem was with the wkhtmltopdf package not installed on my machine using homebrew. So I just installed using homebrew and I haven't gotten this error anymore.

[IliaEremin]

@marcotas man.... this is a miracle. I'm using wkhtmltopdf in the project I work on and this was exactly the problem. After I installed one problem disappeared. Thank you a lot!

[Sun-Woo-Kim]

try this

watchman watch-del-all