AIX: npm install gives IOT/Abort trap

[flynn1973]

Details

installed from here -> https://nodejs.org/dist/v18.13.0/node-v18.13.0-aix-ppc64.tar.gz

npm install gives following error...

root@nimvie: /root # npm install -g n
Assertion failed: __EX, file  ../deps/uv/src/unix/aix.c, line 186
IOT/Abort trap

exporting LIBPATH is of no help

Node.js version

root@nimvie: /root # node -v
v18.13.0

root@nimvie: /root # npm version
{
  npm: '8.19.3',
  node: '18.13.0',
  v8: '10.2.154.23-node.21',
  uv: '1.44.2',
  zlib: '1.2.13',
  brotli: '1.0.9',
  ares: '1.18.1',
  modules: '108',
  nghttp2: '1.51.0',
  napi: '8',
  llhttp: '6.0.10',
  uvwasi: '0.0.13',
  openssl: '3.0.7+quic',
  cldr: '42.0',
  icu: '72.1',
  tz: '2022f',
  unicode: '15.0',
  ngtcp2: '0.8.1',
  nghttp3: '0.7.0'
}

Example code

No response

Operating system

root@nimvie: /root # oslevel -s
7300-01-01-2246

Scope

runtime i guess

Module and version

Not applicable.

[flynn1973]

might be related
libuv/libuv#3465

[flynn1973]

gcc libs

root@nimvie: /home/tremch/GIT/various_python_stuff # rpm -qa | grep libgcc
libgcc10-10.3.0-6.ppc
libgcc-10-2.ppc
root@nimvie: /home/tremch/GIT/various_python_stuff # rpm -qa | grep libstd
libstdc++-10-2.ppc
libstdc++-devel-10-2.ppc
libstdc++10-10.3.0-6.ppc
libstdc++10-devel-10.3.0-6.ppc

[flynn1973]

dbx corefile analysis

seems something is wrong with pthread

root@nimvie: /opt/node-v18.13.0-aix-ppc64 # dbx bin/node  core
Type 'help' for help.
warning: The core file is not a fullcore. Some info may
not be available.
[using memory image in core]
reading symbolic information ...internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c
internal error: assertion failed at line 6693 in file object.c


IOT/Abort trap in pthread_kill at 0x9000000006a92f8
0x9000000006a92f8 (pthread_kill+0x98) e8410028             ld   r2,0x28(r1)
(dbx)

(dbx) where
pthread_kill(??, ??) at 0x9000000006a92f8
_p_raise(??) at 0x9000000006a8b84
raise.raise(??) at 0x900000000229a28
abort() at 0x900000000252108
__assert_c99(??, ??, ??, ??) at 0x9000000002e6510
internal error: 1283-228 expected char ',', found ';-31,448,256;;'
internal error: 1283-228 expected char ';', found ',256;;'
internal error: 1283-232 index("256;;", ':') failed
internal error: unexpected value 120 at line 5206 in file stabstring.c
...
yadayada removed
...
internal error: 1283-228 expected char ',', found '128,64;;'
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
internal error: unexpected value 120 at line 5206 in file stabstring.c
uv_run(loop = 0x000000011030b790, mode = UV_RUN_DEFAULT), line 406 in "core.c"
node::SpinEventLoop(node::Environment*)(??) at 0x100a862e0
node::NodeMainInstance::Run()(??) at 0x101a82a4c
node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*)(??, ??) at 0x100002110
node::Start(int, char**)(??, ??) at 0x100006b50
main(??, ??) at 0x100000580
(dbx)

[sumitradawn]

I had tried to install npm in AIX version "7300-01-00-0000" and it is working fine.

[flynn1973]

i tried it on an older aix level (7.1), even worse than on 7.3

packagebuilder@aixbuildhostng: /home/packagebuilder/test/node-v18.13.0-aix-ppc64/bin # ./node  -v
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument
IOT/Abort trap
packagebuilder@aixbuildhostng: /home/packagebuilder/test/node-v18.13.0-aix-ppc64/bin # ./npm version
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument
IOT/Abort trap

also points to a pthread problem....

[sumitradawn]

I dont have 7.1 machine but I tried it on AIX 7.2 and there also I am not facing this issue. For me it is working fine for both AIX 7.2 and 7.3. I guess this might be an environment issue or gcc, libgcc package installation issue.
I see this in your post:
root@nimvie: /home/tremch/GIT/various_python_stuff # rpm -qa | grep libgcc
libgcc10-10.3.0-6.ppc
libgcc-10-2.ppc
root@nimvie: /home/tremch/GIT/various_python_stuff # rpm -qa | grep libstd
libstdc++-10-2.ppc
libstdc++-devel-10-2.ppc
libstdc++10-10.3.0-6.ppc
libstdc++10-devel-10.3.0-6.ppc
I see two different versions of both libgcc and libstdc++. I am not sure how it is working in your environment but we dont have any "10-2" version in our "AIX toolbox". Can you please uninstall these gcc sub packages and install the latest versions from AIX toolbox and try again.

[flynn1973]

dnf wont let me do this...newest libgcc10 seems to be installed anyway

root@nimvie: /root/bin # dnf search libgcc
Last metadata expiration check: 1 day, 18:51:31 ago on Mon Jan 30 13:25:27 CET 2023.
=============================================================== Name Exactly Matched: libgcc ================================================================
libgcc.ppc : GCC version 10 shared support library
=================================================================== Name Matched: libgcc ====================================================================
libgcc8.ppc : GCC version 8.3.0 shared support library
libgcc10.ppc : GCC version 10.3.0 shared support library
libgcc11.ppc : GCC version 11.3.0 shared support library
root@nimvie: /root/bin # dnf install libgcc10
Last metadata expiration check: 1 day, 18:51:37 ago on Mon Jan 30 13:25:27 CET 2023.
Package libgcc10-10.3.0-6.ppc is already installed.
Dependencies resolved.
Nothing to do.
Complete!
root@nimvie: /root/bin # dnf erase libgcc
Error:
 Problem: The operation would result in removing the following protected packages: dnf
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
root@nimvie: /root/bin # dnf erase libgcc --nobest
Error:
 Problem: The operation would result in removing the following protected packages: dnf
(try to add '--skip-broken' to skip uninstallable packages)

[sumitradawn]

Yes, you are right. We just figured out that "dnf" is having dependency on libgcc that's why it is not allowing you to uninstall it. Anyways, coming to the issue, I see you have given dbx core file analysis. Can you try that with gdb instead of dbx ? Are these freshly installed system because I tried on a fresh system. Now because it's working for me, I am guessing that this is more of and environment issue. Meanwhile, I will try this on AIX 7.1.

Thanks

[flynn1973]

gdb run, as suspected...aix libpthreads.a

root@nimvie: /opt/node-v18.13.0-aix-ppc64 # gdb bin/node core
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-ibm-aix7.1.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin/node...

warning: core file may not match specified executable file.
Core was generated by `node'.
Program terminated with signal SIGABRT, Aborted.
#0  0x09000000006a92f8 in pthread_kill () from /usr/lib/libpthreads.a(_shr_xpg5_64.o)
(gdb) bt
#0  0x09000000006a92f8 in pthread_kill () from /usr/lib/libpthreads.a(_shr_xpg5_64.o)
#1  0x09000000006a8b88 in _p_raise () from /usr/lib/libpthreads.a(_shr_xpg5_64.o)
#2  0x0900000000229a2c in raise () from /usr/lib/libc.a(_shr_64.o)
#3  0x090000000025210c in abort () from /usr/lib/libc.a(_shr_64.o)
#4  0x09000000002e6514 in __assert_c99 () from /usr/lib/libc.a(_shr_64.o)
#5  0x000000010000c6b4 in uv__io_poll (loop=0x11030b790 <_uvcommon.bss_3_+8>, timeout=39) at ../deps/uv/src/unix/aix.c:186
#6  0x0000000100007a60 in uv_run (loop=0x11030b790 <_uvcommon.bss_3_+8>, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:25
#7  0x0000000100a862e4 in node::SpinEventLoop(node::Environment*) ()
#8  0x0000000101a82a50 in node::NodeMainInstance::Run() ()
#9  0x0000000100002114 in node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) ()
#10 0x0000000100006b54 in node::Start(int, char**) ()
#11 0x0000000100000584 in main ()
(gdb)

root@nimvie: /opt/node-v18.13.0-aix-ppc64 # lslpp -l | grep pthread
  bos.rte.libpthreads        7.3.1.0  APPLIED    libpthreads Library
  bos.rte.libpthreads        7.3.1.0  APPLIED    libpthreads Library

root@nimvie: /opt/node-v18.13.0-aix-ppc64 # lslpp -l | grep -i xlc
  xlC.aix61.rte            16.1.0.10  COMMITTED  IBM XL C++ Runtime for AIX 6.1
  xlC.cpp                    9.0.0.0  COMMITTED  C for AIX Preprocessor
  xlC.msg.en_US.cpp          9.0.0.0  COMMITTED  C for AIX Preprocessor
  xlC.msg.en_US.rte        16.1.0.10  COMMITTED  IBM XL C++ Runtime
  xlC.rte                  16.1.0.10  COMMITTED  IBM XL C++ Runtime for AIX
  xlC.sup.aix50.rte          9.0.0.1  COMMITTED  XL C/C++ Runtime for AIX 5.2

[flynn1973]

meanwhile i built node from source...same error...i'm outta here...

export OBJECT_MODE=64
export CC=gcc -maix64
export CXX=g++ -maix
./configure
gmake
gmake install

root@nimvie: /usr/local/bin # ./npm install -g n
Assertion failed: __EX, file  ../deps/uv/src/unix/aix.c, line 186ms
IOT/Abort trap

[mhdawson]

@flynn1973 it's not quite clear to me which version of AIX you were trying to run the binaries on.

For 18.x the community binaries only support AIX 7.2 or later - https://github.com/nodejs/node/blob/main/BUILDING.md#official-binary-platforms-and-toolchains

It is also built with GCC 8 and needs the corresponding glibc installed. glibc versions should really be backward compatible but that has not always been the case from past experience.

[flynn1973]

i did all my testing, installing and building on aix 7.3...just did a quick run on 7.1 but this can be ignored. as seen above a build on 7.3 with gcc8 was succesfull but binaries shows the same problems as the prebuilt ones.

[sumitradawn]

@flynn1973 We had tried to recreate the issue in our system but We are unable to do so, Even We tried with the source code, it is working fine for us in both AIX 7.2 and 7.3.
Can you please run "truss npm install g n" - command in your system and share the output with us, so that we can figure out the issue and work on it.

[flynn1973]

truss_out.txt

[flynn1973]

i skimmed through the truss output and it seems it tries to open "/root/package.json" which of course does not exist. i created an empty file then another error occurs later in the run.

kopen("/root/package.json", 040000000)          = 23
fstatx(23, 0x0FFFFFFFFFFFAEA0, 176, 0)          = 0
kread(23, "\0\0\00110881C  \0\0\0\0".., 8192)   = 0
close(23)                                       = 0
kwrite(21, " 4 4   e r r o r   J S O".., 76)    = 76
npmkwrite(20, "1B [ 3 7 ; 4 0 m n p m1B".., 15)       = 15
 kwrite(20, "  1B [ 0 m", 5)                 = 5
ERR!kwrite(20, "1B [ 3 1 ; 4 0 m E R R !".., 16)      = 16
 kwrite(20, "  1B [ 0 m", 5)                 = 5
JSON.parsekwrite(20, "1B [ 3 5 m J S O N . p a".., 19)   = 19
 Unexpected end of JSON input while parsing empty string
kwrite(20, "   U n e x p e c t e d  ".., 61) = 61
kwrite(21, " 4 5   e r r o r   J S O".., 128)   = 128
npmkwrite(20, "1B [ 3 7 ; 4 0 m n p m1B".., 15)       = 15
 kwrite(20, "  1B [ 0 m", 5)                 = 5
ERR!kwrite(20, "1B [ 3 1 ; 4 0 m E R R !".., 16)      = 16
 kwrite(20, "  1B [ 0 m", 5)                 = 5
JSON.parsekwrite(20, "1B [ 3 5 m J S O N . p a".., 19)   = 19
 Failed to parse JSON data.
kwrite(20, "   F a i l e d   t o   p".., 32) = 32
npmkwrite(20, "1B [ 3 7 ; 4 0 m n p m1B".., 15)       = 15
 kwrite(20, "  1B [ 0 m", 5)                 = 5
ERR!kwrite(20, "1B [ 3 1 ; 4 0 m E R R !".., 16)      = 16
 kwrite(20, "  1B [ 0 m", 5)                 = 5
JSON.parsekwrite(20, "1B [ 3 5 m J S O N . p a".., 19)   = 19
 Note: package.json must be actual JSON, not just JavaScript.
kwrite(20, "   N o t e :   p a c k a".., 66) = 66
kwrite(21, " 4 6   v e r b o s e   e".., 18)    = 18
kwrite(20, 0x0FFFFFFFFFFF6F90, 0)               = 0
kwrite(18, 0x0FFFFFFFFFFF6F90, 0)               = 0
kwrite(21, " 4 7   t i m i n g   n p".., 34)    = 34
kwrite(21, " 4 8   v e r b o s e   u".., 52)    = 52
kwrite(21, " 4 9   v e r b o s e   u".., 62)    = 62
kwrite(21, " 5 0   v e r b o s e   u".., 61)    = 61
kwrite(21, " 5 1   v e r b o s e   c".., 18)    = 18
mprotect(0x0A00000000083000, 245760, 7)         = 0
mprotect(0x0A00000000083000, 245760, 5)         = 0
mprotect(0x0A00000000083000, 245760, 7)         = 0
mprotect(0x0A00000000083000, 245760, 5)         = 0
mprotect(0x0A00000000083000, 245760, 7)         = 0
mprotect(0x0A00000000083000, 245760, 5)         = 0
mprotect(0x0A00000000083000, 245760, 7)         = 0
mprotect(0x0A00000000083000, 245760, 5)         = 0

kwrite(20, "\n", 1)                             = 1
kwrite(21, " 5 2   e r r o r   A   c".., 120)   = 120
npmkwrite(20, "1B [ 3 7 ; 4 0 m n p m1B".., 15)       = 15
 kwrite(20, "  1B [ 0 m", 5)                 = 5
ERR!kwrite(20, "1B [ 3 1 ; 4 0 m E R R !".., 16)      = 16
kwrite(20, "1B [ 3 5 m1B [ 0 m", 9)              = 9
 A complete log of this run can be found in:
kwrite(20, "   A   c o m p l e t e  ".., 49) = 49
npmkwrite(20, "1B [ 3 7 ; 4 0 m n p m1B".., 15)       = 15
 kwrite(20, "  1B [ 0 m", 5)                 = 5
ERR!kwrite(20, "1B [ 3 1 ; 4 0 m E R R !".., 16)      = 16
kwrite(20, "1B [ 3 5 m1B [ 0 m", 9)              = 9
     /root/.npm/_logs/2023-03-09T13_05_32_224Z-debug-0.log
kwrite(20, "           / r o o t / .".., 63) = 63

root@nimvie: /root # cat /root/.npm/_logs/2023-03-09T13_05_32_224Z-debug-0.log
0 verbose cli /opt/node-v18.13.0-aix-ppc64/bin/node /opt/node-v18.13.0-aix-ppc64/bin/npm
1 info using npm@8.19.3
2 info using node@v18.13.0
3 timing npm:load:whichnode Completed in 2ms
4 timing config:load:defaults Completed in 6ms
5 timing config:load:file:/opt/node-v18.13.0-aix-ppc64/lib/node_modules/npm/npmrc Completed in 11ms
6 timing config:load:builtin Completed in 11ms
7 timing config:load:cli Completed in 4ms
8 timing config:load:env Completed in 1ms
9 timing config:load:project Completed in 118ms
10 timing config:load:file:/root/.npmrc Completed in 3ms
11 timing config:load:user Completed in 4ms
12 timing config:load:file:/opt/node-v18.13.0-aix-ppc64/etc/npmrc Completed in 3ms
13 timing config:load:global Completed in 3ms
14 timing config:load:validate Completed in 0ms
15 timing config:load:credentials Completed in 2ms
16 timing config:load:setEnvs Completed in 2ms
17 timing config:load Completed in 151ms
18 timing npm:load:configload Completed in 151ms
19 timing npm:load:mkdirpcache Completed in 10ms
20 timing npm:load:mkdirplogs Completed in 8ms
21 verbose title npm install g n
22 verbose argv "install" "g" "n"
23 timing npm:load:setTitle Completed in 5ms
24 timing config:load:flatten Completed in 14ms
25 timing npm:load:display Completed in 41ms
26 verbose logfile logs-max:10 dir:/root/.npm/_logs
27 verbose logfile /root/.npm/_logs/2023-03-09T13_05_32_224Z-debug-0.log
28 timing npm:load:logFile Completed in 26ms
29 timing npm:load:timers Completed in 1ms
30 timing npm:load:configScope Completed in 0ms
31 timing npm:load Completed in 245ms
32 timing arborist:ctor Completed in 2ms
33 silly logfile start cleaning logs, removing 1 files
34 silly logfile done cleaning log files
35 timing idealTree Completed in 28ms
36 timing command:install Completed in 139ms
37 verbose stack JSONParseError: Unexpected end of JSON input while parsing empty string
37 verbose stack     at /opt/node-v18.13.0-aix-ppc64/lib/node_modules/npm/node_modules/read-package-json-fast/index.js:11:61
37 verbose stack     at async Arborist.buildIdealTree (/opt/node-v18.13.0-aix-ppc64/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:208:7)
37 verbose stack     at async Promise.all (index 1)
37 verbose stack     at async Arborist.reify (/opt/node-v18.13.0-aix-ppc64/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:154:5)
37 verbose stack     at async Install.exec (/opt/node-v18.13.0-aix-ppc64/lib/node_modules/npm/lib/commands/install.js:145:5)
37 verbose stack     at async module.exports (/opt/node-v18.13.0-aix-ppc64/lib/node_modules/npm/lib/cli.js:78:5)
38 verbose cwd /root
39 verbose AIX 7.3
40 verbose node v18.13.0
41 verbose npm  v8.19.3
42 error code EJSONPARSE
43 error path /root/package.json
44 error JSON.parse Unexpected end of JSON input while parsing empty string
45 error JSON.parse Failed to parse JSON data.
45 error JSON.parse Note: package.json must be actual JSON, not just JavaScript.
46 verbose exit 1
47 timing npm Completed in 1229ms
48 verbose unfinished npm timer reify 1678367133267
49 verbose unfinished npm timer reify:loadTrees 1678367133368
50 verbose unfinished npm timer idealTree:init 1678367133371
51 verbose code 1
52 error A complete log of this run can be found in:
52 error     /root/.npm/_logs/2023-03-09T13_05_32_224Z-debug-0.log