investigate flaky test-policy-integrity

[Trott]

• Test: test/pummel/test-policy-integrity.js
• Platform: centos7-arm64-gcc8
• Console Output:

07:01:10 not ok 2971 pummel/test-policy-integrity
07:01:15   ---
07:01:15   duration_ms: 5.728
07:01:15   severity: fail
07:01:15   exitcode: 1
07:01:15   stack: |-
07:01:15     permutation 118 failed
07:01:15     {
07:01:15       config: {
07:01:15         onError: 'exit',
07:01:15         shouldSucceed: true,
07:01:15         entryPath: './worker-spawner.cjs',
07:01:15         willDeletePolicy: true,
07:01:15         preloads: [],
07:01:15         parentPath: './parent.js',
07:01:15         depPath: './dep.js',
07:01:15         resources: {
07:01:15           './dep.js': {
07:01:15             body: '',
07:01:15             integrities: [
07:01:15               'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
07:01:15               'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='
07:01:15             ]
07:01:15           },
07:01:15           './parent.js': {
07:01:15             body: '\n' +
07:01:15               '    if (!process.env.DEP_FILE) {\n' +
07:01:15               '      console.error(\n' +
07:01:15               "        'missing required DEP_FILE env to determine dependency'\n" +
07:01:15               '      );\n' +
07:01:15               '      process.exit(33);\n' +
07:01:15               '    }\n' +
07:01:15               '    require(process.env.DEP_FILE)\n' +
07:01:15               '  ',
07:01:15             integrities: [
07:01:15               'sha256-AzbH3oEA96J/+tLSqOJ+LaIt9vdOY+ikShkvKm0VutU=',
07:01:15               'sha256-R0QVSw9BUI7znwKNvt6BuzkLozhYKTtFNRSENIL5H7k='
07:01:15             ]
07:01:15           },
07:01:15           './worker-spawner.cjs': {
07:01:15             body: '\n' +
07:01:15               "  const path = require('path');\n" +
07:01:15               "  const { Worker } = require('worker_threads');\n" +
07:01:15               '  if (!process.env.PARENT_FILE) {\n' +
07:01:15               '    console.error(\n' +
07:01:15               "      'missing required PARENT_FILE env to determine worker entry point'\n" +
07:01:15               '    );\n' +
07:01:15               '    process.exit(33);\n' +
07:01:15               '  }\n' +
07:01:15               '  if (!process.env.DELETABLE_POLICY_FILE) {\n' +
07:01:15               '    console.error(\n' +
07:01:15               "      'missing required DELETABLE_POLICY_FILE env to check reloading'\n" +
07:01:15               '    );\n' +
07:01:15               '    process.exit(33);\n' +
07:01:15               '  }\n' +
07:01:15               '  const w = new Worker(path.resolve(process.env.PARENT_FILE));\n' +
07:01:15               "  w.on('exit', (status) => process.exit(status === 0 ? 0 : 1));\n",
07:01:15             integrities: [
07:01:15               'sha256-23P4m8edtJAPTH24RJHIHMEuMVgP7QKcgWBSyj6FxPc=',
07:01:15               'sha256-XTRkNC3vxo+zvjv7CD8iapeqc7npNN7TwqRR0TOv5lg='
07:01:15             ]
07:01:15           }
07:01:15         }
07:01:15       },
07:01:15       manifest: {
07:01:15         onerror: 'exit',
07:01:15         resources: {
07:01:15           'file:///home/iojs/node-tmp/.tmp.2970/test-policy-integrity-permutation-118/dep.js': {
07:01:15             integrity: 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
07:01:15             dependencies: true
07:01:15           },
07:01:15           'file:///home/iojs/node-tmp/.tmp.2970/test-policy-integrity-permutation-118/parent.js': {
07:01:15             integrity: 'sha256-AzbH3oEA96J/+tLSqOJ+LaIt9vdOY+ikShkvKm0VutU= sha256-R0QVSw9BUI7znwKNvt6BuzkLozhYKTtFNRSENIL5H7k=',
07:01:15             dependencies: true
07:01:15           },
07:01:15           'file:///home/iojs/node-tmp/.tmp.2970/test-policy-integrity-permutation-118/worker-spawner.cjs': {
07:01:15             integrity: 'sha256-23P4m8edtJAPTH24RJHIHMEuMVgP7QKcgWBSyj6FxPc= sha256-XTRkNC3vxo+zvjv7CD8iapeqc7npNN7TwqRR0TOv5lg=',
07:01:15             dependencies: true
07:01:15           }
07:01:15         }
07:01:15       }
07:01:15     }
07:01:15     exit code: 1 signal: null
07:01:15     stdout: 
07:01:15     stderr: 
07:01:15     /home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc8/test/pummel/test-policy-integrity.js:187
07:01:15             throw e;
07:01:15             ^
07:01:15     
07:01:15     AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
07:01:15     
07:01:15     1 !== 0
07:01:15     
07:01:15         at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc8/test/pummel/test-policy-integrity.js:170:18)
07:01:15         at ChildProcess.emit (node:events:369:20)
07:01:15         at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12) {
07:01:15       generatedMessage: true,
07:01:15       code: 'ERR_ASSERTION',
07:01:15       actual: 1,
07:01:15       expected: 0,
07:01:15       operator: 'strictEqual'
07:01:15     }
07:01:15   ...

• Build Links: https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc8/36782/console

[Trott]

@bmeck Any idea what might be happening to cause this?

[bmeck]

07:01:15     exit code: 1 signal: null
07:01:15     stdout: 
07:01:15     stderr: 

it is highly suspicious that a process would exit without any output and exit code 1. This means the outer process died after seeing the worker die:

07:01:15 " w.on('exit', (status) => process.exit(status === 0 ? 0 : 1));\n",

Likely we should change this harness to output the worker stdout/stderr in this case, but I don't see why parent/dep would die this many permutations down.

[Trott]

It happened again today, and it was again on centos7-arm64-gcc8 and basically all the same symptoms I think:

00:22:08 not ok 2975 pummel/test-policy-integrity
00:22:14   ---
00:22:14   duration_ms: 5.624
00:22:14   severity: fail
00:22:14   exitcode: 1
00:22:14   stack: |-
00:22:14     permutation 118 failed
00:22:14     {
00:22:14       config: {
00:22:14         onError: 'exit',
00:22:14         shouldSucceed: true,
00:22:14         entryPath: './worker-spawner.cjs',
00:22:14         willDeletePolicy: true,
00:22:14         preloads: [],
00:22:14         parentPath: './parent.js',
00:22:14         depPath: './dep.js',
00:22:14         resources: {
00:22:14           './dep.js': {
00:22:14             body: '',
00:22:14             integrities: [
00:22:14               'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
00:22:14               'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='
00:22:14             ]
00:22:14           },
00:22:14           './parent.js': {
00:22:14             body: '\n' +
00:22:14               '    if (!process.env.DEP_FILE) {\n' +
00:22:14               '      console.error(\n' +
00:22:14               "        'missing required DEP_FILE env to determine dependency'\n" +
00:22:14               '      );\n' +
00:22:14               '      process.exit(33);\n' +
00:22:14               '    }\n' +
00:22:14               '    require(process.env.DEP_FILE)\n' +
00:22:14               '  ',
00:22:14             integrities: [
00:22:14               'sha256-AzbH3oEA96J/+tLSqOJ+LaIt9vdOY+ikShkvKm0VutU=',
00:22:14               'sha256-R0QVSw9BUI7znwKNvt6BuzkLozhYKTtFNRSENIL5H7k='
00:22:14             ]
00:22:14           },
00:22:14           './worker-spawner.cjs': {
00:22:14             body: '\n' +
00:22:14               "  const path = require('path');\n" +
00:22:14               "  const { Worker } = require('worker_threads');\n" +
00:22:14               '  if (!process.env.PARENT_FILE) {\n' +
00:22:14               '    console.error(\n' +
00:22:14               "      'missing required PARENT_FILE env to determine worker entry point'\n" +
00:22:14               '    );\n' +
00:22:14               '    process.exit(33);\n' +
00:22:14               '  }\n' +
00:22:14               '  if (!process.env.DELETABLE_POLICY_FILE) {\n' +
00:22:14               '    console.error(\n' +
00:22:14               "      'missing required DELETABLE_POLICY_FILE env to check reloading'\n" +
00:22:14               '    );\n' +
00:22:14               '    process.exit(33);\n' +
00:22:14               '  }\n' +
00:22:14               '  const w = new Worker(path.resolve(process.env.PARENT_FILE));\n' +
00:22:14               "  w.on('exit', (status) => process.exit(status === 0 ? 0 : 1));\n",
00:22:14             integrities: [
00:22:14               'sha256-23P4m8edtJAPTH24RJHIHMEuMVgP7QKcgWBSyj6FxPc=',
00:22:14               'sha256-XTRkNC3vxo+zvjv7CD8iapeqc7npNN7TwqRR0TOv5lg='
00:22:14             ]
00:22:14           }
00:22:14         }
00:22:14       },
00:22:14       manifest: {
00:22:14         onerror: 'exit',
00:22:14         resources: {
00:22:14           'file:///home/iojs/node-tmp/.tmp.2974/test-policy-integrity-permutation-118/dep.js': {
00:22:14             integrity: 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
00:22:14             dependencies: true
00:22:14           },
00:22:14           'file:///home/iojs/node-tmp/.tmp.2974/test-policy-integrity-permutation-118/parent.js': {
00:22:14             integrity: 'sha256-AzbH3oEA96J/+tLSqOJ+LaIt9vdOY+ikShkvKm0VutU= sha256-R0QVSw9BUI7znwKNvt6BuzkLozhYKTtFNRSENIL5H7k=',
00:22:14             dependencies: true
00:22:14           },
00:22:14           'file:///home/iojs/node-tmp/.tmp.2974/test-policy-integrity-permutation-118/worker-spawner.cjs': {
00:22:14             integrity: 'sha256-23P4m8edtJAPTH24RJHIHMEuMVgP7QKcgWBSyj6FxPc= sha256-XTRkNC3vxo+zvjv7CD8iapeqc7npNN7TwqRR0TOv5lg=',
00:22:14             dependencies: true
00:22:14           }
00:22:14         }
00:22:14       }
00:22:14     }
00:22:14     exit code: 1 signal: null
00:22:14     stdout: 
00:22:14     stderr: 
00:22:14     /home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc8/test/pummel/test-policy-integrity.js:187
00:22:14             throw e;
00:22:14             ^
00:22:14     
00:22:14     AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:22:14     
00:22:14     1 !== 0
00:22:14     
00:22:14         at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc8/test/pummel/test-policy-integrity.js:170:18)
00:22:14         at ChildProcess.emit (node:events:369:20)
00:22:14         at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12) {
00:22:14       generatedMessage: true,
00:22:14       code: 'ERR_ASSERTION',
00:22:14       actual: 1,
00:22:14       expected: 0,
00:22:14       operator: 'strictEqual'
00:22:14     }
00:22:14   ...

https://ci.nodejs.org/job/node-test-commit-arm/36974/nodes=centos7-arm64-gcc8/console

[bmeck]

making PR to dump stderr, same permutation number is suspicious

[Trott]

making PR to dump stderr, same permutation number is suspicious

Did that PR land? Here's another one that happened just now:

https://ci.nodejs.org/job/node-test-commit-arm/37067/nodes=centos7-arm64-gcc8/console

08:31:37 not ok 2984 pummel/test-policy-integrity
08:31:43   ---
08:31:43   duration_ms: 5.627
08:31:43   severity: fail
08:31:43   exitcode: 1
08:31:43   stack: |-
08:31:43     permutation 118 failed
08:31:43     {
08:31:43       config: {
08:31:43         onError: 'exit',
08:31:43         shouldSucceed: true,
08:31:43         entryPath: './worker-spawner.cjs',
08:31:43         willDeletePolicy: true,
08:31:43         preloads: [],
08:31:43         parentPath: './parent.js',
08:31:43         depPath: './dep.js',
08:31:43         resources: {
08:31:43           './dep.js': {
08:31:43             body: '',
08:31:43             integrities: [
08:31:43               'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
08:31:43               'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='
08:31:43             ]
08:31:43           },
08:31:43           './parent.js': {
08:31:43             body: '\n' +
08:31:43               '    if (!process.env.DEP_FILE) {\n' +
08:31:43               '      console.error(\n' +
08:31:43               "        'missing required DEP_FILE env to determine dependency'\n" +
08:31:43               '      );\n' +
08:31:43               '      process.exit(33);\n' +
08:31:43               '    }\n' +
08:31:43               '    require(process.env.DEP_FILE)\n' +
08:31:43               '  ',
08:31:43             integrities: [
08:31:43               'sha256-AzbH3oEA96J/+tLSqOJ+LaIt9vdOY+ikShkvKm0VutU=',
08:31:43               'sha256-R0QVSw9BUI7znwKNvt6BuzkLozhYKTtFNRSENIL5H7k='
08:31:43             ]
08:31:43           },
08:31:43           './worker-spawner.cjs': {
08:31:43             body: '\n' +
08:31:43               "  const path = require('path');\n" +
08:31:43               "  const { Worker } = require('worker_threads');\n" +
08:31:43               '  if (!process.env.PARENT_FILE) {\n' +
08:31:43               '    console.error(\n' +
08:31:43               "      'missing required PARENT_FILE env to determine worker entry point'\n" +
08:31:43               '    );\n' +
08:31:43               '    process.exit(33);\n' +
08:31:43               '  }\n' +
08:31:43               '  if (!process.env.DELETABLE_POLICY_FILE) {\n' +
08:31:43               '    console.error(\n' +
08:31:43               "      'missing required DELETABLE_POLICY_FILE env to check reloading'\n" +
08:31:43               '    );\n' +
08:31:43               '    process.exit(33);\n' +
08:31:43               '  }\n' +
08:31:43               '  const w = new Worker(path.resolve(process.env.PARENT_FILE));\n' +
08:31:43               "  w.on('exit', (status) => process.exit(status === 0 ? 0 : 1));\n",
08:31:43             integrities: [
08:31:43               'sha256-23P4m8edtJAPTH24RJHIHMEuMVgP7QKcgWBSyj6FxPc=',
08:31:43               'sha256-XTRkNC3vxo+zvjv7CD8iapeqc7npNN7TwqRR0TOv5lg='
08:31:43             ]
08:31:43           }
08:31:43         }
08:31:43       },
08:31:43       manifest: {
08:31:43         onerror: 'exit',
08:31:43         resources: {
08:31:43           'file:///home/iojs/node-tmp/.tmp.2983/test-policy-integrity-permutation-118/dep.js': {
08:31:43             integrity: 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=',
08:31:43             dependencies: true
08:31:43           },
08:31:43           'file:///home/iojs/node-tmp/.tmp.2983/test-policy-integrity-permutation-118/parent.js': {
08:31:43             integrity: 'sha256-AzbH3oEA96J/+tLSqOJ+LaIt9vdOY+ikShkvKm0VutU= sha256-R0QVSw9BUI7znwKNvt6BuzkLozhYKTtFNRSENIL5H7k=',
08:31:43             dependencies: true
08:31:43           },
08:31:43           'file:///home/iojs/node-tmp/.tmp.2983/test-policy-integrity-permutation-118/worker-spawner.cjs': {
08:31:43             integrity: 'sha256-23P4m8edtJAPTH24RJHIHMEuMVgP7QKcgWBSyj6FxPc= sha256-XTRkNC3vxo+zvjv7CD8iapeqc7npNN7TwqRR0TOv5lg=',
08:31:43             dependencies: true
08:31:43           }
08:31:43         }
08:31:43       }
08:31:43     }
08:31:43     exit code: 1 signal: null
08:31:43     stdout: 
08:31:43     stderr: 
08:31:43     /home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc8/test/pummel/test-policy-integrity.js:187
08:31:43             throw e;
08:31:43             ^
08:31:43     
08:31:43     AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
08:31:43     
08:31:43     1 !== 0
08:31:43     
08:31:43         at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc8/test/pummel/test-policy-integrity.js:170:18)
08:31:43         at ChildProcess.emit (node:events:369:20)
08:31:43         at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12) {
08:31:43       generatedMessage: true,
08:31:43       code: 'ERR_ASSERTION',
08:31:43       actual: 1,
08:31:43       expected: 0,
08:31:43       operator: 'strictEqual'
08:31:43     }
08:31:43   ...

[bmeck]

@Trott local repro shows stderr being logged, got stuff in work stuff this week and cannot find a local repro so it is taking a while. The stuff I have is on #38278 but it should basically act like how worker_threads do normally as documented (though variety of docs seem a bit unclear on timing of events).

[richardlau]

AFAICT the test is failing on test-packetnet-centos7-arm64-2 (I haven't found an instance of the failure on test-packetnet-centos7-arm64-1):

• https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc8/buildTimeTrend
• https://ci.nodejs.org/computer/test-packetnet-centos7-arm64-2/builds

I've logged into test-packetnet-centos7-arm64-1 and test-packetnet-centos7-arm64-2 and it looks like -2 has a package.json file (from 11 Jan 2019) in /home/iojs/node-tmp -- I think I read somewhere that a package.json in a parent directory could affect the policy tests? (Edit: #35600). In any case I've deleted /home/iojs/node-tmp/package.json (and /home/iojs/node-tmp/dep.js), FWIW this is what was in those files:

[iojs@test-packetnet-centos7-arm64-2 node-tmp]$ cat dep.js 
[iojs@test-packetnet-centos7-arm64-2 node-tmp]$ cat package.json 
{"main": "dep.js"}[iojs@test-packetnet-centos7-arm64-2 node-tmp]$

[richardlau]

Test build on master ontest-packetnet-centos7-arm64-2: https://ci.nodejs.org/job/node-test-commit-arm/37084/nodes=centos7-arm64-gcc8/

[richardlau]

Although #35633 looks like it should already have caught the case of a package.json in a parent directory?

[Trott]

Although #35633 looks like it should already have caught the case of a package.json in a parent directory?

Maybe a symlink or mount point somehow results in a difference in the way parent directories are detected?

[richardlau]

After further investigation it looks like the existing check is done for directories above the directory containing the test, but the test is actually staging in the test's tmpdir. Opened #38285 to also check ancestors of the tmpdir for this test.