The performance gap between node16 and node21 changes as the n of the benchmark changes

[Septa2112]

Version

node 16.x and node 21.1

Platform

Linux wpe-icx 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

I tested node-benchmark es/string-concatenations.js with node 21.1 and node 16.x.

When n is the default 1e3, the score difference between the two version reaches 50%.

But when n is set to 5000 or 10000 or other larger numbers, there is no performance gap between the two versions.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

Expected

• When I adjust n to 5e3 or 1e4, the score difference between the two versions should remain around 50%.

Reason

• I think the number of n has not changed much, and the performance gap between node 16.x and node 21.1 should not change so much too.

What do you see instead?

• n = 1e3 (default number)

NODE 16.x  es/string-concatenations.js mode="multi-join" n=1000: 2,275,198.9661495895
NODE 21.1  es/string-concatenations.js mode="multi-join" n=1000: 975,159.7555469525

• n = 5e3

NODE 16.x  es/string-concatenations.js mode="multi-join" n=5000: 1,972,248.1025001006
NODE 21.1  es/string-concatenations.js mode="multi-join" n=5000: 1,983,125.977681107

• n = 1e4

NODE 16.x  es/string-concatenations.js mode="multi-join" n=10000: 2,678,989.303063612
NODE 21.1  es/string-concatenations.js mode="multi-join" n=10000: 2,739,590.924283186

Additional information

I'm not sure whether this problem can be counted as an issue. Or is this a pre-existing problem?

[H4ad]

Probably is just the V8 JIT kick-in and the start of the optimization in that code, maybe the behavior had changed and now they didn't optimize aggressively initially and only started the optimization after more iterations than before.

But this can have a lot of reasons, all of them are related to V8 than Node.js itself.

But you found an interesting case that we can increase the number of iterations to be more precise, are you willing to create a PR to increase that number? Just enough that still be fast but avoid this behavior.

[Septa2112]

Thanks. I have submitted a PR #50585 to increase the number of iterations in benchmark/es/string-concatenations.js.

I plan to test other benchmarks later to see if similar situations exist. If so, I will modify all and create a unified PR.

I wonder if this is acceptable?

[H4ad]

@Septa2112 It would be highly appreciated if you did that!

I suggest you create a commit for each benchmark change.

[Septa2112]

BTW, I tried to analyze the cause of this phenomenon based on your tips. And I run the benchmark with --print-opt-code to see if node generated opt code.

I found that when iterations is default 1e3, neither node-16.x nor node-21.1 generated opt code.

When n increases to 1e5 or larger, I can get the opt code of the two versions of node. Just like:

......
--- Optimized code ---
optimization_id = 0
source_position = 321
kind = TURBOFAN
name = main
stack_slots = 26
compiler = turbofan
address = 0x7f141c054541

Instructions (size = 1228)
0x7f141c0545a0     0  488b59c0           REX.W movq rbx,[rcx-0x40]
0x7f141c0545a4     4  f6430f01           testb [rbx+0xf],0x1
......

Does this mean that the performance gap between node16 and node21 is not a problem caused by the inactive v8 JIT optimization because neither of them generated opt code when iterations is 1e3?

Please correct me if I say something wrong.

[H4ad]

@Septa2112 You are right, at least on 1e3, the slowdown is not caused by V8 JIT.

I used the following code:

let string;

const n = 1e3;
const str = 'abc';
const num = 123;

const start = performance.now();

for (let i = 0; i < n; i++)
  string = ['...', str, ', ', num, ', ', str, ', ', num, '.'].join('');

const end = performance.now();

console.log(`Diff: ${end - start}ms`);
console.log(string);

For 1e3:

• 16.20.1: 0.2531019998714328ms
• 21.1.0: 0.24994500004686415ms

For `1e6:

• 16.20.1: 130.60445600003004ms
• 21.1.0: 123.54395300010219ms

Using 1e6 and --print-opt-code, we have the following comparison:

• 16.20.1: Instructions (size = 1252)
• 21.1.0: Instructions (size = 1112)

Is slower without optimizations but is faster faster the optimizations.

[Septa2112]

@H4ad Thanks for your explanation!

Now I find that some other cases may have similar problems. Should I open a new issue? Or update these cases directly in the current issue and create new PRs directly?

[H4ad]

Create new PRs and just reference this issue as the starting point to looking for those fixes.