[RISCV] Unexpected Latency Pattern for Vector Integer Instructions on Banana Pi BPI-F3 (SpacemiT X60)

[mikhailramalho]

While collecting latency data for integer vector instructions on the Banana Pi BPI-F3 (SpacemiT X60, RVV 1.0), I noticed an unusual pattern I couldn’t explain.

For most instructions, latency scales linearly with LMUL. However, the following instructions consistently exhibit a 1-cycle increase in latency only when LMUL = m4:

• vadd.vi, vadd.vv, vadd.vx
• vmacc.vv, vmacc.vx, vmadd.vv, vmadd.vx
• vmaxu.vv, vmaxu.vx, vmax.vv, vmax.vx
• vminu.vv, vminu.vx, vmin.vv, vmin.vx
• vmulhsu.vv, vmulhsu.vx, vmulhu.vv, vmulhu.vx
• vmulh.vv, vmulh.vx, vmul.vv, vmul.vx
• vnmsac.vv, vnmsac.vx, vnmsub.vv, vnmsub.vx
• vrsub.vi, vrsub.vx, vsub.vv, vsub.vx

This behavior is consistent and reproducible. All other LMUL values behave as expected, and the effect does not depend on SEW.

I’ve uploaded the full latency dataset here for reference:
https://docs.google.com/spreadsheets/d/1u2LF8Uux0BS2_U9zJsG6DE1zUsoPxBd_zH31ze1313o/edit?gid=0#gid=0

If anyone has insight into what might be causing this, especially whether it’s a known hardware characteristic or an issue in measurement, I’d appreciate any thoughts.

Tagging @zqb-all in case you have any thoughts or documentation that might help clarify this.

[llvmbot]

@llvm/issue-subscribers-backend-risc-v

Author: Mikhail R. Gadelha (mikhailramalho)

While collecting latency data for integer vector instructions on the Banana Pi BPI-F3 (SpacemiT X60, RVV 1.0), I noticed an unusual pattern I couldn’t explain.

For most instructions, latency scales linearly with LMUL. However, the following instructions consistently exhibit a 1-cycle increase in latency only when LMUL = m4:

• vadd.vi, vadd.vv, vadd.vx
• vmacc.vv, vmacc.vx, vmadd.vv, vmadd.vx
• vmaxu.vv, vmaxu.vx, vmax.vv, vmax.vx
• vminu.vv, vminu.vx, vmin.vv, vmin.vx
• vmulhsu.vv, vmulhsu.vx, vmulhu.vv, vmulhu.vx
• vmulh.vv, vmulh.vx, vmul.vv, vmul.vx
• vnmsac.vv, vnmsac.vx, vnmsub.vv, vnmsub.vx
• vrsub.vi, vrsub.vx, vsub.vv, vsub.vx

This behavior is consistent and reproducible. All other LMUL values behave as expected, and the effect does not depend on SEW.

I’ve uploaded the full latency dataset here for reference:
https://docs.google.com/spreadsheets/d/1u2LF8Uux0BS2_U9zJsG6DE1zUsoPxBd_zH31ze1313o/edit?gid=0#gid=0

If anyone has insight into what might be causing this, especially whether it’s a known hardware characteristic or an issue in measurement, I’d appreciate any thoughts.

Tagging @zqb-all in case you have any thoughts or documentation that might help clarify this.

[zqb-all]

Wow, detailed data. We will analyze this 1-cycle increase in latency.

[preames]

Wow, detailed data. We will analyze this 1-cycle increase in latency.

For clarity, the goal here is simply making sure we understand the micro-architecture at a rough level. If there's a latency quirk you don't want to explain details on, that's fine. A rough summary of the execution pipe structure that mostly matches the empirical observations is sufficient for the needs of the schedule patch.

[zqb-all]

Take vadd.vx as an example to explain this.
For consecutive vadd.vx instructions, the latency of RAW is 3, while WAW and Bank Conflict make the latency longer, and these also affect the utilization of dual pipe.

For

loop {
vadd.vx v0, v0, a0
vadd.vx v0, v0, a0
}

Both RAW and WAW exist, pipe is

       lmul = 2                    lmul = 4 
     pipe0    pipe1           pipe0    pipe1  
1   inst0-0                  inst0-0
2   inst0-1                  inst0-1
3   inst0-2                  inst0-2
4   inst0-3                  inst0-3
5            inst1-0         inst0-4  
6            inst1-1         inst0-5  inst1-0
7            inst1-2         inst0-6  inst1-1
8            inst1-3         inst0-7  inst1-2
9   inst2-0                           inst1-3
10  inst2-1                           inst1-4
11  inst2-2                  inst2-0  inst1-5
12  inst2-3                  inst2-1  inst1-6
13           inst3-0         inst2-2  inst1-6
14           inst3-1         inst2-3 
15           inst3-2         inst2-4
16           inst3-3         inst2-5  inst3-0
17  ...                      ...      ...
     latency=4cycle            latency=5cycle

As shown above,
When lmul<=2, the main limitation is RAW + WAW and cannot benefit from dual pipe at all, latency=4.
When lmul=4, it can benefit from dual pipe, but there are bubbles , so latency=5.
In addition,
When lmul=8, there are enough split instructions so that WAW is no longer a bottleneck, no bubble in pipe, latency=8.

If no WAW, only RAW:

loop {
vadd.vx v0, v8, a0
vadd.vx v8, v0, a0
}

pipe should be:

       lmul = 2                    lmul = 4
     pipe0    pipe1           pipe0    pipe1  
1   inst0-0                  inst0-0
2   inst0-1                  inst0-1
3   inst0-2                  inst0-2
4   inst0-3  inst1-0         inst0-3  inst1-0
5            inst1-1         inst0-4  inst1-1
6            inst1-2         inst0-5  inst1-2
7   inst2-0  inst1-3         inst0-6  inst1-3
8   inst2-1                  inst0-7  inst1-4
9   inst2-2                  inst2-0  inst1-5
10  inst2-3  inst3-0         inst2-1  inst1-6
11           inst3-1         inst2-2  inst1-7
12           inst3-2         inst2-3  inst3-0
13  inst4-0  inst3-3         inst2-4  inst3-1
14  ...                      inst2-5  inst3-2
15                           inst2-6  inst3-3
16                           inst2-7  inst3-4
17                           inst4-0  inst3-5  
18                           ...      ...
      latency=3cycle           latency=4cycle 

As shown above,
When lmul=2, the main limitation is RAW, slightly benefit from dual pipe, latency=3.
When lmul=4, no bubble in pipe, latency=4.
When lmul=8, no bubble in pipe, latency=8.

@mikhailramalho Hope this can help you understand the current measurement results. And thank you again for your efforts in improving the performance of X60.

[mikhailramalho]

Thanks for the detailed explanation, @zqb-all!

I re-tested the instructions with no WAW, and added the results to the spreadsheet under the tab "instructions + no WAW".

I can confirm that the latency when lmul=4 is 4 with no WAW, but I also saw the latency of other instruction went down to 3, for lmul mf2, mf4, mf8, m1 and m2.

@preames WDYT? Should I proceed by using the latencies with no WAW on the scheduling model?