[RISC-V] Update SpacemiT X60 vector scheduling model with measured latencies

[mikhailramalho]

Updates the SpacemiT X60 scheduling model with actual latencies measured on BPi-F3 hardware.

tl;dr: execution time is neutral on SPEC after this patch. There is a code size regression described in issue #146407

Changes:

• Added 10 new latency classes
• Updated latencies for ~30 instruction categories based on hardware measurements

Completed:

• Basic integer ALU, min/max, saturating/averaging arithmetic
• Carry operations, mask operations, comparisons
• Integer/FP division (split simple/complex based on LMUL)
• Widening operations
• FP operations including add/sub, mul, FMA
• FP conversions (widening/narrowing)
• FP reductions including vfredmax/min/usum (fixed fractional LMUL latencies)
• FP ordered reductions vfredosum (split simple/complex)
• FP widening reductions vfwredosum/vfwredusum (split simple/complex)
• Integer reductions
• Mask manipulation operations
• Permutation operations (gather/compress/slide)
• Narrowing shifts and clips (split simple/complex)

Missing:

• All vector loads/stores uops are missing their actual latency values. The values in this PR are estimations, while I'm still collecting the real numbers

Performance Impact:

• https://lnt.lukelau.me/db_default/v4/nts/674?compare_to=673
• This change is mostly NFC
• The two benchmarks with improvement/regression on execution time are known to be noisy
• There is an increase in code size in two benchmarks: reviewing the code changes, we see a lot more vector load/store instructions

Known Issues:

• Code size regression on two SPEC benchmarks
• Some operations grouped use worst-case latency
• TableGen !cond expressions not working as expected for vector single-width FMA instructions
• All compromises I've made are documented as TODO in the code

Planned follow-up PRs:

• Debug the code size regressions
• Add latencies for vector loads/stores
• Work on the TODOs introduced by this PR. It should require splitting some WriteRes groups and changing other scheduling models, but it should be NFC for them

[preames]

Is there actually a separate VIEU? Or is this a single int and float unit?

[mikhailramalho]

We had the same question, from the C908 manual, this is the VFPU section:

2.2.3 VFPU

FPUs include the floating-point arithmetic logic unit (FALU), floating-point fused multiply-add unit (FMAU), and floating-point divide and square root unit (FDSU). They support half-precision and single-precision operations. The FALU performs operations such as addition, subtraction, comparison, conversion,
register data transmission, sign injection, and classification.

The FMAU performs operations such as common multiplication and fused multiply-add operations. The FDSU performs operations such as floating-point division and square root operations. The vector execution unit is developed by extending the floating-point unit. On the basis of the original scalar floating-point computation, floating-point units can be extended to vector floating-point units Vector floating-point units include the vector floating-point arithmetic logic unit (VFALU), vector floating-point fused multiply-add unit (VFMAU), and vector floating-point divide and square root unit (VFDSU).

Vector floating-point units support vector floating-point computation of different bits. In addition, vector integer units are added. Vector integer units include the vector arithmetic logic unit (VALU), vector shift unit (VSHIFT), vector multiplication unit (VMUL), vector division unit (VDIVU), vector permutation unit (VPERM), vector reduction unit (VREDU), and vector logical operation unit (VMISC).

Bolded part by me. We assumed that it's saying it has both floating point and integer units, rather than that it has FP units that include integer.

@zqb-all, could you help us clarify this?

[zqb-all]

Yes, the x60 has separate VIEU

[preames]

The split here between simple and complex seems mildly complex. This looks like this might actually be NumDLEN * 12?

[preames]

Another batch of mostly stylistic comments, and a few cases where the code structure is mildly suspicious and should be checked against the latency data.

[preames]

Observation triggered by something you said offline.

These look a lot like a dual port, pipelined design with the unit being released after the instruction is complete.

That is, Latency = 4, ReleaseAtCycle =

8 isn't an exact match - we'd expect 7, but it's awfully close. A number of your other sets look like single or double ported pipelined variants too.

[mikhailramalho]

Maybe we can use the data from camel-cdr to set the ReleaseAtCycle for all instructions?

[preames]

This code doesn't appear to match the comment just above.

[mikhailramalho]

The problem here is that the comment is a bit confusing; I'll improve it. These are the latencies for:

• vfwsub/vfwadd.vv, vfwsub/vfwadd.vf: e16mf4=4, e16mf2=4, e16m1=4, e16m2=5, e16m4=8, e32mf2=4, e32m1=4, e32m2=5, e32m4=8
• vfwsub/vfwadd.wv, vfwsub/vfwadd.wf: e16mf4=5, e16mf2=5, e16m1=5, e16m2=9, e16m4=17, e32mf2=5, e32m1=5, e32m2=9, e32m4=17

SMX60GetLMulCycles returns the following: MF4=1, MF2=1, M1=1, M2=2, M=4, which when multiplied by 4, gives us: MF4=4, MF2=4, M1=4, M2=8, M4=16.

vfwsub/vfwadd.vv, vfwsub/vfwadd.vf have greater latency than measured on higher LMUL, unfortunately, because they are all grouped together.

[preames]

Stylistic idea - rather than giving your helper functions names, maybe just do Get6678 and variants? The numbers seem to actually be the unique bits, and your comments approximate.

Another option - have a table lookup helper, and embed the interesting part (the m1 and above) as an array in the callsite? You repeat the numbers in the comment anyways.

[preames]

Is this actually the same for all SEW at fractional LMUL?

[mikhailramalho]

It isn't:

• e16mf4 = 12
• e16mf2 = 24
• e32mf2 = 12

This code generates:

• e16mf4 = 12
• e16mf2 = 24
• e32mf2 = 24 (increased latency)

[preames]

As written, this would seem to say that an m1 is cheaper than an mf2 at SEW64. That seems suspect?

This looks like it might be an unpipelined DLEN unit w/SEW sensitive latency?

[mikhailramalho]

I double-checked the data and re-ran the experiments, but got the same result: https://docs.google.com/spreadsheets/d/1u2LF8Uux0BS2_U9zJsG6DE1zUsoPxBd_zH31ze1313o/edit?gid=2036469797#gid=2036469797&range=1210:1224

[mshockwave]

Personally I kind of agree that we can make multiplication's SchedWrite SEW-dependant