ARM64-SVE: Allow SVE ops to re-use the same registers #107084

a74nh · 2024-08-28T13:42:20Z

Fixes #106866

RMW intrinsics can use the RMW register in other inputs. The only restriction is that those inputs will be overwritten.

Added extra intrinsic testing to cover this. Requires #107036 for all the tests to pass.

a74nh · 2024-08-28T13:52:48Z

@dotnet/arm64-contrib @kunalspathak

a74nh · 2024-08-30T07:07:33Z

Given the discussions over in #107036, I think this PR is correct.

For this case we have:

Generating: N017 (  1,  1) [000033] -----+-----                   t33 =    LCL_VAR   simd16 V07 tmp6         u:2 d0 REG d0 $80
Generating: N019 (  1,  1) [000034] -----+-----                   t34 =    LCL_VAR   simd16 V07 tmp6         u:2 d0 (last use) REG d0 $80
                                                                        /--*  t37    mask
                                                                        +--*  t33    simd16
                                                                        +--*  t34    simd16
Generating: N021 (  6,  6) [000035] -----+-----                   t35 = *  HWINTRINSIC simd16 byte Splice REG d0 $240
							V07 in reg d0 is becoming dead  [000034]
							Live regs: 0000000100000000 {d0} - {d0} => 0000000000000000 {}
							Live vars after [000034]: {V07} -{V07} => {}
IN0004:             splice  z0.b, p0, z0.b, z0.b

Like in the previous PR, we have the same value used twice. But here we are not going to use a MOVPRFX (due to it not being wrapped with conditional select). Therefore we have no requirement to delay free the operands, and can simply allow the same one to be used.

kunalspathak

I am not totally convinced that we should be removing all the asserts without seeing a case that hits it. Its ok to put some of them back for now and relax them as we see cases.

kunalspathak · 2024-09-11T22:30:03Z

src/coreclr/jit/hwintrinsiccodegenarm64.cpp

@@ -1106,7 +1106,6 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
                    }
                    else if (isRMW)
                    {
-                        assert((targetReg == op1Reg) || (targetReg != op2Reg));


as we spoke offline, did you verify if we put this assert back, do we hit them. These asserts were added conservatively with the idea that we understand the scenario under which this cannot be true. I am wondering if we should remove these as we see scenarios that trigger them.

Restored this one

src/coreclr/jit/hwintrinsiccodegenarm64.cpp

kunalspathak · 2024-09-11T22:30:19Z

src/coreclr/jit/hwintrinsiccodegenarm64.cpp

@@ -1124,18 +1123,12 @@ void CodeGen::genHWIntrinsic(GenTreeHWIntrinsic* node)
                    {
                        if (HWIntrinsicInfo::IsExplicitMaskedOperation(intrin.id))
                        {
-                            assert((targetReg == op1Reg) || (targetReg != op1Reg));


these are OK to have.

These ones are hit by the new hwintrinsic tests:

------------------- {'JitStressRegs': '1'} ------------------- Errors: Assert failure(PID 366410 [0x0005974a], Thread: 366410 [0x5974a]): Assertion failed '(targetReg == op1Reg) || (targetReg != op3Reg)' in 'JIT.HardwareIntrinsics.Arm._Sve.SimpleTernaryOpTest__Sve_ConditionalExtractAfterLastActiveElementAndReplicate_float:RunSameClassFldScenario():this' during 'Generate code' (IL size 92; hash 0xa42df59e; FullOpts) File: /home/alahay01/dotnet/runtime_sve_api/src/coreclr/jit/hwintrinsiccodegenarm64.cpp:1118 Image: /home/alahay01/dotnet/runtime_sve_api/artifacts/tests/coreclr/linux.arm64.Checked/Tests/Core_Root/corerun

We need some kind of assert validating the mov is valid.

Since we're doing mov targetReg, op2Reg, then when targetReg == op1Reg and op1Reg != op2Reg, then the we would overwrite op1 and produce in incorrect results.

So, therefore, we expect that targetReg == op2Reg so no mov is emitted -or- op1/op2 have been marked delayFree

The assert therefore needs to be something like assert((targetReg == op2Reg) || ((targetReg != op1Reg) && (targetReg != op3Reg)). Which validates no move or no overwrite

All paths need some kind of similar assert validating no move or no overwrite, if a move is emitted

All paths need some kind of similar assert validating no move or no overwrite, if a move is emitted

Agree.

Done replaced this assert as suggested, and the next one with a similar assert

confirmed stress tests passing.

a74nh · 2024-09-13T11:41:32Z

Stress tests working with the latest version

kunalspathak

LGTM

kunalspathak · 2024-09-13T20:52:21Z

/backport to release/9.0

github-actions · 2024-09-13T20:52:33Z

Started backporting to release/9.0: https://github.com/dotnet/runtime/actions/runs/10855965106

github-actions · 2024-09-13T20:54:12Z

@kunalspathak backporting to release/9.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: ARM64-SVE: Allow SVE ops to re-use the same registers
Using index info to reconstruct a base tree...
M	src/coreclr/jit/hwintrinsiccodegenarm64.cpp
Falling back to patching base and 3-way merge...
Auto-merging src/coreclr/jit/hwintrinsiccodegenarm64.cpp
CONFLICT (content): Merge conflict in src/coreclr/jit/hwintrinsiccodegenarm64.cpp
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 ARM64-SVE: Allow SVE ops to re-use the same registers
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

github-actions · 2024-09-13T20:54:12Z

@kunalspathak an error occurred while backporting to release/9.0, please check the run log for details!

Error: git am failed, most likely due to a merge conflict.

* ARM64-SVE: Allow SVE ops to re-use the same registers * Add Sve.IsSupported * restore an assert * better asserts

…107818) * ARM64-SVE: Allow SVE ops to re-use the same registers (#107084) * ARM64-SVE: Allow SVE ops to re-use the same registers * Add Sve.IsSupported * restore an assert * better asserts * resolve merge conflicts --------- Co-authored-by: Alan Hayward <a74nh@users.noreply.github.com> Co-authored-by: Jeff Schwartz <jeffschw@microsoft.com>

* ARM64-SVE: Allow SVE ops to re-use the same registers * Add Sve.IsSupported * restore an assert * better asserts

ARM64-SVE: Allow SVE ops to re-use the same registers

ba1d58a

dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 28, 2024

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Aug 28, 2024

a74nh marked this pull request as ready for review August 28, 2024 13:52

a74nh mentioned this pull request Aug 28, 2024

ARM64-SVE: Delay free all ops within conditional select #107036

Merged

build-analysis bot mentioned this pull request Aug 28, 2024

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Sep 3, 2024

Add Sve.IsSupported

6b64ef8

a74nh force-pushed the reg3_github branch from c944574 to 6b64ef8 Compare September 9, 2024 11:01

a74nh mentioned this pull request Sep 9, 2024

ARM64-SVE: refactor lsra buildHWIntrinsic #107459

Merged

build-analysis bot mentioned this pull request Sep 11, 2024

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

kunalspathak requested changes Sep 11, 2024

View reviewed changes

a74nh added 2 commits September 13, 2024 09:41

Merge main

41ef196

restore an assert

554d139

better asserts

b8b0fb5

kunalspathak approved these changes Sep 13, 2024

View reviewed changes

kunalspathak merged commit cf9c995 into dotnet:main Sep 13, 2024
117 checks passed

kunalspathak pushed a commit to kunalspathak/runtime that referenced this pull request Sep 13, 2024

ARM64-SVE: Allow SVE ops to re-use the same registers (dotnet#107084)

0ac6b99

* ARM64-SVE: Allow SVE ops to re-use the same registers * Add Sve.IsSupported * restore an assert * better asserts

kunalspathak mentioned this pull request Sep 13, 2024

[release/9.0]: ARM64-SVE: Allow SVE ops to re-use the same registers #107818

Merged

4 tasks

a74nh deleted the reg3_github branch September 16, 2024 08:39

jtschuster pushed a commit to jtschuster/runtime that referenced this pull request Sep 17, 2024

ARM64-SVE: Allow SVE ops to re-use the same registers (dotnet#107084)

c039377

* ARM64-SVE: Allow SVE ops to re-use the same registers * Add Sve.IsSupported * restore an assert * better asserts

sirntar pushed a commit to sirntar/runtime that referenced this pull request Sep 30, 2024

ARM64-SVE: Allow SVE ops to re-use the same registers (dotnet#107084)

412ac21

* ARM64-SVE: Allow SVE ops to re-use the same registers * Add Sve.IsSupported * restore an assert * better asserts

github-actions bot locked and limited conversation to collaborators Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARM64-SVE: Allow SVE ops to re-use the same registers #107084

ARM64-SVE: Allow SVE ops to re-use the same registers #107084

a74nh commented Aug 28, 2024

a74nh commented Aug 28, 2024

a74nh commented Aug 30, 2024

kunalspathak left a comment

kunalspathak Sep 11, 2024

a74nh Sep 13, 2024

kunalspathak Sep 11, 2024

a74nh Sep 13, 2024

tannergooding Sep 13, 2024 •

edited

Loading

tannergooding Sep 13, 2024

kunalspathak Sep 13, 2024

a74nh Sep 13, 2024

a74nh Sep 13, 2024

a74nh commented Sep 13, 2024

kunalspathak left a comment

kunalspathak commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

ARM64-SVE: Allow SVE ops to re-use the same registers #107084

ARM64-SVE: Allow SVE ops to re-use the same registers #107084

Conversation

a74nh commented Aug 28, 2024

a74nh commented Aug 28, 2024

a74nh commented Aug 30, 2024

kunalspathak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tannergooding Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

a74nh commented Sep 13, 2024

kunalspathak left a comment

Choose a reason for hiding this comment

kunalspathak commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

tannergooding Sep 13, 2024 •

edited

Loading