[API Proposal]: Example usages of a VectorSVE API

[a74nh]

Background and motivation

Adding a vector API for Arm SVE/SVE2 would be useful. SVE is a mandatory feature in Arm 9.0 onwards and is an alternative to NEON. Code written in SVE is vector length agnostic and will automatically scale to the vector length of the machine it is running on, and therefore will only require a single implementation per routine. Use of predication in SVE enables loop heads and tails to be skipped, making code shorter, simpler and easier to write.

This issue provides examples of how such an API might be used.

API Proposal

None provided.

API Usage

  /*
    Sum all the values in an int array.
  */
  public static unsafe int sum_sve(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);

    /*
      WhileLessThan comes in two variants:
        VectorSVEPred WhileLessThan(int val, int limit)
        VectorSVEComparison WhileLessThan(VectorSVEPred out predicate, int val, int limit)

      A VectorSVEComparison can be tested using the SVE condition codes (none, any, last, nlast etc).
      `if (cmp.nlast) ....`
      `if (Sve.WhileLessThan(out pred, i, length).first) ....`

      `if (cmp)` is the same as doing `if (cmp.any)`

      Ideally the following will not be allowed:
        auto f = Sve.WhileLessThan(out pred, i, length).first
    */

    /*
      Always using a function call for the vector length instead of assigning to a variable will allow
      easier optimisation to INCW (which is faster than incrementing by a variable).
    */

    for (int i = 0; Sve.WhileLessThan(out pred, i, length); i += Sve.VectorLength<int>())
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);

      /*
        This is the standard sve `add` instruction which uses a merge predicate.
        For each lane in the predicate, add the two vectors. For all other lanes use the first vector.
       */
      total = Sve.MergeAdd(pred, total, vec);
    }

    // No tail call required.
    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Sum all the values in an int array, without predication.
    For performance reasons, it may be better to use an unpredicated loop, followed by a tail.
    Ideally, the user would write the predicated version and the Jit would optimise to this if required.
  */
  public static unsafe int sum_sve_unpredicated_loop(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;

    int i = 0;
    for (i = 0; i+Sve.VectorLength<int>() <= length; i+= Sve.VectorLength<int>() )
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(ref *src, i);
      total = Sve.Add(total, vec);
    }

    // Predicated tail.
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);
    total = Sve.MergeAdd(pred, vec, total);

    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the non zero elements in an int array.
  */
  public static unsafe int CountNonZero_sve(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVEPred true_pred = Sve.CreatePred(true);

    for (int i = 0; Sve.WhileLessThan(out pred, i, length); i += Sve.VectorLength<int>())
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);
      VectorSVEPred cmp_res = Sve.CompareGreaterThan(pred, vec, 0);

      total = Sve.MergeAdd(cmp_res, total, vec);
    }

    // No tail call required.
    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the non zero elements in an int array, without predication.
  */
  public static unsafe int CountNonZero_sve_unpredicated_loop(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVEPred true_pred = Sve.CreatePred(true);

    // Comparisons require predicates. Therefore for a truely non predicated version, use Neon.
    int vector_length = 16/sizeof(int);
    for (int i = 0; i+vector_length <= length; i+=vector_length)
    {
      Vector128<int> vec = AdvSimd.LoadVector128(src);
      Vector128<int> gt = AdvSimd.CompareGreaterThan(vec, zero);
      Vector128<int> bits = AdvSimd.And(gt, one);

      total = AdvSimd.Add(bits, total);
      src += vector_length;
    }

    // Predicated tail.
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src);
    VectorSVEPred cmp_res = Sve.CompareGreaterThan(pred, vec, 0);
    total = Sve.MergeAdd(cmp_res, total, vec);

    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the elements in a null terminated array of unknown size.
  */
  public static unsafe int CountLength_sve(ref int* srcBytes)
  {
    int* src = srcBytes;
    VectorSVEPred pred = Sve.CreatePred(true);
    int ret = 0;

    while (true)
    {
      VectorSVE<int> vec = Sve.LoadUnsafeUntilFault(pred, ref *src); // LD1FF

      /*
        Reading the fault predicate via RDFFRS will also set the condition flags:
          VectorSVEComparison GetFaultPredicate(VectorSVEPred out fault, VectorSVEPred pred)
       */
      VectorSVEPred fault_pred;

      if (Sve.GetFaultPredicate(out fault_pred, pred).last)
      {
        // Last element is set in fault_pred, therefore the load did not fault.

        /*
          Like WhileLessThan, comparisons come in two variants:
            VectorSVEPred CompareEquals(VectorSVEPred pred, VectorSVE a, VectorSVE b)
            VectorSVEComparison CompareEquals(VectorSVEPred out cmp_result, VectorSVEPred pred, VectorSVE a, VectorSVE b)
         */

        // Look for any zeros across the entire vector.
        VectorSVEPred cmp_zero;
        if (Sve.CompareEquals(out cmp_zero, pred, vec, 0).none)
        {
          // No zeroes found. Continue loop.
          ret += Sve.VectorLength<int>();
        }
        else
        {
          // Zero found. Count up to it and return.
          VectorSVEPred matches = Sve.PredFillUpToFirstMatch(pred, cmp_zero); // BRKB
          ret += Sve.PredCountTrue(matches); // INCP
          return ret;
        }
      }
      else
      {
        // Load caused a fault.

        // Look for any zeros across the vector up until the fault.
        VectorSVEPred cmp_zero;
        if (Sve.CompareEquals(out cmp_zero, fault_pred, vec, 0).none)
        {
          // No zeroes found. Clear faulting predicate and continue loop.
          ret += Sve.PredCountTrue(fault_pred); // INCP
          Sve.ClearFaultPredicate(); // SETFFR
        }
        else
        {
          // Zero found. Count up to it and return.
          VectorSVEPred matches = Sve.PredFillUpToFirstMatch(pred, cmp_zero); // BRKB
          ret += Sve.PredCountTrue(matches); // INCP
          return ret;
        }
      }
    }
  }

Alternative Designs

No response

Risks

References

SVE Programming Examples
A64 -- SVE Instructions (alphabetic order)

No response

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Adding a vector API for Arm SVE/SVE2 would be useful. SVE is a mandatory feature in Arm 9.0 onwards and is an alternative to NEON. Code written in SVE is vector length agnostic and will automatically scale to the vector length of the machine it is running on, and therefore will only require a single implementation per routine. Use of predication in SVE enables loop heads and tails to be skipped, making code shorter, simpler and easier to write.

This issue provides examples of how such an API might be used.

API Proposal

None provided.

API Usage

  /*
    Sum all the values in an int array.
  */
  public static unsafe int sum_sve(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);

    /*
      WhileLessThan comes in two variants:
        VectorSVEPred WhileLessThan(int val, int limit)
        VectorSVEComparison WhileLessThan(VectorSVEPred out predicate, int val, int limit)

      A VectorSVEComparison can be tested using the SVE condition codes (none, any, last, nlast etc).
      `if (cmp.nlast) ....`
      `if (Sve.WhileLessThan(out pred, i, length).first) ....`

      `if (cmp)` is the same as doing `if (cmp.any)`

      Ideally the following will not be allowed:
        auto f = Sve.WhileLessThan(out pred, i, length).first
    */

    /*
      Always using a function call for the vector length instead of assigning to a variable will allow
      easier optimisation to INCW (which is faster than incrementing by a variable).
    */

    for (int i = 0; Sve.WhileLessThan(out pred, i, length); i += Sve.VectorLength<int>())
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);

      /*
        This is the standard sve `add` instruction which uses a merge predicate.
        For each lane in the predicate, add the two vectors. For all other lanes use the first vector.
       */
      total = Sve.MergeAdd(pred, total, vec);
    }

    // No tail call required.
    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Sum all the values in an int array, without predication.
    For performance reasons, it may be better to use an unpredicated loop, followed by a tail.
    Ideally, the user would write the predicated version and the Jit would optimise to this if required.
  */
  public static unsafe int sum_sve_unpredicated_loop(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;

    int i = 0;
    for (i = 0; i+Sve.VectorLength<int>() <= length; i+= Sve.VectorLength<int>() )
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(ref *src, i);
      total = Sve.MergeAdd(pred, total, vec);
    }

    // Predicated tail.
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);
    total = Sve.MergeAdd(pred, vec, total);

    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the non zero elements in an int array.
  */
  public static unsafe int CountNonZero_sve(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVEPred true_pred = Sve.CreatePred(true);

    for (int i = 0; Sve.WhileLessThan(out pred, i, length); i += Sve.VectorLength<int>())
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);
      VectorSVEPred cmp_res = Sve.CompareGreaterThan(pred, vec, 0);

      total = Sve.MergeAdd(cmp_res, total, vec);
    }

    // No tail call required.
    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the non zero elements in an int array, without predication.
  */
  public static unsafe int CountNonZero_sve_unpredicated_loop(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVEPred true_pred = Sve.CreatePred(true);

    // Comparisons require predicates. Therefore for a truely non predicated version, use Neon.
    int vector_length = 16/sizeof(int);
    for (int i = 0; i+vector_length <= length; i+=vector_length)
    {
      Vector128<int> vec = AdvSimd.LoadVector128(src);
      Vector128<int> gt = AdvSimd.CompareGreaterThan(vec, zero);
      Vector128<int> bits = AdvSimd.And(gt, one);

      total = AdvSimd.Add(bits, total);
      src += vector_length;
    }

    // Predicated tail.
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src);
    VectorSVEPred cmp_res = Sve.CompareGreaterThan(pred, vec, 0);
    total = Sve.MergeAdd(cmp_res, total, vec);

    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the elements in a null terminated array of unknown size.
  */
  public static unsafe int CountLength_sve(ref int* srcBytes)
  {
    int* src = srcBytes;
    VectorSVEPred pred = Sve.CreatePred(true);
    int ret = 0;

    while (true)
    {
      VectorSVE<int> vec = Sve.LoadUnsafeUntilFault(pred, ref *src); // LD1FF

      /*
        Reading the fault predicate via RDFFRS will also set the condition flags:
          VectorSVEComparison GetFaultPredicate(VectorSVEPred out fault, VectorSVEPred pred)
       */
      VectorSVEPred fault_pred;

      if (Sve.GetFaultPredicate(out fault_pred, pred).last)
      {
        // Last element is set in fault_pred, therefore the load did not fault.

        /*
          Like WhileLessThan, comparisons come in two variants:
            VectorSVEPred CompareEquals(VectorSVEPred pred, VectorSVE a, VectorSVE b)
            VectorSVEComparison CompareEquals(VectorSVEPred out cmp_result, VectorSVEPred pred, VectorSVE a, VectorSVE b)
         */

        // Look for any zeros across the entire vector.
        VectorSVEPred cmp_zero;
        if (Sve.CompareEquals(out cmp_zero, pred, vec, 0).none)
        {
          // No zeroes found. Continue loop.
          ret += Sve.VectorLength<int>();
        }
        else
        {
          // Zero found. Count up to it and return.
          VectorSVEPred matches = Sve.PredFillUpToFirstMatch(pred, cmp_zero); // BRKB
          ret += Sve.PredCountTrue(matches); // INCP
          return ret;
        }
      }
      else
      {
        // Load caused a fault.

        // Look for any zeros across the vector up until the fault.
        VectorSVEPred cmp_zero;
        if (Sve.CompareEquals(out cmp_zero, fault_pred, vec, 0).none)
        {
          // No zeroes found. Clear faulting predicate and continue loop.
          ret += Sve.PredCountTrue(fault_pred); // INCP
          Sve.ClearFaultPredicate(); // SETFFR
        }
        else
        {
          // Zero found. Count up to it and return.
          VectorSVEPred matches = Sve.PredFillUpToFirstMatch(pred, cmp_zero); // BRKB
          ret += Sve.PredCountTrue(matches); // INCP
          return ret;
        }
      }
    }
  }

Alternative Designs

No response

Risks

No response

Author:	a74nh
Assignees:	-
Labels:	api-suggestion, area-CodeGen-coreclr
Milestone:	-

[a74nh]

@kunalspathak @tannergooding @BruceForstall

[a74nh]

@TamarChristinaArm

Tagging subscribers to this area: @dotnet/area-system-runtime-intrinsics
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Adding a vector API for Arm SVE/SVE2 would be useful. SVE is a mandatory feature in Arm 9.0 onwards and is an alternative to NEON. Code written in SVE is vector length agnostic and will automatically scale to the vector length of the machine it is running on, and therefore will only require a single implementation per routine. Use of predication in SVE enables loop heads and tails to be skipped, making code shorter, simpler and easier to write.

This issue provides examples of how such an API might be used.

API Proposal

None provided.

API Usage

  /*
    Sum all the values in an int array.
  */
  public static unsafe int sum_sve(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);

    /*
      WhileLessThan comes in two variants:
        VectorSVEPred WhileLessThan(int val, int limit)
        VectorSVEComparison WhileLessThan(VectorSVEPred out predicate, int val, int limit)

      A VectorSVEComparison can be tested using the SVE condition codes (none, any, last, nlast etc).
      `if (cmp.nlast) ....`
      `if (Sve.WhileLessThan(out pred, i, length).first) ....`

      `if (cmp)` is the same as doing `if (cmp.any)`

      Ideally the following will not be allowed:
        auto f = Sve.WhileLessThan(out pred, i, length).first
    */

    /*
      Always using a function call for the vector length instead of assigning to a variable will allow
      easier optimisation to INCW (which is faster than incrementing by a variable).
    */

    for (int i = 0; Sve.WhileLessThan(out pred, i, length); i += Sve.VectorLength<int>())
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);

      /*
        This is the standard sve `add` instruction which uses a merge predicate.
        For each lane in the predicate, add the two vectors. For all other lanes use the first vector.
       */
      total = Sve.MergeAdd(pred, total, vec);
    }

    // No tail call required.
    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Sum all the values in an int array, without predication.
    For performance reasons, it may be better to use an unpredicated loop, followed by a tail.
    Ideally, the user would write the predicated version and the Jit would optimise to this if required.
  */
  public static unsafe int sum_sve_unpredicated_loop(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;

    int i = 0;
    for (i = 0; i+Sve.VectorLength<int>() <= length; i+= Sve.VectorLength<int>() )
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(ref *src, i);
      total = Sve.MergeAdd(pred, total, vec);
    }

    // Predicated tail.
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);
    total = Sve.MergeAdd(pred, vec, total);

    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the non zero elements in an int array.
  */
  public static unsafe int CountNonZero_sve(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVEPred true_pred = Sve.CreatePred(true);

    for (int i = 0; Sve.WhileLessThan(out pred, i, length); i += Sve.VectorLength<int>())
    {
      VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src, i);
      VectorSVEPred cmp_res = Sve.CompareGreaterThan(pred, vec, 0);

      total = Sve.MergeAdd(cmp_res, total, vec);
    }

    // No tail call required.
    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the non zero elements in an int array, without predication.
  */
  public static unsafe int CountNonZero_sve_unpredicated_loop(ref int* srcBytes, int length)
  {
    VectorSVE<int> total = Sve.Create((int)0);
    int* src = srcBytes;
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVEPred true_pred = Sve.CreatePred(true);

    // Comparisons require predicates. Therefore for a truely non predicated version, use Neon.
    int vector_length = 16/sizeof(int);
    for (int i = 0; i+vector_length <= length; i+=vector_length)
    {
      Vector128<int> vec = AdvSimd.LoadVector128(src);
      Vector128<int> gt = AdvSimd.CompareGreaterThan(vec, zero);
      Vector128<int> bits = AdvSimd.And(gt, one);

      total = AdvSimd.Add(bits, total);
      src += vector_length;
    }

    // Predicated tail.
    VectorSVEPred pred = Sve.WhileLessThan(i, length);
    VectorSVE<int> vec = Sve.LoadUnsafe(pred, ref *src);
    VectorSVEPred cmp_res = Sve.CompareGreaterThan(pred, vec, 0);
    total = Sve.MergeAdd(cmp_res, total, vec);

    return Sve.AddAcross(total).ToScalar();
  }


  /*
    Count all the elements in a null terminated array of unknown size.
  */
  public static unsafe int CountLength_sve(ref int* srcBytes)
  {
    int* src = srcBytes;
    VectorSVEPred pred = Sve.CreatePred(true);
    int ret = 0;

    while (true)
    {
      VectorSVE<int> vec = Sve.LoadUnsafeUntilFault(pred, ref *src); // LD1FF

      /*
        Reading the fault predicate via RDFFRS will also set the condition flags:
          VectorSVEComparison GetFaultPredicate(VectorSVEPred out fault, VectorSVEPred pred)
       */
      VectorSVEPred fault_pred;

      if (Sve.GetFaultPredicate(out fault_pred, pred).last)
      {
        // Last element is set in fault_pred, therefore the load did not fault.

        /*
          Like WhileLessThan, comparisons come in two variants:
            VectorSVEPred CompareEquals(VectorSVEPred pred, VectorSVE a, VectorSVE b)
            VectorSVEComparison CompareEquals(VectorSVEPred out cmp_result, VectorSVEPred pred, VectorSVE a, VectorSVE b)
         */

        // Look for any zeros across the entire vector.
        VectorSVEPred cmp_zero;
        if (Sve.CompareEquals(out cmp_zero, pred, vec, 0).none)
        {
          // No zeroes found. Continue loop.
          ret += Sve.VectorLength<int>();
        }
        else
        {
          // Zero found. Count up to it and return.
          VectorSVEPred matches = Sve.PredFillUpToFirstMatch(pred, cmp_zero); // BRKB
          ret += Sve.PredCountTrue(matches); // INCP
          return ret;
        }
      }
      else
      {
        // Load caused a fault.

        // Look for any zeros across the vector up until the fault.
        VectorSVEPred cmp_zero;
        if (Sve.CompareEquals(out cmp_zero, fault_pred, vec, 0).none)
        {
          // No zeroes found. Clear faulting predicate and continue loop.
          ret += Sve.PredCountTrue(fault_pred); // INCP
          Sve.ClearFaultPredicate(); // SETFFR
        }
        else
        {
          // Zero found. Count up to it and return.
          VectorSVEPred matches = Sve.PredFillUpToFirstMatch(pred, cmp_zero); // BRKB
          ret += Sve.PredCountTrue(matches); // INCP
          return ret;
        }
      }
    }
  }

Alternative Designs

No response

Risks

References

SVE Programming Examples
A64 -- SVE Instructions (alphabetic order)

No response

Author:	a74nh
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.Intrinsics, area-CodeGen-coreclr, untriaged
Milestone:	-

[kunalspathak]

cc: @JulieLeeMSFT

[tannergooding]

Thanks for opening the issue! This is definitely a space we want to improve, just wanting to lay out some baseline information and expectations for this.

---

It's worth noting that this is not actionable as an API proposal in its current setup. Any API proposal needs to follow the API proposal template including defining the full public surface area to be added.

It's also worth noting any work towards SVE/SVE2 is going to require:

1. Hardware we can reliably use and test in CI
2. Significant validation and design work around the exposed vector type, API surface, and usable surface area
3. Additional work to ensure things work correctly for AOT scenarios

---

For 1, there are only some mobile chips (Samsung/Qualcomm) and the AWS Graviton3 that currently support SVE or Armv9 at all. The latter is the only one that would be viable for CI and it would likely require some non-trivial effort to make happen. It may still be some time and require higher market saturation of SVE/SVE2 before work/progress can be made (the same was true for AVX-512). Having more easily accessible hardware that supports the ISAs (both for CI and local development) can accelerate this considerably.

For 2, SVE is a very different programming model from the existing fixed sized vectors exposed by System.Runtime.Intrinsics. It is closer to, but not entirely aligned with, System.Numerics.Vector<T> instead. We had a similar situation with the masking support for AVX-512 and we opted to not expose it for .NET 8. Instead we opted to do pattern recognition and implicit lightup for the existing patterns users already have to use downlevel.

It is entirely possible that SVE will be in a similar boat and in order for the BCL and general user-base to best take advantage of it, we will need to consider a balance between the general programming model most user-code needs for other platforms and what exists in hardware directly.

That is, the simplest way to get SVE support available, without requiring all users to go rewrite their code and add completely separate code paths just for the subset of hardware with SVE support would be to have SVE light-up over Vector<T>. Many of the same predicate/masking patterns will already be getting recognized for AVX-512 and simply involve tracking when a given node returns a mask (such as comparisons) or takes a mask (such as ConditionalSelect)

For 3, native SVE has many limitations around where the SVE vector types can be declared and how they are allowed to be used. Getting the same restrictions in managed would be very complex and not necessarily "pay for play". For the JIT, this is fine since things can be determined at runtime and match exactly. For AOT, this represents a problem since the sizes and other information isn't known until runtime which can complicate a number of scenarios.

[TamarChristinaArm]

@tannergooding Thanks for the initial feedback, just some quick responses:

For 1, there are only some mobile chips (Samsung/Qualcomm) and the AWS Graviton3 that currently support SVE or Armv9 at all. The latter is the only one that would be viable for CI and it would likely require some non-trivial effort to make happen.

There are actually some more that are available. If Linux would be an acceptable target I can send you an email with other options than the ones you have stated here.

It is entirely possible that SVE will be in a similar boat and in order for the BCL and general user-base to best take advantage of it, we will need to consider a balance between the general programming model most user-code needs for other platforms and what exists in hardware directly

Agreed 100%, though the purpose of this ticket is to start the discussions on the SVE intrinsics that would form the basis of Vector<T>. Our expectations is that Vector<T> will provide "good enough" support, but for people who want to use the full capabilities of SVE they would require direct usage of the intrinsics. This pull request was more to highlight some of the difficulties in archiving this and start the discussions around this. In some of the examples for instance we require explicit access to the CC flags which affects which branch gets emitted.

Before doing a full SVE design and proposal, it's better for us to get some input from you folks on how you'd like these kinds of situations to be handled. From our last discussion we highlighted that SVE would need some additional JIT design work for workloads that Vector won't ever cover, such as SME. Therefore I think it's best to focus this issue on how we can even provide core Scalable vector support for current and future Arm architectures rather than how to support them in the generic wrappers. I believe one is the pre-requisite to the other?

For 3, native SVE has many limitations around where the SVE vector types can be declared and how they are allowed to be used. Getting the same restrictions in managed would be very complex and not necessarily "pay for play". For the JIT, this is fine since things can be determined at runtime and match exactly. For AOT, this represents a problem since the sizes and other information isn't known until runtime which can complicate a number of scenarios.

Agreed! one of the questions we had raised last time was what are the constraints on which AOT operates. If AOT is equivalent to a native compiler's -mcpu=native than this won't be an issue as you'd know the vector length you're targetting.

If AOT is supposed to be platform agnostic, than the solution could be the one we discussed 2 years ago, where instead of assuming VLA, we assume a minimum vector length that the user selects. i.e. if they want full compatibility we select VL128 and just simply predicate all SVE operations to this.

This fixes the AOT issues since we'd never use more than VL sized vectors.

But really, to move forward with SVE support we need your input. SVE is much more than just a vector length play, and without support for it .NET the ability to improve will be quite limited. SVE will, and does have capabilities that Advanced SIMD will never have.

[tannergooding]

There are actually some more that are available. If Linux would be an acceptable target I can send you an email with other options than the ones you have stated here.

Linux is great!. We just really need to ensure there are options we can use for both development and validation. This namely means they can't be exclusively mobile devices (i.e. phones). Having a list of viable devices would be good, all the sources I've found for SVE or Armv9 have shown to be phones, android devices, the latest version of Gravitron, or not yet released for general use.

I believe one is the pre-requisite to the other?

It depends on the general approach taken. We could very likely expose some SVE/SVE2 support via Vector<T> or even Vector64/128/256/512<T> without exposing the raw intrinsic APIs.

This model is actually how AVX-512 was designed to be exposed for .NET 8. That is, we planned to ship the general purpose Vector512<T> and then implicit light-up for existing patterns over Vector128/256<T>. The support for VectorMask<T> and the platform specific APIs were an additional stretch goal. We ended up having enough time to do the platform specific APIs and to start work on VectorMask<T>, the latter of which we ultimately determined was not going to be a good fit and was cut for the time being.

We went with this approach because it allowed easier access to the acceleration for a broader audience, worked well when considering all platforms (not just x86/x64), allowed existing code to improve without explicit user action, and allowed developers to iteratively extend their existing algorithms without having to add massive amounts of new platform specific code.

---

If AOT is equivalent to a native compiler's -mcpu=native than this won't be an issue as you'd know the vector length you're targetting.

The default for most AOT would be platform agnostic. For example, x64 AOT still targets the 2004 baseline and has opportunistic light-up (that is allows for a cached runtime based check to access SSE3-SSE4.2). Arm64 AOT still targets the v8.0 baseline and likewise has opportunistic light-up for some scenarios.

predicate all SVE operations to this.

Predicated instructions has an increased cost over non-predicated, correct?

---

In particular, we have at least 3 main platforms to consider support around:

• Arm64
• WASM
• x64

There are also other platforms we support (Arm32, x86), platforms that are being brought online by the community (LoongArch, RISC-V), etc.

Most of these platforms have some amount of SIMD support and giving access to the platform specific APIs gives the greatest amount of power in your algorithms. Exposing this support long term is generally desirable.

However, an algorithm supporting every one of these platforms is also very expensive and most of the time the algorithms are fairly similar to each other regardless of the platform being targeted and there's often only minor differences between them. This is why the cross platform APIs were introduced in .NET 7, as it allowed us to take what was previously a minimum of 3 (Arm64, Fallback, x64) but sometimes more (different vector sizes, per ISA lightup, other platforms, etc) paths and merge them down to just 2 paths (Vector, Fallback) while still maintaining the perf or only losing perf within an acceptable margin.

To that end, the general direction we'd like to maintain is that developers can continue writing (for the most part) more general cross platform code for the bulk of their algorithms. They would then utilize the platform specific intrinsics for opportunistic light-up within these cross platform algorithms where it makes a significant impact. Extreme power users could continue writing individual algorithms that are fine-tuned for each platform/architecture, but it would not necessarily be the "core" scenario we expect most users to be targeting (namely due to the complexity around doing this).

I believe that ensuring SVE/SVE2 can fit into this model will help increase its adoption and usage, particularly among developers who may not have access to the full breadth of "latest hardware" on which to test their code for every platform/architecture. It will also help ensure that users don't necessarily need to go and rewrite their code to benefit, they can simply roll forward to the latest version of .NET and implicitly see performance wins. The BCL and other high performance libraries (including domain specific scenarios, such as ML.NET, Image Processing, etc), would update their own algorithms to make targeted usage of the platform specific APIs where they help achieve significantly more performance than the general purpose code.

---

I do think we should be able to make SVE/SVE2 integrate with the same general model and I believe we should be able to do this around Vector<T> rather than introducing a new Arm64 exclusive vector type.

Given the samples in the OP, there's this callout:

Ideally, the user would write the predicated version and the Jit would optimise to this if required.

I actually think its the inverse and given the cross platform considerations, we want users to write code unpredicated with a tail loop. This is what they already have to do for NEON, it's what they have to do for WASM, it's what they have to do for SSE/AVX. This is also something we considered around AVX-512 masking support and one reason why we opted to not expose VectorMask<T> for right now.

Instead, we could express the code such as:

public static unsafe int sum_sve_unpredicated_loop(int* srcBytes, int length)
{
    Vector<int> total = Vector<int>.Zero;
    (int iterations, int remainder) = int.DivRem(i, Vector<int>.Length);

    for (int i = 0; i < iterations; i++)
    {
        total += Vector.Load(srcBytes);
        srcBytes += Vector<int>.Length;
    }

    // Predicated tail.
    Vector<int> pred = Vector.CreateTrailingMask(remainder);
    total += Vector.MaskedLoad(pred, srcBytes, Vector<int>.Zero);

    return Vector.Sum(total);
}

We would still of course expose Sve.Add, Sve.AddAccross, Sve.LoadVector, Sve.WhileLessThan, etc. However, users wouldn't have to use it, it would just be an additional option if they wanted finer grained control over what codegen they'd get.

There are notably a few other ways this could be done as well, but the general point is that this results in very natural and easy to write code that works on any of the platforms, including on hardware that only supports NEON, without users having to write additional and separate versions of their algorithms to achieve it.

There is a note that CreateTrailingMask, GreaterThan, etc all return a Vector<T> and not a VectorMask<T> (or VectorSVEPred). This is something that we're doing with AVX-512 due to the consideration that VectorMask<T> could not be easily accelerated on downlevel platforms and that it would effectively need to be a Vector<T> in those cases anyways. Such a restriction meant users would have to explicitly rewrite their algorithms to use masking, and that was going to hinder adoption overall.

The JIT handles this difference from hardware in lowering by doing some tracking to know if the value is actually a mask or vector and whether the consumer needs a mask or vector. This then translated very cleanly to existing patterns such as ConditionalSelect(mask, x + y, mergeVector) which can be trivially recognized and emitted as a MergeAdd instead. The same support translates to all other intrinsic APIs that support masking, so we didn't need to expose an additional 3800 API overloads to support masking (this was the minimum actual count of additional API's we'd of had to expose to exactly match what hardware supports).

Noting that this decision also doesn't restrict our ability from adding such support in the future. It just greatly simplified what was required today and made it much faster and more feasible to get the general support out while also keeping the implementation cost low.

We could also add support for the fully predicated loops and optimizing them to be rewritten as unpredicated with a post predicated handler. But that is more complicated and I believe would be better handled as a future consideration after we get the core support out.

I'd like to see us add SVE support and so I'd hope going along the avenues we've found to work and help with adoption and implicit light up would also apply here.

[a74nh]

It's worth noting that this is not actionable as an API proposal in its current setup.

Agreed, this seemed the closest category. Given that this won't turn into a full API proposal, I could just drop the category and move to a normal issue?

Predicated instructions has an increased cost over non-predicated, correct?

For now, yes.

Arm64 AOT still targets the v8.0 baseline and likewise has opportunistic light-up for some scenarios.

Having AOT Arm64 target 8.0 feels sensible. For ReadyToRun, it would then tier appropriately to what is available. For the "no JIT, only AOT" option, maybe a command line option at compile time - essentially the same as the gcc/llvm -march flags.

---

There is a note that CreateTrailingMask, GreaterThan, etc all return a Vector and not a VectorMask (or VectorSVEPred). This is something that we're doing with AVX-512 due to the consideration that VectorMask could not be easily accelerated on downlevel platforms and that it would effectively need to be a Vector in those cases anyways. Such a restriction meant users would have to explicitly rewrite their algorithms to use masking, and that was going to hinder adoption overall.

Without explicit VectorMask types you're losing type checking. Could VectorMask just be a wrapper over Vector?

On downlevel platforms, the tail is being turned into a scalar loop? Therefore on simple loops, the mask is effectively optimised away, right?

I actually think its the inverse and given the cross platform considerations, we want users to write code unpredicated with a tail loop. This is what they already have to do for NEON, it's what they have to do for WASM, it's what they have to do for SSE/AVX. This is also something we considered around AVX-512 masking support and one reason why we opted to not expose VectorMask for right now.

That makes sense when coming at it from the generic side.

From an implementation side, I'm thinking that predication makes things easier. The user writes a single predicated loop. There is enough information in the loop for the jit to 1) create a predicated loop with no tail or 2) create unpredicated main loop and predicated tail or 3) create unpredicated main loop and scalar tail.

I understand users are not used to writing with predication yet. But, in the example, the user still has to use predication for the tail. Is that why VectorMask<T> was dropped for now - to make it easier for users?

Another option maybe would be to never have Vector<T> supporting masks/predication. Tails would always be scalar. All platforms can be supported easily.
Then later Vector<P> is later added which looks similar, but does everything using predication - and the jit turns into the correct thing on the platform.
That would skip ever having unpredicated loops with predicated tail, but maybe that's ok.

[tannergooding]

For the "no JIT, only AOT" option, maybe a command line option at compile time - essentially the same as the gcc/llvm -march flags.

We have this already, and allow targeting specific versions (armv8.1, armv8.2, etc) or specific ISAs that are currently supported.

Could VectorMask just be a wrapper over Vector?

Not cheaply/easily and not without requiring users go and explicitly opt into the use of masking.

The general point is that on downlevel hardware (NEON, SSE-AVX2, etc) there is no masking, you only have Vector<T>. If you want to do masking, you have to use explicit APIs like ConditionalSelect or MaskMove and so this is what developers are already doing.

It is fairly trivial for the JIT to recognize and internally do the correct type tracking. It is also fairly trivial for the JIT to insert the relevant implicit conversions if an API expects a mask and was given a vector; or if it expects a vector and was given a mask. One of the reasons this is cheap and easy to do is because it is completely non-observable to whoever is writing the algorithm and so the JIT can do this in a very "pay for play" manner.

If we inverse things, such that we expose VectorMask publicly then we start having to worry about observable differences in various edge cases. We start having to expose overloads that explicitly take the mask, which means most new APIs now have 2x the overloads. We also have to do significantly more work to ensure that the right conversions and optimizations are being done such that it remains efficient, etc.

On downlevel platforms, the tail is being turned into a scalar loop? Therefore on simple loops, the mask is effectively optimised away, right?

Users do different things for different scenarios. Some users simply defer back to the scalar path. Some will "backtrack" and do 1 more iteration when the operation is idempotent. Some use explicit masked operations to ensure the operation remains idempotent.

From an implementation side, I'm thinking that predication makes things easier. The user writes a single predicated loop. There is enough information in the loop for the jit to 1) create a predicated loop with no tail or 2) create unpredicated main loop and predicated tail or 3) create unpredicated main loop and scalar tail.

I'd disagree on this. The general "how to write vectorized code" steps are:

1. Write your algorithm as you would normally (this is scalar, doing 1 operation per iteration)
2. Unroll your loop to do n operations per iteration (where n is typically sizeof(Vector) / sizeof(T), that is VectorXXX<T>.Count)
3. Squash your n operations into 1 SIMD operation per iteration (e.g. n scalar Add become 1x SIMD Add)

Predication/masking is effectively a more advanced scenario that can be used to handling more complex scenarios such as branching or handling the tail (remaining) elements that don't fill the vector. It's not something that's actually needed super often and generally represents a minority of the overall SIMD code you write.

I understand users are not used to writing with predication yet. But, in the example, the user still has to use predication for the tail. Is that why VectorMask was dropped for now - to make it easier for users?

Right. We opted to drop VectorMask<T> (from the AVX-512 support) for now because it was significantly less work for the JIT and also greatly simplified the end user experience. It made it so that users don't have to change how they are thinking about or writing their code at all, they simply continue doing what they've already been doing and it happens to take advantage of the underlying hardware if the functionality is available.

It also means that we don't have to do more complex handling for downlevel hardware (which is currently the far more common scenario) and can instead restrict it only to the newer hardware. It means that we won't typically encounter unnecessary predication which means we don't have to worry about optimizing the unnecessary bits away. It then also simplifies the general handling of loops as we don't have to consider predication as part of loop cloning, loop hoisting, or other optimizations in general.

Another option maybe would be to never have Vector supporting masks/predication. Tails would always be scalar. All platforms can be supported easily.

Users are already sometimes writing predicated tails today, so I don't see the need to block this. I think it would be better to just expose the couple additional APIs that would help with writing predicated tails instead. Exposing a nearly identical type that supports predication would just make adding predication support harder and would increase the amount of code the typical user needs to maintain.

Without explicit VectorMask types you're losing type checking.

In my opinion, this is largely a non-issue. Downlevel users already need to do their masking/predication using VectorXXX<T> and so they're already used to Vector128.GreaterThan returning a Vector128<T> and not a VectorMask128<T>.

The JIT is then fully capable of knowing which methods return a "mask like value", which we already minimally track on downlevel hardware to allow some special optimizations.

The JIT is also fully capable of correctly tracking the type internally as TYP_MASK (rather than TYP_SIMD16) when it is not just "mask like" but an actual mask (for AVX-512 or SVE capable hardware). This makes the fact that it is actually a mask completely invisible to the end user.

The only requirement for the JIT here is that if it has a TYP_MASK and the API expects a TYP_SIMD16 (or vice-versa) it needs to insert an implicit conversion. On AVX-512, there are dedicated instructions for this. I don't believe there are for SVE, but there are still some options available that can make this very cheap/feasible.

A managed analyzer can then correctly surface to the end user when they are inefficiently handling masking (e.g. passing in a mask to something that doesn't expect one, or vice versa). This is very similar to the [ConstantExpected] analyzer and would likely not trigger for typical code users are writing.