TensorPrimitives improvements in .NET 10.0

[stephentoub]

Regardless of any additional types we may want to add to System.Numerics.Tensors, we would like to expand the set of APIs exposed on the TensorPrimitives static class in a few ways (beyond the work done in .NET 8 in #92219):

• Vectorize TensorPrimitives operations that are currently scalar only #97193
• Alignment improvements for ConvertXx, CosineSimilarity, IndexOfMin, IndexOfMax, IndexOfMinMagnitude, IndexOfMaxMagnitude
• Investigate precision of the vectorized TensorPrimitives implementations. #98861
• Additional operations defined in BLAS / LAPACK that don't currently have representation on TensorPrimitives
• Perform a broader scan of ML.NET APIs, seeking more methods that should be on the post-GA backlog - @michaelgsharp
  • We've already covered all of the shared methods, but there are one-off implementations that might be worth productizing into TensorPrimitives
• Additional operations that would enable completely removing the internal CpuMath class from ML.NET, e.g. Add (with indices), AddScale (with indices), DotProductSparse, MatrixTimesSource, ScaleAdd improvement via AddMultiply or MultipleAdd overloads, SdcaL1UpdateDense, SdcaL1UpdateSparse, and ZeroMatrixItems (might exist in System.Memory).
• Double-check the flow of XML docs -> https://github.com/dotnet/dotnet-api-docs -> docs.microsoft.com/
• Add conceptual docs for TensorPrimitives, maybe near https://github.com/dotnet/docs/blob/main/docs/standard/numerics.md
• Generic overloads of all relevant methods, constrained to the appropriate numerical types
• Get benchmarks added into dotnet/performance
  • Collect baseline results from the time between RC2 and GA right before our alignment improvements went in
  • Collect new results from main after all of the alignment
• Improve performance of Min, Max, MinMagnitude, MaxMagnitude with relation to NaN handling
• Determine for lengths of 0 if we want to throw or return NaN (we consistently throw today when non-0 is required; ML.NET apparently returns 0?) - @tannergooding
  • We currently throw; if we decide not to throw, this could be changed in a minor release in a non-breaking way.
• Additional operations from Math{F} that don't currently have representation on TensorPrimitives, e.g. CopySign, Reciprocal{Sqrt}{Estimate}, Sqrt, Ceiling, Floor, Truncate, Log10, Log(x, y) (with y as both span and scalar), Pow(x, y) (with y as both span and scalar), Cbrt, IEEERemainder, Acos, Acosh, Cos, Asin, Asinh, Sin, Atan. This unmerged commit has a sketch, but it's out-of-date with improvements that have been made to the library since, and all of the operations should be vectorized.
• Refactor the generic TP implementation into multiple source files.
• Additional operations defined in the numerical interfaces that don't currently have representation on TensorPrimitives, e.g. BitwiseAnd, BitwiseOr, BitwiseXor, Exp10, Exp10M1, Exp2, Exp2M1, ExpM1, Atan2, Atan2Pi, ILogB, Lerp, ScaleB, Round, Log10P1, Log2P1, LogP1, Hypot, RootN, AcosPi, AsinPi, AtanPi, CosPi, SinPi, TanPi

We plan to update the System.Numerics.Tensors package alongside .NET 8 servicing releases. When there are bug fixes and performance improvements only, the patch number part of the version will be incremented. When there are new APIs added, the minor version will be bumped. For guidance on how we bump minor/major package versions, see this example.

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Regardless of any additional types we may want to add to System.Numerics.Tensors, we would like to expand the set of APIs exposed on the TensorPrimitives static class in a few ways:

• Additional operations from Math{F} that don't currently have representation on TensorPrimitives, e.g. CopySign, Reciprocal{Sqrt}{Estimate}, Sqrt, Ceiling, Floor, Truncate, Log10, Log(x, y) (with y as both span and scalar), Pow(x, y) (with y as both span and scalar), Cbrt, IEEERemainder, Acos, Acosh, Cos, Asin, Asinh, Sin, Atan. This unmerged commit has a sketch, but it's out-of-date with improvements that have been made to the library since, and all of the operations should be vectorized.
• Additional operations defined in the numerical interfaces that don't currently have representation on TensorPrimitives, e.g. BitwiseAnd, BitwiseOr, BitwiseXor, Exp10, Exp10M1, Exp2, Exp2M1, ExpM1, Atan2, Atan2Pi, ILogB, Lerp, ScaleB, Round, Log10P1, Log2P1, LogP1, Hypot, RootN, AcosPi, AsinPi, AtanPi, CosPi, SinPi, TanPi
• Additional operations defined in BLAS / LAPACK that don't currently have representation on TensorPrimitives
• Additional operations that would enable completely removing the internal CpuMath class from ML.NET, e.g. Add (with indices), AddScale (with indices), DotProductSparse, MatrixTimesSource, ScaleAdd improvement via AddMultiply or MultipleAdd overloads, SdcaL1UpdateDense, SdcaL1UpdateSparse, and ZeroMatrixItems (might exist in System.Memory).
• Generic overloads of all relevant methods, constrained to the appropriate numerical types

Concrete proposal to follow.

Author:	stephentoub
Assignees:	-
Labels:	api-suggestion, area-System.Numerics.Tensors
Milestone:	9.0.0

[Szer]

Could you please elaborate on the advantages of having these APIs in a BCL rather than in a specialized NuGet package (like numpy in Python)? This could provide a valuable perspective for further discussion.

[stephentoub]

Could you please elaborate on the advantages of having these APIs in a BCL rather than in a specialized NuGet package

It is a nuget package today. It's currently not part of netcoreapp. If it were to be pulled into netcoreapp as well, it would be because we'd be using it from elsewhere in netcoreapp, e.g. using it from APIs like Enumerable.Average, BitArray.And, ManagedWebSocket.ApplyMask, etc., which we very well may do in the future (that has no impact on it continuing to be available as a nuget package).

[xoofx]

Hey @stephentoub,

Would it be possible to expose the low level parts of the API instead of only providing Span versions?

e.g

public static Vector128<float> Log2(Vector128<float> value);
public static Vector256<float> Log2(Vector256<float> value);
public static Vector512<float> Log2(Vector512<float> value);
//...etc.

I did that for a prototype for a similar API and it's working great.
One reason to expose these APIs is that you can actually build higher level functions (e.g for tensors, the zoo of the activation functions) and build Span versions on top of them.

These API can then be used for other kind of custom Span batching (not related to tensors), where the packing of the vector is different (e.g 4xfloat chuncked xxxx, yyyy, zzzz)

[tannergooding]

Would it be possible to expose the low level parts of the API instead of only providing Span versions?

Yes, but it needs to be its own proposal and cover all 5 vector types (Vector, Vector64/128/256/512) and consider whether its applicable to Vector2/3/4 as well.

[xoofx]

Yes, but it needs to be its own proposal and cover all 5 vector types (Vector, Vector64/128/256/512)

Cool, I will try to write something.

[xoofx]

Would it be possible to expose the low level parts of the API instead of only providing Span versions?

Follow-up, created the proposal #93513