Better support for UDF when using Dotnet Spark.NET

[GeorgeS2019]

When using UDF or User-Defined Functions with Spark.NET, there are problems
4. User-Defined Functions

Describe the solution you'd like
Now, there is a new initiative to continue support for Spark.NET, we need more compatibility with Dotnet Interativ and Dotnet Spark.NET, especially when working with UDF.

Describe alternatives you've considered
Coordinate with SPARK.NET user community to identify ways to make UDF workable using Interativ.
dotnet/spark#1179 (reply in thread)

dotnet/spark#1199

[GeorgeS2019]

We now have a new Spark.NET 2.3.0 version that support .NET 8

@wudanzy
We shall test if there is still UDF problem with .NET interactive

[grazy27]

@GeorgeS2019 we wtill have this bug, path handling logic needs to be updated.
I shared details in dotnet/spark#1179

[jonsequitur]

I'm open to a call to discuss the state of this and provide any guidance I can. Spark in .NET Interactive has been out of support for a few years because of the deprecation of BinaryFormatter but I think the fundamentals should still be in place for this to work if the appropriate fixes are made in .NET for Spark.

[grazy27]

@jonsequitur thanks!
I'm on a vacation next week, but available afterwards, what is your availability?
Help with dotnet.interactive APIs would be invaluable, currently I'm stuck on this DIsposableDirectory: dotnet/spark#1179 (comment)