[BUG]: UDFs in .NET Interactive: example fails (on .NET 5 / Jupyter / macOS)

[oefe]

Describe the bug
The UDFs in .NET Interactive](https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/dotnet-interactive-udf-issue#define-a-udf-in-net-interactive) fails on macOS Catalina using .NET 5 and Jupyter.

To Reproduce

Steps to reproduce the behavior:

1. Launch Spark using spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-3-0_2.12-1.0.0.jar debug
2. Launch Jupyter using jupyter notebook
3. Create a new notebook
4. Follow the steps in UDFs in .NET Interactive
5. Executing cell [Documentation] Update README.md #5 (particularly df2.Show();) fails with:

[2020-12-14T13:27:09.8868410Z] [Triton-273] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'showString' failed for class '142' when called with 3 arguments ([Index=1, Type=Int32, Value=20], [Index=2, Type=Int32, Value=20], [Index=3, Type=Boolean, Value=False], )
[2020-12-14T13:27:09.8874210Z] [Triton-273] [Error] [JvmBridge] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 28.0 failed 1 times, most recent failure: Lost task 0.0 in stage 28.0 (TID 26, triton-273.fritz.box, executor driver): org.apache.spark.api.python.PythonException: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Spark.Utils.UdfSerDe.<>c.<DeserializeType>b__10_0(TypeData td) in /_/src/csharp/Microsoft.Spark/Utils/UdfSerDe.cs:line 265

(rest of traceback omitted for brevity)

Expected behavior

• Should be able to use UDFs
• The example in UDFs in .NET Interactive should work

Screenshots
NA

Desktop (please complete the following information):

• OS: macOS 10.15.7
• .NET 5 (5.0.100)
• Apache Spark 3.0.1
• .NET for Apache Spark v1.0.0 (netcoreapp3.1, macOS)
• Jupyter:
  • jupyter core : 4.6.3
  • jupyter-notebook : 6.1.1
  • qtconsole : 4.7.5
  • ipython : 7.17.0
  • ipykernel : 5.3.4
  • jupyter client : 6.1.6
  • jupyter lab : not installed
  • nbconvert : 5.6.1
  • ipywidgets : 7.5.1
  • nbformat : 5.0.7
  • traitlets : 4.3.3
• Chrome 86.0.4240.198 (Official Build) (x86_64)

Additional context
NA

[oefe]

User error? I had launched
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-3-0_2.12-1.0.0.jar debug
In a directory that hat the jar, but not the Microsoft.Spark.dll and its dependencies.

If I run spark-submit a directory that contains these (e.g. the bin/Debug/netcoreapp3.1 of a simple Spark app), it works fine.

Shouldn’t it find Microsoft.Spark.dll and its dependencies via DOTNET_WORKER_DIR?

[Niharikadutta]

Hi @oefe , the worker needs to know the path in which to look for the binaries, which it does via the DOTNET_ASSEMBLY_SEARCH_PATHS environment variable that needs to be set to the path where the binaries and its dependencies are, and then passed as a conf via spark-submit. Please refer to this document that explains this in detail.

[Niharikadutta]

Hi @oefe Were you able to resolve this issue?

[Niharikadutta]

Hi, we are going to close this issue as it has been inactive for a while. Please feel free to re-open it if the issue persists and/or there are any new updates. Thank you!