Update needed for Write and call UDFs in .NET Interactive

[GeorgeS2019]

The page: Write and call UDFs in .NET for Apache Spark interactive environments is still using nuget 0.12.1

The latest version is Microsoft.Spark.Extensions.DotNet.Interactive is 2.1.1

Recently, there has been bug complains using UDFs with .NET for Apache Spark including myself.

• Error from many DataFrame methods after UDF called in DataFrame.WithColumn
• [FEATURE REQUEST]: Any idea to call .net UDF in scala spark? Also need some generic parameters
• UDF use deprecated BinaryFormatter, fail in VS Code
• [BUG]: UDF Serialization bug in .Net interactive

Summary

This document needs an update to address the issues reported above:
https://github.com/dotnet/spark/blob/main/docs/udf-guide.md

It is unclear what are the NEW ISSUES associated with using UDFs with .NET for Spark and the document needs an URGENT Update: reflecting Microsoft.Spark.Extensions.DotNet.Interactive : 2.1.1 and NOT 0.12.1

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: a15ae835-9313-7b7a-b8fa-9a8db53c8cb6
• Version Independent ID: 4b650811-6685-742a-55bf-ed8a76920ede
• Content: Write and call UDFs in .NET for Apache Spark interactive environments.
• Content Source: docs/spark/how-to-guides/dotnet-interactive-udf-issue.md
• Product: dotnet-spark
• GitHub Login: @Niharikadutta
• Microsoft Alias: nidutta

[GeorgeS2019]

@Niharikadutta

Use .NET for Apache Spark in Jupyter Notebooks

Please recommend to the users, for .NET Interactive, the same notebook environment used in Azure Synapse, which nuget to use, is there a difference?

#r "nuget:Microsoft.Spark,"
or
#r "Microsoft.Spark.Extensions.DotNet.Interactive"

Topic 2 Update the nuget version recommended from 1.0.0 to 2.1.1

Use .NET for Apache Spark in Jupyter Notebooks

[gewarren]

We're no longer updating the .NET for Apache Spark documentation, and have archived it to https://learn.microsoft.com/previous-versions/dotnet/spark/what-is-spark. As a result, we're closing all the active .NET for Apache Spark issues. I'm sorry we couldn't address your issue.