Native library packaging to support platforms with restricted file-system access

[timyhac]

The way libplctag.NativeImport packages the runtimes is not standard. NativeImport extracts the appropriate native library to disk at runtime.

It was originally done this way to facilitate compliance with libplctag's LGPL license (requiring a way for end users to substitute the native binary). Now libplctag is dual licensed with MPL so this is not required.

This is a problem because some applications can not write into the application directory (UWP apps). As far as I understand, this is a security feature. Snaps (snapcraft.io) also feature an immutable application directory, so the technique of extracting the library at runtime won't work.

Most .NET packages that ship native libraries ship them in a way that the native library is extracted as part of the build process (rather than at runtime). See https://dev.to/jeikabu/nupkg-containing-native-libraries-1576

@timyhac 's expertise in developing nuget packages is limited, and the non-standard way was selected because it wasn't straightforward how to do it the standard way but still keep maximum compatibility:

• Absolutely need example of how to include native libraries NuGet/Home#6648
• Guide for packaging C# library using P/Invoke to per-architecture and/or per-platform C++ native DLLs NuGet/Home#8623

A limitation of the "extract at runtime" procedure is that applications require permissions to write to disk. Depending on the application, this may not be practical (e.g. all UWP applications have highly restricted access to filesystems). If an application requires file system access purely for this feature, it would be an unusual experience for end-users "why does this need access to disk? I can understand why it needs access to local network, but what files is it accessing?!"

The requirements for the "pre-packaged native library" feature are:

• Support any .NET runtime that can run .NET Standard 2.0
  • linux/windows/macos/iOS/Android/etc..
  • both .NET Core and .NET Framework
• Support for over-riding the binary (in case of testing a pre-release or other customised binary)
• Support for platforms where the binary is not packaged (e.g. ARM)
• Support as a referenced package (i.e. should exhibit this behaviour whether it is directly referenced, or indirectly as part of primary libplctag.NET package)
• Updating the nuget package also updates the native library.

Update 27/6/2024: Microsoft has released some guidance on including Native files into packages, including for .NET Framework.

[timyhac]

Doing some research into this again, there seem to be a few approaches to this.

• Embeddeding the unmanaged libraries as a resource into the managed library, extracting only the correct one to disk, then PInvoke loads it to memory from disk.
  • There does appear to be ways at loading the library from specific directories, so there may be some scope to extract the library to somewhere that the application will have permissions.
  • https://spin.atomicobject.com/2019/10/21/nuget-package-native-dll/
  • csproj: document how to properly pack platform-specific native assemblies NuGet/Home#6645
• Embedding the unmanaged library as a resource, and then directly loading to memory, i.e. without writing to disk first.
  • https://github.com/dretax/DynamicDllLoader
• Using the nuget package runtimes/{RID}/native/plctag.dll technique. This means that the deployed application will be platform specific (i.e. the application developer can't use AnyCPU when preparing a build), but it will also mean that the application will as small as it can be because it doesn't contain the full suite of unmanaged libraries.
  • https://docs.microsoft.com/en-us/nuget/guides/create-uwp-packages
  • https://docs.microsoft.com/en-us/nuget/create-packages/supporting-multiple-target-frameworks

Although this doesn't directly solve the issue at hand, there are some other architectural choice that could be made:

• Providing the native API interface in one package, and then shipping different "providers" for each platform. The major SQLite libraries make use of SQLitePCL.raw. I'm not sure how the platform provider is resolved, but I guess it happens at build time.
• More exotic: compiling the C library to be a .NET managed library.
  • https://ericsink.com/entries/sqlite_llama_preview.html
  • https://www.codeproject.com/Articles/1128868/Compiling-Your-C-Code-to-NET-Part
  • https://dotnetfoundation.org/blog/2015/04/14/announcing-llilc-llvm-for-dotnet
  • https://github.com/praeclarum/Iril

Update: In fact, this article by Eric Sink details the problem perfectly, and in fact this was the article that lead to the current solution: https://ericsink.com/entries/native_library.html

The difference now however is that .NET 5 has been released. Unfortunately, those APIs are not available in .NET Standard 2.0.

[GitHubDragonFly]

In your LibraryExtractor.cs file, you are using System.Uri to detect the directory and none of the System.Uri constructors seems to be correctly handling certain paths with certain signs in them.

Instead, you should just try to detect the current directory like this:

        var extractDirectory = "." + System.IO.Path.DirectorySeparatorChar;

which should work for Windows and Linux (and hopefully other OS).

Also, all your plctag libraries within the "runtime" folder are pointers to github location.
If somebody downloads your project as a zip file will not be able to use it as such.

[timyhac]

Thanks @GitHubDragonFly - can you give an example of the kind of path where its not working by any chance?

This issue is more around sandboxed application types where you can't modify the application directory after installation. UWP fits into this category, there is no concept of a Current Working Directory in UWP. Same with snaps. I think Android and iOS apps also fit this.

P.S. we use LFS to store the native libraries. If you do git lfs clone https://github.com/libplctag/libplctag.NET.git you should get the full project including those files.

[GitHubDragonFly]

My project has "#" in the name and that's where uri escapes and directs extraction of the library to the parent folder of the project instead of the Debug folder where the application is.

I used the libraries from the latest libplctag release and all these files in your project amount to 2.2 MB.
Some people might opt to go with a zip file instead of clone and that's when they will realize that it doesn't work properly.

[GitHubDragonFly]

Have you tried using this: var extractDirectory = Directory.GetCurrentDirectory();

[timyhac]

Another reference example that could be used: https://github.com/mono/SkiaSharp/tree/main/nuget