Skip to content

fix: incorrect loading of native library with multiple backends #594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

fix: incorrect loading of native library with multiple backends #594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3d283a8
fix: incorrect loading of native library when having multiple backend…
AsakusaRinne Mar 12, 2024
9630dcd
fix: remove useless code.
AsakusaRinne Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 30 additions & 30 deletions LLama/LLamaSharp.Runtime.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,68 +4,68 @@
</PropertyGroup>
<ItemGroup Condition="'$(IncludeBuiltInRuntimes)' == 'true'">

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama.dll">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/noavx/llama.dll</Link>
<Link>runtimes/win-x64/native/llama-sharp/noavx/llama.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/avx/llama.dll">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/avx/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/avx/llama.dll</Link>
<Link>runtimes/win-x64/native/llama-sharp/avx/llama.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/avx2/llama.dll">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/avx2/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/avx2/llama.dll</Link>
<Link>runtimes/win-x64/native/llama-sharp/avx2/llama.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/avx512/llama.dll">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/avx512/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/avx512/llama.dll</Link>
<Link>runtimes/win-x64/native/llama-sharp/avx512/llama.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu11.7.1/llama.dll">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/cu11.7.1/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/cuda11/llama.dll</Link>
<Link>runtimes/win-x64/native/llama-sharp/cuda11/llama.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu12.1.0/llama.dll">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/cu12.1.0/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/cuda12/llama.dll</Link>
<Link>runtimes/win-x64/native/llama-sharp/cuda12/llama.dll</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/libllama.so">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/noavx/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/noavx/libllama.so</Link>
<Link>runtimes/linux-x64/native/llama-sharp/noavx/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/avx/libllama.so">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/avx/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/avx/libllama.so</Link>
<Link>runtimes/linux-x64/native/llama-sharp/avx/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/avx2/libllama.so">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/avx2/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/avx2/libllama.so</Link>
<Link>runtimes/linux-x64/native/llama-sharp/avx2/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/avx512/libllama.so">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/avx512/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/avx512/libllama.so</Link>
<Link>runtimes/linux-x64/native/llama-sharp/avx512/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu11.7.1/libllama.so">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/cu11.7.1/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/cuda11/libllama.so</Link>
<Link>runtimes/linux-x64/native/llama-sharp/cuda11/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu12.1.0/libllama.so">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/cu12.1.0/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/cuda12/libllama.so</Link>
<Link>runtimes/linux-x64/native/llama-sharp/cuda12/libllama.so</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/osx-arm64/libllama.dylib">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/osx-arm64/libllama.dylib">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/osx-arm64/native/libllama.dylib</Link>
<Link>runtimes/osx-arm64/native/llama-sharp/libllama.dylib</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/osx-arm64/ggml-metal.metal">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/osx-arm64/ggml-metal.metal">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/osx-arm64/native/ggml-metal.metal</Link>
<Link>runtimes/osx-arm64/native/llama-sharp/ggml-metal.metal</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/osx-x64/libllama.dylib">
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama-sharp/osx-x64/libllama.dylib">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/osx-x64/native/libllama.dylib</Link>
<Link>runtimes/osx-x64/native/llama-sharp/libllama.dylib</Link>
</None>
</ItemGroup>
</Project>
35 changes: 29 additions & 6 deletions LLama/Native/NativeApi.Load.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Reflection;
using System.Runtime.InteropServices;
using System.Text.Json;

Expand Down Expand Up @@ -188,23 +189,23 @@ private static List<string> GetLibraryTryOrder(NativeLibraryConfig.Description c
// if check skipped, we just try to load cuda libraries one by one.
if (configuration.SkipCheck)
{
result.Add($"{prefix}cuda12/{libraryNamePrefix}{libraryName}{suffix}");
result.Add($"{prefix}cuda11/{libraryNamePrefix}{libraryName}{suffix}");
result.Add($"{prefix}llama-sharp/cuda12/{libraryNamePrefix}{libraryName}{suffix}");
result.Add($"{prefix}llama-sharp/cuda11/{libraryNamePrefix}{libraryName}{suffix}");
}
else
{
throw new RuntimeError("Configured to load a cuda library but no cuda detected on your device.");
throw new RuntimeError("Configured to load a cuda library strictly but no cuda component detected on your device.");
}
}
else if (cudaVersion == 11)
{
Log($"Detected cuda major version {cudaVersion}.", LogLevel.Information);
result.Add($"{prefix}cuda11/{libraryNamePrefix}{libraryName}{suffix}");
result.Add($"{prefix}llama-sharp/cuda11/{libraryNamePrefix}{libraryName}{suffix}");
}
else if (cudaVersion == 12)
{
Log($"Detected cuda major version {cudaVersion}.", LogLevel.Information);
result.Add($"{prefix}cuda12/{libraryNamePrefix}{libraryName}{suffix}");
result.Add($"{prefix}llama-sharp/cuda12/{libraryNamePrefix}{libraryName}{suffix}");
}
else if (cudaVersion > 0)
{
Expand Down Expand Up @@ -255,6 +256,8 @@ private static IntPtr TryLoadLibrary()
NativeLibraryConfig.LibraryHasLoaded = true;
Log(configuration.ToString(), LogLevel.Information);

Console.WriteLine("########################### begin to find path: " + configuration.Path);

if (!string.IsNullOrEmpty(configuration.Path))
{
// When loading the user specified library, there's no fallback.
Expand All @@ -267,6 +270,8 @@ private static IntPtr TryLoadLibrary()
return result;
}

Console.WriteLine("########################### begin to apply strategy");

var libraryTryLoadOrder = GetLibraryTryOrder(configuration);

var preferredPaths = configuration.SearchDirectories;
Expand All @@ -275,6 +280,24 @@ private static IntPtr TryLoadLibrary()
Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location) ?? ""
};

Console.WriteLine("===============================libraryTryLoadOrder========================================");
foreach(var item in libraryTryLoadOrder)
{
Console.WriteLine(item);
}
Console.WriteLine("================================preferredPaths=======================================");
foreach(var item in preferredPaths)
{
Console.WriteLine(item);
}
Console.WriteLine("================================possiblePathPrefix=======================================");
foreach (var item in possiblePathPrefix)
{
Console.WriteLine(item);
}
Console.WriteLine("=========================================================================================");


string TryFindPath(string filename)
{
foreach (var path in preferredPaths)
Expand Down Expand Up @@ -329,7 +352,7 @@ string TryFindPath(string filename)
if (!supported)
return null;

if (NativeLibrary.TryLoad(path, out var handle))
if (NativeLibrary.TryLoad(path, typeof(NativeApi).Assembly, null, out var handle))
return handle;

return null;
Expand Down
8 changes: 4 additions & 4 deletions LLama/Native/NativeLibraryConfig.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,10 +194,10 @@ internal static string AvxLevelToString(AvxLevel level)
{
return level switch
{
AvxLevel.None => string.Empty,
AvxLevel.Avx => "avx",
AvxLevel.Avx2 => "avx2",
AvxLevel.Avx512 => "avx512",
AvxLevel.None => "llama-sharp/noavx",
AvxLevel.Avx => "llama-sharp/avx",
AvxLevel.Avx2 => "llama-sharp/avx2",
AvxLevel.Avx512 => "llama-sharp/avx512",
_ => throw new ArgumentException($"Unknown AvxLevel '{level}'")
};
}
Expand Down
22 changes: 11 additions & 11 deletions LLama/runtimes/build/LLamaSharp.Backend.Cpu.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,19 @@
<files>
<file src="LLamaSharpBackend.props" target="build/netstandard2.0/LLamaSharp.Backend.Cpu.props" />

<file src="runtimes/deps/llama.dll" target="runtimes\win-x64\native\llama.dll" />
<file src="runtimes/deps/avx/llama.dll" target="runtimes\win-x64\native\avx\llama.dll" />
<file src="runtimes/deps/avx2/llama.dll" target="runtimes\win-x64\native\avx2\llama.dll" />
<file src="runtimes/deps/avx512/llama.dll" target="runtimes\win-x64\native\avx512\llama.dll" />
<file src="runtimes/deps/llama.dll" target="runtimes\win-x64\native\llama-sharp\llama.dll" />
<file src="runtimes/deps/avx/llama.dll" target="runtimes\win-x64\native\llama-sharp\avx\llama.dll" />
<file src="runtimes/deps/avx2/llama.dll" target="runtimes\win-x64\native\llama-sharp\avx2\llama.dll" />
<file src="runtimes/deps/avx512/llama.dll" target="runtimes\win-x64\native\llama-sharp\avx512\llama.dll" />

<file src="runtimes/deps/libllama.so" target="runtimes\linux-x64\native\libllama.so" />
<file src="runtimes/deps/avx/libllama.so" target="runtimes\linux-x64\native\avx\libllama.so" />
<file src="runtimes/deps/avx2/libllama.so" target="runtimes\linux-x64\native\avx2\libllama.so" />
<file src="runtimes/deps/avx512/libllama.so" target="runtimes\linux-x64\native\avx512\libllama.so" />
<file src="runtimes/deps/libllama.so" target="runtimes\linux-x64\native\llama-sharp\libllama.so" />
<file src="runtimes/deps/avx/libllama.so" target="runtimes\linux-x64\native\llama-sharp\avx\libllama.so" />
<file src="runtimes/deps/avx2/libllama.so" target="runtimes\linux-x64\native\llama-sharp\avx2\libllama.so" />
<file src="runtimes/deps/avx512/libllama.so" target="runtimes\linux-x64\native\llama-sharp\avx512\libllama.so" />

<file src="runtimes/deps/osx-x64/libllama.dylib" target="runtimes\osx-x64\native\libllama.dylib" />
<file src="runtimes/deps/osx-arm64/libllama.dylib" target="runtimes\osx-arm64\native\libllama.dylib" />
<file src="runtimes/deps/osx-arm64/ggml-metal.metal" target="runtimes\osx-arm64\native\ggml-metal.metal" />
<file src="runtimes/deps/osx-x64/libllama.dylib" target="runtimes\osx-x64\native\llama-sharp\libllama.dylib" />
<file src="runtimes/deps/osx-arm64/libllama.dylib" target="runtimes\osx-arm64\native\llama-sharp\libllama.dylib" />
<file src="runtimes/deps/osx-arm64/ggml-metal.metal" target="runtimes\osx-arm64\native\llama-sharp\ggml-metal.metal" />

<file src="icon512.png" target="icon512.png" />
</files>
Expand Down
4 changes: 2 additions & 2 deletions LLama/runtimes/build/LLamaSharp.Backend.Cuda11.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
<files>
<file src="LLamaSharpBackend.props" target="build/netstandard2.0/LLamaSharp.Backend.Cuda11.props" />

<file src="runtimes/deps/cu11.7.1/llama.dll" target="runtimes\win-x64\native\cuda11\llama.dll" />
<file src="runtimes/deps/cu11.7.1/libllama.so" target="runtimes\linux-x64\native\cuda11\libllama.so" />
<file src="runtimes/deps/cu11.7.1/llama.dll" target="runtimes\win-x64\native\llama-sharp\cuda11\llama.dll" />
<file src="runtimes/deps/cu11.7.1/libllama.so" target="runtimes\linux-x64\native\llama-sharp\cuda11\libllama.so" />

<file src="icon512.png" target="icon512.png" />
</files>
Expand Down
4 changes: 2 additions & 2 deletions LLama/runtimes/build/LLamaSharp.Backend.Cuda12.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
<files>
<file src="LLamaSharpBackend.props" target="build/netstandard2.0/LLamaSharp.Backend.Cuda12.props" />

<file src="runtimes/deps/cu12.1.0/llama.dll" target="runtimes\win-x64\native\cuda12\llama.dll" />
<file src="runtimes/deps/cu12.1.0/libllama.so" target="runtimes\linux-x64\native\cuda12\libllama.so" />
<file src="runtimes/deps/cu12.1.0/llama.dll" target="runtimes\win-x64\native\llama-sharp\cuda12\llama.dll" />
<file src="runtimes/deps/cu12.1.0/libllama.so" target="runtimes\linux-x64\native\llama-sharp\cuda12\libllama.so" />

<file src="icon512.png" target="icon512.png" />
</files>
Expand Down
6 changes: 3 additions & 3 deletions LLama/runtimes/build/LLamaSharp.Backend.OpenCL.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
<files>
<file src="LLamaSharpBackend.props" target="build/netstandard2.0/LLamaSharp.Backend.OpenCL.props" />

<file src="runtimes/deps/clblast/llama.dll" target="runtimes\win-x64\native\clblast\llama.dll" />
<file src="runtimes/deps/clblast/clblast.dll" target="runtimes\win-x64\native\clblast\clblast.dll" />
<file src="runtimes/deps/clblast/libllama.so" target="runtimes\linux-x64\native\clblast\libllama.so" />
<file src="runtimes/deps/clblast/llama.dll" target="runtimes\win-x64\native\llama-sharp\clblast\llama.dll" />
<file src="runtimes/deps/clblast/clblast.dll" target="runtimes\win-x64\native\llama-sharp\clblast\clblast.dll" />
<file src="runtimes/deps/clblast/libllama.so" target="runtimes\linux-x64\native\llama-sharp\clblast\libllama.so" />

<file src="icon512.png" target="icon512.png" />
</files>
Expand Down