ML.Net is not able to load Tensorflow GPU DLL...

[zeahmed]

I am trying work with TF GPU native dll so that I can speed up the scoring process. I replaced the tensorflow.dll in microsoft.ml.tensorflow.redist\0.5.0\runtimes\win-x64\native with gpu version but ML.Net throws DLL not found exception.

I did the same with TensorflowSharp and it loads the GPU version perfectly fine.

SOLVED

Updating To TensorFlow GPU DLL

If you want to use TF GPU dll during scoring follow the steps below

• In your ML.Net nuget, replace the tensorflow.dll in microsoft.ml.tensorflow.redist\<ML.Net-Version>\runtimes\win-x64\native with gpu version (please keep the name same).
• You can grap the prebuilt gpu dll from https://github.com/fo40225/tensorflow-windows-wheel.
• Update the graphics card drivers, CUDA and CuDNN according the Tensorflow gpu version you are using.
• TensorFlowScorer will automatically pick up the TF GPU version.

[markusweimer]

Can you please paste the full error message here?

[ericstj]

@zeahmed let me know more about what you tried?

Did you get the GPU build of TF and just try a 1:1 replacement? (same name and all) If so, can you share where you got this so I can try it out? That error sounds like an issue with either the DLL name or dependencies, rather than any issue with the bindings in ML.NET.

TF# has some API around listing devices then setting devices. I left out the API that lets you SetDevice at the moment because we weren't using it, but can bring it back.

[zeahmed]

Here is the gpu dll I am using: https://github.com/fo40225/tensorflow-windows-wheel/blob/master/1.9.0/cpp/libtensorflow-gpu-windows-x86_64-1.9.0-avx2cuda92cudnn71.7z

Yes, I did the 1:1 replacement both in TF# and ML.Net (same name and same location).

Now I got it working. Actually, the problem was graphics card driver. I updated it through windows driver update and it worked for sometime with TF# but after restarting windows it showed me the same problem as ML.Net.

I updated my driver to latest from Nvidia site and also updated the CUDA to 9.2 and CuDNN to 7.1 as a requirement to the above dll. It's being loaded correctly now in ML.Net though I am hitting some system specific issues on my end.

@markusweimer and @ericstj, I have added notes in the original issue to elaborate how to work with Tensorflow gpu dll.

[ericstj]

Taking a look at your steps: I wouldn't recommend that folks update the nuget package. The Nuget package cache is shared between all projects. Lots of parts of the build treat it as immutable.

Instead, to use the GPU binary, you can exclude the native assets from the Microsoft.ML.Tensorflow package reference

<PackageReference Include="Microsoft.ML.TensorFlow" ExcludeAssets="native" />

This will tell NuGet not to include the native tensorflow binaries (it applies transitively to the dependencies, and will thus apply to TensorFlow.Redist package).

Then they can "bring their own" copy of tensorflow however they choose. Simply referencing it as content is one way. If we want to officially support this scenario we could consider a packaging story for the GPU build (as well as an official way for someone to choose to use this).

[zeahmed]

@ericstj, totally agree! This was just a work around for my case.

For now, ExcludeAssets="native" option sounds good me. Otherwise, there are lots dependencies for GPU dll based on how it was built e.g. CUDA version, CuDNN version, driver version etc.

[zeahmed]

Tensorflow windows GPU dll is now officially available at https://www.tensorflow.org/install/lang_c

[yaeldekel]

@zeahmed, can this issue be closed?

[zeahmed]

Yes, it can be closed as there is no work item for this right now.

[rossbuggins]

Hi - could you please clarify what now needs doing to enable GPU tensorflow with ML now that

Tensorflow windows GPU dll is now officially available at https://www.tensorflow.org/install/lang_c

I have replaced the tensorflow.dll with the one in the download direct from tensorflow.org, but am getting:

Unhandled exception. System.FormatException: Tensorflow exception triggered while loading model.
---> System.DllNotFoundException: Unable to load DLL 'tensorflow' or one of its dependencies: The specified module could not be found. (0x8007007E)
at Tensorflow.c_api.TF_NewGraph()
at Tensorflow.Graph..ctor()
at Microsoft.ML.Transforms.Dnn.DnnUtils.LoadTFSessionByModelFilePath(IExceptionContext ectx, String modelFile, Boolean metaGraph)
--- End of inner exception stack trace ---
at Microsoft.ML.Transforms.Dnn.DnnUtils.LoadTFSessionByModelFilePath(IExceptionContext ectx, String modelFile, Boolean metaGraph)
at Microsoft.ML.Transforms.Dnn.DnnUtils.GetSession(IHostEnvironment env, String modelPath, Boolean metaGraph)
at Microsoft.ML.Transforms.TensorFlow.TensorFlowUtils.LoadTensorFlowModel(IHostEnvironment env, String modelPath)
at Microsoft.ML.TensorflowCatalog.LoadTensorFlowModel(ModelOperationsCatalog catalog, String modelLocation)
at Classify.Program.GenerateModel(MLContext mlContext) in C:\Users\a\source\repos\Classify\Classify\Program.cs:line 144
at Classify.Program.RunPred() in C:\Users\a\source\repos\Classify\Classify\Program.cs:line 90
at Classify.Program.Main(String[] args) in C:\Users\a\source\repos\Classify\Classify\Program.cs:line 33