Support 'tensorflow::MemmappedEnv' for memory mapped graphs

[feliwir]

Hey, i was trying to load the mozilla deepspeech model, which are usually provided as memory mapped graphs, see Mozilla DeepSpeech as an example:
https://github.com/mozilla/DeepSpeech/releases/tag/v0.9.3

They're loading the file in their native api like this:
https://stackoverflow.com/questions/43533264/how-to-read-tensorflow-memory-mapped-graph-file-in-android

However i couldn't find tensorflow::MemmappedEnv anywhere. How should i load such a model? My current code only works for non-memory mapped files (which i don't have for Mozilla DeepSpeech):

// This only works for regular non-mapped Graphs)
using (var codedInput = new CodedInputStream(File.OpenRead(modelPath)))
{
  var graphDef = new GraphDef();
  codedInput.ReadMessage(graphDef);
}

Thanks in advance!

[Oceania2018]

@feliwir Are you using Xamaria mobile?
We're implementing the Lite (mobile) version in this tflite branch.
Feel free to contribute.

[feliwir]

@Oceania2018 I guess you mean "Xamarin Mobile"? No i'm using desktop .NET 5. I'm currently trying to add the memory mapped filesystem, once i get it working i'll update here

[Oceania2018]

Yes, currently tf.net doesn't support tflite model. Not sure whether it works or not in the way of codedInput.ReadMessage(graphDef);
There is another way to load graph:

var graph = new Graph().as_default();
graph.Import(Path.Join(modelDir, pbFile));

[feliwir]

@Oceania2018 the problem is not the regular tflite model, but the memmapped filesystem:
https://github.com/tensorflow/tensorflow/blob/5dcfc51118817f27fad5246812d83e5dccdc5f72/tensorflow/core/util/memmapped_file_system.cc#L209
It's the file wrapped in another file

[Oceania2018]

The GraphDef is converted to MemmappedFileSystem, and you want to load this format in your application, correct?

[feliwir]

The GraphDef is converted to MemmappedFileSystem, and you want to load this format in your application, correct?

Exactly. I think the extension .pbmm is used to clarify clarify this. I tried implementing this in C# like this:

using System;
using System.IO;
using System.IO.MemoryMappedFiles;
using Google.Protobuf;
using Tensorflow;

namespace Jarvis.TFUtil
{
    public class MemmappedFileSystem
    {
        private MemoryMappedFile _file;
        public MemmappedFileSystem(string path)
        {
            // _file = MemoryMappedFile.CreateFromFile(path, FileMode.Open);

            using (var stream = File.OpenRead(path))
            {
                // Read the offset for the directory
                var offsetData = new byte[8];
                stream.Seek(-8, SeekOrigin.End);
                stream.Read(offsetData, 0, 8);
                var offset = BitConverter.ToUInt64(offsetData, 0);
                Console.WriteLine(offset);
                stream.Seek((long) offset, SeekOrigin.Begin);

                Console.WriteLine("Length: {0}, DirOffset: {1}", stream.Length, offset);

                while(stream.Position < stream.Length - 8)
                {
                    // This is an entry inside the MemMappedFS
                    Console.WriteLine("Reading: {0}",stream.Position);
                    var dir = MemmappedFileSystemDirectory.Parser.ParseFrom(stream);
                    
                }

            }
        }
    }
}

Using the .proto file from tensorflow:

/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/
syntax = "proto3";

package tensorflow;

option cc_enable_arenas = true;

// A message that describes one region of memmapped file.
message MemmappedFileSystemDirectoryElement {
  uint64 offset = 1;
  string name = 2;
  uint64 length = 3;
}

// A directory of regions in a memmapped file.
message MemmappedFileSystemDirectory {
  repeated MemmappedFileSystemDirectoryElement element = 1;
}

However i probably misunderstood parts of the C++ code. since this crashes on me :(

[feliwir]

Yes, currently tf.net doesn't support tflite model. Not sure whether it works or not in the way of codedInput.ReadMessage(graphDef);
There is another way to load graph:

var graph = new Graph().as_default();
graph.Import(Path.Join(modelDir, pbFile));

This does not work with Stream objects. I can't use a path, since the file is only virtual, so i have to use protobuf directly or not? Also the function ImportGraphDef does not accept a GraphDef object :-/

[feliwir]

I got it working like this now:
(MemmappedFileSystem)

using System;
using System.IO;
using System.IO.MemoryMappedFiles;
using System.Linq;
using Tensorflow;

namespace Jarvis.TFUtil
{
    public class MemmappedFileSystem
    {
        public const string MEMMAPPED_PACKAGE_DEFAULT_NAME = "memmapped_package://.";

        private MemoryMappedFile _mmapFile;
        private MemmappedFileSystemDirectory _directory;

        public MemmappedFileSystem(string path)
        {
            using (var stream = File.OpenRead(path))
            {
                // Read the offset for the directory
                var offsetData = new byte[sizeof(ulong)];
                stream.Seek(-sizeof(ulong), SeekOrigin.End);
                stream.Read(offsetData, 0, sizeof(ulong));
                var offset = BitConverter.ToUInt64(offsetData, 0);

                var dirLength = stream.Length - (long) offset - sizeof(ulong);
                if (dirLength < 0)
                {
                    throw new InvalidDataException("Malformed mmapped filesystem!");
                }

                var dirData = new byte[dirLength];

                Console.WriteLine("Length: {0}, DirOffset: {1} DirLength: {2}", stream.Length, offset, dirLength);


                stream.Seek((long) offset, SeekOrigin.Begin);
                stream.Read(dirData, 0, (int) dirLength);

                _directory = MemmappedFileSystemDirectory.Parser.ParseFrom(dirData);

                foreach (var element in _directory.Element)
                {
                    Console.WriteLine("Found mmapped file: {0}, {1})", element.Name, element.Length);
                }
            }

            _mmapFile = MemoryMappedFile.CreateFromFile(path, FileMode.Open);
        }

        public Stream OpenMemmapped(string filename)
        {
            var entry = _directory.Element.FirstOrDefault(x => x.Name == filename);
            if (entry == null)
            {
                throw new FileNotFoundException($"Missing memmaped file entry: {filename}");
            }

            return _mmapFile.CreateViewStream((long) entry.Offset, (long) entry.Length);
        }
    }
}

And then the actual usage:

if (Path.GetExtension(modelPath) == ".pbmm")
            {
                var fs = new MemmappedFileSystem(modelPath);
                using (var stream = fs.OpenMemmapped(MemmappedFileSystem.MEMMAPPED_PACKAGE_DEFAULT_NAME))
                {
                    var buf = new byte[stream.Length];
                    stream.Read(buf, 0, (int)stream.Length);
                    var graphDef = GraphDef.Parser.ParseFrom(buf);
                }
            }
            else
            {
                var graph = new Graph().as_default();
                graph.Import(modelPath));
            }

As you can see i'm now kind of stuck with the GraphDef object which is kind of useless

[Oceania2018]

I add this MemmappedFileSystem proto in this commit.

[feliwir]

@Oceania2018 great thanks! One last question: How can i fetch metadata from my model like here:
https://github.com/mozilla/DeepSpeech/blob/master/native_client/tfmodelstate.cc#L101

I tried using the FeedItem constructor, but with no real success yet :-/

[feliwir]

@Oceania2018 i think i'm using the API correctly like this:

var graph = tf.Graph().as_default();

if (Path.GetExtension(modelPath) == ".pbmm")
{
    var fs = new MemmappedFileSystem(modelPath);
    using (var stream = fs.OpenMemmapped(MemmappedFileSystem.MEMMAPPED_PACKAGE_DEFAULT_NAME))
    {
        var buf = new byte[stream.Length];
        stream.Read(buf, 0, (int) stream.Length);
        if (!graph.Import(buf))
        {
            _logger.Error("Failed to import graph");
            return false;
        }
    }
}
else
{
    if (!graph.Import(modelPath))
    {
        _logger.Error("Failed to import graph");
        return false;
    }
}

_tf_session = tf.Session(graph);
var init = tf.global_variables_initializer();

_tf_session.run(init);
//This crashes with the callstack below
var result = _tf_session.run(new Tensor("metadata_version"));

However this results in an exception:

An exception of type 'System.NullReferenceException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'Object reference not set to an instance of an object.'
   at Tensorflow.Graph._as_graph_element_locked(Object obj, Boolean allow_tensor, Boolean allow_operation)
   at Tensorflow._ElementFetchMapper..ctor(Object[] fetches, Func`2 contraction_fn, Graph graph)
   at Tensorflow._FetchMapper.for_fetch(Object fetch, Graph graph)
   at Tensorflow._FetchHandler..ctor(Graph graph, Object fetches, Dictionary`2 feeds, Action feed_handles)
   at Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict)
   at Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict)
   at Jarvis.Speech.STT.DeepSpeech.<Setup>d__18.MoveNext() in /home/mbits/Development/jarvis/src/Jarvis.Speech/STT/DeepSpeech.cs:line 127

I'm not quite sure which element here should be null exactly, since the graph is not null and the tensor is not null either. Maybe there's a bug with the fetching API?