MediaPipe Unity Plugin

This is a Unity (2020.3.30f1) Native Plugin to use MediaPipe (0.8.9).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

• Write MediaPipe code in C#.
• Run MediaPipe's official solution on Unity.
• Run your custom Calculator and CalculatorGraph on Unity.
  • ⚠️ Depending on the type of input/output, you may need to write C++ code.

😸 Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        var graph = new CalculatorGraph(_ConfigText);
        var poller = graph.AddOutputStreamPoller<string>("out").Value();
        graph.StartRun().AssertOk();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", new StringPacket("Hello World!", new Timestamp(i))).AssertOk();
        }

        graph.CloseInputStream("in");
        var packet = new StringPacket();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone().AssertOk();
    }
}

🛠️ Installation

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc), so you need to build them first.
For step-by-step guide, please refer to the Installation Guide in Wiki.

⚠️ libraries that can be built differ depending on your environment.

Supported Platforms

⚠️ GPU mode is not supported on macOS and Windows.

	Editor	Linux (x86_64)	macOS (x86_64)	macOS (ARM64)	Windows (x86_64)	Android	iOS	WebGL
Linux (AMD64) 1	✔️	✔️				✔️
Intel Mac	✔️		✔️			✔️	✔️
M1 Mac 2	✔️			✔️		✔️	✔️
Windows 10 (AMD64) 34	✔️	✔️			✔️	✔️

🍽️ Try sample app

Example Solutions

Here is a list of solutions that you can try in the sample app.

🔔 The graphs you can run are not limited to the ones in this list.

	Android	iOS	Linux (GPU)	Linux (CPU)	macOS (CPU)	Windows (CPU)	WebGL
Face Detection	✔️	✔️	✔️	✔️	✔️	✔️
Face Mesh	✔️	✔️	✔️	✔️	✔️	✔️
Iris	✔️	✔️	✔️	✔️	✔️	✔️
Hands	✔️	✔️	✔️	✔️	✔️	✔️
Pose	✔️	✔️	✔️	✔️	✔️	✔️
Holistic	✔️	✔️	✔️	✔️	✔️	✔️
Selfie Segmentation
Hair Segmentation	✔️	✔️	✔️	✔️	✔️	✔️
Object Detection	✔️	✔️	✔️	✔️	✔️	✔️
Box Tracking	✔️	✔️	✔️	✔️	✔️	✔️
Instant Motion Tracking	✔️	✔️	✔️	✔️	✔️	✔️
Objectron	✔️	✔️	✔️	✔️	✔️	✔️
KNIFT

UnityEditor

Select Mediapipe/Samples/Scenes/Start Scene and play.

Desktop

If you've built native libraries for CPU (i.e. --desktop cpu), select CPU for inference mode from the Inspector Window.

Android, iOS

Make sure that you select GPU for inference mode before building the app, because CPU inference mode is not supported currently.

📖 Wiki

🚧 https://github.com/homuler/MediaPipeUnityPlugin/wiki

📜 LICENSE

MIT

Note that some files are distributed under other licenses.

• MediaPipe (Apache Licence 2.0
• emscripten (MIT)
• FontAwesome (LICENSE)

Footnotes

1. Tested on Arch Linux. ↩

2. Experimental, because MediaPipe does not support M1 Mac. ↩

3. Windows 11 will be also OK. ↩

4. Running MediaPipe on Windows is experimental. ↩