feat: V3 tensorflow plugin

[mrousavy]

What

TensorFlow Lite Plugin to drop any .tflite model in the JS bundle (src/assets) and just run it in VisionCamera. No native code at all.

This is gonna be a game changer to the mobile camera industry

const model = useTensorflowModel(require('../assets/model.tflite'));

const frameProcessor = useFrameProcessor(
  (frame) => {
    'worklet';
    const results = plugin.model.run(frame);
    console.log(results[0]);
  },
  [plugin.model],
);

The model can be a face detector, object detector, golf swing detector, whatever .tflite model you train. Possibilities are endless.

Example

function App() {
  const plugin = useTensorflowModel(require('../assets/object-detector.tflite'));

  const paint = Skia.Paint();
  paint.setStyle(PaintStyle.Stroke);
  paint.setStrokeWidth(20);
  paint.setColor(Skia.Color('red'));

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      // runs the model on the GPU. The input frame is automatically resized
      const [boundingBoxes, classes, scores, count] = plugin.model.run(frame);

      for (let i = 0; i < scores.length; i++) {
        const confidence = scores[i];
        if (confidence > 0.4) {
          // we're more than 40% confident that this is an object - draw it!
          const top = boundingBoxes[i];
          const left = boundingBoxes[i + 1];
          const bottom = boundingBoxes[i + 2];
          const right = boundingBoxes[i + 3];

          // draws a red box to the screen using Skia - drawn on the GPU!
          frame.drawRect(
            Skia.XYWHRect(left, top, right - left, bottom - top),
            paint,
          );
        }
      }
    },
    [paint, plugin.model],
  );
  
  return <Camera frameProcessor={frameProcessor} />
}

RPReplay_Final1688135872.mov

Changes

Tested on

Related issues

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 13, 2023 0:49am

[philip-bui-goldenowl]

can you help me convert an image with buffer, thanks you @mrousavy @mrousavy

[eledahl]

@mrousavy cool that helped get through that part. Seems to run the model now. Brings up a fun new issue. The camera goes down to like .5 fps running the model. We are using an unofficial tflite blazepose model which might be the cause.

Running an official posenet model gives us just raw data when logged but performance is much improved.

{"0": -2.7642037868499756, "1": 5.746579647064209, "10": 27.31992530822754, "100": 61.20494842529297, "1000": 0, "1001": 0, "1002": 0, "1003": 0, "1004": 0, "1005": 0, "1006": 0, "1007": 0, "1008": 0, "1009": 0, "101": 66.36915588378906, "1010": 0, "1011": 0, "1012": 0, "1013": 0, "1014": 0, "1015": 0, "1016": 0, "1017": 0, "1018": 0, "1019": 0, "102": 7.78696870803833, "1020": 0, "1021": 0, "1022": 0, "1023": 0, "1024": 0, "1025": 0, "1026": 0, "1027": 0, "1028": 0, "1029": 0, "103": 128.09030151367188, "1030": 0, "1031": 0, "1032": 0, "1033": 0, "1034": 0, "1035": 0, "1036": 0, "1037": 0, "1038": 0, "1039": 0, "104": 27.506017684936523, "1040": 0, "1041": 0, "1042": 0

Pose detection is a pain. When I was using the tfjs packages, it was a few steps to running the detection and estimating the poses so I wonder if that is why this is the case here. Did you by chance happen to try any pose detection when building the plugin?

Quick update:

Found an official mediapipe pose detection model. Does the really slow camera feed as well.

[philip-bui-goldenowl]

hi @eledahl you have a solution for detecting pose net ?

[eledahl]

@philip-bui-goldenowl No sir. The pose models I am using are slowing the cameras down or not outputting the keypoints as I intended. I may be using it wrong but that's the point I am at currently.

[philip-bui-goldenowl]

@eledahl I have the same problem, I convert the frame to CGImage but the context CGBitmapContextCreateImage: invalid context 0x0. Backtrace, not get keypoints

[mrousavy]

Hey @eledahl, can you share your .tflite model? if the model itself is slow there's nothing I can do about that.

I can check if the input parsing and output conversions are throttling.

Also, if you want to run something async, VisionCamera V3 introduced a new func: runAsync :) That way the Camera isn't throttled

[eledahl]

@mrousavy Sure yeah here's the few that we were trying to use. Github doesn't like the file type so let me know if you have any issues downloading them from the drive:

Mediapipe blaze

Unofficial blaze

Posenet (gave the raw data)

Oh, that's fancy about the runasync. We'll give that a shot today

[dguari1]

@mrousavy Thank you for working on this; V3 is definitively going to be game-changing.

@eledahl, your pose model is not working for me, but this model is working. I'm getting between 55 and 60 fps!
Using the int8 version, I get a solid 60 fps!

I tried to run it using runAsync, but it skipped the code inside runAsync. Is there anything more that I should do?

Here is a snippet of my code running the model and plotting the nose landmark on the frame if anyone wants to use it

const model = require('./assets/lite-model_movenet_singlepose_lightning_tflite_float16_4.tflite')
const plugin = useTensorflowModel(model);
if (plugin.state === 'loaded') console.log(JSON.stringify(plugin.model, null, 2));

const paint = Skia.Paint();
paint.setStyle(PaintStyle.Stroke);
paint.setStrokeWidth(20);
paint.setColor(Skia.Color('green'));

const inputShape = plugin.model?.inputs[0]?.shape;
const inputFrameSize = {
    width: inputShape?.[1] ?? 0,
    height: inputShape?.[2] ?? 0,
  };

 const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';
      const addition = (frame.height - frame.width)/2
      const scaler = frame.width
      if (plugin.state === 'loaded') {

        const output = plugin.model.run(frame)
        // console.log(output[0][1]*scaler, output[0][0]*scaler + addition)
        frame.drawCircle(output[0][1]*scaler, output[0][0]*scaler + addition, 5,paint)
       } else {
        console.log(`Model state: ${plugin.state}..`);
      }
    },
    [inputFrameSize.height, paint, plugin.model, plugin.state],
  );

@mrousavy two questions:

• What happened to frameProcessorFps. It was a very handy addition to the frame processor; it would be great to have it available.
• Is there any way to adjust the method to convert the input image to a square? Currently, the plugin ignores the top and bottom of the image (for portrait). Would it be possible to add padding as an alternative?

[eledahl]

@dguari1 Ah thanks man. Your code seems to have fixed it for my model. It looks like my model does run if I only do the smaller console.log you did rather than logging the entire output. Do you happen to know how to extract the keypoints from the pose models this way?

@mrousavy Let me know if these questions get too specific for my use case. I am happy to ask them elsewhere if that is better

[mrousavy]

@dguari1 nice to hear you got it running at 60 FPS, this is amazing!!!!

• frameProcessorFps is gone, you can now use runAtTargetFps to throttle something - I think that's a cleaner and more controllable API :)
• Good point on padding, maybe I can add a prop to let the user decide between center-crop and center-scale?
• runAsync should work, that's weird.

[dguari1]

@mrousavy frameProcessorFps works like a charm.
I think it would be great to have more control over the frame pre-processing; choosing between center crop or padding, normalization types ([0,1], Imagenet Normalization, ...), and data types.

Another issue is the data type. The TFLite Movenet works fine, this model requires unit8 input

inputs": [
    {
      "name": "serving_default_input:0",
      "dataType": "uint8",
      "shape": [
        1,
        192,
        192,
        3
      ]
    }
  ],

However, the Mediapipe Pose (all their models are here), does not work, as this model needs float32 input and the frame data type is uint8

"inputs": [
    {
      "name": "input_1",
      "dataType": "float32",
      "shape": [
        1,
        256,
        256,
        3
      ]
    }
  ],

Right now, the output of the MediaPipe pose model is just random numbers. It doesn't process the image.

I made a short Python script to test these models, figure out what they do, and plot the results! It is outside this forum's scope, so I won't post it here.

@eledahl login the data in the terminal takes a lot of time; the MediaPipe Pose model returns a lot of data (it even returns the input image and the heatmaps)

  "outputs": [
    {
      "name": "Identity",
      "dataType": "float32",
      "shape": [
        1,
        195
      ]
    },
    {
      "name": "Identity_1",
      "dataType": "float32",
      "shape": [
        1,
        1
      ]
    },
    {
      "name": "Identity_2",
      "dataType": "float32",
      "shape": [
        1,
        256,
        256,
        1
      ]
    },
    {
      "name": "Identity_3",
      "dataType": "float32",
      "shape": [
        1,
        64,
        64,
        39
      ]
    },
    {
      "name": "Identity_4",
      "dataType": "float32",
      "shape": [
        1,
        117
      ]
    }
  ],

the pose should be the first component of that output. There should be 39 landmarks; you can get with something like this

for (let i = 0; i < output[0].length; i+=5) {
                const cx = output[0][i] * scaler 
                const cy = output[0][i+1] * scaler  + addition
               console.log(`landmark_x = ${cx}, landmark_y = ${cy}`)
              }

but the model doesn't work at the moment due to the data type required.

[mrousavy]

If you can convert the model to uint8_t input that'd be better for performance as well. Converting to float32_t is a bit difficult and not sure how to even test that.

[pvedula7]

I'm not sure if its reasonable to assume most models would have a uint8 input. How difficult would it be to convert to float32_t on the native side? We can help test with blazepose and other mediapipe models.

@dguari1
Could you share the python script to describing TFLite models?

[dguari1]

The issue with input data type and normalization is really annoying. Some models need uint8, others float32, also some models need the data to be normalized, and others don't.

@mrousavy I have been looking for ways to adjust the input data type, but I don't think that is possible without re-training the model, and these models cannot be re-trained as Google doesn't share the data.

@pvedula7 the tflite models can be analyzed in Python using something like this:

# Import TF and TF Hub libraries.
import tensorflow as tf
import matplotlib.pyplot as plt
from PIL import Image, ImageOps

image_path = "PathToYourImage"
im = Image.open(image_path)
im = ImageOps.exif_transpose(im)

# Load the input image.

image = tf.convert_to_tensor(im)
# image = tf.compat.v1.image.decode_jpeg(image)
image = tf.expand_dims(image, axis=0)


interpreter = tf.lite.Interpreter(model_path="PathToYourModel.tflite")
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()


# Resize and pad the image to keep the aspect ratio and fit the expected size.
ExpectedSize = input_details[0]['shape'][2]
imageResized = tf.image.resize_with_pad(image, ExpectedSize, ExpectedSize , antialias=True)

# Adjuste the data type 
ExpectedType = input_details[0]['dtype']
input_image = tf.cast(imageResized, dtype=ExpectedType)

interpreter.set_tensor(input_details[0]['index'], input_image.numpy())
interpreter.invoke()
modelOutput = interpreter.get_tensor(output_details[0]['index'])

[timothygorer]

I am trying to use a text detection model, but I am getting 1 fps unfortunately. This is the model I am trying to use: https://tfhub.dev/sayakpaul/lite-model/east-text-detector/fp16/1 which is directly from the tensorflow example (https://www.tensorflow.org/lite/examples/optical_character_recognition/overview). I've also tried using both float32 and int8 versions of the model but still get 1 fps. Is there something about text models that makes them break when used in the frame processor?

[dguari1]

I am trying to use a text detection model, but I am getting 1 fps unfortunately. This is the model I am trying to use: https://tfhub.dev/sayakpaul/lite-model/east-text-detector/fp16/1 which is directly from the tensorflow example (https://www.tensorflow.org/lite/examples/optical_character_recognition/overview). I've also tried using both float32 and int8 versions of the model but still get 1 fps. Is there something about text models that makes them break when used in the frame processor?

@timothygorer, your model's input type is float32; it would not work properly with the current implementation of the plugin.

[{'name': 'input_images',
  'index': 0,
  'shape': array([  1, 320, 320,   3], dtype=int32),
  'shape_signature': array([  1, 320, 320,   3], dtype=int32),
  'dtype': numpy.float32,
  'quantization': (0.0, 0),
  'quantization_parameters': {'scales': array([], dtype=float32),
   'zero_points': array([], dtype=int32),
   'quantized_dimension': 0},
  'sparsity_parameters': {}}]

Also, your model is huge, so it is probably pretty slow.
Your model takes 220 ms to run on my computer using Python. As a reference, the small pose detection model that I'm using takes 9 ms to run on my computer using Python, and that pose detection model runs at 40 fps using the plugin.

[erwanlpfr]

Hello ! I comment here, since it looks the best place to do so,

I'm actually trying to run models with an output set to float32.
So, every value in are unreadable (negative values).
But According to your implementation, it looks like you care about the float32 or higher. Am I wrong?

Is there any way to handle this?

Models with input/output higher than float16 are common and include mediapipe models, for example.
With int8 we are limited to a few models and since some models are datasetless we can't retrain or quantized them.

The rest looks really stable, smooth and reliable. Awesome !

Thank you !

[mrousavy]

Hey - all the models I worked on are uint8 or int8 - I tried to add some basic float32 conversion but I didn't really test it since that's quite tricky to do. Didn't want to spend more time on it since all the models I had were uint8/int8 :)

[erwanlpfr]

Hey - all the models I worked on are uint8 or int8 - I tried to add some basic float32 conversion but I didn't really test it since that's quite tricky to do. Didn't want to spend more time on it since all the models I had were uint8/int8 :)

Thank you for your answer and your work!
I imagine the work it can be !