Outdated bindings to Google ML Kit Vision Text Regognition api

[deelw]

Because we’re using ML Kit for Firebase (https://firebase.google.com/docs/ml-kit) for on-device text recognition and this is an old version, we need to migrate to the actual version of ML Kit (https://developers.google.com/ml-kit/vision/text-recognition ).

As Dutch government, we develop our apps using Xamarin and we need to migrate as well our Xamarin.iOS as our Xamarin.Android implementation.

For Xamarin.iOS we’re using nuget package Xamarin.Firebase.iOS.MLKit.Vision version 0.21.0 from Microsoft while for Xamarin.Android we’re using nuget package Xamarin.GooglePlayServices.Vision version 120.1.3, also from Microsoft. These packages contain C# bindings to ML Kit for Firebase.

Surprisingly there seems to be no Xamarin nuget packages available for the current GoogleMLKit/TextRecognition (https://developers.google.com/ml-kit/vision/text-recognition). Neither for Xamarin.iOS nor for Xamarin.Android.

We did try to make bindings ourselves. While we did succeed in making and invoking these bindings, text is never recognized. In the console output the MLKit framework did write the following messages:

• [lvl=3] +[MLKITx_CCTClearcutUploader crashIfNecessary] Multiple instances of CCTClearcutUploader were instantiated. Multiple uploaders function correctly but have an adverse affect on battery performance due to lock contention.
• Failed to locate required resources. Please verify the resource bundle is imported.
• [lvl=3] -[MLKITx_GIPPseudonymousIDStore initializeStorage] Shared App Groups unavailable

We ask you to make nuget packages available for the current versions of Google MLKit, specifically for cocoapod GoogleMLKit/TextRecognition and maven repository com.google.android.gms:play-services-mlkit-text-recognition.

Related issue: xamarin/GooglePlayServicesComponents#420

[deelw]

@SotoiGhost you've helped me a year ago with bindings for iOS. I heard the Components team was dissolved. Could you please help us by either making new Google ML Kit Vision TextRecognition (no Firebase) api's available or support us in making our own bindings work?

As you can read in the issue text, we successfully generated bindings. With our bindings we can invoke the MLKTextRecognizers ProcessImage and ResultsInImage methods. That the framework is successfully invoked is confirmed by the fact that the MLKTextRecognitionCallback of the ProcessImage method is called, but text is never recognized while the error object is null and an instance of the MLKText object is returned. We've tried all orientation flags. In the application console there are three messages (mentioned in the initial issue text above).

Could you or an experienced colleague help us please? Thank you very much in advance!

Here some details for your information:

• I made a podfile (see under) because sharpie pod init 'GoogleMLKit/TextRecognition' resulted in a pod file using pod 'GoogleMLKit'
• I used pod update instead of sharpie because sharpie has no option to use the existing podfile
• after installing cocoapods I needed to call pod repo update to be able to successfully call the pod update command
• Before calling sharpie pod bind I had to unselect Mac for each project target (General tab -> deployment info) in the XCode Pods project (./Pods/Pods.xcodeproj)
• In my Xamarin Bindings library I only included MLKitTextRecognition_ApiDefinitions.cs and MLKitVision_ApiDefinitions.cs and I added a native reference to all the framework files (from the ./Bindings folder).
• In my iOS project I had to set the linker behavior to don't link

Podfile

platform   :ios, '14.3'
--
install! 'cocoapods',   :integrate_targets => false
target   'ObjectiveSharpieIntegration' do
use_frameworks!
 
pod   'GoogleMLKit/TextRecognition'
end

[deelw]

@mattleibow could you please help me with this issue? It's enough for me when I have local working bindings (only text recognition), while it would be very nice when the bindings would be globally available as a nuget package.

[deelw]

Here a solved issue i found with the same error, maybe this helps solving the cause?
#382

[mavispuford]

@deelw Were you ever able to get your bindings to work? I'm just starting down that path. Thanks for detailing some of your process in the above comment, by the way.

[deelw]

@mavispuford yes, my bindings did work but without successfull text recognition (the ocr result callback function was called, so the interaction was successfull). However, reproducing the steps now results in a xamarin iOS project with linker errors.

In addition to the above information, I added namespaces to the _ApiDefinitions.cs files and changed the build action to ObjcBindingApiDefinition

Here is my ViewDidLoad implementation with the testing code. Of course I added an bundleresource called identitydocument.png. The object mlkText did have an instance, but the Text property and the Blocks were empty.

    public override void ViewDidLoad()
    {
        base.ViewDidLoad();
        // Perform any additional setup after loading the view, typically from a nib.
        var textRecognizer = MLKTextRecognizer.TextRecognizer();

        var uiImage = UIImage.FromBundle("identitydocument.png");
        if (uiImage == null)
        {
            Console.WriteLine("uiImage IS NULL");
            return;
        }

        var mlkImage = new MLKVisionImage(uiImage);
        if (mlkImage == null)
        {
            Console.WriteLine("mlkImage IS NULL");
            return;
        }

        textRecognizer.ProcessImage(mlkImage, (MLKText mlkText, NSError error) =>
        {
            Console.WriteLine("ProcessImage callback");
            if (mlkText != null)
            {
                Console.WriteLine("mlkText NOT NULL");
                Console.WriteLine($"mlkText.Text {mlkText.Text}");
                Console.WriteLine($"mlkText.Blocks.Length {mlkText.Blocks.Length}");
            }

            if (error != null)
            {
                Console.WriteLine("error NOT NULL");
                Console.WriteLine($"error: {error.DebugDescription}");
            }
        });
        Console.WriteLine("textRecognizer.ProcessImage");
    }

[mavispuford]

@deelw I was able to get iOS bindings working with successful text recognition. My guess is that you were missing the necessary resources in your bindings project that are needed for ML Kit (machine learning models etc.). If you take a look in the MLKitTextRecognition pod folder, you'll see them:

This targets file for the old Firebase ML Kit Vision bindings was a great reference for me, because I later had linking errors in Release mode. Making sure the Frameworks were set up correctly in the bindings project resolved them.

Here's what my csproj file looks like.. The two ApiDefinitions.cs files were generated using the following Sharpie commands:

sharpie bind -o MLKitVisionBindings -sdk iphoneos14.4 -n MLKitVision -framework ./MLKitVision

sharpie bind -o MLKitTextRecognitionBindings -sdk iphoneos14.4 -n MLKitTextRecognition -framework ./MLKitTextRecognition

[deelw]

@mavispuford Thank you very much! I'll try it as soon as possible. I'll let you know

[deelw]

@mavispuford
Thank you for your clear explanation. Because sharpie pod bind fails with the latest ios sdk 14.5, I'm trying to do it exactly your way.

For the Dependency/PodBuilder/Prebuilt contents I tried to use PodBuilder like you did. I did make a XCode iOS app project and workspace only because the PodBuilder requires them. Copied my podfile into the project directory. I was able to run pod_builder init. But pod_builder build_all did fail. Can you tell me how you did that?

Thanks for your help!

[mavispuford]

@deelw

Here are the steps that I took for PodBuilder (but note that I ended up dropping it and simplified my bindings project quite a bit - I'll explain below):

• I created an empty XCode project with my podfile referencing the 'GoogleMLKit/TextRecognition' pod
• I ran pod_builder init to create the PodBuilder folder
• Ran pod_builder build_all to build all pods
• The resulting frameworks were located in PodBuilder/Prebuilt
• Pod-Builder skips any pods that already have frameworks. Those pods are in their normal location in the Pods directory.

So I mentioned above that I stopped using PodBuilder to build my fat binaries. The reason is that even though I had success building my bindings DLL and popping it in a simple Xamarin app, it caused problems with our other app because it was referencing Firebase for analytics/push notifications and the two had conflicts with their native dependencies. I looked around and noticed that the majority of the native dependencies are also in Firebase.Core. So in my bindings project, I removed most of the native references and referenced the Firebase.Core NuGet package instead. This works great for us and doesn't have any conflicts with our other packages references anymore.

Here's what the bindings csproj file looks like now.

[deelw]

@mavispuford
Yes, it's working! Thanks you very much!

I have one question about your latest solution: How did you prevent the Firebase.Core nuget package from being stripped because it's not explicitly referenced from code?
I did so by including a static class with a private static Action reference = Firebase.Core.App.Configure; but maybe you did use a more elegant way?

[mavispuford]

@deelw That's great! I'm glad to hear it.

For Android, we have an r8.cfg to prevent linking with entries like this:

-keep class com.google.firebase.** { *; }
-keep class androidx.camera.core.** { *; }
-keep class com.google.mlkit.vision.** { *; }

As for iOS, I don't think I did anything special to prevent it from being linked out. We were actually already using Firebase.Core in our app, since we use Firebase for push notifications and analytics. Since you're not using it anywhere, you may have to add a Linker.xml entry or just keep that reference around.

[deelw]

Thanks for the info!

[albertruff]

One question: How you defined in MLKitVision_ApiDefinitions.cs the GMLImage_MLKitCompatible Extension/Category, that it extends an Interface?

At the Moment, i create Bindings for the new MLKitBarcodeScanning funktionality. For the MLKitBarcodeScanning.framwork, I successfuly created Bindings. But I stuck At this Point in MLKitVision.

P. S.: Thanks @deelw and @mavispuford for your discussion, it helped me a lot.