Porting ViSP to android #307

akshay326 · 2018-03-12T07:05:51Z

As per as previous conversation #304, I've learnt how to build a C++ package for android platform using android.toolchain.cmake and I'm on my way to build OpenCV for Android(having some 3rd party library issues, you can get the source code here).

Meanwhile I began adding android build flags to ViSP's cmake files. As you said, I'm referring OpenCV's cmake files.

There are few ViSP libraries for which there's no corresponding cmake file in the OpenCV source, which I could've referred to. So it'll take time to find such libraries and edit their cmake files.
Also I wanted to have quick android demo showing any of the ViSP's module. But I don't know which one to start with, which is independent of other modules - so as make least edits.

I'll appreciate any advice from you regarding the edits made and whether I'm going in right direction or not.

Thanks
Akshay

took opencv's cmake files as reference. started editing those VISP cmake files whose corresponding OpenCV cmake file had android build flags

fspindle · 2018-03-12T07:30:18Z

I think that we should start to port ViSP without the support of any 3rd party. There is already apriltag that is embedded in ViSP source code and that could be a good candidate for a first demo (see tutorial).

The pseudo-code of the demo could be:

while(1)
    android_image = acquire_image()
    convert_from_android_to_visp(android_image, visp_image)
    [nbtags, cog, roi] = detect_apriltag(visp_image)
    display_image(android_image)
    for (i=0; i < nbtags)
        display_roi(android_image, roi[i])
        display_cog(android_image, cog[i])

Than in a second time, we could add the support of OpenCV 3rd party.
And only later other 3rd parties like Eigen3 if possible.

defined 3 extra macros

akshay326 · 2018-03-16T17:22:34Z

hi @fspindle

I'm on mid way adding and editing cmake files for android build. Since these files are much similar to the OpenCV analogues, I made a README file where you can refer the file changes I'm making.

Can you check:

whether I'm adding macros at the correct place or not
whether I'm trying to create a cmake script that already exists (with a different name perhaps) and provides the same code/functionality

thanks
akshay

akshay326 · 2018-03-17T06:45:11Z

hi @fspindle

I had some doubts regarding the core module

It seems like the class vpMatrix needs at least one 3rd party library like Eigen, Lapack, OpenCV or GSL for building, refer this line), else it throws an exception. So I'm using the Lapack library. But it'll be creating missing operator errors, since it wasn't build for android.
Also the class vpIOTools requires wordexp.h header file, which is not available in android.

took opencv's cmake files as reference. started editing those VISP cmake files whose corresponding OpenCV cmake file had android build flags

defined 3 extra macros

resolved the wordexp() error in vpIOTools extra CFlags in the android.toolchain file itself

fspindle · 2018-03-17T08:08:55Z

Hi Akshay,

Thanks for the work you are doing. I will check your PR ASAP.

You are right, vpMatrix needs at least one 3d party. But if none is installed and detected we are using clapack that is embedded in ViSP source code. This version is the result of fortran to c conversion. It is not optimized as the native one, but it does the trick.

Concerning, vpIoTools I suggest to do the same as for iOS where wordexp.h is also not available.

#if !defined(_WIN32) && !defined(ANDROID)
#include <wordexp.h>
#endif

and later in the file something similar to

#if (TARGET_OS_IOS == 0) && !defined(ANDROID)

resolved 64 bit offset issues

akshay326 · 2018-03-17T08:36:17Z

hi @fspindle

Thanks for the advice, made the edits in vpIOTools.

But soon after that, few files in clapac library were signalling undefined reference errors while linking. I found the issue and edited a file named fio.h in that folder.

Now the build works well for visp_core module. You can get the install script here

Thanks
akshay

fspindle

Very nice work. I have no time yet to have a trial, but it seems promising.

Minor comment: The logo platforms/android/service/engine/res/drawable/icon.png should be the one of ViSP https://visp.inria.fr/files/2015/12/visp-logo-sans-texte-128x128.png or this one https://visp.inria.fr/files/2015/12/visp-logo-140x140.png

fspindle · 2018-03-17T08:14:16Z

cmake/GenAndroidMK.cmake

                    endif ()
                else ()
-                    ocv_get_libname(lib_name "${_output}")
+                    visp_get_libname(lib_name "${_output}")


Should be here vp_get_libname

fspindle · 2018-03-17T08:20:16Z

CMakeLists.txt

  set(CMAKE_INSTALL_PREFIX "${CMAKE_BINARY_DIR}/install" CACHE PATH "Installation Directory")
-endif()
+endif(NOT CMAKE_TOOLCHAIN_FILE)



Put endif()

fspindle · 2018-03-17T08:21:09Z

CMakeLists.txt

+else(NOT CMAKE_TOOLCHAIN_FILE)
+  #Android: set output folder to ${CMAKE_BINARY_DIR}
+  set( LIBRARY_OUTPUT_PATH_ROOT ${CMAKE_BINARY_DIR} CACHE PATH "root for library output, set this to change where android libs are compiled to" )
  # Crosscompiling


Remove space set(LIBRARY_OUTPUT_PATH_ROOT ...)

fspindle · 2018-03-17T08:23:26Z

cmake/GenConfig.cmake

+if(HAVE_CUDA)
+  vp_cmake_configure("${CMAKE_CURRENT_LIST_DIR}/templates/VISPConfig-CUDA.cmake.in" CUDA_CONFIGCMAKE @ONLY)
+endif()
+


ViSP doesn't use CUDA.
Should be removed as well as templates/VISPConfig-CUDA.cmake.in

I don't see the interest of this new file. There is already cmake/VISPGenerateConfig.cmake

fspindle · 2018-03-17T08:28:21Z

cmake/GenConfig.cmake

+        DESTINATION "${VISP_CONFIG_INSTALL_PATH}" COMPONENT dev)
+  endif()
+endfunction()
+


I see more this function in cmake/VISPUtils.cmake

transferred the function to VISPUtils.cmake

fspindle · 2018-03-17T08:49:43Z

platforms/android/service/engine/build.xml

@@ -0,0 +1,92 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project name="OpenCV Manager" default="help">
+


ViSP Manager

fspindle · 2018-03-17T08:54:52Z

platforms/android/service/engine/res/values/strings.xml

+    <string name="about">About</string>
+    <string name="checkUpdate">Check for update</string>
+    <string name="intro">OpenCV library is used by other applications for image enhancement, panorama stitching, object detection, recognition and tracking and so on. OpenCV Manager provides the best version of the OpenCV for your hardware. See opencv.org for details.</string>
+</resources>


fspindle · 2018-03-17T08:55:23Z

platforms/android/service/engine/src/org/opencv/engine/OpenCVEngineInterface.aidl

+    * @return Returns OpenCV libraries names separated by symbol ";" in loading order
+    */
+    String getLibraryList(String version);
+}


fspindle · 2018-03-17T08:56:03Z

platforms/android/service/engine/src/org/opencv/engine/manager/ManagerActivity.java

+        }
+    };
+
+}


fspindle · 2018-03-17T08:56:19Z

platforms/android/service/readme.txt

+    adb install <path-to-OpenCV-sdk>/apk/OpenCV_<version>_Manager_<app_version>_<platform>.apk
+
+Example: OpenCV_3.4.0-dev_Manager_3.40_armeabi-v7a.apk
+


akshay326 · 2018-03-19T05:28:23Z

Hi @fspindle

Just finished making the OpenCV to VISP edits you mentioned. Also I'm trying to make a demo AprilTag detection app(get the source code here). I'll try to target only latest android version(API>=24) since older versions aren't maintained now and are buggy.

Following the tutorial you referred, creating a vpImage from android bitmap was easy. But I see that AprilTag detection needs a vpImage<unsigned char> while by default android provides uint_16(16 bit RGB-565 or 32 bit ARGB values). The conversion/copying is adding extra overheads while live tracking/detection(fails on high resolution images).

I read that OpenCV android sdk has its own camera framework(they don't use the default one). I'll try to figure out how it works and update you.

Thanks
Akshay

fspindle · 2018-03-19T07:30:04Z

Ok to consider the latest Android version.

In ViSP there is no aRGB to grey conversion, but if SSE3 is available on Android there is an optimized version of RGBa to grey conversion that could be a a good starting to implement a new vpImageConvert::aRGBToGrey() function.

To speed up the Apriltag detection, before reconsidering a new camera framework, you can try to use:

vpDetectorAprilTag::setAprilTagQuadDecimate() function: It operates like a subsampling factor.
vpDetectorAprilTag::setAprilTagNbThreads() function

On 640 by 480 images, I succeeded to make detection realtime on Raspberry Pi 3 setting quad_decimate=4 and nthread=2.

s-trinh · 2018-03-19T13:50:29Z

Small precision, Android is most likely run on ARM platform and SSE intrinsics are x86 only. Maybe this conversion could be done directly with Android API?

akshay326 · 2018-03-23T17:17:27Z

Hi all

Finally got a working demo for single image AprilTag detection on x86 arch, tested on API-24(get the source code and prebuilt apk's here). But there are some issues

The standard bitmap.h header in android NDK provides a lockPixels API to pass ARGB image from java to NDK, but it's not reading pixel values correctly(I checked it against pixel values obtained on my PC). That's the reason I couldn't use vpImageConvert::aRGBToGrey() function. So I'm creating vpImage this way
bitmap(in java) -> byte array(in java) -> uchar array(in c++) -> vpImage (in c++) ->
While creating the image, I've keep the copy flag true vpImage(src, width, height, **true**), else it gives segmentation fault
On armeabi-v7a arch, it was giving some undefined atof symbol error. This is a problem either with the NDK or else we'll need to change the build procedure

I'm trying real time detection(quad_decimate=4 proved to be good advice indeed), but it's still slow. I'll try to tweak a few things like FPS, or vpDetector attributes, etc and look for better results.

Lastly I had some doubts:

What should be the conversion rgb2grey function?
Now should I extend the app to multiple architectures(x86_64, mips) or focus on realtime detection on x86 only?

Looking for your advice
Akshay

akshay326 · 2018-03-24T12:40:31Z

Hi all

Tweaked a few detection parameters(and some changes on Camera Preview on java side) and finally got a demo for realtime vpAprilTag detection with attributes:

x86 arch
API 25
2 FPS, 640x480 greyscale images
nbThreads=4 and quad_decimate=4.0

Get the demo video here and the source code here

Akshay

fspindle · 2018-03-24T15:45:56Z

Thanks for the app demo. Very good work.

Regarding your previous message and questions, I would prefer that you try to improve the framerate of the demo instead of porting to multiple architectures that could be done later.

I think that using AndroidBitmap_lockPixels() and AndroidBitmap_unlockPixels() is the right way. But the conversion between both functions should be more efficient. Should be confirmed, but I think that the bottleneck is in the conversion, not in the tag detection.

I'm not sure to understand your question about rgb2gray(unsigned char *rgb, unsigned char *grey, unsigned int size). This function assumes a rgb bitmap as input [R_1 G_1 B_1 R_2 G_2 B_2 ... R_n G_n B_n] where size = n, meaning the size of the rgb bitmap is equal to 3*size. It updates as output a grey bitmap that is of size n using the formula grey_i = 0.2126 * R_i + 0.7152 * G_i + 0.0722 * B_i.

Some ideas:

The time of your conversion could be compared to the one when using OpenCV functions. This is just to confirm what I suspect.
The conversion should be direct from color bitmap(in java) to grey vpImage<unsigned char>
Why not simply trying to introduce a new argb2grey() function (not tested):

void vpImageConvert::argb2gray(unsigned char *argb, unsigned char *grey, unsigned int size)
{
  unsigned char *pt_input = argb;
  unsigned char *pt_end = argb + size * 4;
  unsigned char *pt_output = grey;
  
  while (pt_input != pt_end) {
    *pt_output = (unsigned char)(0.2126 * (*pt_input+1) + 0.7152 * (*(pt_input + 2)) + 0.0722 * (*(pt_input + 3)));
    pt_input += 4;
    pt_output++;
  }
}

I would be also interested to improve the demo displaying some info in overlay showing that we succeed to localise the tag. I'm thinking about something similar to vpDisplay::displayLine() and vpDisplay::displayFrame(). There could be a button in the app that activates/disables the display.
Later when the demo is working faster, create a new java module in ViSP that will contain some tools (conversions...) that could be used in every Android app.

akshay326 · 2018-03-27T15:40:12Z

Hi @fspindle

While working to improve framerate, I read that we can get the live camera feed in YUV byte[ ] instead of a bitmap. This means that tag detection can work with just the grey scale data found in the Y plane of the preview, so its fast now (~10ms or ~15ms for 4 tags). I haven't used AndroidBitmap_locksPixels since I'm getting preview as YUV byte[ ] not bitmap

And I read that ViSP supports a host of functions for conversion from various formats(YUV, YCbCr, etc) to RGBa. We can get an vpImage<vpRGBa> from it.

Now I'm trying to implement vpDisplay::displayLine() function (or a function to augment something). I had to ask:

Is there any in built function that works like vpDisplay::displayLine() without any 3rd party package
whether to use opengl or OGRE for a demo(NDK supports opengl by default) for 3d augmenting

Thanks
Akshay

akshay326 · 2018-03-31T17:08:53Z

Hi all

Recently I was trying to demonstrate a better tag detection using vpDisplay::displayLine() or something like a 3d axis on the markers(using pose estimation). Since the functions provided in ViSP depend on OpenCV 3rd party, I tried doing so without 3rd party.

But the images are not well calibrated(get the video here, resulting lines across markers are skewed). Even to calibrate the camera we need OpenCV.

So should I try adding OpenCV support(some changes in the build script) or start a visp_java module that can be used to tap ViSP functions that don't require any 3rd party.

Unknown added 2 commits March 12, 2018 11:43

started editing cmake files for android build

6fa593c

took opencv's cmake files as reference. started editing those VISP cmake files whose corresponding OpenCV cmake file had android build flags

ocv macro changes

3de665d

Unknown added 2 commits March 15, 2018 11:49

replaced all ocv macros by vp macros

1bcb5a4

defined 3 extra macros

began adding android template files

1a89e5f

Unknown added 5 commits March 17, 2018 13:37

started editing cmake files for android build

3e63b1a

took opencv's cmake files as reference. started editing those VISP cmake files whose corresponding OpenCV cmake file had android build flags

ocv macro changes

c8c9789

replaced all ocv macros by vp macros

85ad901

defined 3 extra macros

began adding android template files

094b608

building visp_core works. error in linking the lapac 3rd party

1a93661

resolved the wordexp() error in vpIOTools extra CFlags in the android.toolchain file itself

Unknown added 2 commits March 17, 2018 13:39

Merge remote-tracking branch 'origin/android_build' into android_build

2e5f143

libvisp_core.so build. edited clapack 3rd party

ff18229

resolved 64 bit offset issues

fspindle reviewed Mar 17, 2018

View reviewed changes

Unknown added 2 commits March 18, 2018 22:09

made OpenCV to VISP edits

8b98fe4

pthread support for android

5ea4640

Merge branch 'master' into android_build

9280240

akshay326 mentioned this pull request May 15, 2018

Building visp_java module #323

Merged

fspindle merged commit 9280240 into lagadic:master Jul 2, 2018

fspindle mentioned this pull request Jul 11, 2018

Porting ViSP to Android GSoC 2018 general discussion #346

Closed

akshay326 deleted the android_build branch November 5, 2018 12:32

		@@ -0,0 +1,92 @@
		<?xml version="1.0" encoding="UTF-8"?>
		<project name="OpenCV Manager" default="help">

+                      }
+                  };
+              }

		adb install <path-to-OpenCV-sdk>/apk/OpenCV_<version>_Manager_<app_version>_<platform>.apk

		Example: OpenCV_3.4.0-dev_Manager_3.40_armeabi-v7a.apk

Porting ViSP to android #307

Porting ViSP to android #307

Uh oh!

Conversation

akshay326 commented Mar 12, 2018

Uh oh!

fspindle commented Mar 12, 2018

Uh oh!

akshay326 commented Mar 16, 2018

Uh oh!

akshay326 commented Mar 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fspindle commented Mar 17, 2018

Uh oh!

akshay326 commented Mar 17, 2018

Uh oh!

fspindle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akshay326 commented Mar 19, 2018

Uh oh!

fspindle commented Mar 19, 2018

Uh oh!

s-trinh commented Mar 19, 2018

Uh oh!

akshay326 commented Mar 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akshay326 commented Mar 24, 2018

Uh oh!

fspindle commented Mar 24, 2018

Uh oh!

akshay326 commented Mar 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akshay326 commented Mar 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

akshay326 commented Mar 17, 2018 •

edited

Loading

akshay326 commented Mar 23, 2018 •

edited

Loading

akshay326 commented Mar 27, 2018 •

edited

Loading