Support the Android NDK with libtorch

[Suhail]

🚀 Feature

Hey folks, I think the best path forward if you want to truly support mobile production environments is to make it easier to allow people to compile libtorch such that it can be used with Android Studio's NDK.

Right now, given the current state of things, I'm forced to fall back on the protobuf, caffe2 solution. I do understand that things are still unstable & a work in progress.

Motivation

Current shared/static libs offered in the docs are pre-built for specific architectures & thereby do not allow you to utilize libtorch on a robust set of production devices easily in the real-world.

For example, in my case, I can't run libtorch even a rooted Android phone because the architecture (aarch64) still has differences in compilation due to the OS/ABI I suspect. The NDK has invested in solving this for developers already via a fairly robust toolchain. It's not obvious how to build pytorch via it & then utilize those various static/shared libs to compile with the NDK.

Additionally, it's not simple to just compile libtorch on an arm64 machine and run the binary on the phone.

Pitch

NDK support will dramatically reduce the numerous pipeline of tools required for a mobile production environment.

This will also help people who do not have the priviledge of developing on a rooted phone & enable fast, client-side inference.

Alternatives

pytorch -> export to onnx -> caffe2 -> Write C++ w/ NDK -> Java

[soumith]

currently the official / supported solution for mobile is pytorch -> ONNX -> {Caffe2/CoreML/nnapi}.

Even if libtorch does compile for Android (which it does, i'll get to that in a sec), it's not really optimized for ARM64 right now, and the binary size will be pretty large.

That being said, @t-vi wants to take a crack at porting libtorch to run with android ndk. Read https://lernapparat.de/pytorch-android/ for more details.

cc: @dreiss

[ezyang]

By the way, c10, a lower level core library (which doesn't have much stuff, but will eventually contain a tensor class but no kernels) has been designed from the start to work on mobile, and is currently used as part of the Caffe2 mobile build and is absorbing bits from libtorch.

[zhyj3038]

I can just get libtorch.so under x86

[xxradon]

@t-vi From your latest pytorch-jit-android/ blog,I know you have done libtorch compiled for Android,but I don't kown where to find your tools/build_pytorch_libs_android.sh. Can you share the code?And that will be my appreciate.

[t-vi]

The android port is not ready for general consumption yet, unfortunately, and the build script is a part of why it isn't. The proof of concept is based on a very (in PyTorch time) old version of build_pytorch_libs.sh that the colleagues did a great job of cleaning up after I copied it. I guess I have added a bit of "to be cleaned up" instead. Basically, you set all the android-relevant flags.

The crux is that I cannot publish something that people will complain about being unfinished and at the same time nobody I know seems to have enough commercial interest in getting this into a nice enough state (which the blog post Soumith linked that discussed to do in a "B2B-Crowdfunding", but it didn't work out, yet).

I'm cautiously optimistic that we might be in a better situation next year, but for the time being, the PyTorch Android port isn't releasable. If you have ideas, I'm all ears.

[xxradon]

@t-vi Thanks for your reply,As you said,there is a bit of "to be cleaned up" in pytorch code for compiling,but I think you and your colleagues had made very greatful job,maybe you have met so many problems,and nobody enough commercial interest in getting this into a nice enough state.But actually,we just need a guide to explore and the more we participate in the more related problems will be solved.

So,can you just share your concept libtorch code or ideas for android,and that will be much appreciate.
Thanks,and Give you best wish Merry Christmas in advance!

And @soumith ,is there a way to build full ATen API on android?

[t-vi]

In addition to Soumith's PR #15509, having a fix for #9704 would be really helpful to have the build+install succeed and get the includes in a sane way.
After that it's ~100 lines of build fixes to a working android build, I just need to find time to get this cleaned up a bit.

[xxradon]

@t-vi Thank you,nice work.

[Oktai15]

Any progress here?

[soumith]

Fixed on master via @ljk53 's PRs