[C++] Build C++ project including thirdparty dependencies from local tarballs

[asfimport]

Could there be a flag or similar to make offline building of the project easier (or maybe that is in place and I don't see it)?

Something that perhaps in CMakeLists.txt that has the option for the URL to be a file:// path?

This would help me in that I could download all relevant tarballs and have the same build happen offline or online.

Reporter: Luke Higgins
Assignee: Wes McKinney / @wesm

Related issues:

• Possible to reduce size of release tarballs? (is related to)

Original Issue Attachments:

• CMakeLists.txt

PRs and other links:

• GitHub Pull Request #2141

_{Note: This issue was originally created as ARROW-902. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
You can already accomplish this by building an external toolchain and setting the *_HOME environment variables. We could also provide environment variables for setting the local tarball locations. Patch would be welcome for this

[asfimport]

Luke Higgins:
I have a CMakeLists.txt (attached) that implements this. How do I do propose this change in the way you want for this project?

[asfimport]

Wes McKinney / @wesm:
See https://github.com/apache/arrow#how-to-contribute

I am not sure what's the best way to do this. Environment variables (one per dependency for the URL) might be more flexible. Do you know another open source project that provides for offline builds to get some guidance about best practices?

[asfimport]

Luke Higgins:
I do not know of other projects that have this feature to model on (although I have wanted this feature for a lot of projects!)

I don't have a preference. If preferred changing to check for an enviroment variable for each third party dependency seems fine to me too.

This feature would help a lot. I believe there are quite a few people that are a little stuck from interacting with projects because of being in environments not connected to the internet (directly).

On a different note, is there a reason the boost dependency is not done like the other third party dependencies?

[asfimport]

Wes McKinney / @wesm:
Boost is much more complex than the other dependencies, and it's available almost everywhere, so I'm not sure what would be the benefit.

This feature would help a lot. I believe there are quite a few people that are a little stuck from interacting with projects because of being in environments not connected to the internet (directly).

It sounds like you are working in an airgapped environment, or behind some other firewall. In this environments, I would recommend building the thirdparty toolchain separately and using the *_HOME variables. I personally do some work in a firewalled environment and have been successfully building these libraries without giving cmake outbound internet access, using a separately-managed toolchain (using conda packages).

On some contemplation, I'm unsure if we want to support offline builds via ExternalProject due to the extra maintenance burden. I'm interested to hear from other users how they are handling builds in environments without direct internet access.

[asfimport]

Luke Higgins:
Yes, behind a firewall or airgapped is where I am often finding myself and wanting to do things in the latest pydata stack.

I will give the thirdparty toolchain method a whirl. Having a cmake,make process helps with my ansible deployments so was working towards that independent of shipping around a toolchain environment.

[asfimport]

Luke Higgins:
so on the thirdparty toolchain doc, development.rst, I had to do the following for the build:
export ARROW_HOME=/[blah blah]/envs/pyarrow-dev/
export LD_LIBRARY_PATH=/[blah blah]/envs/pyarrow-dev/lib64/

[asfimport]

Wes McKinney / @wesm:
Good point; we have to add CMAKE_INSTALL_LIBDIR in conda recipes for this reason. I opened https://issues.apache.org/jira/browse/ARROW-912 to fix this in the documentation

[asfimport]

Luke Higgins:
I like the whole conda environment build, but how does that help me with offline deployments? I was thinking there was a simple bundle environment kind of functionality in conda environments and that I could then copy the bundle. It doesn't appear to be anything that straightforward. I am okay with this feature not being in the master but for now I will be cloning the master at build and making modifications like above to the CMakeLists.txt so I have a pretty smooth ansible build in an offline cluster. My guess is that this won't be an issue as much as subsequent versions are released and key features aren't needed as much in between releases. Thanks for all the hard work!

[asfimport]

Wes McKinney / @wesm:
I've been having to build the project on airplanes a lot this year and there are some problematic thirdparty projects, like ORC, which have larger tarballs that take a long time to download over plane wifi. I'm adding this to 0.10.0 and I will look into allowing the user to configure the build to work in an environment with no internet assuming you have saved all the thirdparty package source tarballs in some particular location (which can be set using environment variables)

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 2141
#2141