ARROW-3180: [C++] Add docker-compose setup to simulate Travis CI run locally #2572

kszucs · 2018-09-16T20:34:01Z

Also resolves:

ARROW-3078: [Python] Docker integration tests should not contaminate the local Python development environment

Working:

The rest:

js (gulp error...)
matlab
R
dask integration
spark integration
gen_apidocs
hiveserver2
iwyu

Perhaps resolve them in follow-up PRs.

kszucs · 2018-09-17T14:43:01Z

@wesm could You take a look at this docker-compose setup?

Executing python tests: docker-compose run python
Executing cpp tests: docker-compose run cpp
Executing c_glib tests: docker-compose run c_glib
Executing hdfs tests: docker-compose run hdfs-integration
etc.

It's also possible to mount arrow externally for an editable setup:

docker-compose run -v local/path/to/arrow:/arrow python

Additionally docker-compose up runs all of the containers in parallel (discouraged of course, because it doesn't stop the hdfs containers).

Ideally "the rest" of the containers should be ported too.

wesm · 2018-09-18T21:11:08Z

@kszucs thanks a lot for working on this. I will take it for a spin when I can and report back feedback

kszucs · 2018-09-19T08:40:10Z

Sadly we can't chain the builds to depend on each other purely with docker compose e.g.: conda -> cpp -> c_glib -> ruby, so chaining builds in a Makefile would improve the build times.

kszucs · 2018-09-19T16:38:05Z

TODO: crossbow should have its own image

wesm · 2018-09-24T10:52:15Z

I'm working on code reviews, but hope to get to this soon

wesm

+1, thanks, this is good progress modularizing these dockerfiles! I think the hope of getting everything 100% working on one patch is a lot to ask, but we can work toward it in follow-up PRs. Will merge once the build passes

xhochy · 2018-09-29T10:14:35Z

We should add a small word about these files to the docs.

xhochy · 2018-09-29T10:39:04Z

The JS build fails for me:

(node:81) [DEP0097] DeprecationWarning: Using a domain property in MakeCallback is deprecated. Use the async_context variant of MakeCallback or the AsyncResource class instead.
[10:20:45] The following tasks did not complete: build, build:apache-arrow, build:es5:umd, build:es2015:umd, build:esnext:umd, <parallel>, build:es5:umd, build:es2015:umd, build:es2015:umd:task, build:es2015:umd:task, build:esnext:umd:task, build:es5:umd:task, build:es5:umd:task
[10:20:45] Did you forget to signal async completion?
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! apache-arrow@0.3.0 build: `gulp build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the apache-arrow@0.3.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-09-29T10_20_45_774Z-debug.log

wesm · 2018-09-29T11:01:46Z

He mentioned that the JS build is broken in the PR description

kszucs · 2018-09-29T19:33:44Z

Well, I'm a JS noob :)

Strange that the js build works outside of the docker container, I've tried to tune the dockerignore file to include all required packaging files in the image, but sadly with no luck.

Created issues for the follow-up tasks:

Documentation https://issues.apache.org/jira/browse/ARROW-3365
JS Dockerfile https://issues.apache.org/jira/browse/ARROW-3365
R Dockerfile https://issues.apache.org/jira/browse/ARROW-3366
Spark integration https://issues.apache.org/jira/browse/ARROW-3367
Dask integration https://issues.apache.org/jira/browse/ARROW-3368

kszucs · 2018-09-29T21:02:21Z

@wesm @xhochy ready for merge

xhochy · 2018-09-30T09:26:45Z

The Plasma tests all fail for me, I tried adding shm_size to the compose file but it seems to be ignored:

diff --git a/docker-compose.yml b/docker-compose.yml
index 93ec11ac..394d3fda 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.

-version: '3'
+version: '3.5'
 services:

   # we can further improve the caching mechanism for go rust and js via
@@ -59,6 +59,7 @@ services:
     build:
       context: .
       dockerfile: python/Dockerfile
+      shm_size: '2gb'
       args:
         PYTHON_VERSION: ${PYTHON_VERSION:-3.6}

kszucs · 2018-09-30T09:42:20Z

It indeed ignores shm_size. What's your docker engine? Are You using docker on mac or docker-machine?

xhochy · 2018-09-30T09:53:19Z

Docker on mac: 18.06.1-ce-mac73

xhochy · 2018-09-30T09:53:44Z

When running the manylinux images using docker run --shm-size… it works.

kszucs · 2018-09-30T09:57:57Z

Same for me, it can be a docker on mac issue. I'm spinning up a linux box...

kszucs · 2018-09-30T10:12:37Z

docker-compose.yml

+
+  python:
+    image: arrow:python-${PYTHON_VERSION:-3.6}
+    shm_size: 2G


@xhochy docker has a "great" documentation, it should't be placed under build

xhochy

+1, Python tests now also pass for me.

kszucs · 2018-09-30T10:21:41Z

Cool!

kszucs · 2018-09-30T11:33:43Z

@xhochy CI error is related to the wheel issue

Change-Id: I355c38a0aac222a000a471674a43af95ab802989

codecov-io · 2018-09-30T13:12:09Z

Codecov Report

Merging #2572 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2572      +/-   ##
==========================================
+ Coverage   87.22%   87.23%   +0.01%     
==========================================
  Files         381      381              
  Lines       59431    59432       +1     
==========================================
+ Hits        51836    51847      +11     
+ Misses       7521     7515       -6     
+ Partials       74       70       -4

Impacted Files	Coverage Δ
python/pyarrow/tests/test_parquet.py	`97.41% <100%> (ø)`	⬆️
go/arrow/math/float64_sse4_amd64.go	`0% <0%> (-100%)`	⬇️
go/arrow/math/uint64_sse4_amd64.go	`0% <0%> (-100%)`	⬇️
go/arrow/memory/memory_sse4_amd64.go	`0% <0%> (-100%)`	⬇️
go/arrow/math/int64_sse4_amd64.go	`0% <0%> (-100%)`	⬇️
go/arrow/math/float64_amd64.go	`33.33% <0%> (ø)`	⬆️
go/arrow/math/int64_amd64.go	`33.33% <0%> (ø)`	⬆️
go/arrow/math/uint64_amd64.go	`33.33% <0%> (ø)`	⬆️
go/arrow/math/math_amd64.go	`36.84% <0%> (+5.26%)`	⬆️
go/arrow/memory/memory_amd64.go	`42.85% <0%> (+14.28%)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74e98f8...41cd810. Read the comment docs.

wesm · 2018-09-30T17:50:44Z

Thanks everyone. This will help a lot

@romainfrancois

Thank you to @romainfrancois and others who have pushed forward the R side of this project! This PR is my attempt to address [ARROW-3336](https://issues.apache.org/jira/projects/ARROW/issues/ARROW-3366), providing a testing container for the R package. This follows up on work done by @kszucs in #2572 in an R-specific way. **NOTE:** This PR is a WIP. `R CMD INSTALL` currently fails because it cannot find wherever I installed `arrow` to. But I felt that this is far enough along to put up for review. Author: James Lamb <jaylamb20@gmail.com> Author: Krisztián Szűcs <szucs.krisztian@gmail.com> Closes #2770 from jameslamb/r_dockerfile and squashes the following commits: 9fcebe5 <James Lamb> removed failing test in the R package dd26d4b <James Lamb> fixed failing test in the R package 343456a <James Lamb> more changes 0db3369 <James Lamb> added bit64 dependency to R package f7c050e <Krisztián Szűcs> R_CHECK_FORCE_SUGGESTS db9ee5b <Krisztián Szűcs> pass CXXFLAGS 8a8376d <James Lamb> Put R build steps into a separate CI script 64ea840 <James Lamb> Updated LD_LIBRARY_PATH in R container 4d7f02c <James Lamb> fixed a few issues in R container 94242b3 <James Lamb> Add explict cpp build in usage for R docker-compose 4a7112b <James Lamb> Fixed comment text in R Dockerfile d8e55d7 <James Lamb> removed todo about R containers in docker-compose 210a774 <James Lamb> ARROW-3366: Dockerfile for docker-compose setup

kszucs added WIP PR is work in progress and removed WIP PR is work in progress labels Sep 16, 2018

kszucs force-pushed the ARROW-3180 branch from af25303 to 7b7812a Compare September 24, 2018 16:43

kszucs mentioned this pull request Sep 27, 2018

ARROW-3327: [Python] Use local Arrow checkout instead of separate clone #2642

Closed

wesm approved these changes Sep 29, 2018

View reviewed changes

xhochy mentioned this pull request Sep 30, 2018

ARROW-3363: [C++/Python] Add helper functions to detect scalar Python types #2659

Closed

kszucs commented Sep 30, 2018

View reviewed changes

xhochy approved these changes Sep 30, 2018

View reviewed changes

kszucs added 5 commits September 30, 2018 14:32

docker-compose setup; go, rust, cpp, python, hdfs-integration

2900810

comment fixes

1737ef4

license headers

26fe1f6

missing license headers

f505105

mixed whitespaces in hdfs Dockerfile

99ce42e

kszucs and others added 14 commits September 30, 2018 14:32

mixed whitespaces in cpp Dockerfile

c8d3f26

mixed whitespaces in python Dockerfile

9e51cb9

js dockerfile

09a3e6f

c_glib dockerfile

210d4eb

meson install c_glib; run ruby tests

e287919

update dockerignore to reduce the build context's size

a5385ad

improve layer caching

4503116

organize js dockerfile

0d11fc6

[skip ci] java docker image

8ea75a6

[skip ci] add integration data to java image

5074cf5

Add glog to C++ env

52c0d89

Change-Id: I355c38a0aac222a000a471674a43af95ab802989

[skip ci] remove js lock files from dockerignore

83929c9

set shm_size

c3b73e7

shm_size

41cd810

kszucs force-pushed the ARROW-3180 branch from b257423 to 41cd810 Compare September 30, 2018 12:32

wesm closed this in 1649864 Sep 30, 2018

jameslamb mentioned this pull request Oct 16, 2018

ARROW-3366: [R] Dockerfile for docker-compose setup #2770

Closed

This was referenced Mar 14, 2019

[Doc] Document docker compose setup #19695

Closed

[R] Dockerfile for docker-compose setup #19697

Closed

[Integration/CI/Python] Add dask integration test to docker-compose setup #19699

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-3180: [C++] Add docker-compose setup to simulate Travis CI run locally #2572

ARROW-3180: [C++] Add docker-compose setup to simulate Travis CI run locally #2572

kszucs commented Sep 16, 2018 •

edited

Loading

kszucs commented Sep 17, 2018 •

edited

Loading

wesm commented Sep 18, 2018

kszucs commented Sep 19, 2018 •

edited

Loading

kszucs commented Sep 19, 2018 •

edited

Loading

wesm commented Sep 24, 2018

wesm left a comment

xhochy commented Sep 29, 2018

xhochy commented Sep 29, 2018

wesm commented Sep 29, 2018

kszucs commented Sep 29, 2018

kszucs commented Sep 29, 2018

xhochy commented Sep 30, 2018

kszucs commented Sep 30, 2018

xhochy commented Sep 30, 2018

xhochy commented Sep 30, 2018

kszucs commented Sep 30, 2018 •

edited

Loading

kszucs Sep 30, 2018 •

edited

Loading

xhochy Sep 30, 2018

xhochy left a comment

kszucs commented Sep 30, 2018

kszucs commented Sep 30, 2018

codecov-io commented Sep 30, 2018

wesm commented Sep 30, 2018

ARROW-3180: [C++] Add docker-compose setup to simulate Travis CI run locally #2572

ARROW-3180: [C++] Add docker-compose setup to simulate Travis CI run locally #2572

Conversation

kszucs commented Sep 16, 2018 • edited Loading

kszucs commented Sep 17, 2018 • edited Loading

wesm commented Sep 18, 2018

kszucs commented Sep 19, 2018 • edited Loading

kszucs commented Sep 19, 2018 • edited Loading

wesm commented Sep 24, 2018

wesm left a comment

Choose a reason for hiding this comment

xhochy commented Sep 29, 2018

xhochy commented Sep 29, 2018

wesm commented Sep 29, 2018

kszucs commented Sep 29, 2018

kszucs commented Sep 29, 2018

xhochy commented Sep 30, 2018

kszucs commented Sep 30, 2018

xhochy commented Sep 30, 2018

xhochy commented Sep 30, 2018

kszucs commented Sep 30, 2018 • edited Loading

kszucs Sep 30, 2018 • edited Loading

Choose a reason for hiding this comment

xhochy Sep 30, 2018

Choose a reason for hiding this comment

xhochy left a comment

Choose a reason for hiding this comment

kszucs commented Sep 30, 2018

kszucs commented Sep 30, 2018

codecov-io commented Sep 30, 2018

Codecov Report

wesm commented Sep 30, 2018

kszucs commented Sep 16, 2018 •

edited

Loading

kszucs commented Sep 17, 2018 •

edited

Loading

kszucs commented Sep 19, 2018 •

edited

Loading

kszucs commented Sep 19, 2018 •

edited

Loading

kszucs commented Sep 30, 2018 •

edited

Loading

kszucs Sep 30, 2018 •

edited

Loading