Reduce size of Docker containers

[jameslamb]

What do these changes do?

Hello,

Thank you for this excellent project!

I've been reading through this repo trying to familiarize myself with the project, and saw something I may be able to help with. In this PR, I propose a few changes to the Dockerfiles in this repo to reduce container size and build time.

This PR reduces the size of base-deps by about 70 MB.

Proposals related to build time / size:

• chaining together run commands
  • less build layers --> smaller image
• removing vim from the dependencies
  • Users who want to run vim in the container to mess with files are welcome to add it in their own spin-off dockerfiles, but I believe the project should make the published container used to run Ray as small as possible

Proposals related to style and administration:

• Putting dependency installation (e.g. with apt-get) on one line makes the git blame more usable and the Dockerfile more human-readable

Testing

I can confirm I was able to build the deploy and examples containers.

To test, I built the base-deps container as follows and then changed the deploy container to look at ray_new_deps (which it found locally):

docker build -t ray_deps_new -f Dockerfile .

I built deploy and examples using install-docker.sh at the repo root.

Future PR

I also want to propose removing git from these containers (since it isn't used in the build process, AFAICT), but didn't want to put that in this PR since it would involve changing some of the example documentation. If you are open to that change, I will make it in a follow-up PR.

Thank you for your consideration,

-James

Related issue number

This does not relate to any open issues.

[jameslamb]

it looked like the old comment on this file had been copy-pasted from the deploy container

[jssmith]

Good catch.

[robertnishihara]

Thanks @jameslamb this looks good to me!

I think we do use git in the build process, e.g., we git clone Arrow and then checkout specific commits.

cc @jssmith in case you want to take a look.

[jameslamb]

@robertnishihara ahhhhhh ok I get it now, I see build.sh (which I know will drop down to build_arrow and build_parquet and other stuff using git) is getting run when I pip install . inside deploy (

ray/python/setup.py

Line 54 in f1e5789

subprocess.check_call(["../build.sh", sys.executable])

)

that's a cool trick. Glad I decided to leave git out of this PR!

[robertnishihara]

Yeah, it feels a bit messy in some ways... inside of setup.py we're running the build script, which in turn runs setup.py for pyarrow..

Anyway, thanks for the help!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4166/
Test PASSed.

[jssmith]

These changes all look like sensible cleanup and enhancements me.

[jssmith]

Good catch.