Skip to content

Commit 26b78c6

Browse files
andygroveandygrove
authored
Instructions for cross-compiling Ballista to the Raspberry Pi (#263)
1 parent ed92673 commit 26b78c6

File tree

4 files changed

+190
-0
lines changed

4 files changed

+190
-0
lines changed

‎dev/build-ballista-docker-arm64.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/bin/bash
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
if [ -z "${DOCKER_REPO}" ]; then
21+
echo "DOCKER_REPO env var must be set"
22+
exit -1
23+
fi
24+
cargo install cross
25+
cross build --release --target aarch64-unknown-linux-gnu
26+
rm -rf temp-ballista-docker
27+
mkdir temp-ballista-docker
28+
cp target/aarch64-unknown-linux-gnu/release/ballista-executor temp-ballista-docker
29+
cp target/aarch64-unknown-linux-gnu/release/ballista-scheduler temp-ballista-docker
30+
cp target/aarch64-unknown-linux-gnu/release/tpch temp-ballista-docker
31+
mkdir temp-ballista-docker/queries/
32+
cp benchmarks/queries/*.sql temp-ballista-docker/queries/
33+
docker buildx build --push -t $DOCKER_REPO/ballista-arm64 --platform=linux/arm64 -f dev/docker/ballista-arm64.Dockerfile temp-ballista-docker
34+
rm -rf temp-ballista-docker

‎dev/docker/ballista-arm64.Dockerfile‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
FROM arm64v8/ubuntu
18+
19+
ADD ballista-scheduler /
20+
ADD ballista-executor /
21+
22+
# Add benchmarks
23+
ADD tpch /
24+
RUN mkdir /queries
25+
ADD queries/*.sql /queries/
26+
27+
ENV RUST_LOG=info

‎docs/user-guide/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
- [Docker](distributed/standalone.md)
3232
- [Docker Compose](distributed/docker-compose.md)
3333
- [Kubernetes](distributed/kubernetes.md)
34+
- [Raspberry Pi](distributed/raspberrypi.md)
3435
- [Ballista Configuration](distributed/configuration.md)
3536
- [Clients](distributed/clients.md)
3637
- [Rust](distributed/client-rust.md)

‎docs/user-guide/src/distributed/raspberrypi.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
# Running Ballista on Raspberry Pi
20+
21+
The Raspberry Pi single-board computer provides a fun and relatively inexpensive way to get started with distributed
22+
computing.
23+
24+
These instructions have been tested using an Ubuntu Linux desktop as the host, and a
25+
[Raspberry Pi 4 Model B](https://www.raspberrypi.org/products/raspberry-pi-4-model-b/) with 4 GB RAM as the target.
26+
27+
## Preparing the Raspberry Pi
28+
29+
We recommend installing the 64-bit version of [Ubuntu for Raspberry Pi](https://ubuntu.com/raspberry-pi).
30+
31+
The Rust implementation of Arrow does not work correctly on 32-bit ARM architectures
32+
([issue](https://github.com/apache/arrow-rs/issues/109)).
33+
34+
## Cross Compiling DataFusion for the Raspberry Pi
35+
36+
We do not yet publish official Docker images as part of the release process, although we do plan to do this in the
37+
future ([issue #228](https://github.com/apache/arrow-datafusion/issues/228)).
38+
39+
Although it is technically possible to build DataFusion directly on a Raspberry Pi, it really isn't very practical.
40+
It is much faster to use [cross](https://github.com/rust-embedded/cross) to cross-compile from a more powerful
41+
desktop computer.
42+
43+
Docker must be installed and the Docker daemon must be running before cross-compiling with cross. See the
44+
[cross](https://github.com/rust-embedded/cross) project for more detailed instructions.
45+
46+
Run the following command to install cross.
47+
48+
```bash
49+
cargo install cross
50+
```
51+
52+
From the root of the DataFusion project, run the following command to cross-compile for ARM 64 architecture.
53+
54+
```bash
55+
cross build --release --target aarch64-unknown-linux-gnu
56+
```
57+
58+
It is even possible to cross-test from your desktop computer:
59+
60+
```bash
61+
cross test --target aarch64-unknown-linux-gnu
62+
```
63+
64+
## Deploying the binaries to Raspberry Pi
65+
66+
You should now be able to copy the executable to the Raspberry Pi using scp on Linux. You will need to change the IP
67+
address in these commands to be the IP address for your Raspberry Pi. The easiest way to find this is to connect a
68+
keyboard and monitor to the Pi and run `ifconfig`.
69+
70+
```bash
71+
scp ./target/aarch64-unknown-linux-gnu/release/ballista-scheduler ubuntu@10.0.0.186:
72+
scp ./target/aarch64-unknown-linux-gnu/release/ballista-executor ubuntu@10.0.0.186:
73+
```
74+
75+
Finally, ssh into the Pi and make the binaries executable:
76+
77+
```bash
78+
ssh ubuntu@10.0.0.186
79+
chmod +x ballista-scheduler ballista-executor
80+
```
81+
82+
It is now possible to run the Ballista scheduler and executor natively on the Pi.
83+
84+
## Docker
85+
86+
Using Docker's `buildx` cross-platform functionality, we can also build a docker image targeting ARM64
87+
from any desktop environment. This will require write access to a Docker repository
88+
on [Docker Hub](https://hub.docker.com/) because the resulting Docker image will be pushed directly
89+
to the repo.
90+
91+
```bash
92+
DOCKER_REPO=myrepo ./dev/build-ballista-docker-arm64.sh
93+
```
94+
95+
On the Raspberry Pi:
96+
97+
```bash
98+
docker pull myrepo/ballista-arm64
99+
```
100+
101+
Run a scheduler:
102+
103+
```bash
104+
docker run -it myrepo/ballista-arm64 /ballista-scheduler
105+
```
106+
107+
Run an executor:
108+
109+
```bash
110+
docker run -it myrepo/ballista-arm64 /ballista-executor
111+
```
112+
113+
Run the benchmarks:
114+
115+
```bash
116+
docker run -it myrepo/ballista-arm64 \
117+
/tpch benchmark --query=1 --path=/path/to/data --format=parquet \
118+
--concurrency=24 --iterations=1 --debug --host=ballista-scheduler --port=50050
119+
```
120+
121+
Note that it will be necessary to mount appropriate volumes into the containers and also configure networking
122+
so that the Docker containers can communicate with each other. This can be achieved using Docker compose or Kubernetes.
123+
124+
## Kubernetes
125+
126+
With Docker images built using the instructions above, it is now possible to deploy Ballista to a Kubernetes cluster
127+
running on one of more Raspberry Pi computers. Refer to the instructions in the [Kubernetes](kubernetes.md) chapter
128+
for more information, and remember to change the Docker image name to `myrepo/ballista-arm64`.

0 commit comments

Comments
 (0)