Delta Lake Quickstart Docker

This folder contains instructions and materials to get new users started with Delta Lake and work through the quickstart materials using a self-contained Docker image.

Note: The basic prerequisite for following along using Delta Lake Docker image is having Docker installed on your machine. Please follow the steps from the Docker website to install Docker locally. Based on your local machine operating system, please choose the appropriate option listed on the Get Docker page.

Follow the steps below to build an Apache Spark^TM image with Delta Lake installed, run a container, and follow the quickstart in an interactive notebook or shell with any of the options like Python, PySpark, Scala Spark or even Rust.

Working with Docker
1. Build the image
2. Docker Hub
Choose an interface

Note: Python version available in this Docker image is 3.9.2 and is available as python3.

Working with Docker

Build the Image

Clone this repo
Navigate to the cloned folder
Navigate to the quickstart_docker folder
Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)

Execute the following from the static/quickstart_docker folder

docker build -t delta_quickstart -f Dockerfile_delta_quickstart .

Build Entry Point

Your entry point for this locally built docker file is

docker run --name delta_quickstart --rm -it --entrypoint bash delta_quickstart

Docker Hub

You can also download the image from DockerHub at Delta Lake DockerHub

Note, there are different versions of the Delta Lake docker

Tag	Platform	Python	Rust	Delta-Spark	Spark	JupyterLab	Pandas	ROAPI
0.8.1_2.3.0	amd64	0.8.1	latest	2.3.0	3.3.2	3.6.3	1.5.3	0.9.0
0.8.1_2.3.0_arm64	arm64	0.8.1	latest	2.3.0	3.3.2	3.6.3	1.5.3	0.9.0
1.0.0_3.0.0	amd64	0.12.0	latest	3.0.0	3.5.0	3.6.3	1.5.3	0.9.0
1.0.0_3.0.0_arm64	arm64	0.12.0	latest	3.0.0	3.5.0	3.6.3	1.5.3	0.9.0
latest	amd64	0.12.0	latest	3.0.0	3.5.0	3.6.3	1.5.3	0.9.0
latest	arm64	0.12.0	latest	3.0.0	3.5.0	3.6.3	1.5.3	0.9.0

Note, the arm64 version is built for ARM64 platforms like Mac M1

Download the appropriate tag, e.g.:

docker pull deltaio/delta-docker:latest for the standard Linux docker
docker pull deltaio/delta-docker:latest_arm64 for running this optimally on your Mac M1

Image Entry Point

Your entry point for the Docker Hub image is:

# Running locally on Mac M1
docker run --name delta_quickstart --rm -it --entrypoint bash deltaio/delta-docker:latest_arm64

# Running on Linux VM
docker run --name delta_quickstart --rm -it --entrypoint bash deltaio/delta-docker:latest

Once the image has been built or you have downloaded the correct image, you can then move on to running the quickstart in a notebook or shell.

Choose the Delta Package version

In the following instructions, the variable ${DELTA_PACKAGE_VERSION} refers to the Delta Lake Package version.

The current version is delta-spark_2.12:3.0.0 which corresponds to Apache Spark 3.5.x release line.

delta-rs Python

Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)
Run a container from the image with a bash entrypoint (build | DockerHub)
Launch a python interactive shell session with python3
```
python3
```
Note: The Delta Rust Python bindings are already installed in this docker. To do this manually in your own environment, run the command: pip3 install deltalake==0.12.0

Run some basic commands in the shell to write to and read from Delta Lake with Pandas

import pandas as pd
from deltalake.writer import write_deltalake
from deltalake import DeltaTable

# Create a Pandas DataFrame
df = pd.DataFrame({"data": range(5)})

# Write to the Delta Lake table
write_deltalake("/tmp/deltars_table", df)

# Append new data
df = pd.DataFrame({"data": range(6, 11)})
write_deltalake("/tmp/deltars_table", df, mode="append")

# Read the Delta Lake table
dt = DeltaTable("/tmp/deltars_table")

# Show the Delta Lake table
dt.to_pandas()

## Output
   data
0     0
1     1
2     2
...
8     9
9    10

Review the files

# List files for the Delta Lake table
dt.files()

## Output
['0-6944fddf-60e3-4eab-811d-1398e9f64073-0.parquet', '1-66c7ee6e-6aab-4c74-866d-a82790102652-0.parquet']

Review history

# Review history
dt.history()

## Output
[{'timestamp': 1698002214493, 'operation': 'WRITE', 'operationParameters': {'mode': 'Append', 'partitionBy': '[]'}, 'clientVersion': 'delta-rs.0.17.0', 'version': 1}, {'timestamp': 1698002207527, 'operation': 'CREATE TABLE', 'operationParameters': {'mode': 'ErrorIfExists', 'protocol': '{"minReaderVersion":1,"minWriterVersion":1}', 'location': 'file:///tmp/deltars_table', 'metadata': '{"configuration":{},"created_time":1698002207525,"description":null,"format":{"options":{},"provider":"parquet"},"id":"bf749aab-22b6-484b-bd73-dc1680ee4384","name":null,"partition_columns":[],"schema":{"fields":[{"metadata":{},"name":"data","nullable":true,"type":"long"}],"type":"struct"}}'}, 'clientVersion': 'delta-rs.0.17.0', 'version': 0}]

Time Travel (load older version of table)

# Load initial version of table
dt.load_version(0)

# Show table
dt.to_pandas()

## Output
   data
0     0
1     1
2     2
3     3
4     4

Follow the delta-rs Python documentation here
To verify that you have a Delta Lake table, you can list the contents within the folder of your Delta Lake table. For example, in the previous code, you saved the table in /tmp/deltars-table. Once you close your python3 process, run a list command in your Docker shell and you should get something similar to below.
```
$ ls -lsgA /tmp/deltars_table
```
```
total 12
4 -rw-r--r-- 1 NBuser 1689 Oct 22 19:16 0-6944fddf-60e3-4eab-811d-1398e9f64073-0.parquet
4 -rw-r--r-- 1 NBuser 1691 Oct 22 19:16 1-66c7ee6e-6aab-4c74-866d-a82790102652-0.parquet
4 drwxr-xr-x 2 NBuser 4096 Oct 22 19:16 _delta_log
```
[Optional] Skip ahead to try out the Delta Rust API and ROAPI

JupyterLab Notebook

Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)

Run a container from the image with a JuypterLab entrypoint

# Build entry point
docker run --name delta_quickstart --rm -it -p 8888-8889:8888-8889 delta_quickstart

# Image entry point (M1)
docker run --name delta_quickstart --rm -it -p 8888-8889:8888-8889 -entrypoint bash deltaio/delta-docker:latest_arm64

Running the above command gives a JupyterLab notebook URL, copy that URL and launch a browser to follow along the notebook and run each cell.

Note that you may also launch the pyspark or scala shells after launching a terminal in JupyterLab

PySpark Shell

Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)
Run a container from the image with a bash entrypoint (build | DockerHub)

Launch a pyspark interactive shell session

$SPARK_HOME/bin/pyspark --packages io.delta:${DELTA_PACKAGE_VERSION} \
--conf spark.driver.extraJavaOptions="-Divy.cache.dir=/tmp -Divy.home=/tmp" \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

Note: DELTA_PACKAGE_VERSION is set in ./startup.sh

Run some basic commands in the shell

# Create a Spark DataFrame
data = spark.range(0, 5)

# Write to a Delta Lake table
(data
   .write
   .format("delta")
   .save("/tmp/delta-table")
)

# Read from the Delta Lake table
df = (spark
        .read
        .format("delta")
        .load("/tmp/delta-table")
        .orderBy("id")
      )

# Show the Delta Lake table
df.show()

## Output
+---+
| id|
+---+
|  0|
|  1|
|  2|
|  3|
|  4|
+---+

Continue with the quickstart here

To verify that you have a Delta Lake table, you can list the contents within the folder of your Delta Lake table. For example, in the previous code, you saved the table in /tmp/delta-table. Once you close your pyspark process, run a list command in your Docker shell and you should get something similar to below.

$ ls -lsgA /tmp/delta-table

total 52
4 drwxr-xr-x 2 NBuser 4096 Oct 22 19:23 _delta_log
4 -rw-r--r-- 1 NBuser  296 Oct 22 19:23 part-00000-dc0fd6b3-9c0f-442f-a6db-708301b27bd2-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:23 .part-00000-dc0fd6b3-9c0f-442f-a6db-708301b27bd2-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:23 part-00001-d379441e-1ee4-4e78-8616-1d9635df1c7b-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:23 .part-00001-d379441e-1ee4-4e78-8616-1d9635df1c7b-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:23 part-00003-c08dcac4-5ea9-4329-b85d-9110493e8757-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:23 .part-00003-c08dcac4-5ea9-4329-b85d-9110493e8757-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:23 part-00005-5db8dd16-2ab1-4d76-9b4d-457c5641b1c8-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:23 .part-00005-5db8dd16-2ab1-4d76-9b4d-457c5641b1c8-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:23 part-00007-cad760e0-3c26-4d22-bed6-7d75a9459a0f-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:23 .part-00007-cad760e0-3c26-4d22-bed6-7d75a9459a0f-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:23 part-00009-b58e8445-07b7-4e2a-9abf-6fea8d0c3e3f-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:23 .part-00009-b58e8445-07b7-4e2a-9abf-6fea8d0c3e3f-c000.snappy.parquet.crc

Scala Shell

Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)
Run a container from the image with a bash entrypoint (build | DockerHub)

Launch a scala interactive shell session

$SPARK_HOME/bin/spark-shell --packages io.delta:${DELTA_PACKAGE_VERSION} \
--conf spark.driver.extraJavaOptions="-Divy.cache.dir=/tmp -Divy.home=/tmp" \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

Run some basic commands in the shell

note: if you've already written to the Delta table in the python shell example, use .mode("overwrite") to overwrite the current delta table. You can always time-travel to rewind.

// Create a Spark DataFrame
val data = spark.range(0, 5)

// Write to a Delta Lake table

(data
   .write
   .format("delta")
   .save("/tmp/delta-table")
)

// Read from the Delta Lake table
val df = (spark
            .read
            .format("delta")
            .load("/tmp/delta-table")
            .orderBy("id")
         )

// Show the Delta Lake table
df.show()

## Output
+---+
| id|
+---+
|  0|
|  1|
|  2|
|  3|
|  4|
+---+

Follow the quickstart here

To verify that you have a Delta Lake table, you can list the contents within the folder of your Delta Lake table. For example, in the previous code, you saved the table in /tmp/delta-table. Once you close your Scala Spark process [spark-shell], run a list command in your Docker shell and you should get something similar to below.

$ ls -lsgA /tmp/delta-table

total 52
4 drwxr-xr-x 2 NBuser 4096 Oct 22 19:28 _delta_log
4 -rw-r--r-- 1 NBuser  296 Oct 22 19:28 part-00000-f1f417f7-df64-4c7c-96f2-6a452ae2b49e-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:28 .part-00000-f1f417f7-df64-4c7c-96f2-6a452ae2b49e-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:28 part-00001-b28acb6f-f08a-460f-a24e-4d9c1affee86-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:28 .part-00001-b28acb6f-f08a-460f-a24e-4d9c1affee86-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:28 part-00003-29079c58-d1ad-4604-9c04-0f00bf09546d-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:28 .part-00003-29079c58-d1ad-4604-9c04-0f00bf09546d-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:28 part-00005-04424aa7-48e1-4212-bd57-52552c713154-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:28 .part-00005-04424aa7-48e1-4212-bd57-52552c713154-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:28 part-00007-e7a54a4f-bee4-4371-a35d-d284e28eb9f8-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:28 .part-00007-e7a54a4f-bee4-4371-a35d-d284e28eb9f8-c000.snappy.parquet.crc
4 -rw-r--r-- 1 NBuser  478 Oct 22 19:28 part-00009-086e6cd9-e8c6-4f16-9658-b15baf22905d-c000.snappy.parquet
4 -rw-r--r-- 1 NBuser   12 Oct 22 19:28 .part-00009-086e6cd9-e8c6-4f16-9658-b15baf22905d-c000.snappy.parquet.crc

Delta Rust API

Note: Use a docker volume in case of running into limits "no room left on device" docker volume create rustbuild > docker run --name delta_quickstart -v rustbuild:/tmp --rm -it --entrypoint bash deltaio/delta-docker:3.0.0

Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)
Run a container from the image with a bash entrypoint (build | DockerHub)

Execute examples/read_delta_table.rs to review the Delta Lake table metadata and files of the covid19_nyt Delta Lake table.

cd rs
cargo run --example read_delta_table

You can also use a different location to build and run the examples

cd rs
CARGO_TARGET_DIR=/tmp cargo run --example read_delta_table

If using Delta Lake DockerHub, sometimes the Rust environment hasn't been configured. To resolve this, run the command source "$HOME/.cargo/env"

=== Delta table metadata ===
DeltaTable(/opt/spark/work-dir/rs/data/COVID-19_NYT)
   version: 0
   metadata: GUID=7245fd1d-8a6d-4988-af72-92a95b646511, name=None, description=None, partitionColumns=[], createdTime=Some(1619121484605), configuration={}
   min_version: read=1, write=2
   files count: 8


=== Delta table files ===
[Path { raw: "part-00000-a496f40c-e091-413a-85f9-b1b69d4b3b4e-c000.snappy.parquet" }, Path { raw: "part-00001-9d9d980b-c500-4f0b-bb96-771a515fbccc-c000.snappy.parquet" }, Path { raw: "part-00002-8826af84-73bd-49a6-a4b9-e39ffed9c15a-c000.snappy.parquet" }, Path { raw: "part-00003-539aff30-2349-4b0d-9726-c18630c6ad90-c000.snappy.parquet" }, Path { raw: "part-00004-1bb9c3e3-c5b0-4d60-8420-23261f58a5eb-c000.snappy.parquet" }, Path { raw: "part-00005-4d47f8ff-94db-4d32-806c-781a1cf123d2-c000.snappy.parquet" }, Path { raw: "part-00006-d0ec7722-b30c-4e1c-92cd-b4fe8d3bb954-c000.snappy.parquet" }, Path { raw: "part-00007-4582392f-9fc2-41b0-ba97-a74b3afc8239-c000.snappy.parquet" }]

Execute examples/read_delta_datafusion.rs to query the covid19_nyt Delta Lake table using datafusion

cargo run --example read_delta_datafusion

=== Datafusion query ===
[RecordBatch { schema: Schema { fields: [Field { name: "cases", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "county", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: "date", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }], metadata: {} }, columns: [PrimitiveArray<Int32>
[
   1,
   1,
   1,
   1,
   1,
], StringArray
[
   "Snohomish",
   "Snohomish",
   "Snohomish",
   "Cook",
   "Snohomish",
], StringArray
[
   "2020-01-21",
   "2020-01-22",
   "2020-01-23",
   "2020-01-24",
   "2020-01-24",
]], row_count: 5 }]

[Optional] ROAPI

You can query your Delta Lake table with Apache Arrow and Datafusion using ROAPI which are pre-installed in this docker.

Note: If you need to do this in your own environment, run the command pip3 install roapi==0.9.0

Open a bash shell (if on windows use git bash, WSL, or any shell configured for bash commands)
Run a container from the image with a bash entrypoint (build | DockerHub)
Start the roapi API using the following command. Notes:
- the API calls are pushed to the nohup.out file.
- if you haven't created the deltars_table in your container create it via the delta-rs Python option above. Alternatively you may omit the following from the command: --table 'deltars_table=/tmp/deltars_table/,format=delta' as well as any steps that call the deltars_table
```
nohup roapi --addr-http 0.0.0.0:8080 --table 'deltars_table=/tmp/deltars_table/,format=delta' --table 'covid19_nyt=/opt/spark/work-dir/rs/data/COVID-19_NYT,format=delta' &
```

Check the schema of the two Delta Lake tables

curl localhost:8080/api/schema

{
   "covid19_nyt":{"fields":[
      {"name":"date","data_type":"Utf8","nullable":true,"dict_id":0,"dict_is_ordered":false},
      {"name":"county","data_type":"Utf8","nullable":true,"dict_id":0,"dict_is_ordered":false},
      {"name":"state","data_type":"Utf8","nullable":true,"dict_id":0,"dict_is_ordered":false},
      {"name":"fips","data_type":"Int32","nullable":true,"dict_id":0,"dict_is_ordered":false},
      {"name":"cases","data_type":"Int32","nullable":true,"dict_id":0,"dict_is_ordered":false},
      {"name":"deaths","data_type":"Int32","nullable":true,"dict_id":0,"dict_is_ordered":false}
   ]},
   "deltars_table":{"fields":[
      {"name":"0","data_type":"Int64","nullable":true,"dict_id":0,"dict_is_ordered":false}
   ]}
}

Query the deltars_table

curl -X POST -d "SELECT * FROM deltars_table"  localhost:8080/api/sql

# output
[{"0":0},{"0":1},{"0":2},{"0":3},{"0":4},{"0":6},{"0":7},{"0":8},{"0":9},{"0":10}]

Query the covid19_nyt table

curl -X POST -d "SELECT cases, county, date FROM covid19_nyt ORDER BY cases DESC LIMIT 5" localhost:8080/api/sql

[
   {"cases":1208672,"county":"Los Angeles","date":"2021-03-11"},
   {"cases":1207361,"county":"Los Angeles","date":"2021-03-10"},
   {"cases":1205924,"county":"Los Angeles","date":"2021-03-09"},
   {"cases":1204665,"county":"Los Angeles","date":"2021-03-08"},
   {"cases":1203799,"county":"Los Angeles","date":"2021-03-07"}
]

Name		Name	Last commit message	Last commit date
Latest commit History 77 Commits
.vscode		.vscode
0.8.1_2.3.0		0.8.1_2.3.0
0.9.0_2.3.0		0.9.0_2.3.0
rs		rs
.gitignore		.gitignore
Dockerfile_delta_quickstart		Dockerfile_delta_quickstart
README.md		README.md
quickstart.ipynb		quickstart.ipynb
startup.sh		startup.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Delta Lake Quickstart Docker

Working with Docker

Build the Image

Build Entry Point

Docker Hub

Image Entry Point

Choose the Delta Package version

Choose an Interface

delta-rs Python

JupyterLab Notebook

PySpark Shell

Scala Shell

Delta Rust API

[Optional] ROAPI

About

Releases

Packages

Contributors 10

Languages

delta-io/delta-docker

Folders and files

Latest commit

History

Repository files navigation

Delta Lake Quickstart Docker

Working with Docker

Build the Image

Build Entry Point

Docker Hub

Image Entry Point

Choose the Delta Package version

Choose an Interface

delta-rs Python

JupyterLab Notebook

PySpark Shell

Scala Shell

Delta Rust API

[Optional] ROAPI

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 10

Languages

Packages