Add automated tooling and a guide for building and installing Kedro on a Databricks cluster #2369

jmholzer · 2023-02-27T20:05:28Z

Description

Resolves #2318.

This PR adds tooling for building a local version of Kedro and uploading it to the developer's Databricks cluster. This greatly speeds up development time and is an important step towards improving Kedro for Databricks.

It also adds documentation for using this tooling.

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

astrojuanlu · 2023-03-01T10:48:28Z

🤖 Rendered version: https://kedro--2369.org.readthedocs.build/en/2369/contribution/development_for_databricks.html

docs/source/contribution/development_for_databricks.md

stichbury · 2023-03-01T11:16:08Z

docs/source/contribution/development_for_databricks.md

+
+1. Python **version >=3.8** installed.
+2. A Python virtual environment activated in which the [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html) is installed with [authentication for your workspace](https://docs.databricks.com/dev-tools/cli/index.html#set-up-the-cli).
+3. Access to a Databricks workspace with an [existing cluster](https://docs.databricks.com/clusters/create-cluster.html).


Suggested change

3. Access to a Databricks workspace with an [existing cluster](https://docs.databricks.com/clusters/create-cluster.html).

3. Access to a Databricks workspace on an [existing cluster](https://docs.databricks.com/clusters/create-cluster.html).

No mention of type of cluster, e.g. AWS? We have a deployment guide and it seems to only support AWS at present, so is that a prerequisite or can any cloud provider cluster be used? Probably worth saying or pointing to a relevant databricks link?

This is a good point, though creating a cluster on Databricks is actually a cloud-agnostic process on Databricks. The linked documentation actually covers the topic really well and the user can switch the cloud provider in the top right to view the (very minor) differences for their specific cloud. For this reason I don't think it's important that we specify a cloud here.

docs/source/contribution/development_for_databricks.md

…ro-org/kedro into feat/add-databricks-build-automation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

jmholzer · 2023-03-01T16:04:58Z

Thanks @jmholzer this is really good. I made a fair few comments because I think this is the kind of page that needs crystal clarity, but really it was almost there anyway, so apologies if it looks a bit mangled now.

Awesome, thanks so much for the feedback @stichbury. I have addressed the comments you left in my latest round of commits.

docs/source/contribution/development_for_databricks.md

stichbury · 2023-03-02T10:16:13Z

docs/source/contribution/development_for_databricks.md

+This page is for people developing Kedro, users working on their own Kedro projects should see the documentation for [deploying Kedro projects on Databricks](../deployment/databricks.md).
+```
+
+This guide describes how to efficiently develop features and fixes for Kedro on Databricks. Using this guide, you will be able to quickly test your locally modified version of Kedro on Databricks as part of a build-and-test development cycle.


I'd be tempted to move these two sentences to precede the note rather than follow it, but will leave it up to you.

That's fair, it's probably bad form to start a section with a note. Fixed this.

docs/source/contribution/development_for_databricks.md

stichbury · 2023-03-02T10:19:24Z

docs/source/contribution/development_for_databricks.md

+4. Queues your updated wheel file for installation
+5. Restarts your cluster to apply the changes.
+
+Note that your cluster will be unavailable while it restarts. You can poll the status of the cluster using the Databricks CLI command `databricks clusters get --cluster-id <your-cluster-id> | grep state` (macOS, Linux).


Still not clear what Windows users do

You're right about this, I'm sorry I missed it. I updated both this command and the previous Windows one with PowerShell commands that I verified as working.

stichbury

Nice! Thanks for the updates. Just a couple of minor comments, but this looks great to me.

SajidAlamQB

This looks awesome @jmholzer! 🌟 Left some minor comments, and have tested it manually and it all seems to work. Haven't gone into the DataBricks API code that deeply but it all looks good and will look into it further during the session.

Makefile

docs/source/contribution/development_for_databricks.md

tools/databricks_build.py

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

merelcht · 2023-03-02T13:56:13Z

docs/source/contribution/development_for_databricks.md

+}
+```
+
+Any runs of a Kedro project on this cluster will now reflect your latest local changes to Kedro. You can now test your changes to Kedro by using your cluster to run a Kedro project.


I'm guessing that with any new change you'd have to run make databricks-build again?

That's right, this is a one-off run. The alternative would be to create a process which watches /dist, using e.g. dbx sync.

I think this makes sense for now. A continues sync process might be handy if we're building the kedro-databricks plugin, but by that time there might already be another way to do that.

SajidAlamQB · 2023-03-02T14:00:01Z

tools/databricks_build.py

+    """Get the path of the whl file to install. If multiple whl files are found,
+    return the file with the highest version number.
+    """
+    dist_path = Path(__file__).resolve().parent.parent / "dist"


Maybe it would be better to clean dist after each run of make databricks-build

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

…ro-org/kedro into feat/add-databricks-build-automation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com>

SajidAlamQB

Awesome job with this, thank you! 🌟

…n a Databricks cluster (kedro-org#2369) * Add databricks_build.py Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add databricks-build target to Makefile Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add new documentation page Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Complete first draft of the documentation page for databricks-build Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add verification step Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove spurious newlines at end of file Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Modify subtitle capitalisation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Lint Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add databricks guide to index Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add release note Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add note directing Kedro users to Databricks deployment docs Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add note for Kedro developers to project deployment guide Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove 'prerequisites' Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Modify numbers to bullet points Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove spurious reference to 'version' Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove spurious 'installed' Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Modify git repo requirement Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add Windows env variable instructions and cluster ID example Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add 'In your terminal,' to instruction Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add final sentence Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Reformulate final sentence Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Reformulate python requirement Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Change order of introductory note Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add PowerShell commands Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> * Update tools/databricks_build.py Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> --------- Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Danny Farah <danny.farah@quantumblack.com>

jmholzer added 3 commits February 27, 2023 20:03

Add databricks_build.py

33f3d58

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Add databricks-build target to Makefile

8059f7c

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Add new documentation page

da6a97c

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

jmholzer changed the title ~~Feat/add databricks build automation~~ Add automated tooling for building and installing Kedro on a Databricks cluster Feb 27, 2023

jmholzer added 5 commits February 27, 2023 22:22

Complete first draft of the documentation page for databricks-build

cf783d2

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Add verification step

3d44e3b

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Remove spurious newlines at end of file

a1bb716

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Modify subtitle capitalisation

356bf69

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Lint

bbf5c7d

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

jmholzer mentioned this pull request Feb 28, 2023

Add build-and-install tooling for developing Kedro for Databricks and documentation on how to use it #2318

Closed

jmholzer and others added 3 commits February 28, 2023 15:48

Add databricks guide to index

6c8a830

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Add release note

458e946

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Merge branch 'main' into feat/add-databricks-build-automation

23458a4

jmholzer marked this pull request as ready for review February 28, 2023 15:52

jmholzer requested review from yetudada and idanov as code owners February 28, 2023 15:52

jmholzer requested review from merelcht and stichbury and removed request for idanov and yetudada February 28, 2023 15:53

jmholzer changed the title ~~Add automated tooling for building and installing Kedro on a Databricks cluster~~ Add automated tooling and a guide for building and installing Kedro on a Databricks cluster Feb 28, 2023

jmholzer requested a review from SajidAlamQB February 28, 2023 18:25

stichbury reviewed Mar 1, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 1, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 1, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 1, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 1, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 1, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

jmholzer and others added 4 commits March 1, 2023 15:45

Merge branch 'feat/add-databricks-build-automation' of github.com:ked…

c601025

…ro-org/kedro into feat/add-databricks-build-automation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Reformulate final sentence

6ee2598

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Reformulate python requirement

0d622ac

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Merge branch 'main' into feat/add-databricks-build-automation

960f8c2

stichbury reviewed Mar 2, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 2, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 2, 2023

View reviewed changes

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

stichbury reviewed Mar 2, 2023

View reviewed changes

stichbury approved these changes Mar 2, 2023

View reviewed changes

SajidAlamQB reviewed Mar 2, 2023

View reviewed changes

Makefile Show resolved Hide resolved

docs/source/contribution/development_for_databricks.md Outdated Show resolved Hide resolved

docs/source/contribution/development_for_databricks.md Show resolved Hide resolved

tools/databricks_build.py Outdated Show resolved Hide resolved

jmholzer and others added 4 commits March 2, 2023 13:30

Update docs/source/contribution/development_for_databricks.md

411b01b

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

Change order of introductory note

f60981b

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Update docs/source/contribution/development_for_databricks.md

083b09c

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

Update docs/source/contribution/development_for_databricks.md

d992474

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

merelcht reviewed Mar 2, 2023

View reviewed changes

SajidAlamQB reviewed Mar 2, 2023

View reviewed changes

jmholzer and others added 5 commits March 2, 2023 14:36

Add PowerShell commands

7a70801

Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Merge branch 'feat/add-databricks-build-automation' of github.com:ked…

89bef49

…ro-org/kedro into feat/add-databricks-build-automation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com>

Update docs/source/contribution/development_for_databricks.md

e9195f4

Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com>

Update tools/databricks_build.py

ee657ab

Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com>

Merge branch 'main' into feat/add-databricks-build-automation

a3cd72a

SajidAlamQB approved these changes Mar 6, 2023

View reviewed changes

Merge branch 'main' into feat/add-databricks-build-automation

4df8187

jmholzer merged commit 48b8dbc into main Mar 6, 2023

jmholzer deleted the feat/add-databricks-build-automation branch March 6, 2023 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automated tooling and a guide for building and installing Kedro on a Databricks cluster #2369

Add automated tooling and a guide for building and installing Kedro on a Databricks cluster #2369

jmholzer commented Feb 27, 2023 •

edited

Loading

astrojuanlu commented Mar 1, 2023

stichbury Mar 1, 2023

stichbury Mar 1, 2023

jmholzer Mar 1, 2023

jmholzer commented Mar 1, 2023

stichbury Mar 2, 2023

jmholzer Mar 2, 2023

stichbury Mar 2, 2023

jmholzer Mar 2, 2023

stichbury left a comment

SajidAlamQB left a comment

merelcht Mar 2, 2023

jmholzer Mar 2, 2023

merelcht Mar 2, 2023

SajidAlamQB Mar 2, 2023

SajidAlamQB left a comment

	3. Access to a Databricks workspace with an [existing cluster](https://docs.databricks.com/clusters/create-cluster.html).
	3. Access to a Databricks workspace on an [existing cluster](https://docs.databricks.com/clusters/create-cluster.html).

Add automated tooling and a guide for building and installing Kedro on a Databricks cluster #2369

Add automated tooling and a guide for building and installing Kedro on a Databricks cluster #2369

Conversation

jmholzer commented Feb 27, 2023 • edited Loading

Description

Checklist

astrojuanlu commented Mar 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmholzer commented Mar 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stichbury left a comment

Choose a reason for hiding this comment

SajidAlamQB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SajidAlamQB left a comment

Choose a reason for hiding this comment

jmholzer commented Feb 27, 2023 •

edited

Loading