-
Notifications
You must be signed in to change notification settings - Fork 893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Databricks docs #3360
Comments
And also dbx:
https://docs.databricks.com/en/archive/dev-tools/dbx/index.html |
I'll take a look at this in an upcoming sprint -- we did some updates for the asset bundles recently as suggested by Harmony. |
Another couple of things I found in our Databricks workspace guide
It's true that creating a Databricks Repo synced with a GH repo gives some nice advantages, like being able to edit the code in an actual IDE (whether a local editor or a cloud development environment like Gitpod or GitHub Codespaces). And it's also true that Databricks recommends in different places that data should live in the DBFS root. However, it would be nice to consider what's the shortest and simplest guide we can write for users to get started with Kedro on Databricks, and then build from there. |
To clarify on the initial comment:
Both things above require the Databricks CLI version 0.205 and above. Apart from that, the commands haven't changed, so what we should do in this regard is making sure we're not sending users to legacy docs and that's it. |
To summarise:
|
@astrojuanlu Is this something that the team can pick up or do we need to ask for time from Jannic or another databricks expert (maybe @deepyaman could ultimately review)? What are we prioritising this? I'm guessing it's relatively high importance to keep databricks docs tracking with their tech. |
We need to build Databricks expertise in the team, so I hope we don't need to ask external experts to do it (it's OK if they give assistance, but we need to own this). |
Added this to the Inbox so that we prioritise. |
At this point, it's been almost 4 years since I've used Databricks (and don't currently have any interest in getting back into it), so I'd defer to somebody else. 🙂 |
More than fair enough @deepyaman! Good to confirm though. |
I'm adding one more item:
Every time I give a talk or workshop, invariably somebody from the audience asks "how does the Kedro Catalog play along with Databricks Unity Catalog?". Our reference docs for And there's one subtle mention of it in databricks.ManagedTableDataset ("the name of the catalog in Unity". The broader question of Delta datasets is a topic for kedro-org/kedro-plugins#542. |
Maybe this could help: |
This looks really cool. @JenspederM do you want to share a bit more insight on how far do you intend to go with your project? |
Hey @astrojuanlu Actually I don't really know if there's more to do. I almost want the project to be as barebone as possible. The way I left it now is with a very simple As for the DAB resource generator, I'm considering if I could find a better way for users to set defaults such as which job clusters, instance pools, etc.. One thing that is generally lacking is the documentation so that will definitely receive some attention once I have the time. Do you have any suggestions? |
I gave two Kedro on Databricks demos yesterday, so I'm sharing that very simple notebook here https://github.com/astrojuanlu/kedro-databricks-demo hopefully it can be the basis of what I proposed in #3360 (comment) (still no Kedro Framework there) |
@JenspederM I gave your |
@astrojuanlu Go for it! I've been a bit busy these last few days and haven't had the chance to make any progress. But it's always nice to have some concrete issues to address. 😉 |
@astrojuanlu Just fyi, I'll merge a quite big PR soon so hopefully that will address most issues that you found. The substitution algorithm was a bit more cumbersome than first anticipated.. |
There is now a community plugin exists: https://github.com/JenspederM/kedro-databricks We need to update the documentation according to the readme to walkthrough the steps to setup on Databricks. |
Description
It looks like Databricks have deprecated their cli tools, which has had the knock on effect of breaking our docs. A quick fix in #3358 adds the necessary
/archive/
to the broken links, but maybe we should rethink the section as a whole?CC: @stichbury
docs: https://docs.kedro.org/en/stable/deployment/databricks/databricks_ide_development_workflow.html
The text was updated successfully, but these errors were encountered: