[Question] A question about the planned future of DBX

[allebacco]

DBX is a great tool, and I love it. It has simplified my Databricks workflows.

However, Databricks tools evolved since the introduction of DBX and now there is a new CLI (written in Go) and a new Python SDK.

So, considering that, what is the planned future of DBX?

• will it be actively developed, by, e.g., using the Databricks Python SDK instead of the old Python Databricks CLI?
• will it be replaced by the new Databricks Bundle concept (which uses Terraform under the hood)?
• will it be archived?

[allebacco]

@crash-g FYI

[copdips]

hello,

could you please provide the tools links for now there is a new CLI (written in Go) and a new Python SDK

[allebacco]

Hello @copdips.

You can reach the Databricks platform release notes for June 2023 for more details about the new CLI and SDKs.

[copdips]

thx for sharing, one of the drawbacks of databricks-cli IMHO is that it should support py2 that some developers don't want code py2 anymore. I think building dbx on top of databricks-sdk could be a good choice, using terraform limits the CICD capability.

[pspeter]

After talking to our Databricks representative, it sounds like dbx will be superseded by Databricks Asset Bundles (currently in private preview) eventually.

[aviadshimoni]

would be any easy way to migrate workloads to databricks assets api from dbx and deployment.yaml?

any more information about the TF architecture?

I'm starting to feel that going with dbx was a terrible mistake, as we built our entire CI/CD solution around this, in a hope that it will improve in the future.

[copdips]

@aviadshimon
Dbx is as simple as a single command dbx deploy, why you said it's a terrible mistake in terms of cicd?

[aviadshimoni]

We're using dbx for the CI/CD, and it works great, but if it won't be maintained and replaced, we'll have to migrate.
the issue now is that we've multiple deployment.yaml files accross multiple repos that need to be migrated one-by-one to meet the new structure of databricks bundle, correct me if I'm wrong.
If we knew that it will be replaced after 6M since we've started to use it, we would stick to the UI and wait to build the solution around databricks bundle.

[copdips]

Just reviewed the schema generated by the new databricks bundle command. Well, the format has changed significantly. Maybe dbx or cli will introduce a feature to facilitate the migration in the future. BTW the new cli is written in go, I have no idea if databricks bundle deploy will upload the generated wheel file to artifact or not, or perform any smart name-to-id conversions like dbx does.

[aviadshimoni]

Thank you @copdips , I'm starting a POC on DAB now, maybe will update on this thread.

[pietern]

Hey folks, we’ve been working on Databricks Asset Bundles (or DABs) and while we do expect DABs to supersede DBX, it’s not quite there yet. DBX is a great tool and is supported by the Databricks Labs team. In fact, DABs are inspired by DBX and many of its capabilities form the basis of the approach in DABs.

If you’re using DBX, please continue doing so. DABs won’t be generally available until we have migration guidance in place. As we’re getting closer to this point, we’ll be actively looking for feedback. If you have specific concerns, please reach out dabs-preview@databricks.com.

If you are starting off with a new project, we encourage you to try out DABs (you can reach out to your account team or contact us at the same email address). Please keep in mind that DABs is still in preview and that not all features you’re used to from DBX have an equivalent in DABs yet. For example, automatic Python wheel builds and uploads are not yet available in DABs, but will be in the near future.