Databricks Asset Bundles (expect DABs to supersede DBX!!)

[LauJohansson]

DABs will supersede DBX: databrickslabs/dbx#806 (comment)

Projects that already uses dbx can continue that for now... (But new projects should use DAB)

what is it: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/
deploy notebooks: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/how-to/use-bundles-with-jobs
deploy python wheel: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/python-wheel
run CICD pipeline in github: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/ci-cd

Example of creating a python wheel

# yaml-language-server: $schema=bundle_config_schema.json
bundle:
  name: my-wheel-bundle

artifacts:
  my-wheel:
    type: whl
    path: ./my_package

resources:
  jobs:
    wheel-job:
      name: wheel-job
      tasks:
        - task_key: wheel-task
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_DS3_v2
            data_security_mode: USER_ISOLATION
            num_workers: 1
          python_wheel_task:
            entry_point: run
            package_name: my_package
          libraries:
            - whl: ./my_package/dist/my_package-*.whl

environments:
  development:
    workspace:
    host: <workspace-url>

Validate configurations with:

databricks bundle validate.

Build the wheel locally, deploy the built wheel to your workspace, deploy the notebook to your workspace, and create the job in your workspace:

databricks bundle deploy

Run the deployed job, which uses the deployed notebook to call the deployed wheel:

databricks bundle run wheel-job

[LauJohansson]

@mrmasterplan - maybe this can be utilized to execute a test job using the wheel and run unittests?