Update Databricks cli version and Databricks deploy process (#73)

fabio-padua · Copilot · web-flow · commit c75f355ba302 · 2025-06-04T11:12:50.000-03:00
* Rebase (#3) * chore: enviornments variables adjustment * chore: schema update * fix: add SP version on job variable * chore: force powershell version * chore: change task order to publish DB secrets * chore: Adjustments on Databricks Secrets deploy * chore: adjust variables info on parameter * chore: adjustment on Databricks scripts * chore: dataLakeName variable adjustment * chore: dataComputeRS variable change * chore: documentation adjustment att image * chore: change keyvalt set permissions on SP * chore: Databricks task order * chore: databricks adjustments * chore: var output display removal * chore: databricks adjustments * chore: documentation adjustment * chore: documentation change * chore: script adjustment * chore: add command to list scope content * chore: add SP into Admin role * chore: adjust Administrator Role SP * chore: add connect to AD first * chore: Remove connection do Azure AD * chore: documentation adjustment * chore: fix uppercase extensions on some images (#42) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * Issue 43 - Templates Changes (#44) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * Issue #43 - Adjust the pipeline run to use first time SP secret (#45) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * chore: pipeline execution change * chore: documentation adjustments * feat: Issue templates creations - Bug Issue template - New Feature request issue template * Update issue templates * Issue#35 - Change the store of sample-data to current repo (#46) * Rebase (#2) * chore: enviornments variables adjustment * chore: schema update * fix: add SP version on job variable * chore: force powershell version * chore: change task order to publish DB secrets * chore: Adjustments on Databricks Secrets deploy * chore: adjust variables info on parameter * chore: adjustment on Databricks scripts * chore: dataLakeName variable adjustment * chore: dataComputeRS variable change * chore: documentation adjustment att image * chore: change keyvalt set permissions on SP * chore: Databricks task order * chore: databricks adjustments * chore: var output display removal * chore: databricks adjustments * chore: documentation adjustment * chore: documentation change * chore: script adjustment * chore: add command to list scope content * chore: add SP into Admin role * chore: adjust Administrator Role SP * chore: add connect to AD first * chore: Remove connection do Azure AD * chore: documentation adjustment * chore: fix uppercase extensions on some images (#42) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * Issue 43 - Templates Changes (#44) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * Issue #43 - Adjust the pipeline run to use first time SP secret (#45) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * chore: pipeline execution change * chore: documentation adjustments * chore: sample-data upload to main-stream * chore: unzip and upload sample-data * fix: adjustment on pipe run * Issue#47 - Change Databricks version (#48) * Rebase (#2) * chore: enviornments variables adjustment * chore: schema update * fix: add SP version on job variable * chore: force powershell version * chore: change task order to publish DB secrets * chore: Adjustments on Databricks Secrets deploy * chore: adjust variables info on parameter * chore: adjustment on Databricks scripts * chore: dataLakeName variable adjustment * chore: dataComputeRS variable change * chore: documentation adjustment att image * chore: change keyvalt set permissions on SP * chore: Databricks task order * chore: databricks adjustments * chore: var output display removal * chore: databricks adjustments * chore: documentation adjustment * chore: documentation change * chore: script adjustment * chore: add command to list scope content * chore: add SP into Admin role * chore: adjust Administrator Role SP * chore: add connect to AD first * chore: Remove connection do Azure AD * chore: documentation adjustment * chore: fix uppercase extensions on some images (#42) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * Issue 43 - Templates Changes (#44) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * Issue #43 - Adjust the pipeline run to use first time SP secret (#45) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * chore: pipeline execution change * chore: documentation adjustments * chore: sample-data upload to main-stream * chore: unzip and upload sample-data * fix: adjustment on pipe run * chore: changed databrick version * PowerShell version changed when run pipeline (#50) * Rebase (#2) * chore: enviornments variables adjustment * chore: schema update * fix: add SP version on job variable * chore: force powershell version * chore: change task order to publish DB secrets * chore: Adjustments on Databricks Secrets deploy * chore: adjust variables info on parameter * chore: adjustment on Databricks scripts * chore: dataLakeName variable adjustment * chore: dataComputeRS variable change * chore: documentation adjustment att image * chore: change keyvalt set permissions on SP * chore: Databricks task order * chore: databricks adjustments * chore: var output display removal * chore: databricks adjustments * chore: documentation adjustment * chore: documentation change * chore: script adjustment * chore: add command to list scope content * chore: add SP into Admin role * chore: adjust Administrator Role SP * chore: add connect to AD first * chore: Remove connection do Azure AD * chore: documentation adjustment * chore: fix uppercase extensions on some images (#42) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * Issue 43 - Templates Changes (#44) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * Issue #43 - Adjust the pipeline run to use first time SP secret (#45) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * chore: pipeline execution change * chore: documentation adjustments * chore: sample-data upload to main-stream * chore: unzip and upload sample-data * fix: adjustment on pipe run * chore: changed databrick version * chore: changes into PS Version on Pipe execution * [Fix] - Remove logs (#51) * Rebase (#2) * chore: enviornments variables adjustment * chore: schema update * fix: add SP version on job variable * chore: force powershell version * chore: change task order to publish DB secrets * chore: Adjustments on Databricks Secrets deploy * chore: adjust variables info on parameter * chore: adjustment on Databricks scripts * chore: dataLakeName variable adjustment * chore: dataComputeRS variable change * chore: documentation adjustment att image * chore: change keyvalt set permissions on SP * chore: Databricks task order * chore: databricks adjustments * chore: var output display removal * chore: databricks adjustments * chore: documentation adjustment * chore: documentation change * chore: script adjustment * chore: add command to list scope content * chore: add SP into Admin role * chore: adjust Administrator Role SP * chore: add connect to AD first * chore: Remove connection do Azure AD * chore: documentation adjustment * chore: fix uppercase extensions on some images (#42) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * Issue 43 - Templates Changes (#44) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * Issue #43 - Adjust the pipeline run to use first time SP secret (#45) * chore: fix uppercase extensions on some images * chore: directory changes * chore: directory rename * chore: Service Principal secrets first adjustments * fix: adjustment of SP * chore: save secret on output * chore: syntax fix * chore: token adjustment * chore: store secret inside output hol file * chore: put SP inside parameters * chore: template adjustment * chore: lower parameter for fix * chore: syntax adjustment * chore: Add logs * chore: add log info * chore: log info * chore: pipeline execution change * chore: documentation adjustments * chore: sample-data upload to main-stream * chore: unzip and upload sample-data * fix: adjustment on pipe run * chore: changed databrick version * chore: changes into PS Version on Pipe execution * chore: tests adjustments * chore: log adjustments * chore: documentation change * fix: change the Databricks cli installation and version * Update infrastructure-as-code/scripts/DatabricksSecrets.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update infrastructure-as-code/scripts/DatabricksClusters.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Fabio Padua <61516935+fabiohaifa@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
diff --git a/azure-pipelines/databricks/databricks-ci.yml b/azure-pipelines/databricks/databricks-ci.yml
@@ -4,18 +4,18 @@ variables:
   NOTEBOOK_WORKING_DIR: $(System.DefaultWorkingDirectory)/data-platform/notebooks
 
 pool:
-  vmImage: 'Ubuntu-18.04'
+  vmImage: 'ubuntu-latest'
 
 jobs:
 - job: 'validate_notebooks'
   displayName: 'Validate Databricks Notebooks'
   steps:
     - task: UsePythonVersion@0
       inputs:
-        versionSpec: 3.6
+        versionSpec: '3.9'
         addToPath: true
         architecture: 'x64'
-      displayName: 'Use Python Version: 3.6'
+      displayName: 'Use Python Version: 3.9'
 
     - script: |
         python -m pip install --upgrade pip
diff --git a/azure-pipelines/databricks/templates/databricks-deploy-library-job-template.yml b/azure-pipelines/databricks/templates/databricks-deploy-library-job-template.yml
@@ -8,13 +8,12 @@ parameters:
 
 stages:
 - stage: publish_library_${{ parameters.environment }}
-  #condition: eq(variables['Build.SourceBranch'], '${{ parameters.branch }}')
   displayName: 'Deploy to ${{ parameters.environment }} Databricks'
   jobs:
   - deployment: publish_library_${{ parameters.environment }}
     displayName: 'Deploy to ${{ parameters.environment }} Databricks'
     pool:
-      vmImage: 'Ubuntu-18.04'
+      vmImage: 'ubuntu-latest'
     environment: databricks-${{ parameters.environment }}
     variables:
     - group: dataops-iac-cd-output-${{ parameters.environment }}
diff --git a/azure-pipelines/databricks/templates/databricks-deploy-notebooks-job-template.yml b/azure-pipelines/databricks/templates/databricks-deploy-notebooks-job-template.yml
@@ -13,7 +13,7 @@ stages:
   - deployment: publish_static_artifacts_${{ parameters.environment }}
     displayName: 'Deploy to ${{ parameters.environment }} Databricks'
     pool:
-      vmImage: 'Ubuntu-18.04'
+      vmImage: 'ubuntu-latest'
     environment: databricks-${{ parameters.environment }}
     variables:
     - group:  ${{ parameters.iacCdVariableGroupPrefix }}-${{ parameters.environment }}
diff --git a/azure-pipelines/databricks/templates/databricks-setup-environment-template.yml b/azure-pipelines/databricks/templates/databricks-setup-environment-template.yml
@@ -2,12 +2,17 @@ steps:
 
 - task: UsePythonVersion@0
   inputs:
-    versionSpec: 3.6
+    versionSpec: '3.9'
     addToPath: true
     architecture: 'x64'
-  displayName: 'Use Python Version: 3.6'
-  
+  displayName: 'Use Python Version: 3.9'
+
 - script: |
-    python -m pip install --upgrade pip
-    pip install databricks-cli
-  displayName: 'Setup Agent'
+    echo "Downloading Databricks CLI last version..."
+    curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
+    chmod +x databricks
+    sudo mv databricks /usr/local/bin/databricks
+
+    echo "Verifying Databricks CLI version..."
+    databricks version
+  displayName: 'Install Databricks CLI last version (Binary)'
diff --git a/azure-pipelines/iac/templates/stage.acceptance-test.yml b/azure-pipelines/iac/templates/stage.acceptance-test.yml
@@ -13,7 +13,7 @@ stages:
   - job: test
     displayName: 'Acceptance Test for ${{ parameters.environment }}'
     pool:
-      vmImage: 'Ubuntu-20.04'
+      vmImage: 'ubuntu-latest'
     variables:
       azPowershellVersion: 7.5.0
     steps:
@@ -39,4 +39,4 @@ stages:
         testResultsFormat: 'NUnit'
         testResultsFiles: '**/testResults.xml' 
         failTaskOnFailedTests: true
-        testRunTitle: 'Pester Acceptance Tests (${{ parameters.environment }})'
+        testRunTitle: 'Pester Acceptance Tests (${{ parameters.environment }})'
diff --git a/azure-pipelines/iac/templates/stage.deploy.yml b/azure-pipelines/iac/templates/stage.deploy.yml
@@ -16,7 +16,7 @@ stages:
     displayName: 'Deploy ARM to ${{ parameters.environment }}'
     condition: succeeded()
     pool:
-      vmImage: 'Ubuntu-20.04'
+      vmImage: 'ubuntu-latest'
     variables:
       azPowershellVersion: 7.5.0
     environment: ${{ parameters.environment }}
@@ -85,10 +85,11 @@ stages:
     condition: succeeded()
     dependsOn: deploy_arm
     pool:
-      vmImage: 'Ubuntu-20.04'
+      vmImage: 'ubuntu-latest'
     environment: ${{ parameters.environment }}
     variables:
     - group: dataops-iac-cd-output-${{ parameters.environment }}
+    - group: databricks-sp-auth-${{ parameters.environment }}
     strategy:
       runOnce:
         deploy:
@@ -109,8 +110,9 @@ stages:
                 -DataLakeName $(dataLakeName)
                 -DatabricksName $(databricksName)
                 -KeyVaultName $(keyVaultName)
-                -DATABRICKS_TOKEN $(DATABRICKS_TOKEN)
               azurePowerShellVersion: latestVersion
+            env:
+              DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
 
           - task: PowerShell@2
             displayName: Deploy DBW Clusters
@@ -121,9 +123,16 @@ stages:
               arguments: >
                 -Environment ${{ parameters.environment }}
                 -DeploymentOutputFile "dbwOutput.json"
+            # Service Principal authentication (preferred)
             env:
               DATABRICKS_HOST: https://$(databricksWorkspaceUrl)
-              DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
+              ARM_CLIENT_ID: $(ARM_CLIENT_ID)
+              ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
+              ARM_TENANT_ID: $(ARM_TENANT_ID)
+            # OR Token-based authentication (not both)
+            # env:
+            #   DATABRICKS_HOST: https://$(databricksWorkspaceUrl)
+            #   DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
           - task: PowerShell@2
             displayName: 'Publish Outputs'
             inputs:
@@ -137,4 +146,4 @@ stages:
               pwsh: true
               showWarnings: true
             env:
-              AzureDevOpsPAT: $(System.AccessToken)
+              AzureDevOpsPAT: $(System.AccessToken)
diff --git a/azure-pipelines/iac/templates/stage.plan.yml b/azure-pipelines/iac/templates/stage.plan.yml
@@ -15,7 +15,7 @@ stages:
   - job: plan
     displayName: 'Plan for ${{ parameters.environment }}'
     pool:
-      vmImage: 'Ubuntu-20.04'
+      vmImage: 'ubuntu-latest'
     variables:
       azPowershellVersion: 7.5.0
     steps:
diff --git a/azure-pipelines/iac/templates/stage.validate.yml b/azure-pipelines/iac/templates/stage.validate.yml
@@ -5,7 +5,7 @@ stages:
   - job: lint
     displayName: 'Lint'
     pool:
-      vmImage: 'Ubuntu-20.04'
+      vmImage: 'ubuntu-latest'
     steps:
     - template: step.install-arm-template-toolkit.yml
       parameters:
diff --git a/azure-pipelines/iac/templates/step.install-databricks-cli.yml b/azure-pipelines/iac/templates/step.install-databricks-cli.yml
@@ -3,30 +3,65 @@ parameters:
   type: string
 
 steps:
-
 - task: UsePythonVersion@0
   inputs:
-    versionSpec: 3.6
+    versionSpec: 3.9
     addToPath: true
     architecture: 'x64'
-  displayName: 'Use Python Version: 3.6'
-  
+  displayName: 'Use Python Version: 3.9'
+
 - script: |
-    python -m pip install --upgrade pip
-    pip install databricks-cli
-  displayName: 'Setup Agent'
-  
-- task: AzureCLI@2
-  displayName: Get Databricks token
-  inputs:
-    azureSubscription: ${{ parameters.azureServiceConnection }}
-    scriptType: bash
-    scriptLocation: inlineScript
-    inlineScript: |
-      databricks_resource_id="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d" # This is the official databricks resource id
-      accessToken=$(curl -X GET -H 'Content-Type: application/x-www-form-urlencoded' \
-        -d "grant_type=client_credentials&client_id=$servicePrincipalId&resource=$databricks_resource_id&client_secret=$servicePrincipalKey" \
-        https://login.microsoftonline.com/$tenantId/oauth2/token \
-        | jq -r .access_token)
-      echo "##vso[task.setvariable variable=DATABRICKS_TOKEN;isSecret=true]$accessToken"
-    addSpnToEnvironment: true
+    # Use the latest stable version from GitHub releases
+    CLI_VERSION="0.252.0"
+    
+    echo "Downloading Databricks CLI v${CLI_VERSION} (v2.x)..."
+    
+    # Download with proper error handling
+    curl -fsSL --output databricks.tar.gz "https://github.com/databricks/cli/releases/download/v${CLI_VERSION}/databricks_cli_${CLI_VERSION}_linux_amd64.tar.gz"
+
+    if [ $? -ne 0 ]; then
+      echo "##vso[task.logissue type=error]Failed to download Databricks CLI"
+      exit 1
+    fi
+    
+    # Extract and install
+    tar -xzf databricks.tar.gz
+    chmod +x databricks
+    sudo mv databricks /usr/local/bin/
+    
+    # Verify installation
+    echo "Verifying Databricks CLI version..."
+    databricks version
+    
+    if [ $? -ne 0 ]; then
+      echo "##vso[task.logissue type=error]Databricks CLI installation failed"
+      exit 1
+    fi
+  displayName: 'Install Databricks CLI v2.x (binary)'
+
+- script: |
+    echo "Configuring Databricks CLI authentication..."
+    
+    # Create config directory if it doesn't exist
+    mkdir -p ~/.databricks
+    
+    # Create config file with service principal credentials
+    cat > ~/.databrickscfg << EOF
+    [DEFAULT]
+    host = https://$(databricksWorkspaceUrl)
+    azure_client_id = $(ARM_CLIENT_ID)
+    azure_client_secret = $(ARM_CLIENT_SECRET)
+    azure_tenant_id = $(ARM_TENANT_ID)
+    azure_use_msi = false
+    EOF
+    
+    # Test authentication 
+    databricks workspace list /
+    
+    if [ $? -ne 0 ]; then
+      echo "##vso[task.logissue type=error]Databricks CLI authentication failed"
+      exit 1
+    fi
+    
+    echo "Authentication successful!"
+  displayName: 'Authenticate with Databricks CLI'
diff --git a/infrastructure-as-code/databricks/dev/interactive.json b/infrastructure-as-code/databricks/dev/interactive.json
@@ -1,27 +1,21 @@
 {
     "cluster_name": "interactive-cluster",
-    "autoscale": {
-        "min_workers": 2,
-        "max_workers": 4
-    },
-    "spark_version": "9.1.x-scala2.12",
-    "spark_conf": {
-        "spark.databricks.delta.preview.enabled": "true"
-    },
+    "spark_version": "15.4.x-scala2.12",
     "azure_attributes": {
         "first_on_demand": 1,
         "availability": "ON_DEMAND_AZURE",
         "spot_bid_max_price": -1
     },
     "node_type_id": "Standard_DS3_v2",
-    "driver_node_type_id": "Standard_DS3_v2",
-    "ssh_public_keys": [],
-    "custom_tags": {},
     "spark_env_vars": {
         "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
     },
-    "autotermination_minutes": 60,
-    "enable_elastic_disk": true,
-    "cluster_source": "API",
-    "init_scripts": []
-}
+    "autotermination_minutes": 120,
+    "single_user_name": "alopezmoreno@mngenvmcap602086.onmicrosoft.com",
+    "data_security_mode": "SINGLE_USER",
+    "runtime_engine": "PHOTON",
+    "autoscale": {
+        "min_workers": 2,
+        "max_workers": 4
+    }
+}
diff --git a/infrastructure-as-code/databricks/prod/interactive.json b/infrastructure-as-code/databricks/prod/interactive.json
@@ -1,27 +1,21 @@
 {
     "cluster_name": "interactive-cluster",
-    "autoscale": {
-        "min_workers": 2,
-        "max_workers": 4
-    },
-    "spark_version": "9.1.x-scala2.12",
-    "spark_conf": {
-        "spark.databricks.delta.preview.enabled": "true"
-    },
+    "spark_version": "15.4.x-scala2.12",
     "azure_attributes": {
         "first_on_demand": 1,
         "availability": "ON_DEMAND_AZURE",
         "spot_bid_max_price": -1
     },
     "node_type_id": "Standard_DS3_v2",
-    "driver_node_type_id": "Standard_DS3_v2",
-    "ssh_public_keys": [],
-    "custom_tags": {},
     "spark_env_vars": {
         "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
     },
-    "autotermination_minutes": 60,
-    "enable_elastic_disk": true,
-    "cluster_source": "API",
-    "init_scripts": []
-}
+    "autotermination_minutes": 120,
+    "single_user_name": "alopezmoreno@mngenvmcap602086.onmicrosoft.com",
+    "data_security_mode": "SINGLE_USER",
+    "runtime_engine": "PHOTON",
+    "autoscale": {
+        "min_workers": 2,
+        "max_workers": 4
+    }
+}
diff --git a/infrastructure-as-code/databricks/qa/interactive.json b/infrastructure-as-code/databricks/qa/interactive.json
@@ -1,27 +1,21 @@
 {
     "cluster_name": "interactive-cluster",
-    "autoscale": {
-        "min_workers": 2,
-        "max_workers": 4
-    },
-    "spark_version": "9.1.x-scala2.12",
-    "spark_conf": {
-        "spark.databricks.delta.preview.enabled": "true"
-    },
+    "spark_version": "15.4.x-scala2.12",
     "azure_attributes": {
         "first_on_demand": 1,
         "availability": "ON_DEMAND_AZURE",
         "spot_bid_max_price": -1
     },
     "node_type_id": "Standard_DS3_v2",
-    "driver_node_type_id": "Standard_DS3_v2",
-    "ssh_public_keys": [],
-    "custom_tags": {},
     "spark_env_vars": {
         "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
     },
-    "autotermination_minutes": 60,
-    "enable_elastic_disk": true,
-    "cluster_source": "API",
-    "init_scripts": []
-}
+    "autotermination_minutes": 120,
+    "single_user_name": "alopezmoreno@mngenvmcap602086.onmicrosoft.com",
+    "data_security_mode": "SINGLE_USER",
+    "runtime_engine": "PHOTON",
+    "autoscale": {
+        "min_workers": 2,
+        "max_workers": 4
+    }
+}
diff --git a/infrastructure-as-code/scripts/DatabricksClusters.ps1 b/infrastructure-as-code/scripts/DatabricksClusters.ps1
diff --git a/infrastructure-as-code/scripts/DatabricksSecrets.ps1 b/infrastructure-as-code/scripts/DatabricksSecrets.ps1
