Description
Description:
When running dbt-databricks on Databricks LTS clusters (12.2, 14.3, and 15.4), your databricks-sql-python library fails due to incompatibility with the version of urllib3 shipped with these runtimes. Specifically, the library’s retry logic assumes the presence of features from urllib3 2.x (e.g., Retry.backoff_jitter), which do not exist in the older version provided by the runtime. This results in errors such as:
AttributeError: type object 'Retry' has no attribute 'backoff_jitter'
Steps to Reproduce:
- Deploy and run dbt-databricks on a Databricks cluster using one of the LTS versions: 12.2, 14.3, or 15.4.
- Execute operations that trigger the retry logic in
databricks-sql-python(e.g., connection establishment or query execution). - Observe the error:
AttributeError: type object 'Retry' has no attribute 'backoff_jitter'
Additional Context:
A potential workaround mentioned in the repository is to manually upgrade urllib3 to version 2.3.0. However, this solution is problematic because:
- Dependency Conflicts: Upgrading urllib3 might break other dependencies in the Databricks runtime that rely on an older version.
- Runtime Stability: Altering core libraries in the runtime environment can lead to unforeseen issues and impact overall stability.
Given that the Databricks LTS runtimes have been confirmed to ship with older versions of urllib3, a robust fix would be to update the retry logic in databricks-sql-python to provide backward compatibility. This could involve conditionally using the newer attributes when available or implementing an alternative approach for environments that rely on older versions of urllib3.
Desired Solution:
Instead of recommending a manual upgrade of urllib3, it would be preferable for databricks-sql-python to detect the installed urllib3 version and adjust its retry logic accordingly. This would avoid dependency conflicts and maintain runtime stability across various Databricks environments.
For reference, see Issue #516 where this problem has been noted.
Thank you for your attention to this matter. I look forward to a solution that ensures compatibility with Databricks LTS clusters without compromising runtime stability.