DatabricksSqlOperator fails with IndexError: list index out of range

[yershovkiril]

Apache Airflow Provider(s)

databricks

Versions of Apache Airflow Providers

6.0.0

Apache Airflow version

2.6.3

Operating System

composer-2.5.4-airflow-2.6.3

Deployment

Google Cloud Composer

Deployment details

Google Cloud Composer composer-2.5.4-airflow-2.6.3

What happened

While running SQL with CREATE IF NOT EXISTS statement task fails with IndexError: list index out of range error:

[2024-01-16, 08:51:48 UTC] {taskinstance.py:1826} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/common/sql/operators/sql.py", line 282, in execute
    output = hook.run(
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/databricks/hooks/databricks_sql.py", line 256, in run
    result = self._make_common_data_structure(raw_result)
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/databricks/hooks/databricks_sql.py", line 283, in _make_common_data_structure
    rows_fields = rows[0].__fields__
IndexError: list index out of range

What you think should happen instead

Task should succeed if table has been created or fail if it's not, it previously worked up to 5.0.1 (incl)

How to reproduce

Run DatabricksSqlOperator with CREATE TABLE IF NOT EXISTS statement

Example:

DatabricksSqlOperator(
    task_id='run_sql',
    sql=['create table if not exists schema.test_table as select 1 as res'],
    databricks_conn_id=<your_conn_id>
)

Anything else

Occurs every time, worked fine before 6.0.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

cc @alexott