Remove api prefix from update_job_permission in databricks hook

[fjmacagno]

---

Fixes a problem with the update_job_permission databricks hook method where it crashes with a 404. The fix is to remove api/ from the start of the path, which is automatically added by the databricks base hook. I am assuming this is a safe change because all the other endpoints do not include api/ at the start, implying this one fell through the cracks.

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 672, in _do_api_call
    for attempt in self._get_retry_object():
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line 443, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 691, in _do_api_call
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://<>/api/api/2.0/permissions/jobs/807588172815468
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 768, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 734, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 424, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 402, in execute
    self._hook.update_job_permission(job_id, normalise_json_content(acl_json))
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks.py", line 773, in update_job_permission
    return self._do_api_call(("PATCH", f"api/2.0/permissions/jobs/{job_id}"), json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py", line 698, in _do_api_call
    raise AirflowException(msg)
airflow.exceptions.AirflowException: Response: {"error":"Bad Target: /api/api/2.0/permissions/jobs/807588172815468"}
, Status Code: 404

Fixes problem introduced in #52385.

[gopidesupavan]

can you pls add tests

[kyungjunleeme]

@fjmacagno Sorry, It's my fault,

The test code definitely passed, but I’m wondering how I missed this. I’ll try to explore if there’s another way to address this issue aside from this PR.

[kyungjunleeme]

• providers.databricks.tests.unit.databricks.operators.test_databricks.TestDatabricksCreateJobsOperator.test_exec_update_job_permission

@mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook")
def test_exec_update_job_permission(self, db_mock_class):

In this case, since the entire DatabricksHook is mocked, the internal _do_api_call() method is never actually invoked. Therefore, you cannot verify whether the path is "api/2.0/..." or "2.0/...".

I think that
This is a good example of one of the risks of relying too heavily on mocking in tests. Since the actual implementation isn't exercised, critical issues like incorrect API paths can easily slip through unnoticed.

It would be great if you could add a test case for this part! In the meantime, I’ll think about whether there are any other areas where additional tests might be helpful.

[kyungjunleeme]

@fjmacagno hello? I think that this issue caused by my work. so I feel very sorry for everyone. So could you accelerate this work? plz?

[fjmacagno]

Sorry i was on vacation for a while, I will try to write a test today.

[fjmacagno]

Added a test which i confirmed fails on master.