Use new job search API for triggering Databricks job by name

[alexott]

Existing implementation uses list API to find all possible jobs and filtering out them by name. This leads to huge load on the API backend for workspaces with big number of jobs (> 1k). Databricks Workflows team extended list API with search functionality.

[alexott]

Instead of explicit parameter we can check for presence of an environment variable. Not sure what is the best way, especially when we depreciate old implementation completely

[o-nikolas]

Instead of explicit parameter we can check for presence of an environment variable. Not sure what is the best way, especially when we depreciate old implementation completely

Could you not attempt to make the new API request and then handle the error/exception you get back if it isn't supported by the backend being used, then automatically fall back to the old api in that case?

[alexott]

@o-nikolas it's not really a completely new API, it's a new parameter to that API. If API doesn't support that parameter it just returns all data as before...

[o-nikolas]

@o-nikolas it's not really a completely new API, it's a new parameter to that API. If API doesn't support that parameter it just returns all data as before...

If that's the case, then it should be easy to build this in a backwards compatible way without the use_old_list_impl flag right?

[alexott]

I've reimplemented it differently, so old review comments aren't much relevant

[alexott]

Hmmm, not sure if it fails in static check... Looks like mypy inferred type incorrectly

[alexott]

@ephraimbuddy comments are addressed & mypy error has been fixed. Can you please review again?
thank you