chore: Upgrade acryl-datahub-airflow-plugin to v1.0.0.2

[sean-rose]

Description

We received the following message from Acryl today (emphasis mine):

We have upgraded your instance to the Acryl DataHub SaaS release v0.3.10.2-acryl. This release brings in a lot of fixes and new features from OSS DataHub to you. Full changelog can be found in the release notes here for v0.3.10.2-acryl release.
If you are using remote executor please upgrade your image version to be v0.3.10.2-acryl. This is required so you are running the latest versions and do not face issues in Ingestion or Observability.
The recommended CLI to use with this release is 1.0.0.2. This applies for all CLI usages, if you are using it through your terminal, github actions, airflow, in python SDK somewhere etc. This is a strong recommendation to upgrade as we keep on pushing fixes in the CLI and it helps us support you better

Also, after upgrading WTMO to Airflow 2.10 (#2179) there's been an issue with some task tries not showing in the main UI, and I believe I've identified that the acryl-datahub-airflow-plugin's custom retry handler code is somehow the culprit. I'm hoping upgrading to this newer version of acryl-datahub-airflow-plugin might solve that issue, as it doesn't add a custom retry handler.

It's worth noting that when I previously tried upgrading acryl-datahub-airflow-plugin to v0.15.0.5 (#2181) it had dependency conflicts with Airflow and seemed to cause an Airflow outage when deployed, so I reverted that attempt (#2183).

However, with this PR there were no dependency conflicts with Airflow, and when testing in stage the bqetl_status_check, bqetl_cjms_nonprod, and bqetl_google_search_console DAGs all ended up running fine, except for a non-critical DataHub plugin error about not being able to render templates, and with https://github.com/mozilla-it/dataservices-infra/pull/725 we'll avoid such errors.

Related Tickets & Documents

• DENG-5881: Upgrade WTMO to Airflow 2.10
• chore: Upgrade acryl-datahub-airflow-plugin to v0.15.0.5 #2181
• fix: Revert "chore: Upgrade acryl-datahub-airflow-plugin to v0.15.0.5" #2183
• https://github.com/mozilla-it/dataservices-infra/pull/725

[sean-rose]

Also, after upgrading WTMO to Airflow 2.10 (#2179) there's been an issue with some task tries not showing in the main UI, and I believe I've identified that the acryl-datahub-airflow-plugin's custom retry handler code is somehow the culprit. I'm hoping upgrading to this newer version of acryl-datahub-airflow-plugin might solve that issue, as it doesn't add a custom retry handler.

Unfortunately this didn't resolve the problem with some task tries not showing in the main UI. 😞