merge from upstream #2
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
--------- Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
This is a follow-up on #18616 where we introduced retries on the occassional deadlocks when rendered task fields have been deleted by parallel threads (this is not a real deadlock, it's because MySQL locks too many things when queries are executed and will deadlock when one of those queries wait too much). Adding retry - while not perfect - should allow to handle the problem and significantly decrease the likelihood of such deadlocks. We can probably think about different approach for rendered fields, but for now retrying is - I think - acceptable short-term fix. Fixes: #32294 Fixes: #29687
Co-authored-by: Amogh Desai <adesai@adesai-MBP16.local>
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Co-authored-by: Amogh Desai <adesai@adesai-MBP16.local>
Typo in final dag, merge_data.
…of FAB auth manager (#32217)
--------- Co-authored-by: Amogh <adesai@cloudera.com> Co-authored-by: Amogh Desai <adesai@adesai-MBP16.local>
The fields that are sensitive (i.e password field is used to show them in Connection edit view) should also be hidden when they are stored as "extra" in the form extra field. This PR handles both - replacing such values with a placeholder as well as not updating the value if the placeholder has not been modified.
…1904) * reproduce the problem in a unit test * Fix operator_extra_links serialization * replace fget by __get__
* Logging from all containers in KubernetesOperatorPod * review comments from uranusjs * Fixing init logic * Fixing docs * Fixing sphinx logging * Addressing review comments from vincbeck * nits from potiuk * nits from potiuk * reverting return type * comment from uranusjr * fixing tests * fixing tests * review comments from hussein * handling nits from hussein * fixing tests --------- Co-authored-by: Amogh <adesai@cloudera.com> Co-authored-by: Amogh Desai <adesai@adesai-MBP16.local>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com> --------- Co-authored-by: Amogh Desai <adesai@adesai-MBP16.local> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
The change #29519 added experimental MariaDB client support for MySQL for ARM images but we missed that in the changelog.
* Fix logic of the skip_all_except method to work correctly with a mapped branch operator * Address feadback * Add an unit test * Skipp empty tasks list * Fix static checks * Address feadback * Use fully qualified import --------- Co-authored-by: Zhyhimont Dmitry <dzhigimont@gmail.com> Co-authored-by: zhyhimont <zhyhimont@gmail.com> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
The standard regexp library can consume > O(n) in certain circumstances. The re2 library does not have this issue.
Click 8.1.4 breaks our mypy checks. The upper limit can be lifted when the #32412 issue is resolved
* Allow FileTaskHandler to delegate to instances of logging.Handler FileTaskHandler is the base class for logging handlers, including those that don't log to files via delegating to logging.FileHandler, e.g. in the CloudwatchTaskHandler at https://github.com/apache/airflow/blob/2940b9fa55a6a72c60c2162e541631addec3d6b8/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py#L67 It is suspected that type checking is not enabled in this part of the CloudwatchTaskHandler otherwise it would have already been failing. This change adjusts the base class so if type checking is enabled in the task handler, if it delegates to a logging.Handler that is not a logging.FileHandler as the CloudWatchHandler, then the type checking should pass. This was originally part of #31855 and split out. related: #31834 * Add Redis task handler This stores log lines in Redis up to a configured maximum log lines, always keeping the most recent, up to a configured TTL. This deviates from other existing task handlers in that it accepts a connection ID. This allows it to be used in addition to other handlers, and so allows a graceful/reversible transition from one logging system to another. This is particularly useful in situations that use Redis as a message broker, where additional infrastructure isn't desired. closes: #31834
small readability change Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
The test has been recently failing with deadlock (see #32778) and needs thorough looking at if we want to find the root cause/remedium. In the meantime it looks like a niche case connected with Dask Executor that is rather obsure and we have no expertise in solving problems with and diagnosing, so until the problem is diagnosed it might be a long time (and maybe even we decide not to care about it and let Dask community take a look and either fix or ignore it. We aim to have a very low number of those Quarantined tests (currently we have 1 and we have not run it for a while as this was a mysql test run on Postgres) but we have now the opportunity to also improve the quarantined tests framework. This test will be run together with other (1) quarantined test and: * they will not be run in our regular tests * they are run sequentially not in parallel with all other tests * they are run for all 4 backends but only for the default versions of those backends * failure of the quarantined tests will not cause failure of the whole job or limit constraints from being generated and updated
Co-authored-by: Zhyhimont Dmitry <dzhigimont@gmail.com>
* add error handling in case of 404 errors to protm script Signed-off-by: Michael Robinson <merobi@gmail.com> * add error handling in case of missing score when using pickle file Signed-off-by: Michael Robinson <merobi@gmail.com> * reformat Signed-off-by: Michael Robinson <merobi@gmail.com> * add UknownObjectException class to error handling Signed-off-by: Michael Robinson <merobi@gmail.com> * fix github imports Signed-off-by: Michael Robinson <merobi@gmail.com> * fix interaction_score Signed-off-by: Michael Robinson <merobi@gmail.com> * remove attribute error handling Signed-off-by: Michael Robinson <merobi@gmail.com> * remove second attribute error handling block Signed-off-by: Michael Robinson <merobi@gmail.com> * change api endpoint for pull requests in pr script in stats Signed-off-by: Michael Robinson <merobi@gmail.com> * reformat Signed-off-by: Michael Robinson <merobi@gmail.com> * Update dev/stats/get_important_pr_candidates.py Co-authored-by: Wei Lee <weilee.rx@gmail.com> * reformat nested loop and add new line Signed-off-by: Michael Robinson <merobi@gmail.com> * consolidate funs to decrease number of api calls Signed-off-by: Michael Robinson <merobi@gmail.com> * add rate limit parameter and limit info to output Signed-off-by: Michael Robinson <merobi@gmail.com> * refactor after linting Signed-off-by: Michael Robinson <merobi@gmail.com> * refactor after linting contd Signed-off-by: Michael Robinson <merobi@gmail.com> --------- Signed-off-by: Michael Robinson <merobi@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com>
…32513) * Minor code refactoring * Add type annotations * update doc string about default value of deferrable
Allow auth managers to override the security manager
…#32775) * Quarantine test_backfill_integration in dask executor The test has been recently failing with deadlock (see #32778) and needs thorough looking at if we want to find the root cause/remedium. In the meantime it looks like a niche case connected with Dask Executor that is rather obsure and we have no expertise in solving problems with and diagnosing, so until the problem is diagnosed it might be a long time (and maybe even we decide not to care about it and let Dask community take a look and either fix or ignore it. We aim to have a very low number of those Quarantined tests (currently we have 1 and we have not run it for a while as this was a mysql test run on Postgres) but we have now the opportunity to also improve the quarantined tests framework. This test will be run together with other (1) quarantined test and: * they will not be run in our regular tests * they are run sequentially not in parallel with all other tests * they are run for all 4 backends but only for the default versions of those backends * failure of the quarantined tests will not cause failure of the whole job or limit constraints from being generated and updated * Add pre-Airflow-2-7 hardcoded defaults for config for older providers During thorough testing and review of moving configuration to provoders I realised that there was a case that was not handled properly. In some cases some providers and DAGs could rely on some default values being available as default, but when we move them from core, and use older version of provider those defaults were not available: * they were remove as defaults in core * the old providers did not have "config" section to contribute the defaults This would be a breaking change and old providers (Celery, K8s) could fail - as it happened in some tests. This PR implements a nice solution to that, also allowing to remove some manual fallbacks in Celery and Kubernetes executor code. The solution is to add a hard-coded "pre-2.7" configuration which would only contain "provider" pre-2.7 hard-coded defaults and make it a fallback option if the values are neither set nor defaults contributed by the providers. We do not have to maintain those - the defaults are "frozen" effectively at the values available just before 2.7. The nice side effect is that we can remove a number of fallbacks, because this hard-coded configuration becomes the fallback automatically, That entirely solves the case where you want to install older providers on 2.7 where config.yml does not contain those provider values. * Update airflow/configuration.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
We are also using webserver config when we start worker and in the future the internal-api server - which was not happening after #32766. This PR brings the webserver config back for those commands.
Atlas does not have its own provider, the only thing we have with atlas extra is an extra dependency to atlasclient. Abd we also had the atlas configuration for some reason that even the eldest don't know. With hard-coded defaults, we are still keeping backwards compatibility if someone used those values. However we should remove it from "core" configuration because it has absolutely no meaning for Airflow. Users are free to define their own sections and variables following Airflow schema and this squarely fits into this category.
There was a code comment warning that the fallbacks for celery cli commands should not be removed until some conditions are met, but the fallbacks were removed in #32775
…cated DatabricksRunNowDeferrableOperator (#32806) As mentioned in #27837, we should do something similar to DatabricksRunOperator instead of running the original executing logic in DatabricksRunNowDeferrableOperator This is due to a bug that causes issues with empty job IDs. The DatabricksRunNowOperator operates smoothly and supports deferrable mode, so this PR removes the malfunctioning execute method.
* Fix displaying execution_date in graph view task instance tooltip. * Fix tests.
…2716) Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.