PR#18991 introduced directly navigating to a DAG when selecting one
from the typeahead search results. Unfortunately, the search results
also includes DAG owner names, and selecting one of those navigates to
a DAG with that name, which almost certainly doesn't exist.

This extends the autocompletion endpoint to return the type of result,
and adjusts the typeahead selection to use this to know which way to
navigate.

Fixes: #23867

The foldable parts were not good. They made links not to work as
well as they were not too discoverable.

Fixes: #23174

Fixes: #23868

…23877)

Currently there are string-value evaluations of `compute`, `nodegroup_role_arn`,  and `fargate_pod_execution_role_arn` args in the constructor of `EksCreateClusterOperator`.  These args are all listed as a template fields so it's entirely possible that the value(s) passed in to the operator is a Jinja expression or an `XComArg`. Either of these value types could cause a false-negative `ValueError` (in the case of unsupported `compute` values) or a `false-positive` (in the the cases of explicit checks for the *arn values) since the values themselves have not been rendered.

This PR moves the evaluations of these args to the `execute()` scope.

String instead of Int see https://docs.readthedocs.io/en/stable/config-file/v2.html

The --verbose and --dry-dun commands caused n --files command to fail
and the flag was "artifficial" -it was equivalent to bool flag.
the actual files were taken  from arguments.

This PR fixes it by turning the arguments into multiple ``--file``
commands  - each with its own completioin for local files.

Simple refactoring to make the jobs more consistent.

The rest of the page has consistently alphabetized tables. This commit fixes three `extras` that were not alphabetized.

When patching as "checked", we have to use the current version of the pod otherwise we may get an error when trying to patch it, e.g.:

```
Operation cannot be fulfilled on pods \"test-kubernetes-pod-db9eedb7885c40099dd40cd4edc62415\": the object has been modified; please apply your changes to the latest version and try again"
```

This error would not cause a failure of the task, since errors in `cleanup` are suppressed.  However, it would fail to patch.

I believe one scenario when the pod may be updated is when retrieving xcom, since the sidecar is terminated after extracting the value.

Concerning some changes in the tests re the "already_checked" label, it was added to a few "expected pods" recently, when we changed it to patch even in the case of a successful pod.

Since we are changing the "patch" code to patch with the latest read on the pod that we have (i.e. using the `remote_pod` variable), and no longer the pod object stored on `k.pod`, the label no longer shows up in those tests (that's because in k.pod isn't actually a read of the remote pod, but just happens to get mutated in the patch function before it is used to actually patch the pod).

Further, since the `remote_pod` is a local variable, we can't observe it in tests.  So we have to read the pod using k8s api. _But_, our "find pod" function excludes "already checked" pods!  So we have to make this configurable.

So, now we have a proper integration test for the "already_checked" behavior (there was already a unit test).

It was not clear to me what this really means

main breaks with
`Traceback:
  /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  tests/providers/amazon/aws/hooks/test_cloud_formation.py:31: in <module>
      class TestCloudFormationHook(unittest.TestCase):
  tests/providers/amazon/aws/hooks/test_cloud_formation.py:67: in TestCloudFormationHook
      @mock_cloudformation
  /usr/local/lib/python3.7/site-packages/moto/__init__.py:30: in f
      module = importlib.import_module(module_name, "moto")
  /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  /usr/local/lib/python3.7/site-packages/moto/cloudformation/__init__.py:1: in <module>
      from .models import cloudformation_backends
  /usr/local/lib/python3.7/site-packages/moto/cloudformation/models.py:18: in <module>
      from .parsing import ResourceMap, OutputMap
  /usr/local/lib/python3.7/site-packages/moto/cloudformation/parsing.py:17: in <module>
      from moto.apigateway import models  # noqa  # pylint: disable=all
  /usr/local/lib/python3.7/site-packages/moto/apigateway/__init__.py:1: in <module>
      from .models import apigateway_backends
  /usr/local/lib/python3.7/site-packages/moto/apigateway/models.py:9: in <module>
      from openapi_spec_validator import validate_spec
  E   ModuleNotFoundError: No module named 'openapi_spec_validator'
  `
  Fix is already in placed in moto getmoto/moto#5165 but version 3.1.11 wasn't released yet

New release of Azure Cosmos library has added typing information
and it broke main builds with mypy verification.

…3886)

The change of the release workflow in #23928 removed the reason
why we should have rebase workflow possible. We only needed to
do rebase when we merged test branch into stable branch and
since we are doing it manually, there is no more reeason to
have it in the GitHub UI.

* UI fix for null task instances

* improve tests without global vars

* fix test data

* Show details button and wrap on LegendRow.

* Update following brent review

* Fix display on small width

* Rotate icon for a 'ReadLess' effect

This fetches all TIs for a given task across dag runs, leading to
signifincatly faster response times. It also fixes a bug where Nones
were being passed to the UI when a new task was added to a DAG with
exiting runs.

There's are two code blocks with identical text in the helm-chart docs. This commit removes one of them.

This turned out to be mistake in manual submission. Fixed
on JPype1 side.

This reverts commit 3699be4.

Bulk delete does not work if the cascade behaviour of a foreignkey
is set on python side(relationship configuration). To allow bulk delete of dags
we need to setup cascade deletion in the DB.

The warning on query.delete at
https://docs.sqlalchemy.org/en/14/orm/session_basics.html#selecting-a-synchronization-strategy
stated that:

The operations do not offer in-Python cascading of relationships - it is assumed that ON UPDATE CASCADE and/or ON DELETE CASCADE is configured for any foreign key references which require it, otherwise the database may emit an integrity violation if foreign key references are being enforced.

Another alternative is avoiding bulk delete of dags but I prefer we support bulk deletes.

This will break offline sql generation for mssql(already broken before now :) ). Also, since there's only one foreign key
in `dag_tag` table, I assume that the foreign key would be named `dag_tag_ibfk_1` in `mysql`. This
avoided having to query the db for the name.

The foreignkey is explicitly named now, would be easy for future upgrades

…ocess not a session leader (#23872)

This PR fixes inverted levels in the sections added to the "Best Practices" document in #21879.

* Add support to specify language name in PapermillOperator

* Replace getattr() with simple attribute access

* fix links
* added right link to breeze

#23797)

Now that we support dynamic task mapping, we should have the 'REMOVED'
state of task instances as a finished state because
for dynamic tasks with a removed task instance, the dagrun would be stuck in
running state if 'REMOVED' state is not in finished states.

Latest version of buildx removed -f as shorthand for --force flag.

…23982)

The --volume flag is an old style of specifying mounts used by docker,
the newer and more explicit version is --mount where you have to
specify type, source, destination in the form of key/value pairs.

This is more explicit and avoids some guesswork when volumes are
mounted (for example seems that on WSL2 volume name might be
guessed as path wrongly). The change explicitly specifies which
of the mounts are bind mounts and which are volume mounts.

Another nice side effect of this change is that when source is
missing, docker will not automatically create directories with the
missing name but it will fail. This is nicer because before it
led to creating directories when they were missing (for example
.bash_aliases and similar). This allows us to avoid some cleanups
to account for those files being created - instead we simply
skip those mounts if the file/folder does not exist.

If you have no buildx plugin installed on Mac (for example when
you use colima instead of Docker Desktop) the breeze check was
failing - but buildx in fact is not needed to run typical breeze
commands, and breeze already has support for it - it was just
wrongly handled.

The pre-commit to generate docker volumes in docker compose
file is now written in Python and it also uses the newer "volume:"
syntax to define the volumes mounted in the docker-compose.

* Replace `use_task_execution_date` with `use_task_logical_date`
We have some operators/sensors that use `*_execution_date` as the class parameters. This PR deprecate the usage of these parameters and replace it with `logical_date`.
There is no change in functionality, under the hood the functionality already uses `logical_date` this is just about the parameters name as exposed to the users.

We have now moto 3.1.9+ in constraints so we should remove the limit.

Fixes: #23576

…23994)

When we yanked cncf.kubernetes provider, we pinned 3.1.2
temporarily for provider generation. This removes the pinning as
we are already at 4.0.2 version

…23980)

The pre-commits that require CI image run docker command under
the hood that is highly optimized for performance (only mounts
files that are necessary to be mounted) - in order to improve
performance on Mac OS and make sure that artifacts are not left
in the source code of Airflow.

However that makes the command slightly more difficult to debug
because they generate dynamically the docker command used,
including the volumens that should be mounted when the docker
command is run.

This PR adds better diagnostics to the pre-commit scripts
allowing VERBOSE="true" and DRY_RUN="true" variables that can
help with diagnosing problems such as running the scripts on
WSL2.

It also fixes a few documentation bugs that have been missed
after changing names of the image-related static checks and
thanks to separating the common code to utility function
it allows to set SKIP_IMAGE_PRE_COMMITS variable to true
which will skip running all pre-commit checks that require
breeze image to be available locally.

There is an issue with pushing cache to docker registry that
is connected to containerd bug but started to appear more
frequently recently (as evidenced for example by
https://github.community/t/buildx-failed-with-error-cannot-reuse-body-request-must-be-retried/253178
). The issue is still open in containerd:
containerd/containerd#5978.

Until it if fixed, we disable fail-fast on pushing cache
so that even if it happens, we just have to re-run that single
python version that actually failed. Currently there is a much
lower chance of success because all 4 build have to succeed.

#24006)

There is a flakiness in pushing cache images to ghcr.io, therefore
we want to add automated retries when the images fail intermittently.

The root cause of the problem is tracked in containerd:
containerd/containerd#5978

…ges in CI (#24006)" (#24016)

This reverts commit 7cf0e43.

* Cleanup BranchDayOfWeekOperator example dag
There is no need for `dag=dag` when using context manager.

Apparently pushing multi-platform images when building cache on CI
has some problems recently, connected with ghcr.io being more
vulnerable to race condition described in this issue:

containerd/containerd#5978

Apparently when two, different platform layers are pushed about
the same time to ghcr.io, the error
"cannot reuse body, request must be retried" is generated.

However we actually do not even need to build the multiplatform
latest images because as of recently we have separate cache for each
platform, and the ghcr.io/:latest images are not used any more
not even for docker builds. We we always build images rather than
pull and we use --from-cache for that - specific per platform. The only
image pulling we do is when we pull the :COMMIT_HASH images in CI- but
those are single-platform images (amd64) and even if we add tests for
arm, they will have different tag.

Hopefully we can still build release images without causing the
race condition too frequently - this is more likely because when
we build images for cache we use machines with different performance
characteristics and the same layers are pushed at different times
from different platforms.

The previous version of buildx cache preparation implied --push-image
flag, but now this is completely separated (we do not push image,
we just prepare cache), so when mutli-platform buildx preparation is
run we should also allow the cache to run without --push-image flag.

* "get_partition" to retrieve a Partition
* "create_partition" to create a Partition

This is follow-up after #24026 which added capability of selectively
deciding for each breeze command, whether the output of the command
should be "foldable" group. All CI output has been reviewed, and
the commands which "need" it were identified.

This also fixes a problem introduced there - that the command itself
was not "foldable" group itself.

Our ARM cache builds started to hang recently at yarn prod step.
The most likely reason are limited resources we had for the ARM
instance to run the docker build - it was rather small instance
with 2GB RAM and it is likely not nearly enought to cope with
recent changes related to Grid View where we likely need much
more memory during the yarn build step.

This change increases the instance memory to 8 GB (c6g.xlarge).
Also this instance type gives 70% cost saving and has very low
probability of being evicted (it's not in high demand in Ohio
Region of AWS.

Also the AMI used is refreshed with latest software (docker)

…23961)

* Removing magic numbers from exceptions

* Running pre-commit

Pip has been upgraded to version 22.1.2 12 minutes ago. Time to
catch up.

Preparing airflow packages and provider packages does not
need to be done on ARM and actually the ARM instance is idle
while they are prepared during cache building.

This change moves preparation of the packages to before
the ARM instance is started which saves about 3 minutes of ARM
instance time.

…vider (#23891)

)

Add support for k8s hook in KPO; use it always (even when no conn id); continue to consider the core k8s settings that KPO already takes into account but emit deprecation warning about them.

KPO historically takes into account a few settings from core airflow cfg (e.g. verify ssl, tcp keepalive, context, config file, and in_cluster). So to use the hook to generate the client, somehow the hook has to take these settings into account. But we don't want the hook to consider these settings in general.  So we read them in KPO and if necessary patch the hook and warn.

After #24052 we also need to add --force-build flag as for
Python 3.7 rebuilding CI cache would have been silently ignored as
no image building would be needed

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Co-authored-by: Tzu-ping Chung <tp@astronomer.io>

This is broken since 2.3.0. that's if a user specifies a migration_timeout
of 0 then no migration is run at all.

Bumps [eventsource](https://github.com/EventSource/eventsource) from 1.0.7 to 1.1.1.
- [Release notes](https://github.com/EventSource/eventsource/releases)
- [Changelog](https://github.com/EventSource/eventsource/blob/master/HISTORY.md)
- [Commits](EventSource/eventsource@v1.0.7...v1.1.1)

---
updated-dependencies:
- dependency-name: eventsource
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

The certifi limitation was introduced to keep snowflake happy while
performing eager upgrade because it added limits on certifi. However
seems like it is not limitation any more in latest versions of
snowflake python connector, so we can safely remove it from here.

The only remaining limit is dill but this one still holds.

Fixes: #23639

closes: #22470

closes: #22448 , #22431

Bitnami's CloudFront CDN is seemingly having issues, so point at github
direct instead until it is resolved.

We started to experience "Internal Error" when installing
Helm chart and apperently bitnami "solved" the problem by
removing from their index software older than 6 months(!).

This makes our CI fail but It is much worse. This
renders all our charts useless for people to install
This is terribly wrong, and I raised this in the issue
here:

bitnami/charts#10539 (comment)

- Trivial typo fix in the command to run static checks on the last commit
- Update "run all tests" to "run all checks" where applicable for consistency