feat(core): make elasticsearch compatible with remote_task_log#62121
Draft
Owen-CH-Leung wants to merge 3 commits intoapache:mainfrom
Draft
feat(core): make elasticsearch compatible with remote_task_log#62121Owen-CH-Leung wants to merge 3 commits intoapache:mainfrom
Owen-CH-Leung wants to merge 3 commits intoapache:mainfrom
Conversation
eladkal
reviewed
Feb 18, 2026
| elif ELASTICSEARCH_HOST: | ||
| ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get_mandatory_value("elasticsearch", "END_OF_LOG_MARK") | ||
| ELASTICSEARCH_FRONTEND: str = conf.get_mandatory_value("elasticsearch", "frontend") | ||
| from airflow.providers.elasticsearch.log.es_task_handler import ElasticsearchRemoteLogIO |
Contributor
There was a problem hiding this comment.
I think we need also min elastic version in
Because we know this won't work with older version of elastic provider
Need to verify with @potiuk this is the right way to proceed
Member
There was a problem hiding this comment.
I think it depends on whether there is min version that will work here.
Contributor
Author
There was a problem hiding this comment.
I added "elasticsearch": parse_version("6.5.0") which should be the next es provider version release after the provider mr is merged.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes: #50349
Fixes: #51456
This is the
airflow-corecounterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.airflow_local_settings.pypreviously configured Elasticsearch logging by injectingElasticsearchTaskHandlerdirectly into the logging handler dict. In Airflow 3, remote log writing goes through aREMOTE_TASK_LOGobject implementing the RemoteLogIO interface. This PR updates the config template to setREMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)The minimum required version of
apache-airflow-providers-elasticsearchis also bumped to>=6.5.0(the release containing ElasticsearchRemoteLogIO) in the dependency constraints.