-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Add feature to read log from opensearch #41799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add feature to read log from opensearch #41799
Conversation
|
@eladkal can I seek your review for this PR ? Thanks |
|
@kaxil @ephraimbuddy can you review the core parts of this PR? Given the work for Airflow 3 I am not sure about the provider<->core log integrations. To my understanding it is still coupled with core to some extant |
|
Noting: I set Airflow 3 milestone for this PR. |
* Add feature to read log from opensearch
Closes #33619
Airflow currently only supports reading remote log from elasticsearch. This PR adds feature to also allow reading remote logs from opensearch.
Similar to reading remote logs from elasticsearch, users need to use other tools to import logs into Opensearch.
The set up is very similar to setting up remote logging for Elasticsearch. In the airflow.cfg:
Depending on your Opensearch config, you may also add the following in airflow.cfg :
If you set up everything successfully (And correctly ship your logs to Opensearch), webserver should output logs like the following: