Skip to content

Conversation

@Owen-CH-Leung
Copy link
Contributor

@Owen-CH-Leung Owen-CH-Leung commented Aug 27, 2024

Closes #33619

Airflow currently only supports reading remote log from elasticsearch. This PR adds feature to also allow reading remote logs from opensearch.

Similar to reading remote logs from elasticsearch, users need to use other tools to import logs into Opensearch.

The set up is very similar to setting up remote logging for Elasticsearch. In the airflow.cfg:

remote_logging = True
remote_log_conn_id = opensearch_default

[opensearch]
host = [OS host name] (e.g. my-opensearch-01)
port = [port number] (e.g. 9200)
username = 
password = 

Depending on your Opensearch config, you may also add the following in airflow.cfg :

[opensearch_configs]
http_compress = False
use_ssl = False
verify_certs = False
ssl_assert_hostname = False
ssl_show_warn = False
ca_certs = 

If you set up everything successfully (And correctly ship your logs to Opensearch), webserver should output logs like the following:

[2024-08-27T15:02:54.611+0000] {base.py:258} INFO - POST http://my-opensearch-01:9200/_all/_count [status:200 request:0.034s]
[2024-08-27T15:02:54.654+0000] {base.py:258} INFO - POST http://my-opensearch-01:9200/_all/_search?size=1000&sort=log&from=0 [status:200 request:0.040s]
192.168.65.1 - - [27/Aug/2024:15:02:54 +0000] "GET /api/v1/dags/consume_1_or_2_with_dataset_expressions/dagRuns/manual__2024-08-27T15:00:00.024040+00:00/taskInstances/consume_1_or_2_with_dataset_expressions/logs/1?full_content=false HTTP/1.1" 200 4219 "http://localhost:28080/dags/consume_1_or_2_with_dataset_expressions/grid?dag_run_id=manual__2024-08-27T15%3A00%3A00.024040%2B00%3A00&task_id=consume_1_or_2_with_dataset_expressions&tab=logs"

@Owen-CH-Leung Owen-CH-Leung marked this pull request as ready for review August 30, 2024 01:17
@Owen-CH-Leung
Copy link
Contributor Author

@eladkal can I seek your review for this PR ? Thanks

@eladkal
Copy link
Contributor

eladkal commented Sep 20, 2024

@kaxil @ephraimbuddy can you review the core parts of this PR? Given the work for Airflow 3 I am not sure about the provider<->core log integrations. To my understanding it is still coupled with core to some extant

@eladkal
Copy link
Contributor

eladkal commented Sep 20, 2024

Noting: I set Airflow 3 milestone for this PR.
The provider part will be released in next wave after it's merged but for the core parts we can't release it in 2.10.x as this is feature, we can't also release it in 2.11 as this is a feature that does not serve the porpuse of bridge release as we agreed (reference to https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#developing-for-airflow-3-and-210x--211x )

@eladkal eladkal added this to the Airflow 3.0.0 milestone Sep 20, 2024
@eladkal eladkal merged commit ba1c602 into apache:main Sep 21, 2024
joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
* Add feature to read log from opensearch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a compatibility with Opensearch search for log management

3 participants