Skip to content

Commit

Permalink
[Airflow] Formatting fixes (#6866)
Browse files Browse the repository at this point in the history
* formating fixes

* adding missing space
  • Loading branch information
Pierre Guceski authored Jun 12, 2020
1 parent 0bd2fd2 commit bebbc9e
Showing 1 changed file with 55 additions and 56 deletions.
111 changes: 55 additions & 56 deletions airflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,62 +161,61 @@ _Available for Agent versions >6.0_
```

2. Uncomment and edit this configuration block at the bottom of your `airflow.d/conf.yaml`:

Change the `path` and `service` parameter values and configure them for your environment.

a. Configuration for DAG processor manager and Scheduler logs:

```yaml
logs:
- type: file
path: '<PATH_TO_AIRFLOW>/logs/dag_processor_manager/dag_processor_manager.log'
source: airflow
service: '<SERVICE_NAME>'
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
- type: file
path: '<PATH_TO_AIRFLOW>/logs/scheduler/*/*.log'
source: airflow
service: '<SERVICE_NAME>'
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
```

Regular clean up is recommended for scheduler logs with daily log rotation.

b. Additional configuration for DAG tasks logs:

```yaml
logs:
- type: file
path: '<PATH_TO_AIRFLOW>/logs/*/*/*/*.log'
source: airflow
service: '<SERVICE_NAME>'
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
```

Caveat: By default Airflow uses this log file template for tasks: `log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log`. The number of log files will grow quickly if not cleaned regularly. This pattern is used by Airflow UI to display logs individually for each executed task.

If you do not view logs in Airflow UI, Datadog recommends this configuration in `airflow.cfg`: `log_filename_template = dag_tasks.log`. Then log rotate this file and use this configuration:

```yaml
logs:
- type: file
path: '<PATH_TO_AIRFLOW>/logs/dag_tasks.log'
source: airflow
service: '<SERVICE_NAME>'
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
```
Change the `path` and `service` parameter values and configure them for your environment.

- Configuration for DAG processor manager and Scheduler logs:

```yaml
logs:
- type: file
path: "<PATH_TO_AIRFLOW>/logs/dag_processor_manager/dag_processor_manager.log"
source: airflow
service: "<SERVICE_NAME>"
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
- type: file
path: "<PATH_TO_AIRFLOW>/logs/scheduler/*/*.log"
source: airflow
service: "<SERVICE_NAME>"
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
```

Regular clean up is recommended for scheduler logs with daily log rotation.

- Additional configuration for DAG tasks logs:

```yaml
logs:
- type: file
path: "<PATH_TO_AIRFLOW>/logs/*/*/*/*.log"
source: airflow
service: "<SERVICE_NAME>"
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
```

Caveat: By default Airflow uses this log file template for tasks: `log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log`. The number of log files will grow quickly if not cleaned regularly. This pattern is used by Airflow UI to display logs individually for each executed task.

If you do not view logs in Airflow UI, Datadog recommends this configuration in `airflow.cfg`: `log_filename_template = dag_tasks.log`. Then log rotate this file and use this configuration:

```yaml
logs:
- type: file
path: "<PATH_TO_AIRFLOW>/logs/dag_tasks.log"
source: airflow
service: "<SERVICE_NAME>"
log_processing_rules:
- type: multi_line
name: new_log_start_with_date
pattern: \[\d{4}\-\d{2}\-\d{2}
```

3. [Restart the Agent][7].

Expand Down

0 comments on commit bebbc9e

Please sign in to comment.