This issue has been migrated from the CC Search Catalog repository

Author: mathemancer
Date: Fri Dec 06 2019
Labels: 🔒 staff only,🙅 status: discontinued

Currently, we store airflow logs in the running Docker Container. The logs are ~650-900MB per day, and storing them in the container actually requires approximately double the space on the host machine. This equates to more than 1GB of logs being stored on the ec2 instance disk per day, quickly filling the disk.

Airflow includes out-of-the-box logging to s3 functionality, and we should reconfigure the airflow daemon to use that ability.

Original Comments:

mathemancer commented on Tue Feb 25 2020:

Marking this for CC staff, since it requires access to AWS to implement/test
source