CloudWatch task handler doesn't fall back to local logs when Amazon CloudWatch logs aren't found

[hankehly]

Closes #27396

Summary

The CloudWatch task log handler does not default to local logs (like the documentation says it should) when remote logs are unavailable. This PR fixes the issue.

Todo

• Update CloudWatch log handler
• Add/update unit tests
• Manual check (see below for details)

Manual check

Procedure

1. Run a test DAG with AWS CloudWatch logging configured
2. Verify that logs are sent to CloudWatch
3. Delete CloudWatch log stream associated with task
4. Create log file locally echo "test" >> {log file path}
5. Refresh the task log page

Expectations

• "Unable to read remote logs from Cloudwatch" should appear in the log because we deleted the log stream associated with the task.
• "Reading local file: ..." should appear in the log to show that the log handler falls back to local logs.
• The contents of the local log file should appear (in this case the contents are "test")

Screenshot

[vincbeck]

Nice!

[hankehly]

@vincbeck
Thanks for the early review. Once I've handled the failing tests I'll click the "Ready for review" button.
Ready for review

[hankehly]

I deleted these 2 test cases for the following reasons.

• They overlap with the one I added (testing for generic failure to fetch remote logs)
• AWS error messages are now propagated to the task log. These strings could change in the future and break our unit tests.

[hankehly]

I am propagating the AWS error message to the task log. I think this will be useful for debugging IAM/connection issues.

[hankehly]

Log position seems irrelevant in the context of the CloudWatch handler because we fetch/return all logs in one request. I'm following the example I found in the GCS task handler here.

airflow/tests/providers/google/cloud/log/test_gcs_task_handler.py

Line 108 in 2106326

assert metadata == {"end_of_log": False, "log_pos": 31 + len(self.local_log_location)}

[o-nikolas]

🚀