AWS logs. Exit fast when 3 consecutive responses are returned from AWS Cloudwatch logs #30756
Conversation
…S Cloudwatch logs
boring-cyborg
bot
added
area:providers
provider:amazon
| # The issue with this approach is, it can take a huge amount of time (e.g. 20 seconds) to retrieve logs | ||
| # As an intermediate solution, we decided to stop fetching logs if 3 consecutive responses are empty |
There was a problem hiding this comment.
if that happens what user is expected to do / how would user know that this is the issue ?
There was a problem hiding this comment.
He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs
Co-authored-by: Niko Oliveira <onikolas@amazon.com>
| token_arg = {} | ||
|
|
||
| response = self.get_conn().get_log_events( | ||
| response = self.conn.get_log_events( |
There was a problem hiding this comment.
Thanks for including this update ❤️
ferruzzi
left a comment
ferruzzi
left a comment
There was a problem hiding this comment.
I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.
4 participants
An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.