AWS logs. Exit fast when 3 consecutive responses are returned from AWS Cloudwatch logs

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.

[vincbeck]

An issue had been introduced by #20814. Now, it takes customers 20 seconds or more to load task logs from past executions. As a solution we decided to exit when 3 consecutive empty responses are returned from AWS Cloudwatch logs

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

if that happens what user is expected to do / how would user know that this is the issue ?

[vincbeck]

He would not be able to know. Before #20814, we were stopping fetching logs after one response from CW is empty. After #20814 being introduced, we now wait that the next token is the same in 2 consecutive responses. We now shoot in the middle to satisfy a bit of the two worlds. This fix is very empirical based on user experiences. They get a very frustrated experience where, sometimes, they need to wait 20 seconds to retrieve logs

[ferruzzi]

Thanks for including this update ❤️

[ferruzzi]

I guess we could wrap it in a flag to let the user decide whether to wait or quit early, defaulting to quit early, but this seems like a decent compromise to me.