Don't mistakenly take a lock on DagRun via ti.refresh_from_fb by ashb · Pull Request #25312 · apache/airflow

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb · 2022-07-26T15:57:55Z

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine
for most cases, but for refresh_from_db we don't need that (we don't
access anything under ti.dag_run) and it's possible that when
lock_for_update=True is passed we are locking more than we want to and
might cause deadlocks.

Even if it doesn't, selecting more than we need is wasteful.

Idea inspired by @dstaple, except rather than not locking it changed to not even select against the DagRun table in the first place.

Might fix#23361

ashb · 2022-07-26T15:58:45Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

ashb · 2022-07-26T15:59:21Z

@RNHTTR Can you see if this change does anything for the deadlock you are able to reproduce?

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

In 2.2.0 we made TI.dag_run be automatically join-loaded, which is fine for most cases, but for `refresh_from_db` we don't need that (we don't access anything under ti.dag_run) and it's possible that when `lock_for_update=True` is passed we are locking more than we want to and _might_ cause deadlocks. Even if it doesn't, selecting more than we need is wasteful. (cherry picked from commit be2b53e)

…#25312)"

dstaple · 2023-10-18T15:43:22Z

/cc @potiuk Even if this doesn't fix the deadlock it's probably worth doing

For future reference, this PR did indeed fix the specific deadlock issue from #23361 , thanks again folks.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in apache#25312 when we converted to query with the column attrs instead of the TI object (apache#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

Fundamentally what's going on here is we need a TaskInstance object instead of a Row object when sending over the wire in RPC call. But the full story on this one is actually somewhat complicated. It was back in 2.2.0 in #25312 when we converted to query with the column attrs instead of the TI object (#28900 only refactored this logic into a function). The reason was to avoid locking the dag_run table since TI newly had a dag_run relationship attr. Now, this causes a problem with AIP-44 because the RPC api does not know how to serialize a Row object. This PR switches back to querying a TaskInstance object, but avoids locking dag_run by using lazy_load option. Meanwhile, since try_number is a horrible attribute (which gives you a different answer depending on the state), we have to switch it back to look at the underlying private attr instead of the public accesor.

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

ashb requested review from XD-DENG and kaxil as code owners July 26, 2022 15:57

potiuk reviewed Jul 26, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

potiuk approved these changes Jul 26, 2022
View reviewed changes

ashb force-pushed the possible-deadlock-fix branch from 87ab6b6 to 2073e1aCompare July 27, 2022 14:40

potiuk approved these changes Jul 27, 2022
View reviewed changes

ashb added 2 commits July 28, 2022 11:30

fixup! Don't mistakenly take a lock on DagRun via ti.refresh_from_fb
0984660

ashb force-pushed the possible-deadlock-fix branch from 2073e1a to 0984660Compare July 28, 2022 10:56

RNHTTR mentioned this pull request Jul 28, 2022
DAG Run fails when chaining multiple empty mapped tasks #25200
Closed
2 tasks

potiuk mentioned this pull request Jul 28, 2022
Fix Deadlock on refresh from DB by local task run #25266
Closed

jedcunningham approved these changes Jul 28, 2022
View reviewed changes

Comment threadairflow/models/taskinstance.py Outdated

Update airflow/models/taskinstance.py
5cc7608
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

jedcunningham added the type:bug-fix Changelog: Bug Fixes label Jul 28, 2022

jedcunningham added this to the Airflow 2.3.4 milestone Jul 28, 2022

Merge branch 'main' into possible-deadlock-fix
aa18f18

ashb merged commit be2b53e into apache:mainAug 9, 2022

ashb deleted the possible-deadlock-fix branch August 9, 2022 14:17

ephraimbuddy added a commit that referenced this pull request Aug 19, 2022
Revert "Don't mistakenly take a lock on DagRun via ti.refresh_from_fb (…
0d08201
…#25312)"

ephraimbuddy mentioned this pull request Aug 20, 2022
Status of testing of Apache Airflow 2.3.4rc1 #25846
Closed
53 tasks

dstaple mentioned this pull request Aug 24, 2022
Scheduler crashes with psycopg2.errors.DeadlockDetected exception #23361
Closed
2 tasks

dstaple mentioned this pull request Nov 2, 2022
Deadlock when updating DagRun last_scheduling_decision and TaskInstance state=scheduled #27473
Closed
2 tasks

dstandish mentioned this pull request Mar 30, 2024
Remove select_column option in TaskInstance.get_task_instance #38571
Merged

Uh oh!

Conversation

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022

Uh oh!

Uh oh!

Uh oh!

dstaple commented Oct 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022

Uh oh!

Uh oh!

Uh oh!

dstaple commented Oct 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022

Uh oh!

Uh oh!

Uh oh!

dstaple commented Oct 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Jul 26, 2022

Uh oh!

Uh oh!

Uh oh!

dstaple commented Oct 18, 2023

Uh oh!

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading

ashb commented Jul 26, 2022•
edited
Loading