Adding run_id column to log table by SamWheating · Pull Request #37731 · apache/airflow

SamWheating · 2024-02-26T23:31:17Z

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

SamWheating · 2024-02-26T23:31:17Z

closes: #37597

This adds the run_id column to the log table (without a backfill, see discussion) and exposes it in all of the different surfaces:

the /eventLogs API
the logs view (/logs/list)
the inset / react audit logs DagRun details page
the legacy (non-react) audit logs view in the DAG details page

Screenshots:

Dag-scoped audit logs:

Dagrun-scoped audit logs:

Open questions:

Should we be pre-populating the run_id filter in the inset audit logs view (similar to what we do with task_id when a single task is selected)? This would be more precise than just filtering on task_id + dates, but it would exclude relevant events which aren't tagged with a run_id (for example, pausing a DAG mid-execution)

bbovenzi · 2024-02-27T05:42:32Z

I think prepopulating the runId filter works. After this I will open up the before/after fields to be user-editable but with some default options like a dag run's entire durations. The before/after range doesn't capture changing the state or a note. so either approach has pros and cons

Also, after this and #37734. We probably want to make sure we are populating run_id everywhere we can.

potiuk

LGTM . But would love another maintainer's review

potiuk · 2024-02-27T19:35:27Z

Just a watch-out - we have two migrations in-progress. I think we have a protection that will make the secon one not mergable if the first is but there will be some re-work needed once the other mighration is merged #37708

jedcunningham · 2024-02-27T20:45:13Z

We actually have 3 in flight 😅: #37734

potiuk · 2024-02-27T21:24:08Z

We actually have 3 in flight 😅: #37734

😱 😱 😱 😱 😱 😱 😱 😱 😱

SamWheating · 2024-02-27T22:08:48Z

Just a watch-out - we have two migrations in-progress

Thanks for the heads up - I'll keep a close 👀 on the other two and try to keep this PR in a mergeable state.

bbovenzi · 2024-02-28T16:50:49Z

We actually have 3 in flight 😅: #37734

Since we have two migrations touching Log. Should we do both at once?

SamWheating · 2024-02-28T17:58:58Z

Since we have two migrations touching Log. Should we do both at once?

This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

bbovenzi · 2024-02-29T15:08:22Z

Since we have two migrations touching Log. Should we do both at once?
This is a pretty good idea. Which PR do you want to merge first? If needed I can port the DB changes from the other PR into this one.

The other Audit Log one still has errors in the CI. So let's merge this first. The only change I had to make over there is to increase the max size of the event string.

SamWheating · 2024-02-29T17:35:47Z

Looks like the other TaskInstance migration PR was merged, I'll get this one updated.

bbovenzi · 2024-03-04T17:43:33Z

Since we have two migrations touching Log. Should we do both at once?

@jedcunningham @potiuk What do you prefer? Two separate event log table migrations, or combine them?

bbovenzi

lgtm. We'll merge the two migrations in the second PR

boring-cyborgBot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues kind:documentation labels Feb 26, 2024

SamWheating force-pushed the sw-add-run-id-to-log branch 2 times, most recently from 6656f1e to b04aa58Compare February 27, 2024 00:26

SamWheating marked this pull request as ready for review February 27, 2024 00:27

SamWheating requested review from XD-DENG, ashb, bbovenzi, ephraimbuddy, kaxil, pierrejeambrun, potiuk and ryanahamilton as code owners February 27, 2024 00:27

SamWheating force-pushed the sw-add-run-id-to-log branch from b04aa58 to 1052412Compare February 27, 2024 00:29

SamWheating changed the title ~~WIP: adding run_id column to log table~~Adding run_id column to log tableFeb 27, 2024

potiuk approved these changes Feb 27, 2024
View reviewed changes

potiuk mentioned this pull request Feb 27, 2024
Add migration for rendered_map_index column in task_instance table. #37708
Merged

SamWheating force-pushed the sw-add-run-id-to-log branch 2 times, most recently from 9f2400a to 45e11e4Compare February 29, 2024 18:03

adding run_id column to log table
ebbc05f

SamWheating force-pushed the sw-add-run-id-to-log branch from 45e11e4 to ebbc05fCompare March 1, 2024 18:39

bbovenzi approved these changes Mar 5, 2024
View reviewed changes

bbovenzi added this to the Airflow 2.9.0 milestone Mar 5, 2024

bbovenzi merged commit c41d194 into apache:mainMar 5, 2024

utkarsharma2 added the type:improvement Changelog: Improvements label Mar 6, 2024

boraberke mentioned this pull request Jun 7, 2024
Add run_id to event logs API #34211
Closed
2 tasks