Update dag reserialize command

[ephraimbuddy]

Now that we have versioning, users must specify that they want to delete history before we do it in airflow dags reserialize command.

serialize dags are no longer deleted as part of this command. Users should use airflow db-clean command

Also updated the _reserialize function at DB upgrade so that it doesn't delete the serializedDag since that won't be necessary.

[potiuk]

Approved. We need it now. Though I think it's quite a big limitation. Possibly (@jedcunningham ) - with DAG bundles we could actually in the future to reserialize also history ? It sounds feasible, we would just have to go through version history and checkout the bundle in each version and reserialize it then. It also might (or might not) work - depends for example on the third-party package versions installed now vs then but we could at least try.

But yeah, that's likely Airflow 3.5+ or smth :)

[potiuk]

Maybe also we should attempt to keep old serialized DAGS and attempt to load them (Assuming that our serialization is forward-compatible) . That could also be done on a "best effort" case.

[ephraimbuddy]

Maybe also we should attempt to keep old serialized DAGS and attempt to load them (Assuming that our serialization is forward-compatible) . That could also be done on a "best effort" case.

Yeah, old serialized dags won't be deleted except users want to do so. We would keep a serialization version that when deserializing, we could check the version it was serialized with and use that to deserialize it. It should be forward-compatible

[jedcunningham]

with DAG bundles we could actually in the future to reserialize also history ? It sounds feasible, we would just have to go through version history and checkout the bundle in each version and reserialize it then.

This can already be pretty slow. Doing it for every historical version isn't an option imo.

Maybe also we should attempt to keep old serialized DAGS and attempt to load them (Assuming that our serialization is forward-compatible) . That could also be done on a "best effort" case.

I think something like this is the right way to handle this. I think Kaxil is going to look into it as part of AIP-72 stuff: #43648

[ashb]

Given users often run this as part of upgrades (or db upgrade does it itself) we'll need to stop deleting things very soon

[ephraimbuddy]

I'm thinking of renaming this command as airflow dags re-version or any better name other than reserialize. Thoughts?

[ephraimbuddy]

Will appreciate another review @jedcunningham@pierrejeambrun

[pierrejeambrun]

I am not yet super familiar with the Versioning AIP but this makes sense to me.

Thanks.

[dstandish]

I'm thinking of renaming this command as airflow dags re-version or any better name other than reserialize. Thoughts?

I think re-version is probably not a good name. Shall we chat about it?

[dstandish]

think we may want to revisit some of the language changes. blocking merge @ephraimbuddy has chance to respond

old

[dstandish]

couple suggestions