Update conf column in dag_run table type from bytes to JSON

[vatsrahul1001]

closes: #43933

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vatsrahul1001]

While Testing this I noticed an issue in the downgrade case. We are not removing the data from downgrade and just perform below conversion.

 ALTER TABLE dag_run
ALTER COLUMN conf TYPE BYTEA
USING CASE
WHEN conf IS NOT NULL THEN CONVERT_TO(conf::TEXT, 'UTF8')
ELSE NULL
END

I am getting UnpicklingError error with this when I try to open DAG page.

Should we also move data to the archive table in case of a downgrade as well?
cc: @kaxil

[kaxil]

While Testing this I noticed an issue in the downgrade case. We are not removing the data from downgrade and just perform below conversion.

 ALTER TABLE dag_run
ALTER COLUMN conf TYPE BYTEA
USING CASE
WHEN conf IS NOT NULL THEN CONVERT_TO(conf::TEXT, 'UTF8')
ELSE NULL
END

I am getting UnpicklingError error with this when I try to open DAG page.

Should we also move data to the archive table in case of a downgrade as well? cc: @kaxil

Yeah, it is different than XCom, because XCom code had handling of both JSON & pickle type -- for dagrun conf it would be different. For downgrade you might just want to insert all records from the archive table back here.