[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) - #67129

Merged
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test
May 19, 2026
Merged

[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016)#67129
vatsrahul1001 merged 1 commit into
v3-2-testfrom
backport-c8a6c55-v3-2-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • fix: migrate existing deadline rows in migration 0080 upgrade and downgrade

Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:

  • upgrade: NotNullViolation when adding callback JSON NOT NULL to a
    non-empty table (existing rows have no value for the new
    column).
  • downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
    adding callback VARCHAR(500) NOT NULL after dropping the
    JSON column, crashing the subsequent 0094 upgrade with
    json.loads(None).

Fix both paths with the same pattern used by migration 0094:

  1. Read the existing data before any schema change.
  2. Add the new column(s) as nullable so the DDL succeeds on all
    supported databases.
  3. Back-fill the column using a typed SA table clause (so each dialect
    handles JSON serialisation correctly).
  4. Enforce NOT NULL only after every row has a valid value.

Upgrade serialises the old (path, kwargs) pair into the
{"data": {"path": ..., "kwargs": ...}, "classname": ...,
"version": 0} format expected by migration 0094.

Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.

Made-with: Cursor
Co-authored-by: Cursor cursoragent@cursor.com

  • fix(deadline): repair NULL callback rows in 0094 upgrade

Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).

Defensive handling in 0094:

  • _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
    empty envelope.
  • _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
    doesn't propagate into callback.data.

Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.

Addresses ephraimbuddy's review comment on #66016.

  • fix(deadline): address review nits
  • 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
    (e.g. from hand-edited DBs) is also normalized to {} -- matches the
    MySQL/SQLite defensive normalization.
  • 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
    single summary line after the loop to avoid log spam on deployments
    with many legacy NULL rows.
  • 0080 downgrade: log a WARNING when an existing row's kwargs is not a
    dict (instead of silently resetting), mirroring the VARCHAR truncate
    warning pattern.
  • 0080 tests: add test_upgrade_exact_batch_boundary covering the case
    where rows == batch_size to exercise the loop-continuation path.
  • test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch

_upgrade_mysql_sqlite declares the deadline id column as sa.Uuid().
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via str(uuid.uuid4())
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.

Insert ids via uuid.uuid4().hex so read and write round-trip cleanly.

  • fix(deadline): drop redundant empty-rows break in 0080 batch loops

  • fix(deadline): restore empty-rows break in 0080 batch loops

Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.


(cherry picked from commit c8a6c55)

Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com
Co-authored-by: Cursor cursoragent@cursor.com

…rade and downgrade (#66016)
* fix: migrate existing deadline rows in migration 0080 upgrade and downgrade
Both the upgrade and downgrade paths of migration 0080
(808787349f22 - Modify deadline callback schema) added NOT NULL columns
to the deadline table without first populating them from the existing
data, causing:
* upgrade: NotNullViolation when adding callback JSON NOT NULL to a
non-empty table (existing rows have no value for the new
column).
* downgrade: NotNullViolation on PostgreSQL / silent NULL on MySQL when
adding callback VARCHAR(500) NOT NULL after dropping the
JSON column, crashing the subsequent 0094 upgrade with
json.loads(None).
Fix both paths with the same pattern used by migration 0094:
1. Read the existing data before any schema change.
2. Add the new column(s) as nullable so the DDL succeeds on all
supported databases.
3. Back-fill the column using a typed SA table clause (so each dialect
handles JSON serialisation correctly).
4. Enforce NOT NULL only after every row has a valid value.
Upgrade serialises the old (path, kwargs) pair into the
{"__data__": {"path": ..., "kwargs": ...}, "__classname__": ...,
"__version__": 0} format expected by migration 0094.
Downgrade extracts path/kwargs back from that same JSON envelope
before restoring the VARCHAR callback column.
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(deadline): repair NULL callback rows in 0094 upgrade
Legacy MySQL deployments that ran the original (pre-#66016) 0080
migration silently wrote NULL into deadline.callback. Alembic is already
stamped past 808787349f22 on those deployments, so the fixed 0080 won't
re-run -- 0094 then crashes on json.loads(None).
Defensive handling in 0094:
- _upgrade_mysql_sqlite: detect raw_cb is None, log warning, default to
empty envelope.
- _upgrade_postgresql: COALESCE on cb_path / cb_kwargs so NULL jsonb
doesn't propagate into callback.data.
Regression test starts from a post-0080 schema with callback=NULL and
verifies 0094 produces an empty envelope without crashing, plus that a
mixed NULL+valid batch migrates both rows correctly.
Addresses ephraimbuddy's review comment on #66016.
* fix(deadline): address review nits
- 0094 PG path: wrap kwargs COALESCE with NULLIF so JSON-literal null
(e.g. from hand-edited DBs) is also normalized to {} -- matches the
MySQL/SQLite defensive normalization.
- 0094 MySQL/SQLite: aggregate per-row NULL-callback WARNING into a
single summary line after the loop to avoid log spam on deployments
with many legacy NULL rows.
- 0080 downgrade: log a WARNING when an existing row's kwargs is not a
dict (instead of silently resetting), mirroring the VARCHAR truncate
warning pattern.
- 0080 tests: add test_upgrade_exact_batch_boundary covering the case
where rows == batch_size to exercise the loop-continuation path.
* test(deadline): insert deadline ids in hex form to avoid SA Uuid type mismatch
`_upgrade_mysql_sqlite` declares the deadline `id` column as `sa.Uuid()`.
On SQLite the SQLAlchemy write path serializes UUID objects as a hex string
without dashes. The previous test inserted ids via `str(uuid.uuid4())`
(dashed form), so the SELECT correctly parsed them back to UUID objects
but the subsequent UPDATE's WHERE clause produced the hex form -- no rows
matched, the WHERE callback_id IS NULL filter never narrowed, and the
loop spun until the 60s execution timeout.
Insert ids via `uuid.uuid4().hex` so read and write round-trip cleanly.
* fix(deadline): drop redundant empty-rows break in 0080 batch loops
* fix(deadline): restore empty-rows break in 0080 batch loops
Without this guard, an empty deadline table causes executemany with an
empty parameter list, which SQLAlchemy rejects with 'A value is required
for bind parameter'. Reverts the removal from e50f106.
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@boring-cyborgboring-cyborgBot added area:db-migrations PRs with DB migration area:deadline-alerts AIP-86 (former AIP-57) labels May 18, 2026
@potiuk
potiuk marked this pull request as ready for review May 18, 2026 19:37
@potiuk
potiuk requested a review from ephraimbuddy as a code ownerMay 18, 2026 19:37
@vatsrahul1001vatsrahul1001 added this to the Airflow 3.2.2 milestone May 19, 2026
@vatsrahul1001vatsrahul1001 added the type:bug-fix Changelog: Bug Fixes label May 19, 2026
@vatsrahul1001
vatsrahul1001 merged commit e33c99f into v3-2-testMay 19, 2026
105 checks passed
@vatsrahul1001
vatsrahul1001 deleted the backport-c8a6c55-v3-2-test branch May 19, 2026 04:34
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 20, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request May 21, 2026
…rade and downgrade (#66016) (#67129)
[v3-2-test] fix: migrate existing deadline rows in migration 0080 upgrade and downgrade (#66016) (#67129)
---------
(cherry picked from commit c8a6c55)
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:db-migrationsPRs with DB migrationarea:deadline-alertsAIP-86 (former AIP-57)type:bug-fixChangelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@vatsrahul1001