Skip to content

Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301

Closed
Closed
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
Labels
area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release

Description

@hkc-8010
hkc-8010
opened on May 3, 2026

Apache Airflow version

3.2.0

What happened?

We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.

In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.

There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.

We also saw repeated API/UI-side errors for the same DAG version while this was happening:

2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

What you think should happen instead?

If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.

At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.

How to reproduce

We do not yet have a minimal standalone repro, but this is the observed pattern:

  1. Run Airflow with versioned or changing DAG bundle paths.
  2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
  3. Move the same DAG to a new bundle path.
  4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
  5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.

Operating System

Linux

Versions of Apache Airflow Providers

Not central to the issue

Deployment

Custom / Astro Hosted runtime based on Airflow 3.2.0

Deployment details

Observed on Airflow 3.2.0 with versioned DAG bundle paths.

Anything else?

A few concrete observations from the incident:

  • DagModel.fileloc was current and pointed at the active bundle path.
  • DagModel.last_parsed_time was current.
  • SerializedDagModel.last_updated was still old.
  • SerializedDagModel still pointed at the old bundle path.
  • The old bundle path no longer existed on the current pod.
  • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
  • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.

The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:

if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse

But fileloc is set separately from dag.fileloc when building the serialized row:

self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)

So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.

Are you willing to submit PR?

Yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      , 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
       blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
      Skip to content
      Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
      Closed
      Closed
      Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
      Labels
      area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
      Description
      @hkc-8010
      hkc-8010
      opened on May 3, 2026
      Apache Airflow version
      3.2.0
      What happened?
      We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
      In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
      There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
      We also saw repeated API/UI-side errors for the same DAG version while this was happening:
      2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

      What you think should happen instead?
      If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
      At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
      How to reproduce
      We do not yet have a minimal standalone repro, but this is the observed pattern:
      1. Run Airflow with versioned or changing DAG bundle paths.
      2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
      3. Move the same DAG to a new bundle path.
      4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
      5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
      Operating System
      Linux
      Versions of Apache Airflow Providers
      Not central to the issue
      Deployment
      Custom / Astro Hosted runtime based on Airflow 3.2.0
      Deployment details
      Observed on Airflow 3.2.0 with versioned DAG bundle paths.
      Anything else?
      A few concrete observations from the incident:
      • DagModel.fileloc was current and pointed at the active bundle path.
      • DagModel.last_parsed_time was current.
      • SerializedDagModel.last_updated was still old.
      • SerializedDagModel still pointed at the old bundle path.
      • The old bundle path no longer existed on the current pod.
      • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
      • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
      The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
      if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
      But fileloc is set separately from dag.fileloc when building the serialized row:
      self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
      So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
      Are you willing to submit PR?
      Yes
      Metadata
      Metadata
      Assignees
      No one assigned
        Labels
        area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
        Type
        No type
        Projects
        No projects
          Milestone
          No milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
          Skip to content
          Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
          Closed
          Closed
          Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
          Labels
          area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
          Description
          @hkc-8010
          hkc-8010
          opened on May 3, 2026
          Apache Airflow version
          3.2.0
          What happened?
          We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
          In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
          There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
          We also saw repeated API/UI-side errors for the same DAG version while this was happening:
          2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

          What you think should happen instead?
          If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
          At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
          How to reproduce
          We do not yet have a minimal standalone repro, but this is the observed pattern:
          1. Run Airflow with versioned or changing DAG bundle paths.
          2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
          3. Move the same DAG to a new bundle path.
          4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
          5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
          Operating System
          Linux
          Versions of Apache Airflow Providers
          Not central to the issue
          Deployment
          Custom / Astro Hosted runtime based on Airflow 3.2.0
          Deployment details
          Observed on Airflow 3.2.0 with versioned DAG bundle paths.
          Anything else?
          A few concrete observations from the incident:
          • DagModel.fileloc was current and pointed at the active bundle path.
          • DagModel.last_parsed_time was current.
          • SerializedDagModel.last_updated was still old.
          • SerializedDagModel still pointed at the old bundle path.
          • The old bundle path no longer existed on the current pod.
          • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
          • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
          The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
          if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
          But fileloc is set separately from dag.fileloc when building the serialized row:
          self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
          So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
          Are you willing to submit PR?
          Yes
          Metadata
          Metadata
          Assignees
          No one assigned
            Labels
            area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
            Type
            No type
            Projects
            No projects
              Milestone
              No milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
              Skip to content
              Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
              Closed
              Closed
              Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
              Labels
              area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
              Description
              @hkc-8010
              hkc-8010
              opened on May 3, 2026
              Apache Airflow version
              3.2.0
              What happened?
              We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
              In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
              There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
              We also saw repeated API/UI-side errors for the same DAG version while this was happening:
              2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

              What you think should happen instead?
              If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
              At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
              How to reproduce
              We do not yet have a minimal standalone repro, but this is the observed pattern:
              1. Run Airflow with versioned or changing DAG bundle paths.
              2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
              3. Move the same DAG to a new bundle path.
              4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
              5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
              Operating System
              Linux
              Versions of Apache Airflow Providers
              Not central to the issue
              Deployment
              Custom / Astro Hosted runtime based on Airflow 3.2.0
              Deployment details
              Observed on Airflow 3.2.0 with versioned DAG bundle paths.
              Anything else?
              A few concrete observations from the incident:
              • DagModel.fileloc was current and pointed at the active bundle path.
              • DagModel.last_parsed_time was current.
              • SerializedDagModel.last_updated was still old.
              • SerializedDagModel still pointed at the old bundle path.
              • The old bundle path no longer existed on the current pod.
              • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
              • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
              The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
              if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
              But fileloc is set separately from dag.fileloc when building the serialized row:
              self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
              So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
              Are you willing to submit PR?
              Yes
              Metadata
              Metadata
              Assignees
              No one assigned
                Labels
                area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                Type
                No type
                Projects
                No projects
                  Milestone
                  No milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions
                  
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
                  Skip to content
                  Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
                  Closed
                  Closed
                  Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
                  Labels
                  area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                  Description
                  @hkc-8010
                  hkc-8010
                  opened on May 3, 2026
                  Apache Airflow version
                  3.2.0
                  What happened?
                  We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
                  In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
                  There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
                  We also saw repeated API/UI-side errors for the same DAG version while this was happening:
                  2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

                  What you think should happen instead?
                  If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
                  At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
                  How to reproduce
                  We do not yet have a minimal standalone repro, but this is the observed pattern:
                  1. Run Airflow with versioned or changing DAG bundle paths.
                  2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
                  3. Move the same DAG to a new bundle path.
                  4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
                  5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
                  Operating System
                  Linux
                  Versions of Apache Airflow Providers
                  Not central to the issue
                  Deployment
                  Custom / Astro Hosted runtime based on Airflow 3.2.0
                  Deployment details
                  Observed on Airflow 3.2.0 with versioned DAG bundle paths.
                  Anything else?
                  A few concrete observations from the incident:
                  • DagModel.fileloc was current and pointed at the active bundle path.
                  • DagModel.last_parsed_time was current.
                  • SerializedDagModel.last_updated was still old.
                  • SerializedDagModel still pointed at the old bundle path.
                  • The old bundle path no longer existed on the current pod.
                  • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
                  • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
                  The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
                  if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
                  But fileloc is set separately from dag.fileloc when building the serialized row:
                  self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
                  So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
                  Are you willing to submit PR?
                  Yes
                  Metadata
                  Metadata
                  Assignees
                  No one assigned
                    Labels
                    area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                    Type
                    No type
                    Projects
                    No projects
                      Milestone
                      No milestone
                      Relationships
                      None yet
                      Development
                      No branches or pull requests
                      Issue actions
                      
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
                      Skip to content
                      Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
                      Closed
                      Closed
                      Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
                      Labels
                      area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                      Description
                      @hkc-8010
                      hkc-8010
                      opened on May 3, 2026
                      Apache Airflow version
                      3.2.0
                      What happened?
                      We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
                      In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
                      There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
                      We also saw repeated API/UI-side errors for the same DAG version while this was happening:
                      2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

                      What you think should happen instead?
                      If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
                      At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
                      How to reproduce
                      We do not yet have a minimal standalone repro, but this is the observed pattern:
                      1. Run Airflow with versioned or changing DAG bundle paths.
                      2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
                      3. Move the same DAG to a new bundle path.
                      4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
                      5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
                      Operating System
                      Linux
                      Versions of Apache Airflow Providers
                      Not central to the issue
                      Deployment
                      Custom / Astro Hosted runtime based on Airflow 3.2.0
                      Deployment details
                      Observed on Airflow 3.2.0 with versioned DAG bundle paths.
                      Anything else?
                      A few concrete observations from the incident:
                      • DagModel.fileloc was current and pointed at the active bundle path.
                      • DagModel.last_parsed_time was current.
                      • SerializedDagModel.last_updated was still old.
                      • SerializedDagModel still pointed at the old bundle path.
                      • The old bundle path no longer existed on the current pod.
                      • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
                      • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
                      The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
                      if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
                      But fileloc is set separately from dag.fileloc when building the serialized row:
                      self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
                      So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
                      Are you willing to submit PR?
                      Yes
                      Metadata
                      Metadata
                      Assignees
                      No one assigned
                        Labels
                        area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                        Type
                        No type
                        Projects
                        No projects
                          Milestone
                          No milestone
                          Relationships
                          None yet
                          Development
                          No branches or pull requests
                          Issue actions
                          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
                          Skip to content
                          Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
                          Closed
                          Closed
                          Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
                          Labels
                          area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                          Description
                          @hkc-8010
                          hkc-8010
                          opened on May 3, 2026
                          Apache Airflow version
                          3.2.0
                          What happened?
                          We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
                          In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
                          There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
                          We also saw repeated API/UI-side errors for the same DAG version while this was happening:
                          2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

                          What you think should happen instead?
                          If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
                          At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
                          How to reproduce
                          We do not yet have a minimal standalone repro, but this is the observed pattern:
                          1. Run Airflow with versioned or changing DAG bundle paths.
                          2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
                          3. Move the same DAG to a new bundle path.
                          4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
                          5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
                          Operating System
                          Linux
                          Versions of Apache Airflow Providers
                          Not central to the issue
                          Deployment
                          Custom / Astro Hosted runtime based on Airflow 3.2.0
                          Deployment details
                          Observed on Airflow 3.2.0 with versioned DAG bundle paths.
                          Anything else?
                          A few concrete observations from the incident:
                          • DagModel.fileloc was current and pointed at the active bundle path.
                          • DagModel.last_parsed_time was current.
                          • SerializedDagModel.last_updated was still old.
                          • SerializedDagModel still pointed at the old bundle path.
                          • The old bundle path no longer existed on the current pod.
                          • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
                          • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
                          The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
                          if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
                          But fileloc is set separately from dag.fileloc when building the serialized row:
                          self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
                          So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
                          Are you willing to submit PR?
                          Yes
                          Metadata
                          Metadata
                          Assignees
                          No one assigned
                            Labels
                            area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                            Type
                            No type
                            Projects
                            No projects
                              Milestone
                              No milestone
                              Relationships
                              None yet
                              Development
                              No branches or pull requests
                              Issue actions
                              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles · Issue #66301 · apache/airflow · GitHub
                              Skip to content
                              Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles #66301
                              Closed
                              Closed
                              Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles#66301
                              Labels
                              area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                              Description
                              @hkc-8010
                              hkc-8010
                              opened on May 3, 2026
                              Apache Airflow version
                              3.2.0
                              What happened?
                              We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
                              In our case, DAG-level on_failure_callback behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
                              There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
                              We also saw repeated API/UI-side errors for the same DAG version while this was happening:
                              2026-05-01T17:58:24.234995Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.233481Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=8557f53b-6c9a-4482-ba94-bc70c3623e36 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.071765Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:58:24.069796Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=e21b1ab4-042d-4b77-b2c6-36bd4e05e8cc version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.513703Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:57:01.511211Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=ece1848c-2ea4-4355-9757-440b8eb725fd version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.651543Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1
2026-05-01T17:56:44.650352Z [error ] No serialized dag found [airflow.api_fastapi.core_api.routes.ui.grid] dag_id=fail_check loc=grid.py:113 request_id=6cbfde23-d947-45c9-a95c-5d071e00ff09 version_id=UUID('019db6f1-958c-7317-98c2-472393931510') version_number=1

                              What you think should happen instead?
                              If a DAG's fileloc changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
                              At minimum, fileloc should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
                              How to reproduce
                              We do not yet have a minimal standalone repro, but this is the observed pattern:
                              1. Run Airflow with versioned or changing DAG bundle paths.
                              2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
                              3. Move the same DAG to a new bundle path.
                              4. Observe that DagModel.fileloc and current parsing reflect the new path, but SerializedDagModel may still point to the old path.
                              5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
                              Operating System
                              Linux
                              Versions of Apache Airflow Providers
                              Not central to the issue
                              Deployment
                              Custom / Astro Hosted runtime based on Airflow 3.2.0
                              Deployment details
                              Observed on Airflow 3.2.0 with versioned DAG bundle paths.
                              Anything else?
                              A few concrete observations from the incident:
                              • DagModel.fileloc was current and pointed at the active bundle path.
                              • DagModel.last_parsed_time was current.
                              • SerializedDagModel.last_updated was still old.
                              • SerializedDagModel still pointed at the old bundle path.
                              • The old bundle path no longer existed on the current pod.
                              • DAG-level failure callbacks did not appear to fire for failed runs before refresh.
                              • After deleting the serialized DAG row and letting Airflow recreate it, the serialized fileloc updated to the current path and the next customer test worked.
                              The serialization logic appears relevant here. In SerializedDagModel.write_dag, the row is skipped when dag_hash and processor_subdir are unchanged:
                              if (
serialized_dag_dbisnotNoneandserialized_dag_db.dag_hash==new_serialized_dag.dag_hashandserialized_dag_db.processor_subdir==new_serialized_dag.processor_subdir
):
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to DB", dag.dag_id)
returnFalse
                              But fileloc is set separately from dag.fileloc when building the serialized row:
                              self.fileloc=dag.filelocself.fileloc_hash=DagCode.dag_fileloc_hash(self.fileloc)
                              So if only fileloc changes while the serialized DAG content hash stays the same, the row may never be rewritten.
                              Are you willing to submit PR?
                              Yes
                              Metadata
                              Metadata
                              Assignees
                              No one assigned
                                Labels
                                area:corearea:serializationkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release
                                Type
                                No type
                                Projects
                                No projects
                                  Milestone
                                  No milestone
                                  Relationships
                                  None yet
                                  Development
                                  No branches or pull requests
                                  Issue actions