Skip to content

Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004

Closed
#65594
Closed
Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
#65594
Assignees
ashb
Labels
affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release

Description

@wolfier
wolfier
opened on Aug 28, 2025

Apache Airflow version

3.0.5

If "Other Airflow 2 version" selected, which one?

No response

What happened?

A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.

No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.

This means the following:

  • Task log is pushed to the remote log location
  • Task attempt is NOT updated in task_instance_history
  • Task log of the missing attempt is not displayed on the Airflow UI
  • Task is reset mid-execution and retried in a way that is NOT expected by a user

What you think should happen instead?

Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.

I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).

How to reproduce

  1. Create a dag with a long running task instance
  2. Lock the task instance by running a locking query
  3. Kill the scheduler and confirm the task instance is reset

Operating System

Debian

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    , 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
     blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
    Skip to content
    Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
    Closed
    #65594
    Closed
    Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
    #65594
    Assignees
    ashb
    Labels
    affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
    Description
    @wolfier
    wolfier
    opened on Aug 28, 2025
    Apache Airflow version
    3.0.5
    If "Other Airflow 2 version" selected, which one?
    No response
    What happened?
    A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
    No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
    This means the following:
    • Task log is pushed to the remote log location
    • Task attempt is NOT updated in task_instance_history
    • Task log of the missing attempt is not displayed on the Airflow UI
    • Task is reset mid-execution and retried in a way that is NOT expected by a user
    What you think should happen instead?
    Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
    I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
    How to reproduce
    1. Create a dag with a long running task instance
    2. Lock the task instance by running a locking query
    3. Kill the scheduler and confirm the task instance is reset
    Operating System
    Debian
    Versions of Apache Airflow Providers
    No response
    Deployment
    Astronomer
    Deployment details
    No response
    Anything else?
    No response
    Are you willing to submit PR?
    •  Yes I am willing to submit a PR!
    Code of Conduct
    Metadata
    Metadata
    Assignees
    Labels
    affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
    Type
    No type
    Projects
    No projects
      Milestone
      No milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions
      
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
      Skip to content
      Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
      Closed
      #65594
      Closed
      Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
      #65594
      Assignees
      ashb
      Labels
      affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
      Description
      @wolfier
      wolfier
      opened on Aug 28, 2025
      Apache Airflow version
      3.0.5
      If "Other Airflow 2 version" selected, which one?
      No response
      What happened?
      A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
      No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
      This means the following:
      • Task log is pushed to the remote log location
      • Task attempt is NOT updated in task_instance_history
      • Task log of the missing attempt is not displayed on the Airflow UI
      • Task is reset mid-execution and retried in a way that is NOT expected by a user
      What you think should happen instead?
      Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
      I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
      How to reproduce
      1. Create a dag with a long running task instance
      2. Lock the task instance by running a locking query
      3. Kill the scheduler and confirm the task instance is reset
      Operating System
      Debian
      Versions of Apache Airflow Providers
      No response
      Deployment
      Astronomer
      Deployment details
      No response
      Anything else?
      No response
      Are you willing to submit PR?
      •  Yes I am willing to submit a PR!
      Code of Conduct
      Metadata
      Metadata
      Assignees
      Labels
      affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
      Type
      No type
      Projects
      No projects
        Milestone
        No milestone
        Relationships
        None yet
        Development
        No branches or pull requests
        Issue actions
        
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
        Skip to content
        Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
        Closed
        #65594
        Closed
        Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
        #65594
        Assignees
        ashb
        Labels
        affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
        Description
        @wolfier
        wolfier
        opened on Aug 28, 2025
        Apache Airflow version
        3.0.5
        If "Other Airflow 2 version" selected, which one?
        No response
        What happened?
        A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
        No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
        This means the following:
        • Task log is pushed to the remote log location
        • Task attempt is NOT updated in task_instance_history
        • Task log of the missing attempt is not displayed on the Airflow UI
        • Task is reset mid-execution and retried in a way that is NOT expected by a user
        What you think should happen instead?
        Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
        I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
        How to reproduce
        1. Create a dag with a long running task instance
        2. Lock the task instance by running a locking query
        3. Kill the scheduler and confirm the task instance is reset
        Operating System
        Debian
        Versions of Apache Airflow Providers
        No response
        Deployment
        Astronomer
        Deployment details
        No response
        Anything else?
        No response
        Are you willing to submit PR?
        •  Yes I am willing to submit a PR!
        Code of Conduct
        Metadata
        Metadata
        Assignees
        Labels
        affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
        Type
        No type
        Projects
        No projects
          Milestone
          No milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
          Skip to content
          Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
          Closed
          #65594
          Closed
          Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
          #65594
          Assignees
          ashb
          Labels
          affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
          Description
          @wolfier
          wolfier
          opened on Aug 28, 2025
          Apache Airflow version
          3.0.5
          If "Other Airflow 2 version" selected, which one?
          No response
          What happened?
          A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
          No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
          This means the following:
          • Task log is pushed to the remote log location
          • Task attempt is NOT updated in task_instance_history
          • Task log of the missing attempt is not displayed on the Airflow UI
          • Task is reset mid-execution and retried in a way that is NOT expected by a user
          What you think should happen instead?
          Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
          I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
          How to reproduce
          1. Create a dag with a long running task instance
          2. Lock the task instance by running a locking query
          3. Kill the scheduler and confirm the task instance is reset
          Operating System
          Debian
          Versions of Apache Airflow Providers
          No response
          Deployment
          Astronomer
          Deployment details
          No response
          Anything else?
          No response
          Are you willing to submit PR?
          •  Yes I am willing to submit a PR!
          Code of Conduct
          Metadata
          Metadata
          Assignees
          Labels
          affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
          Type
          No type
          Projects
          No projects
            Milestone
            No milestone
            Relationships
            None yet
            Development
            No branches or pull requests
            Issue actions
            
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
            Skip to content
            Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
            Closed
            #65594
            Closed
            Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
            #65594
            Assignees
            ashb
            Labels
            affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
            Description
            @wolfier
            wolfier
            opened on Aug 28, 2025
            Apache Airflow version
            3.0.5
            If "Other Airflow 2 version" selected, which one?
            No response
            What happened?
            A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
            No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
            This means the following:
            • Task log is pushed to the remote log location
            • Task attempt is NOT updated in task_instance_history
            • Task log of the missing attempt is not displayed on the Airflow UI
            • Task is reset mid-execution and retried in a way that is NOT expected by a user
            What you think should happen instead?
            Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
            I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
            How to reproduce
            1. Create a dag with a long running task instance
            2. Lock the task instance by running a locking query
            3. Kill the scheduler and confirm the task instance is reset
            Operating System
            Debian
            Versions of Apache Airflow Providers
            No response
            Deployment
            Astronomer
            Deployment details
            No response
            Anything else?
            No response
            Are you willing to submit PR?
            •  Yes I am willing to submit a PR!
            Code of Conduct
            Metadata
            Metadata
            Assignees
            Labels
            affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
            Type
            No type
            Projects
            No projects
              Milestone
              No milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
              Skip to content
              Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
              Closed
              #65594
              Closed
              Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
              #65594
              Assignees
              ashb
              Labels
              affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
              Description
              @wolfier
              wolfier
              opened on Aug 28, 2025
              Apache Airflow version
              3.0.5
              If "Other Airflow 2 version" selected, which one?
              No response
              What happened?
              A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
              No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
              This means the following:
              • Task log is pushed to the remote log location
              • Task attempt is NOT updated in task_instance_history
              • Task log of the missing attempt is not displayed on the Airflow UI
              • Task is reset mid-execution and retried in a way that is NOT expected by a user
              What you think should happen instead?
              Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
              I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
              How to reproduce
              1. Create a dag with a long running task instance
              2. Lock the task instance by running a locking query
              3. Kill the scheduler and confirm the task instance is reset
              Operating System
              Debian
              Versions of Apache Airflow Providers
              No response
              Deployment
              Astronomer
              Deployment details
              No response
              Anything else?
              No response
              Are you willing to submit PR?
              •  Yes I am willing to submit a PR!
              Code of Conduct
              Metadata
              Metadata
              Assignees
              Labels
              affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
              Type
              No type
              Projects
              No projects
                Milestone
                No milestone
                Relationships
                None yet
                Development
                No branches or pull requests
                Issue actions
                
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Task instances are reset when its first executor event is ignored (missing `external_executor_id`) and the scheduler dies · Issue #55004 · apache/airflow · GitHub
                Skip to content
                Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies #55004
                Closed
                #65594
                Closed
                Task instances are reset when its first executor event is ignored (missing external_executor_id) and the scheduler dies#55004
                #65594
                Assignees
                ashb
                Labels
                affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
                Description
                @wolfier
                wolfier
                opened on Aug 28, 2025
                Apache Airflow version
                3.0.5
                If "Other Airflow 2 version" selected, which one?
                No response
                What happened?
                A task instance is queued and the celery executor submitted an executor event with state=TaskInstanceState.QUEUED. However, the scheduler did not fetch the event's corresponding task instance because it was locked at the time. You may be wondering what locked the task instance when the scheduler ran the query. I do not think it is critical to identify the source as task instances are locked for various reasons in Airflow.
                No task instance fetched means the task instance's external_executor_id is not set. Usually, the external_executor_id is NOT critical to the completion of the task execution; however, if the scheduler died after the event is essentially ignored then it is an issue as the orphaned task is NOT adopted without an external_executor_id. The task instance is reset instead.
                This means the following:
                • Task log is pushed to the remote log location
                • Task attempt is NOT updated in task_instance_history
                • Task log of the missing attempt is not displayed on the Airflow UI
                • Task is reset mid-execution and retried in a way that is NOT expected by a user
                What you think should happen instead?
                Instead of ignoring the executor event when its corresponding task instance is locked, the executor event should be returned to the executor's event buffer to be processed at the next scheduler loop. Currently, all executor events are removed from the event buffer first to be processed so any ignored executor events are consumed and not retried.
                I want to highlight that external_executor_id is important when an orphaned task is attempted to be adopted. The external_executor_id is used to query celery for the celery task and, only if successful, is the corresponding task instance adopted and added to the scheduler's running set (see source for more information).
                How to reproduce
                1. Create a dag with a long running task instance
                2. Lock the task instance by running a locking query
                3. Kill the scheduler and confirm the task instance is reset
                Operating System
                Debian
                Versions of Apache Airflow Providers
                No response
                Deployment
                Astronomer
                Deployment details
                No response
                Anything else?
                No response
                Are you willing to submit PR?
                •  Yes I am willing to submit a PR!
                Code of Conduct
                Metadata
                Metadata
                Assignees
                Labels
                affected_version:3.0Issues Reported for 3.0area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release
                Type
                No type
                Projects
                No projects
                  Milestone
                  No milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions