Skip to content

fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061

Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom
bujjibabukatta:fix/#68045
Closed

fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom
bujjibabukatta:fix/#68045

Conversation

@bujjibabukatta

@bujjibabukattabujjibabukatta commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable
or CronPartitionTimetable intermittently produce duplicate DagRuns for the
same asset event.

PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets
flow through a separate code path — _create_dagruns_for_partitioned_asset_dags
— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.
Two schedulers can read the same unprocessed rows within ~40ms of each other
and independently create a DagRun for each.

Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers
and 2+ consumer DAGs on the same asset.

Fix

Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun
query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.

SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the
APDR rows it is processing, Scheduler B's identical query returns zero rows
and exits cleanly — no duplicate DagRun is created.

with_row_locks is already imported in this file so no new imports are needed.

Testing

Related

@boring-cyborgboring-cyborgBot added the area:Scheduler including HA (high availability) scheduler label Jun 5, 2026
jroachgolf84
jroachgolf84 reviewed Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py

@jroachgolf84jroachgolf84Jun 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?

@jroachgolf84jroachgolf84Jun 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes

@potiuk

potiuk commented Jun 9, 2026

Copy link
Copy Markdown
Member

@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@Mercury2699

Mercury2699 commented Jun 10, 2026

Copy link
Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.

This PR can be closed as superseded by #64571.

@Mercury2699Mercury2699 mentioned this pull request Jun 10, 2026
Closed
@kaxilkaxil closed this Jun 12, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner
1 more reviewer
@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773

6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… by bujjibabukatta · Pull Request #68061 · apache/airflow · GitHub
Skip to content
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent… - #68061
Closed
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Closed
fix(scheduler): add row lock to AssetPartitionDagRun fetch to prevent…#68061
bujjibabukatta wants to merge 4 commits into
apache:mainfrom 
bujjibabukatta:fix/#68045
Conversation
@bujjibabukatta
@bujjibabukattabujjibabukatta
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
When running 2+ schedulers (HA mode), DAGs using PartitionedAssetTimetable

or CronPartitionTimetable intermittently produce duplicate DagRuns for the

same asset event.
PR #60773 fixed this race for non-partitioned asset scheduling by adding
with_row_locks to the AssetDagRunQueue fetch. However, partitioned assets

flow through a separate code path — _create_dagruns_for_partitioned_asset_dags

— which reads AssetPartitionDagRun rows with a plain SELECT, no lock.

Two schedulers can read the same unprocessed rows within ~40ms of each other

and independently create a DagRun for each.
Confirmed reproducible on MWAA 3.2.1 (which ships #60773) with 2 schedulers

and 2+ consumer DAGs on the same asset.
Fix
Add with_row_locks(..., skip_locked=True) to the AssetPartitionDagRun

query in _create_dagruns_for_partitioned_asset_dags, mirroring exactly what
#60773 did for AssetDagRunQueue.
SELECT ... FOR UPDATE SKIP LOCKED ensures that when Scheduler A locks the

APDR rows it is processing, Scheduler B's identical query returns zero rows

and exits cleanly — no duplicate DagRun is created.
with_row_locks is already imported in this file so no new imports are needed.
Testing
Related
@boring-cyborgboring-cyborgBot
added
the 
area:Scheduler
including HA (high availability) scheduler
label
Jun 5, 2026
jroachgolf84
jroachgolf84
reviewed
Jun 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/jobs/scheduler_job_runner.py
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bujjibabukatta - should we add unit-testing to this?
@jroachgolf84jroachgolf84Jun 7, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out the tests that were written for this PR (which yours mirrors): https://github.com/apache/airflow/pull/60773/changes
@potiuk
potiuk
commented
Jun 9, 2026

Copy link


Copy Markdown

Member
@bujjibabukatta — There are 1 unresolved review thread(s) on this PR from @jroachgolf84. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (@jroachgolf84) for a final look. Thanks!
Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.
@Mercury2699
Mercury2699
commented
Jun 10, 2026

Copy link


Copy Markdown

This fix is already on main — PR #64571 ("AIP-76: Hold Dag run until all upstream partitions arrive", merged Jun 5 2026 by @Lee-W) rewrote _create_dagruns_for_partitioned_asset_dags from scratch with with_row_locks(skip_locked=True, key_share=False) built in from the start. The race condition described in #68045 is fully addressed by that rewrite.
This PR can be closed as superseded by #64571.
@Mercury2699Mercury2699
mentioned this pull request
Jun 10, 2026

Closed

@kaxilkaxil
closed this
Jun 12, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbAwaiting requested review from ashbashb is a code owner
@XD-DENGXD-DENGAwaiting requested review from XD-DENGXD-DENG is a code owner

1 more reviewer

@jroachgolf84jroachgolf84jroachgolf84 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

AI Spamarea:Schedulerincluding HA (high availability) scheduler

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate DAG runs for PartitionedAssetTimetable with multiple schedulers (HA) — race not covered by #60773


6 participants

@bujjibabukatta@potiuk@Mercury2699@jroachgolf84@kaxil@Ramakrishna698