Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Nuking SequentialExecutor from the in-tree executors - #48528

Merged
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor
Apr 1, 2025
Merged

Nuking SequentialExecutor from the in-tree executors#48528
amoghrajesh merged 11 commits into
apache:mainfrom
astronomer:nuke-sequential-executor

Conversation

@amoghrajesh

@amoghrajeshamoghrajesh commented Mar 29, 2025

Copy link
Copy Markdown
Contributor

closes: #48522

Let me do the honors!

Tried generating the image using AI, the spellings is what is hallucinates with! Anyway, worthwhile to add i'd say!

RIP

The alternative here is to go and use LocalExecutor -- which is a similar one and it works with sqlite. This was a PR that enabled it #44839


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Guess this is what's left as of now:
image

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

I know the failures here, but unfortunately wont have access to computer till tomorrow evening. I will fix those then..

But its ready for review id say 😀

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
@jscheffl

Copy link
Copy Markdown
Contributor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@eladkal

Copy link
Copy Markdown
Contributor

Was #44243 resolved?

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

I have access now, thanks @jscheffl. Just looking at the CI and missing occurences now.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Was #44243 resolved?

Looks like #44243 (comment) is a solution that works. Although i'd want someone to test it again if possible, I know its important but we've had more successes than not in that migration. LocalExec has been tested plenty :)

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Besides the leftovers if you search for "SequentialExecutor" (and also there are some hits when you search for "Sequential Executor" with space!) and some CI fails I think this is looking pretty good. If you lag access to a PC, let me know if I can help. Would be cool to remove before RC0.

@jscheffl it has been handled now 🗡️
Thanks for finding that!

Comment threadairflow-core/docs/core-concepts/executor/index.rst Outdated
Comment threadairflow-core/docs/howto/set-up-database.rst
Comment threadairflow-core/src/airflow/cli/commands/standalone_command.py

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally with a couple of DAGs and "small load test" (wow, was able to saturate 80% of my 20 cores) and looks good on my view. Would be cautious making this for a production system but much much better than SequentialExecutor. Cool out-of-the-box experience possible with this!

#protm

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Thanks for the reviews folks, merging this one.

@amoghrajesh
amoghrajesh merged commit 8a784a7 into apache:mainApr 1, 2025
@amoghrajesh
amoghrajesh deleted the nuke-sequential-executor branch April 1, 2025 09:32
aaron-y-chen pushed a commit to aaron-y-chen/airflow that referenced this pull request Apr 4, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
diogotrodrigues pushed a commit to diogotrodrigues/airflow that referenced this pull request Apr 6, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
* Nuking SequentialExecutor
* some occurences
* fixing ci failures -- docs, mypy and tests
* comments from jens
* fixing tests
* fixing occurences - both SequentialExecutor and Sequential Executor
* last occurence
* comments from niko
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit to astronomer/airflow that referenced this pull request Apr 10, 2025
…nificant.rst
reading apache#48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
Lee-W added a commit that referenced this pull request Apr 10, 2025
…49048)
* docs(newsfragments): check action items that has been done
* feat(config_command): add "core.dag_default_view" and "core.dag_orientation"
* docs(newsfragments): fix wrong description in newsfragments/48528.significant.rst
reading #48528, it suggest users to use dag.test() instead of replacing the value as dag.test()
* feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint
* Revert "feat(config_command): add scheduler.create_cron_data_intervals and scheduler.create_delta_data_intervals to config lint"
This reverts commit e60bfe3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:Executors-coreLocalExecutor & SequentialExecutorarea:loggingarea:Schedulerincluding HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nuke SequentialExecutor

5 participants

@amoghrajesh@jscheffl@eladkal@kaxil@o-nikolas