Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Avoid extra_dejson in ADF and Synapse async hooks by Vamsi-klu · Pull Request #72130 · apache/airflow · GitHub
Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Merged
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Aug 29, 2026
Merged

Avoid extra_dejson in ADF and Synapse async hooks#72130
potiuk merged 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-kluVamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.
closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
ContributorAuthor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

@dabla

Copy link
Copy Markdown
Contributor

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla@eladkal@Lee-W

Drafted-by: Cursor Grok 4.6 (no human review before posting)

There is no async extra_dejson counterpart, so yes I think it makes senses. We could add one but then we also need to add a backward compat function for the providers, which probably would be the neatest solution.

The is also a PR open which addresses another generic issue regarding the async path with connections as well.

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Simple to accept without proof of running it on real Synapse

@potiuk

Copy link
Copy Markdown
Member

But @Vamsi-klu - please test it when we release the provider - you will be `@-mentioned'

@potiuk
potiuk merged commit 39b4804 into apache:mainAug 29, 2026
83 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

3 participants

@Vamsi-klu@dabla@potiuk