Sync DAG specific permissions when parsing - #15311

Merged
kaxil merged 13 commits into
apache:masterfrom
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged

Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom
astronomer:perm_sync_on_parse

Conversation

@jedcunningham

@jedcunninghamjedcunningham commented Apr 9, 2021•
edited
Loading

Copy link
Copy Markdown
Member

This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.

With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.

This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.

Closes#8609

jedcunningham
jedcunningham commented Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?

@jhtimminsjhtimminsApr 13, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.

@jedcunningham
jedcunningham marked this pull request as draft April 9, 2021 18:03
@jedcunninghamjedcunningham mentioned this pull request Apr 9, 2021
Closed
@jedcunninghamjedcunningham mentioned this pull request Apr 10, 2021
Closed
jhtimmins
jhtimmins previously requested changes Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py Outdated
Comment threadairflow/models/serialized_dag.py Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.

kaxil
kaxil reviewed Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py Outdated
kaxil
kaxil reviewed Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py Outdated
@jedcunningham
jedcunningham marked this pull request as ready for review April 15, 2021 19:26
kaxil
kaxil reviewed Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md Outdated
kaxil
kaxil reviewed Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py Outdated
@kaxilkaxil changed the title WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil added this to the Airflow 2.1 milestone Apr 16, 2021
@kaxil
kaxil merged commit d52ad87 into apache:masterApr 19, 2021
@kaxil
kaxil deleted the perm_sync_on_parse branch April 19, 2021 11:50
kaxil pushed a commit to astronomer/airflow that referenced this pull request Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc

chodankarcc commented Jun 30, 2021

Copy link
Copy Markdown

which airflow version has this fixed change?

@jedcunningham

jedcunningham commented Jun 30, 2021

Copy link
Copy Markdown
MemberAuthor

2.1.0:

airflow/CHANGELOG.txt

Line 100 in e3f5eb9

- Sync DAG specific permissions when parsing (#15311)

@chodankarcc

chodankarcc commented Jun 30, 2021

Copy link
Copy Markdown

2.1.0:

airflow/CHANGELOG.txt

Line 100 in e3f5eb9

- Sync DAG specific permissions when parsing (#15311)

Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,

@jedcunningham

jedcunningham commented Jun 30, 2021

Copy link
Copy Markdown
MemberAuthor

I believe running airflow sync-perm should do it as well.

@chodankarcc

chodankarcc commented Jun 30, 2021

Copy link
Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.

@jedcunningham

jedcunningham commented Jun 30, 2021

Copy link
Copy Markdown
MemberAuthor

Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.

airflow/airflow/bin/cli.py

Lines 2075 to 2080 in 5786dcd

print('Updating permission on all DAG views')
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values()
fordagindags:
appbuilder.sm.sync_perm_for_dag(
dag.dag_id,
dag.access_control)

@chodankarcc

chodankarcc commented Jul 1, 2021

Copy link
Copy Markdown

store_serialized_dags

I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer

@ashb

ashb commented Jul 1, 2021

Copy link
Copy Markdown
Member

If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.

@ali-hafidz

ali-hafidz commented Jul 16, 2021

Copy link
Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.

@kosteevkosteev mentioned this pull request Apr 20, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees

No one assigned

Labels

area:serialization

Projects

None yet

Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI

6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Sync DAG specific permissions when parsing - #15311
Merged
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Apr 19, 2021
Merged
Sync DAG specific permissions when parsing#15311
kaxil merged 13 commits into
apache:masterfrom 
astronomer:perm_sync_on_parse
Conversation
@jedcunningham
@jedcunninghamjedcunningham
commented
Apr 9, 2021•

edited

Loading

Copy link


Copy Markdown

Member
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.
Closes#8609
jedcunningham
jedcunningham
commented
Apr 9, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

@jedcunninghamjedcunninghamApr 9, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?
@jhtimminsjhtimminsApr 13, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.
@jedcunningham
jedcunningham
marked this pull request as draft
April 9, 2021 18:03
@jedcunninghamjedcunningham
mentioned this pull request
Apr 9, 2021
 Closed

@jedcunninghamjedcunningham
mentioned this pull request
Apr 10, 2021

Closed

jhtimmins
jhtimmins
previously requested changes
Apr 13, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

Comment threadairflow/models/serialized_dag.py
Outdated

@jhtimminsjhtimminsApr 13, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.
kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/models/serialized_dag.py
Outdated

kaxil
kaxil
reviewed
Apr 14, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@jedcunningham
jedcunningham
marked this pull request as ready for review
April 15, 2021 19:26
kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadUPDATING.md
Outdated

kaxil
kaxil
reviewed
Apr 15, 2021
View reviewed changes
Comment threadairflow/www/security.py
Outdated

@kaxilkaxil
changed the title
WIP: Sync DAG specific permissions when parsingSync DAG specific permissions when parsingApr 15, 2021
@kaxilkaxil
added this to the Airflow 2.1 milestone
Apr 16, 2021
@kaxil
kaxil
merged commit d52ad87
into
apache:masterApr 19, 2021
@kaxil
kaxil
deleted the

perm_sync_on_parse

branch
April 19, 2021 11:50
kaxil pushed a commit
to astronomer/airflow
that referenced
this pull request
Apr 26, 2021
@jedcunningham@kaxil
Sync DAG specific permissions when parsing (apache#15311)
4bedb7b
This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli `sync-perm`.
With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.
This also fixed a bug where the default webserver DAG specific syncing didn't handle `access_control`.
Closesapache#8609
(cherry picked from commit d52ad87)
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

which airflow version has this fixed change?
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
2.1.0: 
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

2.1.0:
airflow/CHANGELOG.txt

Line 100
in
e3f5eb9
 - Sync DAG specific permissions when parsing (#15311) 
Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
I believe running airflow sync-perm should do it as well.
@chodankarcc
chodankarcc
commented
Jun 30, 2021

Copy link


Copy Markdown

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.
@jedcunningham
jedcunningham
commented
Jun 30, 2021

Copy link


Copy Markdown

MemberAuthor
Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.
airflow/airflow/bin/cli.py

Lines 2075 to 2080
in
5786dcd
print('Updating permission on all DAG views') 
dags=DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values() 
fordagindags: 
appbuilder.sm.sync_perm_for_dag( 
dag.dag_id, 
dag.access_control) 
@chodankarcc
chodankarcc
commented
Jul 1, 2021

Copy link


Copy Markdown

store_serialized_dags
I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer
@ashb
ashb
commented
Jul 1, 2021

Copy link


Copy Markdown

Member
If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.
@ali-hafidz
ali-hafidz
commented
Jul 16, 2021

Copy link


Copy Markdown

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.
@kosteevkosteev
mentioned this pull request
Apr 20, 2022
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kaxilkaxilkaxil approved these changes
@ashbashbAwaiting requested review from ashb
@turbaszekturbaszekAwaiting requested review from turbaszek
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@jhtimminsjhtimminsAwaiting requested review from jhtimmins

Assignees


No one assigned


Labels

area:serialization

Projects


None yet


Milestone

Airflow 2.1

Development

Successfully merging this pull request may close these issues.

DAG's parameter access_control is not refreshing in the UI


6 participants

@jedcunningham@chodankarcc@ashb@ali-hafidz@jhtimmins@kaxil