Create CustomJob and Datasets operators for Vertex AI service - #20077

Merged
potiuk merged 20 commits into
apache:mainfrom
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged

Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom
lwyszomi:vertex-ai-operators

Conversation

@MaksYermak

@MaksYermakMaksYermak commented Dec 6, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.

Co-authored-by: Wojciech Januszek januszek@google.com
Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com
Co-authored-by: Maksim Yermakou maksimy@google.com

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborgboring-cyborgBot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Dec 6, 2021
josh-fell
josh-fell reviewed Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py Outdated
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py Outdated
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed the vertex-ai-operators branch 2 times, most recently from 100781f to 3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak commented Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py Outdated
@potiuk

potiuk commented Dec 13, 2021

Copy link
Copy Markdown
Member

Tests are failing - please fix and rebase.

@MaksYermak
MaksYermakforce-pushed the vertex-ai-operators branch 5 times, most recently from 4303c1e to 3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak

MaksYermak commented Jan 10, 2022

Copy link
Copy Markdown
ContributorAuthor

@potiuk could we merge this PR ?

@MaksYermak

MaksYermak commented Jan 13, 2022

Copy link
Copy Markdown
ContributorAuthor

@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?

josh-fell
josh-fell reviewed Jan 13, 2022
View reviewed changes

@josh-felljosh-fell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.

Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py Outdated
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py Outdated
@MaksYermak
MaksYermakforce-pushed the vertex-ai-operators branch 3 times, most recently from 11b1d45 to abc7475CompareJanuary 18, 2022 09:40
@MaksYermak

MaksYermak commented Jan 19, 2022

Copy link
Copy Markdown
ContributorAuthor

@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?

@josh-fell

josh-fell commented Jan 19, 2022

Copy link
Copy Markdown
Contributor

@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?

Just a small comment on a possible clarification but LGTM.

@josh-fell

josh-fell commented Jan 20, 2022

Copy link
Copy Markdown
Contributor

@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?

@ashb

ashb commented Jan 20, 2022

Copy link
Copy Markdown
Member

Yes please!

@MaksYermak

MaksYermak commented Jan 20, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?

@josh-fell I have deleted :type in last commit

@ashb

ashb commented Jan 20, 2022

Copy link
Copy Markdown
Member

@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63

Sorry about that.

@MaksYermak

MaksYermak commented Jan 21, 2022

Copy link
Copy Markdown
ContributorAuthor

@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63

Sorry about that.

@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.

@MaksYermak

MaksYermak commented Jan 28, 2022

Copy link
Copy Markdown
ContributorAuthor

@ashb@josh-fell@potiuk Hi all, could we merge this PR?

potiuk
potiuk approved these changes Jan 28, 2022
View reviewed changes

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.

@github-actions

github-actionsBot commented Jan 28, 2022

Copy link
Copy Markdown
Contributor

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actionsgithub-actionsBot added the full tests needed We need to run full set of tests for this PR to merge label Jan 28, 2022
@potiuk
potiuk merged commit 640c0b6 into apache:mainJan 28, 2022
@jedcunninghamjedcunningham mentioned this pull request Jan 28, 2022
Merged
jedcunningham added a commit to astronomer/airflow that referenced this pull request Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit that referenced this pull request Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham

jedcunningham commented Jan 29, 2022

Copy link
Copy Markdown
Member

This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.

@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!

@potiuk

potiuk commented Jan 29, 2022•
edited
Loading

Copy link
Copy Markdown
Member

Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )

@potiukpotiuk mentioned this pull request Jan 29, 2022
Merged
potiuk added a commit to potiuk/airflow that referenced this pull request Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit that referenced this pull request Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak mentioned this pull request Feb 1, 2022
Merged
@MaksYermak

MaksYermak commented Feb 1, 2022

Copy link
Copy Markdown
ContributorAuthor

@potiuk@jedcunningham I have created a new PR #21253

@jedcunninghamjedcunningham added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Mar 1, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees

No one assigned

Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Create CustomJob and Datasets operators for Vertex AI service - #20077
Merged
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Jan 28, 2022
Merged
Create CustomJob and Datasets operators for Vertex AI service#20077
potiuk merged 20 commits into
apache:mainfrom 
lwyszomi:vertex-ai-operators
Conversation
@MaksYermak
@MaksYermakMaksYermak
commented
Dec 6, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
Create operators for working with Custom Job and Datasets for Vertex AI service. Includes operators, hooks, example dags, tests and docs.
Co-authored-by: Wojciech Januszek januszek@google.com

Co-authored-by: Lukasz Wyszomirski wyszomirski@google.com

Co-authored-by: Maksim Yermakou maksimy@google.com
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in UPDATING.md.
@boring-cyborgboring-cyborgBot
added

area:providers

kind:documentation

provider:google
Google (including GCP) related issues
labels
Dec 6, 2021
josh-fell
josh-fell
reviewed
Dec 7, 2021
View reviewed changes
Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
2 times, most recently
from
100781f to
3787260CompareDecember 9, 2021 10:45
MaksYermak
MaksYermak
commented
Dec 9, 2021
View reviewed changes
Comment threadscripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
Outdated

@potiuk
potiuk
commented
Dec 13, 2021

Copy link


Copy Markdown

Member
Tests are failing - please fix and rebase.
@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
5 times, most recently
from
4303c1e to
3c1b4caCompareDecember 15, 2021 16:24
@MaksYermak
MaksYermak
commented
Jan 10, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk could we merge this PR ?
@MaksYermak
MaksYermak
commented
Jan 13, 2022

Copy link


Copy Markdown

ContributorAuthor
@josh-fell@mik-laj@turbaszek@vikramkoka@ashb@potiuk hi guys what about this PR, could we merge it?
josh-fell
josh-fell
reviewed
Jan 13, 2022
View reviewed changes
@josh-felljosh-fell
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the Custom Job operators had docstrings for params/types so they are exposed in the Airflow documentation.
Comment threadairflow/providers/google/cloud/operators/vertex_ai/custom_job.py
Outdated

Comment threadairflow/providers/google/cloud/example_dags/example_vertex_ai.py
Outdated

@MaksYermak
MaksYermakforce-pushed
the
vertex-ai-operators
branch
3 times, most recently
from
11b1d45 to
abc7475CompareJanuary 18, 2022 09:40
@MaksYermak
MaksYermak
commented
Jan 19, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
@josh-fell
josh-fell
commented
Jan 19, 2022

Copy link


Copy Markdown

Contributor
@potiuk@josh-fell hi guys could you look on my PR one more time and approve/merge if all looks good?
Just a small comment on a possible clarification but LGTM.
@josh-fell
josh-fell
commented
Jan 20, 2022

Copy link


Copy Markdown

Contributor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
Yes please!
@MaksYermak
MaksYermak
commented
Jan 20, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@ashb Should the :type: directives in the docstrings here be removed preemptively because of #20951?
@josh-fell I have deleted :type in last commit
@ashb
ashb
commented
Jan 20, 2022

Copy link


Copy Markdown

Member
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@MaksYermak
MaksYermak
commented
Jan 21, 2022

Copy link


Copy Markdown

ContributorAuthor
@MaksYermak You might have to add some local spelling dictionaries to your hook file too -- check out 1149e63
Sorry about that.
@ashb I have run this command for check ./breeze build-docs -- --spellcheck-only --package-filter apache-airflow-providers-google and I haven´t seen any spell errors in this code.
@MaksYermak
MaksYermak
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@ashb@josh-fell@potiuk Hi all, could we merge this PR?
potiuk
potiuk
approved these changes
Jan 28, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big. But looks like really all things are there. Was it - in big parts auto-generated? It certainly looks like (and we used to have airflow-munchkin for that :)), so I wonder if this is some kind of successor.
@github-actions
github-actionsBot
commented
Jan 28, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jan 28, 2022
@potiuk
potiuk
merged commit 640c0b6
into
apache:mainJan 28, 2022
@jedcunninghamjedcunningham
mentioned this pull request
Jan 28, 2022
 Merged

jedcunningham added a commit
to astronomer/airflow
that referenced
this pull request
Jan 28, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
50aec8b
…apache#20077)"
This reverts commit 640c0b6.
jedcunningham added a commit
that referenced
this pull request
Jan 29, 2022
@jedcunningham
Revert "Create CustomJob and Datasets operators for Vertex AI service (…
1d4b709
…#20077)" (#21203)
This reverts commit 640c0b6.
@jedcunningham
jedcunningham
commented
Jan 29, 2022

Copy link


Copy Markdown

Member
This had a bunch of failing mypy issues once it landed in main. #21203 reverted it.
@MaksYermak, can you rebase your changes on current main, fix the mypy issues, and open a new PR? Thanks!
@potiuk
potiuk
commented
Jan 29, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Yeah. So the up-to-date checker did not prevent it because it was really last run 8 days ago :( . Up-to-date was pretty useless actually (or maybe not :) - this was the ONLY change with this problem. I wonder how many we prevented )
@potiukpotiuk
mentioned this pull request
Jan 29, 2022
 Merged

potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python
de3418a
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the apache#20077 change that was approved before the up-to-date checker
was enabled in apache#21016
potiuk added a commit
that referenced
this pull request
Jan 29, 2022
@potiuk
Remove up-to-date checks for Python (#21208)
c93cb9e
The up-to-date check for Python run for ~week so all the PRs
raised in the last week will need to be rebased to account for
MyPy changes. Hopefully there will be no more PRs from before,
that have not been rebased (we had one serious MyPy problem for
the #20077 change that was approved before the up-to-date checker
was enabled in #21016
@MaksYermakMaksYermak
mentioned this pull request
Feb 1, 2022
 Merged

@MaksYermak
MaksYermak
commented
Feb 1, 2022

Copy link


Copy Markdown

ContributorAuthor
@potiuk@jedcunningham I have created a new PR #21253
@jedcunninghamjedcunningham
added
the 
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Mar 1, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@josh-felljosh-felljosh-fell left review comments
@potiukpotiukpotiuk approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj
@turbaszekturbaszekAwaiting requested review from turbaszek
@vikramkokavikramkokaAwaiting requested review from vikramkoka
@ashbashbAwaiting requested review from ashbashb is a code owner

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@MaksYermak@potiuk@josh-fell@ashb@jedcunningham