Skip to content

cleanup: Upgrade Composer samples to Airflow 2.0 - #5782

Merged
leahecole merged 31 commits into
masterfrom
airflow_migration
May 27, 2021
Merged

cleanup: Upgrade Composer samples to Airflow 2.0#5782
leahecole merged 31 commits into
masterfrom
airflow_migration

Conversation

@leahecole

@leahecoleleahecole commented May 3, 2021

Copy link
Copy Markdown
Collaborator

Description

This PR follows best practices for upgrading DAGs from Airflow 1 to Airflow 2. Changes included:

  • Adding an additional conftest.py to the blogs folder. For now, that DAG will stay at using Airflow 1.0, and we need to initialize a compatible DB for those tests separate from the workflows DB.
  • Use of the newly released unit test util in workflows/requirements-test.txt
  • Imports changed from contrib to providers
  • Removal of samples that showcased running Python 2 code
  • Referring to the officially supported Airflow constraints file instead of the piecemeal one I had
  • Making the appropriate changes on anything previously throwing a Deprecation Warning (i.e. bash_operator -> bash)

If you would like access to my Composer environment to see the DAGs deployed, please let me know. Otherwise, here is a screenshot showing successful dag runs across the board

Fixes#5171
DataflowTemplateOperator tutorial update in CL/371792521 - removed instructions for setting gce_region Airflow variable (@sofisl )

CHANGE NOT ADDRESSED - I need to look further at the DAG example we have triggered by GCF. That will be addressed in an upcoming PR.

Note: It's a good idea to open an issue first for discussion.

Checklist

@product-auto-labelproduct-auto-labelBot added the samples Issues that are directly related to samples. label May 3, 2021
@snippet-bot

snippet-botBot commented May 3, 2021

Copy link
Copy Markdown
You are about to delete the following sample browser pages.

Here is the summary of changes.

You are about to add 1 region tag.
You are about to delete 3 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@google-clagoogle-claBot added the cla: yes This human has signed the Contributor License Agreement. label May 3, 2021
@leahecole

Copy link
Copy Markdown
CollaboratorAuthor

YAY #5806 made the tests pass!! This is only red because I'm bad at remembering to run lint locally!!!!!!!!!!!!!

@leahecoleleahecole mentioned this pull request May 19, 2021
8 tasks
@leahecoleleahecole added the status: blocked Resolving the issue is dependent on other work. label May 21, 2021
@leahecoleleahecole mentioned this pull request May 24, 2021
8 tasks
@leahecole
leahecole marked this pull request as ready for review May 24, 2021 21:44
@leahecole
leahecole requested review from a team and rachael-ds as code ownersMay 24, 2021 21:44
@leahecoleleahecole changed the title [WIP] Upgrade Composer samples to Airflow 2.0cleanup: Upgrade Composer samples to Airflow 2.0May 24, 2021
@leahecoleleahecole removed the status: blocked Resolving the issue is dependent on other work. label May 24, 2021
@leahecole
leahecole requested a review from sofislMay 24, 2021 21:58
@sofisl

Copy link
Copy Markdown
Contributor

Just looked at the dataflow/dataproc tutorials, LGTM!

Comment threadcomposer/conftest.py
Comment threadcomposer/workflows/bq_notify.py Outdated
Comment threadcomposer/workflows/bq_copy_eu_to_us_sample.csv Outdated
Comment threadcomposer/workflows/bq_notify.py Outdated
Comment threadcomposer/workflows/hadoop_tutorial.py
Comment threadcomposer/workflows/connections.py

@rachael-dsrachael-ds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@leahecole

Copy link
Copy Markdown
CollaboratorAuthor

omg I have no idea how I made this mess with adding these requirements files. I'll fix later today 🤦🏻‍♀️

@leahecole
leahecoleforce-pushed the airflow_migration branch 2 times, most recently from affa66f to 207371fCompareMay 27, 2021 16:18
@leahecoleleahecole added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label May 27, 2021
@leahecole

Copy link
Copy Markdown
CollaboratorAuthor

I have been talking to TWs and need to move the Airflow 1 versions to a "legacy samples" folder

@leahecoleleahecole removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label May 27, 2021
@leahecole

Copy link
Copy Markdown
CollaboratorAuthor

Actually, I'll do the legacy samples thing in a separate PR. I'm ready for you, @dinagraves ! :)

Comment threadcomposer/workflows/bq_notify.py
Comment threadcomposer/workflows/unit_testing_cycle.py
@leahecole
leahecole merged commit 675e55f into masterMay 27, 2021
@leahecole
leahecole deleted the airflow_migration branch May 27, 2021 23:16
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yesThis human has signed the Contributor License Agreement.samplesIssues that are directly related to samples.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update composer workflow samples

4 participants

@leahecole@sofisl@rachael-ds@dinagraves