Skip to content

fix(samples): fix column name in DagModel of Airflow workflow to clean up database - #6532

Merged
gcf-merge-on-green[bot] merged 10 commits into
GoogleCloudPlatform:masterfrom
kosteev:db-clean-up-airflow2-fix
Aug 6, 2021
Merged

fix(samples): fix column name in DagModel of Airflow workflow to clean up database#6532
gcf-merge-on-green[bot] merged 10 commits into
GoogleCloudPlatform:masterfrom
kosteev:db-clean-up-airflow2-fix

Conversation

@kosteev

@kosteevkosteev commented Aug 5, 2021

Copy link
Copy Markdown
Contributor

Description

Fixes#6531

Checklist

@kosteev
kosteev requested review from a team, leahecole and rachael-ds as code ownersAugust 5, 2021 12:31
@google-clagoogle-claBot added the cla: yes This human has signed the Contributor License Agreement. label Aug 5, 2021
@product-auto-labelproduct-auto-labelBot added the samples Issues that are directly related to samples. label Aug 5, 2021
@leahecole

leahecole commented Aug 5, 2021•
edited
Loading

Copy link
Copy Markdown
Collaborator

(edited) - I see in the issue comment this only affects Airflow 2+. Thanks for the fix!

leahecole
leahecole previously approved these changes Aug 5, 2021
View reviewed changes
@leahecoleleahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 5, 2021
@kokoro-teamkokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 5, 2021
@leahecole
leahecole dismissed their stale reviewAugust 5, 2021 16:28

tests not passing - I thought they were

@leahecole

leahecole commented Aug 5, 2021

Copy link
Copy Markdown
Collaborator

Hey @kosteev - I now see tests aren't passing. Are they passing for you locally? (See the Testing Guidelines for instructions on running tests locally)

@kosteev

kosteev commented Aug 5, 2021

Copy link
Copy Markdown
ContributorAuthor

@leahecole I've bumped version of apache-airflow in requirements.txt file.

I wasn't able to run locally tests with nox, but I have installed pypi requirements* manually and ran py.test manually, and airflow_db_cleanup_test has passed successfully.

Can you, please trigger Kokoro. I believe it has to be triggered manually, as I do not see it is executed automatically.

@kosteev
kosteev requested a review from leahecoleAugust 5, 2021 21:06
@dandhleedandhlee added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 6, 2021
@kokoro-teamkokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 6, 2021
@kosteev

kosteev commented Aug 6, 2021

Copy link
Copy Markdown
ContributorAuthor

I see it failed again. Trying actually to run all tests locally and fix.

@kosteev

kosteev commented Aug 6, 2021

Copy link
Copy Markdown
ContributorAuthor

Ok.
I was able to setup local environment and run tests with nox.

I have updated constraints.txt file to have constraints (I didn't realized we have it) to Airflow 2.1.2 constraints (as I bumped it in requirements.txt) and it was the reason for failing tests with new Airflow version. Now hadoop_tutorial_test.test_dag_import passes for me locally.

Please, run Kokoro again.

@kosteev

kosteev commented Aug 6, 2021

Copy link
Copy Markdown
ContributorAuthor

Side question: can I get permissions for running Kokoro CI tests for this specific folder?

@leahecoleleahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 6, 2021
@kokoro-teamkokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 6, 2021
@kosteev

kosteev commented Aug 6, 2021

Copy link
Copy Markdown
ContributorAuthor

New DataflowTemplatedJobStartOperator requires apache.beam extra, I have added it and tests are passing.
PTAL.

leahecole
leahecole approved these changes Aug 6, 2021
View reviewed changes
Comment threadcomposer/workflows/airflow_db_cleanup.py Outdated
@leahecole

leahecole commented Aug 6, 2021•
edited
Loading

Copy link
Copy Markdown
Collaborator

ready for review 2 when you are @dandhlee 😄

dandhlee
dandhlee reviewed Aug 6, 2021
View reviewed changes
Comment threadcomposer/workflows/airflow_db_cleanup.py
}, {
"airflow_db_model": DagModel,
"age_check_column": DagModel.last_scheduler_run,
"age_check_column": DagModel.last_parsed_time, # prior to Airflow 2.0.2 this column was named last_scheduler_run

@dandhleedandhleeAug 6, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for mentioning this!

dandhlee
dandhlee approved these changes Aug 6, 2021
View reviewed changes

@dandhleedandhlee left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@dandhleedandhlee changed the title Fix column name in DagModel of Airflow workflow to clean up databasefix(samples): fix column name in DagModel of Airflow workflow to clean up databaseAug 6, 2021
@dandhleedandhlee added the automerge Merge the pull request once unit tests and other checks pass. label Aug 6, 2021
@gcf-merge-on-green
gcf-merge-on-greenBot merged commit 9fa2a7c into GoogleCloudPlatform:masterAug 6, 2021
@gcf-merge-on-greengcf-merge-on-greenBot removed the automerge Merge the pull request once unit tests and other checks pass. label Aug 6, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leahecoleleahecoleleahecole approved these changes
@dandhleedandhleedandhlee approved these changes
@rachael-dsrachael-dsAwaiting requested review from rachael-ds

Assignees

@dandhleedandhlee

Labels

cla: yesThis human has signed the Contributor License Agreement.samplesIssues that are directly related to samples.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Airflow DB clean up DAG not working with recent versions

4 participants

@kosteev@leahecole@dandhlee@kokoro-team