Skip to content

Relax mandatory requirement for start_date when schedule=None - #35356

Merged
eladkal merged 6 commits into
apache:mainfrom
vishnucoder1:make-start-date-optional
Nov 28, 2023
Merged

Relax mandatory requirement for start_date when schedule=None#35356
eladkal merged 6 commits into
apache:mainfrom
vishnucoder1:make-start-date-optional

Conversation

@vishnucoder1

@vishnucoder1vishnucoder1 commented Nov 1, 2023•
edited by eladkal
Loading

Copy link
Copy Markdown
Contributor

As a part of this PR, existing start_date validation is removed to handle schedule=None

Closes: #35199

eladkal
eladkal reviewed Nov 1, 2023
View reviewed changes
Comment threadtests/models/test_dag.py Outdated

@eladkaleladkalNov 1, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can DagRunType be scheduled ? With scheduled runs we must have start_date

@hussein-awalahussein-awalaNov 1, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can DagRunType be scheduled ? With scheduled runs we must have start_date

IMHO it should be ok if catchup is False 🤔

@eladkaleladkalNov 1, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scheduler creates runs with type scheduled. The fix in this PR is for manual runs that are created by the user.

catchup parameter has no affect on manual runs

@vishnucoder1vishnucoder1Nov 1, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated DagRunType.SCHEDULED to DagRunType.MANUAL

@hussein-awalahussein-awalaNov 1, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I wonder if we can do the same thing for dags with schedule!=None if the catchup is set to False, wdyt?

@eladkaleladkalNov 3, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think so because the first run depends on the start_date. If you set it in the past and interval passed when dag is activated it will create a run. If you set it for a future date then interval is not completed thus no run will be created when dag is set to active.

@hussein-awalahussein-awalaNov 8, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, but we can consider that this condition is always met when the user does not provide a start date.

@eladkaleladkal added this to the Airflow 2.7.4 milestone Nov 3, 2023
@kaxilkaxil added the full tests needed We need to run full set of tests for this PR to merge label Nov 5, 2023
@uranusjr

uranusjr commented Nov 6, 2023

Copy link
Copy Markdown
Member

Do we have a test for a non-None schedule raising an exception?

@vishnucoder1

vishnucoder1 commented Nov 8, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@uranusjr Do we need to fail explicitly for non None Schedule and empty start date? In this case, it will start the job on the next schedule

@vishnucoder1

vishnucoder1 commented Nov 10, 2023

Copy link
Copy Markdown
ContributorAuthor

Also want to mention that previously start date and task date were checked to throw the start date error which is part of add task. But schedule will not be empty then as it will be the default schedule interval value.

@vishnucoder1

vishnucoder1 commented Nov 16, 2023

Copy link
Copy Markdown
ContributorAuthor

@uranusjr@eladkal@hussein-awala
Need your inputs here. Do we need to fail explicitly for non None Schedule and empty start date? Should task date also be considered. But considering both start date and task date and schedule is not possible as schedule takes default interval value when task is added.

@uranusjr

uranusjr commented Nov 17, 2023•
edited
Loading

Copy link
Copy Markdown
Member

If I remember correctly, we currently fail explicitly for that combination, so it’s better to continue the behaviour. Having a DAG created silently but can never fire can be confusing for users.

Task start dates are fine, just consider the DAG date.

@vishnucoder1

vishnucoder1 commented Nov 17, 2023

Copy link
Copy Markdown
ContributorAuthor

Thanks for the input. Changes are added.

@eladkal

eladkal commented Nov 17, 2023

Copy link
Copy Markdown
Contributor

Tests are failing:

ERROR tests/providers/amazon/aws/operators/test_athena.py::TestAthenaOperator::test_return_value - ValueError: DAG is missing the start_date parameter
ERROR tests/providers/amazon/aws/operators/test_datasync.py::TestDataSyncOperatorCreate::test_return_value - ValueError: DAG is missing the start_date parameter
ERROR tests/providers/amazon/aws/operators/test_datasync.py::TestDataSyncOperatorGetTasks::test_return_value - ValueError: DAG is missing the start_date parameter
ERROR tests/providers/amazon/aws/operators/test_datasync.py::TestDataSyncOperatorUpdate::test_return_value - ValueError: DAG is missing the start_date parameter
ERROR tests/providers/amazon/aws/operators/test_datasync.py::TestDataSyncOperator::test_return_value - ValueError: DAG is missing the start_date parameter
ERROR tests/providers/amazon/aws/operators/test_datasync.py::TestDataSyncOperatorDelete::test_return_value - ValueError: DAG is missing the start_date parameter
ERROR tests/providers/amazon/aws/operators/test_dms.py::TestDmsDescribeTasksOperator::test_describe_tasks_return_value - ValueError: DAG is missing the start_date parameter
========== 394 passed, 1861 skipped, 35 warnings, 7 errors in 48.17s ===========

@vishnucoder1

vishnucoder1 commented Nov 18, 2023

Copy link
Copy Markdown
ContributorAuthor

Fixed the failing tests

uranusjr
uranusjr reviewed Nov 22, 2023
View reviewed changes
Comment threadairflow/utils/helpers.py Outdated
@eladkaleladkal added the type:improvement Changelog: Improvements label Nov 25, 2023
@eladkal
eladkal merged commit 930f165 into apache:mainNov 28, 2023
@vishnucoder1
vishnucoder1 deleted the make-start-date-optional branch November 29, 2023 17:34
ephraimbuddy pushed a commit that referenced this pull request Dec 5, 2023
@vishnucoder1@ephraimbuddy
Relax mandatory requirement for start_date when schedule=None (#35356)
bc95360
* Relax mandatory requirement for start_date when schedule=None
* Updated run_type in unit tests
* Added check for empty start_date and non empty schedule
* Fix the build failures
* Fix the build failures
* Update based on review comments
(cherry picked from commit 930f165)
@romsharon98romsharon98 mentioned this pull request Jul 17, 2024
Closed
2 tasks
jedcunningham added a commit to astronomer/airflow that referenced this pull request Jul 19, 2024
@jedcunningham
Fix tasks API endpoint when DAG doesn't have start_date
e52b81b
`start_date` is now optional, if there is no schedule for the DAG (apache#35356). Tasks,
however, inherit the `start_date` of their DAG by default. So it's now
possible for tasks to have no `start_date` as well.
Closes: apache#40828
@jedcunninghamjedcunningham mentioned this pull request Jul 19, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala left review comments
@uranusjruranusjruranusjr approved these changes
@eladkaleladkaleladkal approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@ashbashbAwaiting requested review from ashb

Assignees

No one assigned

Labels

full tests neededWe need to run full set of tests for this PR to mergetype:improvementChangelog: Improvements

Projects

None yet

Milestone

Airflow 2.8.0

Development

Successfully merging this pull request may close these issues.

Relax mandatory requirement for start_date when schedule=None

6 participants

@vishnucoder1@uranusjr@eladkal@hussein-awala@ephraimbuddy@kaxil