Skip to content

openlineage: defensively check for provided datetimes in listener - #33343

Merged
mobuchowski merged 1 commit into
mainfrom
openlineage_fix_getting_dates_from_tis
Aug 13, 2023
Merged

openlineage: defensively check for provided datetimes in listener#33343
mobuchowski merged 1 commit into
mainfrom
openlineage_fix_getting_dates_from_tis

Conversation

@mobuchowski

Copy link
Copy Markdown
Contributor

This fixes following bug:

[2023-08-11, 18:27:35 UTC] {trigger_dagrun.py:209} INFO - Waiting for trigger_target_dag on 2023-08-11T18:23:35.355250+00:00 to become allowed state ['success'] ...
[2023-08-11, 18:28:35 UTC] {trigger_dagrun.py:209} INFO - Waiting for trigger_target_dag on 2023-08-11T18:23:35.355250+00:00 to become allowed state ['success'] ...
[2023-08-11, 18:29:35 UTC] {taskinstance.py:1943} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/airflow/operators/trigger_dagrun.py", line 220, in execute
raise AirflowException(f"{self.trigger_dag_id} failed with failed states {state}")
airflow.exceptions.AirflowException: trigger_target_dag failed with failed states failed
[2023-08-11, 18:29:35 UTC] {utils.py:401} ERROR - 'NoneType' object has no attribute 'isoformat'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 399, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/listener.py", line 145, in on_failure
end_time=task_instance.end_date.isoformat(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'isoformat'

Additionally, add fallback for any similar situation, where it's getting datetime from Airflow object, checking if it's not null first.

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski
mobuchowskiforce-pushed the openlineage_fix_getting_dates_from_tis branch from 88e60b2 to d0cf447CompareAugust 12, 2023 16:49
@mobuchowski
mobuchowski merged commit 23d5076 into mainAug 13, 2023
@kaxil
kaxil deleted the openlineage_fix_getting_dates_from_tis branch August 14, 2023 16:40
ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 17, 2023
…ache#33343)
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@mobuchowski@ephraimbuddy