Uh oh!
There was an error while loading. Please reload this page.
Add OpenLineage support to AthenaSQLHook - #66844
Conversation
Signed-off-by: Rahul Madan <madan.rahul9@gmail.com>
Uh oh!
There was an error while loading. Please reload this page.
kacpermuda
left a comment
There was a problem hiding this comment.
Looks good, let's maybe add some tests, even static, that would check the expected output, just to make sure the region extraction works, the columns do not change by accident etc.
rahul-madaan
commented
May 14, 2026
Skipped them initially to match convention — TrinoHook, PostgresHook, MySQLHook and MsSqlHook all have these same get_openlineage_* methods without dedicated unit tests. Adding static tests now per your request. |
Signed-off-by: Rahul Madan <madan.rahul9@gmail.com>
a9b4cad to
37a9f7aComparekacpermuda
commented
May 14, 2026
lgtm, will leave for @mobuchowski to merge. Thanks ! |
mobuchowski
left a comment
There was a problem hiding this comment.
Two small changes, looks good otherwise.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Signed-off-by: Rahul Madan <madan.rahul9@gmail.com>
Uh oh!
There was an error while loading. Please reload this page.
AthenaSQLHook (PyAthena-based SQL hook for Amazon Athena) had no OpenLineage
methods, so SQL executed through it via SQLExecuteQueryOperator or the common
SQL machinery did not emit input/output dataset lineage.
This adds the three standard OpenLineage hook methods, following the pattern
used by TrinoHook and PostgresHook:
get_openlineage_database_info— returns DatabaseInfo withscheme="awsathena"andauthority="athena.{region}.amazonaws.com",matching the namespace already emitted by AthenaOperator
(
operators/athena.py:get_openlineage_dataset). The default catalog isAwsDataCatalog, overridable via the connection'sextra.catalog.get_openlineage_database_dialect— returns"trino". Athena's SQL enginev3 is Trino-based, and OpenLineage has no separate "presto" dialect parser.
get_openlineage_default_schema— returns the connection's schema, or"default"as Athena's fallback database.No new unit tests added — this matches the convention used by TrinoHook,
PostgresHook, MySQLHook and MsSqlHook, none of which have dedicated unit
tests for these trivial wrapper methods. Existing AthenaSQLHook tests
continue to pass.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.7 following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.