[OPENLINEAGE] Introduce AirflowJobFacet and AirflowStateRunFacet

[kacpermuda]

closes: #39467

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[kacpermuda]

I had to add some words to spelling wordlist, due to the docs error in CI here. Let me know if there is a better solution to that problem, but part of these words have already been present, just with different capitalization.

[kacpermuda]

I also added json specs for the facets, in a new directory. I was not sure if i should add the "$id" key, as in the OL repo, as these facets will not be uploaded to openlineage.io, so i skipped it.

[kacpermuda]

@potiuk Are the 2.7 compatibility tests run somehow different on main? I keep getting the common.io error in this PR (as it's not available in 2.7) and am pretty sure that's not on me 😉 but the same test looks green on main

[potiuk]

Yes. When you run tests in PR "selective" checks are run, but when you run them in main "all" tests are run.

In this case, there is an inconsistenty between what is excluded and what is included. On one hand selective checks want to run common.io tests - because openlineage has some common.io import or the other way (whenever openlineage provider gets modified, we run tests for all providers that are cross-dependend on openlineage). But also we want to ignore common.io

Look at the command executed:

Starting the tests with those pytest arguments: tests/providers/common/io tests/providers/common/sql tests/providers/dbt/cloud tests/providers/ftp tests/providers/mysql tests/providers/openlineage tests/providers/postgres tests/providers/sftp tests/providers/snowflake tests/providers/trino
--verbosity=0 --strict-markers --durations=100 --maxfail=50 --color=yes --junitxml=/files/test_result-providers_common_io_common_sql-sqlite.xml --timeouts-order moi --setup-timeout=60 --execution-timeout=60 --teardown-timeout=60 --disable-warnings -rfEX --ignore=tests/system --ignore=tests/integration --warning-output-path=/files/warnings-providers_common_io_common_sql-sqlite.txt --ignore=helm_tests --with-db-init --no-cov --ignore=tests/providers/common/io --ignore=tests/providers/fab

From selective checks:

affected-providers-list-as-string = amazon common.io common.sql dbt.cloud ftp google mysql openlineage postgres sftp snowflake trino

From compatibility check we have --ignore=tests/providers/common/io.

Apparently pytest overrides the ignore when we explicitly have tests/providers/common/io on the list of folders to disover tests in. The fix (#40037) is to remove such provider from pytest args when it is also specified as --ignore.

[potiuk]

This is the command that will be used after #40037 is merged and you rebase your PR (tests/providers/common/io is gone as explicit pytest parameter):

tests/providers/common/sql tests/providers/dbt/cloud tests/providers/ftp tests/providers/mysql tests/providers/openlineage tests/providers/postgres tests/providers/sftp tests/providers/snowflake tests/providers/trino --verbosity=0 --strict-markers --durations=100 --maxfail=50 --color=yes --junitxml=/files/test_result-providers_common_io_common_sql-sqlite.xml --timeouts-order moi --setup-timeout=60 --execution-timeout=60 --teardown-timeout=60 --disable-warnings -rfEX --ignore=tests/system --ignore=tests/integration --warning-output-path=/files/warnings-providers_common_io_common_sql-sqlite.txt --ignore=helm_tests --with-db-init --ignore tests/providers/apache/beam --ignore tests/system/providers/apache/beam --ignore tests/integration/providers/apache/beam --ignore tests/providers/papermill --ignore tests/system/providers/papermill --ignore tests/integration/providers/papermill --no-cov
--ignore=tests/providers/common/io --ignore=tests/providers/fab

[potiuk]

Rebased after merging #40037 to see if it helps.

[potiuk]

OK. The original problem is fixed. Now the remaining problem is to add compatibility so that the openlineage tests can be run on Airflow 2.7 as well (following the different cases of how to handle compatibility described in the unit_tests.rst).

[kacpermuda]

I fixed all the compatibility tests and adjusted the code itself so it works on all Airflow 2.7+ versions now without any issues. Imho it's ready to be merged