AIP-84 Add Lists Jobs with Filters API

[jason810496]

Closes: #43660
Related: #43657

Feature

I’ve placed the endpoint in airflow/api_fastapi/core_api/routes/public/job.py, but I’m not certain if this is the most appropriate location—perhaps airflow/api_fastapi/core_api/routes/cli/job.py would also fine ?
The query parameters, response data models, and implementation are consistent with the rest of the public API, utilizing common.parameters and paginated_select.

Tests

The test cases are refactored from the original airflow/tests/cli/commands/test_jobs_command.py, as the CLI should yield the same results when leveraging the API instead of direct DB access.

Question

Hi @bugraoz93, I’m curious about the distinction between this PR’s endpoint (List Jobs with filters) and #43661 (List Jobs), as both endpoints seem to provide job listings. From my perspective, calling this "List Jobs with filters" endpoint without query parameters would yield the same response as the "List Jobs" endpoint.
cc @josix

[bugraoz93]

Hey @jason810496,

That’s a good question, and I’m glad you raised it! @rawwar and I had a similar discussion, and it seems we all have similar concerns.

At first, I thought keeping "List Jobs" and "List Jobs with Filters" as separate endpoints made sense for clarity. But after thinking it through more and taking a closer look, I believe combining them into a single "List Jobs" endpoint with optional query parameters for filters might be a better approach. This way, the endpoint would return all jobs by default, with filtering options (as a QueryParameter) as needed. It would make it both straightforward and easy to maintain.

Thanks for raising this! I am happy to keep the discussion going if others have thoughts on this approach. Otherwise, let’s go ahead and merge it into one endpoint.

cc: @josix@rawwar

[bugraoz93]

I’ve placed the endpoint in airflow/api_fastapi/core_api/routes/public/job.py, but I’m not certain if this is the most appropriate location—perhaps airflow/api_fastapi/core_api/routes/cli/job.py would also fine ?
The query parameters, response data models, and implementation are consistent with the rest of the public API, utilizing common.parameters and paginated_select.

@jason810496 I missed including this part in my previous answer. I responded from my mind but forgot to put it into text. :)

I think it makes more sense to keep the endpoint under Core API. Since most CLI calls will still rely on Core API, creating a separate category for just a few endpoints could add unnecessary complexity. By sticking to Core API, we stay consistent with the rest of the API and follow the same rules across the board. This way, we avoid extra maintenance overhead and keep things simple as we scale.

[pierrejeambrun]

Nice, looking good overall.

A couple of suggestions and we can merge.

[pierrejeambrun]

Tests need fixing, but looking god to me.

[bugraoz93]

Looks good! Thanks for the changes!

[pierrejeambrun]

I think rebasing went wrong on the genrated file part.

When you have any conflicts on those (openapi spec, or front end generated code). You can just re-run manually the hooks to refresh the files:

pre-commit run generate-openapi-spec --all-files
pre-commit run ts-compile-format-lint-ui --all-files

[pierrejeambrun]

Nice one minor thing need to be adjusted, then we can merge.

[jason810496]

Just fix the trailing endpoint naming.
I will work on follow up PR to refactor the test case with manual replace to datetime_zulu_format utility function 👍

[pierrejeambrun]

Well I just tried the conflict resolution via the github interface. (because it was a really small conflict), quite handy for quick and simple conflicts.

[jason810496]

Hi @pierrejeambrun , I have ran the pre-commit locally and didn't not encounter current CI issue, not sure about what cause the Generate the FastAPI API spec error.

[pierrejeambrun]

This is due to the recent pydantic release that breaks some stuff. Main lower bound has been added and should be better, you most likely not have the appropriate pydantic version locally. Now it should be >= 2.10.1(#44284)

[pierrejeambrun]

Most likely the CI image used to run the pre-commit hook (in_container script) still has an old pydantic version.

You can run:

breeze ci-image build --python 3.9

To refresh the image. And you should get the appropriate generated spec locally then.

[pierrejeambrun]

One unrelated provider failure, merging:

 from google.cloud.translate_v3.types import automl_translation
E ImportError: cannot import name 'automl_translation' from 'google.cloud.translate_v3.types' (/usr/local/lib/python3.9/site-packages/google/cloud/translate_v3/types/__init__.py)
----- generated xml file: /files/test_result-providers_google-postgres.xml -----