Remove offset-based pagination from list_jobs function in DatabricksHook

[oleksiidav]

Databricks has deprecated the offset parameter from ListJobs API, and to soften and control the inevitable breaking change for Airflow users, as well as to conceal the implementation details we are removing the offset-based pagination from all connectors.

This is a follow-up to #33472. Airflow users passing offsets to list_jobs should be receiving deprecation warnings already starting from this release.

[oleksiidav]

@alexott could you please take a look and assign a maintainer to review if you can?

[eladkal]

This is a follow-up to #33472. Airflow users passing offsets to list_jobs have been receiving deprecation warnings for the last month.

#33472 have not been released yet so no have been warned

[oleksiidav]

#33472 have not been released yet so no have been warned

I see @eladkal ! In any case, there is at least a month of time between that PR and this one. So this change can surely go into the following release after this one, assuming monthly release cadence

[Lee-W]

Do we need to do something like #34738 (comment) in this PR as well? Looks like a breaking change.

[uranusjr]

Yes we do. Or to not make this a breaking change, we can continue to accept the argument but just ignore it with a warning. I would prefer dropping it entirely though.

[oleksiidav]

Thanks @uranusjr, I have added the breaking change note the changelog

[oleksiidav]

Hi @eladkal, could you ptal? This is ready to be merged

[eladkal]

Please also add 5.0.0 entry to versions in provider.yaml:

airflow/airflow/providers/databricks/provider.yaml

Lines 25 to 28 in e9987d5

	versions:
	- 4.6.0
	- 4.5.0
	- 4.4.0

[oleksiidav]

Thanks @eladkal ! Applied changes

[harishkesavarao]

Looks good, agree with removing the offset altogether.

[oleksiidav]

I have spent some time but I'm really not sure where the bad revision error is coming from in provider docs GH workflow. I suspect it's the different commit introducing v4.7.0 in between before I added 5.0.0. @eladkal could I ask you to take over and merge? Or at least provide advice how to solve this?

[potiuk]

You need to wait until 4.7.0 is released. It is being voted currently - but not yet released and the error you see is when documentation is generated to show the difference between released 4.7.0 and your 5.0.0 - but since 4.7.0 is not yet released (and tag not set) this is impossible to generate the difference from 4.7.0.

Just wait until 4.7.0 is released (will be announced on the devlist - likely tomorrow) and rebase. It should get fixed.

See https://lists.apache.org/thread/mpbkbdwftq7bmfx5vdnd9tz36xmq8rwc

[oleksiidav]

Ah, thank you very much @potiuk! Will wait for a few days

[oleksiidav]

Hi @potiuk@eladkal, let's merge!