Add Pagination To List Apis

[rahil-c]

Implemented pagination in list apis based on the spec: #9660
@jackye1995@nastra@danielcweeks@geruh

Testing

• Ran gradle build
• Manual test passed using spark sql and rest catalog

[rahil-c]

@jackye1995@rdblue@danielcweeks@nastra If you can take a look at the revision for this whenever you guys get a chance would be appreciated.

[danielcweeks]

I just want to add a few high-level comments here:

1. It looks like pageSize is currently required, but I believe the default should be that if it is not set, no pagination should be requested and all results are returned. This is going to be the typical behavior for any REST catalog that does not support paging.
2. If a REST server has a specific page limit, it should return that as part of the config route properties as an override. Implementations may have different limits and we should not require users to configure the client to correspond to the limitations of the server.

As a follow up to #2, it seems there may be a chase were some servers require pagination, so they will probably need to return a 400 BadRequest if a client is asking for all values in a request and the server requires they support pagination.

[rahil-c]

I just want to add a few high-level comments here:

1. It looks like pageSize is currently required, but I believe the default should be that if it is not set, no pagination should be requested and all results are returned. This is going to be the typical behavior for any REST catalog that does not support paging.
2. If a REST server has a specific page limit, it should return that as part of the config route properties as an override. Implementations may have different limits and we should not require users to configure the client to correspond to the limitations of the server.

As a follow up to #2, it seems there may be a chase were some servers require pagination, so they will probably need to return a 400 BadRequest if a client is asking for all values in a request and the server requires they support pagination.

@danielcweeks@nastra@rdblue@jackye1995@sachet

1. Will fix this
2. im curious if this page limit is to be returned by the rest server, should this be apart of the capabilities pr? I think from the pr OpenAPI: Express server capabilities via /config endpoint #9940 we did not want to have pagination listed there but wanted to bring this up.

[nastra]

I've left a few minor comments, but this LGTM now. Thanks @rahil-c. I'll wait a bit with merging to also give @danielcweeks a chance to take another look

[rahil-c]

Thanks @nastra for the help I appreciate it, will wait on @danielcweeks review.

[rahil-c]

@danielcweeks@nastra Have made recent revisions, hoping we can land this if no other concerns and if CI run is green.

[danielcweeks]

Approved, pending checks. There's one that failed, but may have been a transient failure.

[rahil-c]

Approved, pending checks. There's one that failed, but may have been a transient failure.

@danielcweeks yea I think it is transient, since every time I go to check this link it still keeps running.

[rahil-c]

@danielcweeks Seems to be green now

[nastra]

Thanks for working on this @rahil-c

[rahil-c]

Thanks @nastra and @danielcweeks for the help on this!