Add pagination to open api spec for listing of namespaces, tables, views by rahil-c · Pull Request #9660 · apache/iceberg

rahil-c · 2024-02-05T20:00:25Z

Dev List discussion thread around adding support for pagination in list namespaces, tables, and views: https://lists.apache.org/thread/lql05h02qtp8mgq74ovhb0ndd76ck4f3

Credit to @emkornfield for creating this google doc for listing all cases which this change is based off: https://docs.google.com/document/d/1bbfoLssY1szCO_Hm3_93ZcN0UAMpf7kjmpwHQngqQJ0/edit

cc @jackye1995 @rdblue @danielcweeks

Testing

Ran make install, make lint, and python3 rest-catalog-open-api.py without issues.`

jackye1995

I think we are missing the ListViews API?

jackye1995

looks good to me!

nastra · 2024-02-22T10:09:04Z

+ Clients will initiate the first paginated request by sending an empty `pageToken` e.g. `GET /tables?pageToken` or `GET /tables?pageToken=`
+ signaling to the service that the response should be paginated.
+
+ For servers that support pagination, they will recognize `pageToken` and return a `next-page-token` in response if there are more results available.


I still think the wording can be improved in this section to avoid using they to refer to servers. I did a previous suggestion which avoids using they

additionally it's confusing to use pageToken and next-page-token (vs nextPageToken), so which one is correct here?

I still think the wording can be improved in this section to avoid using they to refer to servers. I did a previous suggestion which avoids using they
+1
additionally it's confusing to use pageToken and next-page-token (vs nextPageToken), so which one is correct here?
I think we probably have to use next-page-token instead of nextPageToken, because it is the actual field name.

+1 for using next-page-token, @nastra removed usage of they, let me know if everything looks good and we can land this?

rahil-c · 2024-02-27T17:14:11Z

@jackye1995 can we merge this?

jackye1995 · 2024-02-27T18:39:56Z

Looks like it's still pending review from @danielcweeks? Is there any further comments?

danielcweeks

LGTM, Thanks @rahil-c !

…ews (apache#9660) * Add pagination to open api spec for listing of namespaces, tables, views * Rev for pagination in open api spec * Fix writing in pagination descriptions * Use opaque token * Rev on pagination pr writing * Rev on pagination pr writing py file * Rev on description for pageSize * revise pageToken description * minor rev and make int for pageSize * minor rev and remove emtpy val true for pageSize * use only PageToken instead of NextPageToken * address jack and nastra comments * address final comments * remove they and use min 1 for pageSize --------- Co-authored-by: Rahil Chertara <rchertar@amazon.com>

rdblue · 2024-03-03T18:16:17Z

+
+ Servers that support pagination will recognize `pageToken` and return a `next-page-token` in response if there are more results available.
+ After the initial request, it is expected that the value of `next-page-token` from the last response is used in the subsequent request.
+ Servers that do not support pagination will ignore `next-page-token` and return all results.


Servers that do not support pagination will ignore pageToken right? This should be the query param not the response field.

Will fix this.

rdblue · 2024-03-03T18:17:25Z

+ PageToken:
+ description:
+ An opaque token which allows clients to make use of pagination for a list API (e.g. ListTables). 
+ Clients will initiate the first paginated request by sending an empty `pageToken` e.g. `GET /tables?pageToken` or `GET /tables?pageToken=`


@rahil-c, in a spec you always want to be specific, not give options. Options make the spec more complicated. Here, only the second form should be added. The first form with no = is not recommended.

Trying to remember what was decided during the community sync discussion, https://www.youtube.com/watch?v=uAQVGd5zV4I, starting at 45:04, seems like you said we should support both there. But maybe it's a misunderstanding of what Rahil meant there when he was talking about the first option of value not present.

@rdblue @jackye1995 Yea I think based on the community sync I thought we were suppose to be flexible for what users can do hence I included the two options in the description.
I can remove the first form if needed.

…ews (apache#9660) * Add pagination to open api spec for listing of namespaces, tables, views * Rev for pagination in open api spec * Fix writing in pagination descriptions * Use opaque token * Rev on pagination pr writing * Rev on pagination pr writing py file * Rev on description for pageSize * revise pageToken description * minor rev and make int for pageSize * minor rev and remove emtpy val true for pageSize * use only PageToken instead of NextPageToken * address jack and nastra comments * address final comments * remove they and use min 1 for pageSize --------- Co-authored-by: Rahil Chertara <rchertar@amazon.com>

Add pagination to open api spec for listing of namespaces, tables, views
231b4dd

github-actionsBot added the OPENAPI label Feb 5, 2024

jackye1995 reviewed Feb 6, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 6, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 6, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 6, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 6, 2024
View reviewed changes

danielcweeks reviewed Feb 6, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

geruh reviewed Feb 7, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

Rev for pagination in open api spec
a3dc720

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Feb 9, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

Rahil Chertara added 2 commits February 11, 2024 16:06

Fix writing in pagination descriptions
a1e20ad

Use opaque token
4128211

jackye1995 reviewed Feb 12, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 12, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 12, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 12, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 12, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 12, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.py Outdated

Rahil Chertara added 2 commits February 13, 2024 11:30

Rev on pagination pr writing
25656b2

Rev on pagination pr writing py file
e0307a1

jackye1995 reviewed Feb 13, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

Rev on description for pageSize
c78f251

jackye1995 reviewed Feb 14, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

nastra reviewed Feb 20, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

nastra reviewed Feb 20, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

address jack and nastra comments
eed29cc

jackye1995 reviewed Feb 21, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 21, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

jackye1995 reviewed Feb 21, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml Outdated

address final comments
918a1ad

jackye1995 approved these changes Feb 21, 2024
View reviewed changes

jackye1995 requested review from danielcweeks, geruh and nastra February 21, 2024 18:26

geruh reviewed Feb 21, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml

geruh approved these changes Feb 22, 2024
View reviewed changes

nastra reviewed Feb 22, 2024
View reviewed changes

rahil-c mentioned this pull request Feb 22, 2024
Add Pagination To List Apis #9782
Merged

geruh reviewed Feb 22, 2024
View reviewed changes

Comment threadopen-api/rest-catalog-open-api.yaml

remove they and use min 1 for pageSize
ef4af36

nastra approved these changes Feb 26, 2024
View reviewed changes

danielcweeks approved these changes Feb 29, 2024
View reviewed changes

danielcweeks merged commit acbf96f into apache:mainFeb 29, 2024

rdblue reviewed Mar 3, 2024
View reviewed changes

rahil-c mentioned this pull request Mar 5, 2024
Fix pagination spec description #9866
Closed

jayceslesar mentioned this pull request Jun 11, 2025
rest catalog does not support pagination apache/iceberg-python#2084
Open
3 tasks

Uh oh!

Conversation

rahil-c commented Feb 5, 2024

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nastraFeb 22, 2024

Choose a reason for hiding this comment

Uh oh!

nastraFeb 22, 2024

Choose a reason for hiding this comment

Uh oh!

jackye1995Feb 23, 2024

Choose a reason for hiding this comment

Uh oh!

rahil-cFeb 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rahil-c commented Feb 27, 2024

Uh oh!

jackye1995 commented Feb 27, 2024

Uh oh!

danielcweeks left a comment

Choose a reason for hiding this comment

Uh oh!

rdblueMar 3, 2024

Choose a reason for hiding this comment

Uh oh!

rahil-cMar 4, 2024

Choose a reason for hiding this comment

Uh oh!

rdblueMar 3, 2024

Choose a reason for hiding this comment

Uh oh!

jackye1995Mar 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahil-cMar 4, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

rahil-cFeb 25, 2024•
edited
Loading

jackye1995Mar 4, 2024•
edited
Loading