Skip to content

Narrow a list response with fields, limit and page - #22

Open
pallaoro wants to merge 1 commit into
mainfrom
list-pagination
Open

pallaoro wants to merge 1 commit into
mainfrom
list-pagination

Conversation

@pallaoro

@pallaoro pallaoro commented Sep 2, 2026

Copy link
Copy Markdown
Member

Why

A listed entry carries every field it has, richtext bodies included. So fetching a library to render an index page, or to let an agent pick an entry to edit, transfers the entire corpus to read a column of titles.

Measured on a live library: 28 entries, 505,457 bytes, 18KB apiece. The same list as ?fields=title,slug is 3,380 bytes, a 149x difference. It grows linearly with the collection, and the list routes had no way to ask for less. GET /api/entries/:plural was SELECT * over the whole table.

What

Three optional query params on both list routes, the public /api/entries/:plural and the editor's /api/admin/entries/:plural:

Param Description
fields Comma-separated columns, e.g. title,slug,status. id always rides along so entries stay addressable
limit Page size, 1 to 500
page 1-based page number. Implies limit=25 when limit is absent
curl "$BASE/api/entries/posts?fields=title,slug,post_date&limit=25"

fields works on the single-entry routes as well. Accepting a param on the list and ignoring it one route over is the quietest way for an API to lie about what it takes.

When limit or page is present the response carries X-Total-Count with the unpaged total, added to the CORS expose list so a cross-origin reader can actually read it. Paging itself never depends on that header. A short page means the end.

Why these names, and why not the rest of the convention

page and its default page size match the shared paging helper the sibling templates already use, so the names mean one thing across the family.

Only the names transfer. Not one of those list routes is declared public: they are internal APIs behind an app's own UI, which is what lets them default to a page and return a {items, total, page} envelope. This route is declared public in clawnify.json and read by outside sites and sync tools, where a default cap silently truncates an index page and an envelope turns every consumer's .map() into a TypeError. Same-looking route, different contract class.

Compatibility

All three are opt-in, and a request that sends none of them gets byte-for-byte the response it got before.

The README now states what that costs instead of implying it is free. Returning everything is what a sync tool wiring up a whole collection wants, and what an index page does not. A library is one unbounded read in that mode, so past a few thousand entries a list call should always carry fields or page.

Safety

fields is validated against the content type's own attributes plus the platform columns, so an unknown name is a 400 with the known field list rather than a SQL error. notes is deliberately not selectable: ?fields=notes is refused rather than silently empty, and the author's brief stays on /api/notes/*, off the public route. On a single entry, status is still read when a content type gates drafts and dropped again unless asked for, so selecting fields cannot expose a draft.

Everything is in the OpenAPI spec (params, the X-Total-Count header, the 400), which is what an agent introspects before calling.

Verified

Exercised end to end against the real Hono app on SQLite. 42 assertions, all passing:

  • no params returns the unchanged response, drafts still hidden, notes still stripped, no header added
  • ?fields=title,slug on 6 entries: 21,231 bytes to 211 bytes
  • a full paged walk covers every entry with no overlap, last page is short, page without limit uses the default of 25
  • 400 on non-numeric, fractional, zero and over-cap limit, on page=0 and non-numeric page, on empty fields, unknown field, and fields=notes
  • public paging stays live-only and its total ignores drafts, admin paging sees drafts and counts them
  • selecting fields on a single entry returns only those fields, still 404s a draft, and does not leak status
  • the params, the header and the 400 all appear in /api/openapi.json

Client builds. tsc --noEmit is clean apart from the pre-existing routes-uploads.ts error.

Also

The README's endpoint table still documented the /api/posts routes that /api/entries/:plural replaced. Corrected, and the new params documented there and in the in-app API dialog.

A listed entry carries every field it has, richtext bodies included. So
fetching a library to render an index, or to let an agent pick an entry to
edit, transfers the whole corpus to read a column of titles. Measured on a
live library: 28 entries, 505,457 bytes, 18KB apiece. The same list as
`?fields=title,slug` is 3,380 bytes.

Both list routes now take three optional query params:

  fields  comma-separated columns. `id` always rides along
  limit   page size, 1-500
  page    1-based, implies limit=25 when limit is absent

`fields` works on the single-entry routes too. Accepting it on the list and
ignoring it one route over is the quietest way for an API to lie about what
it takes. `status` is still read when a content type gates drafts, and
dropped again unless it was asked for, so selecting fields cannot expose a
draft.

`page` and its default page size match the shared paging helper the sibling
templates already use, so the names mean one thing across the family. Only
the names transfer. Those routes are internal APIs behind an app's own UI,
so they can default to a page and return a `{items, total, page}` envelope.
This route is declared public and read by outside sites and sync tools,
where a default cap silently truncates an index and an envelope turns every
consumer's `.map()` into a TypeError.

So all three params are opt-in: a request sending none of them gets exactly
the response it got before. The README now states what that costs instead of
implying it is free. A library is one unbounded read in that mode, so past a
few thousand entries a list call should always carry `fields` or `page`.

When limit or page is present the response carries `X-Total-Count` with the
unpaged total, exposed through CORS so a cross-origin reader can see it.
Paging never depends on it. A short page means the end.

`fields` validates against the content type's own attributes, so an unknown
name is a 400 rather than a SQL error, and `notes` is not selectable. The
author's brief stays on /api/notes/*.

Also corrects the README's endpoint table, which still described the
`/api/posts` routes these replaced.
@pallaoro pallaoro changed the title Narrow a list response with fields, limit and offset Narrow a list response with fields, limit and page Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant