feat(tools): add auto-pagination with Redis cache for paginated tool outputs

[abhinavDhulipala]

Summary

Paginate tools in the workflow executor as opposed to having the user explicitly managing cursor state.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• Other: ___________

Testing

Unit testing done. Staging test will involve running the ZenDesk v2 tool on a previously larger than mem operation and observing a passing execution.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

Video of cache ref working e2e:

Screen.Recording.2026-04-04.at.19.23.47.mov

[cursor]

PR Summary

Medium Risk
Adds Redis-backed auto-pagination into the core tool execution path and block output hydration, which can impact workflow correctness/performance and introduces new failure modes when Redis is unavailable or cache entries are missing.

Overview
Adds an auto-pagination + Redis caching layer for tools that return cursor/offset paginated data: tools can now declare a pagination config, and executeTool will fetch subsequent pages, store each page in Redis, and return a lightweight paginated_cache_ref instead of a potentially huge array.

Introduces a new paginated-cache module (RedisPaginatedCache, reference types, hydration + cleanup utilities) with unit tests, and updates block execution to transparently hydrate any cache references back into full arrays before downstream blocks run.

Wires cache lifecycle cleanup into workflow execution shutdown paths (SSE, streaming API, and queued/direct execution) and migrates Zendesk “Get Tickets” to a new zendesk_get_tickets_v2 tool that opts into auto-pagination (UI block now routes get_tickets to v2).

^{Reviewed by Cursor Bugbot for commit 658038b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Apr 5, 2026 2:17am

[greptile-apps]

Greptile Summary

This PR introduces a storage-agnostic auto-pagination framework for paginated tool outputs — pages are flushed individually to Redis as they are fetched, a lightweight PaginatedCacheReference is stored in block state instead of the full dataset, and downstream blocks transparently hydrate from Redis before execution. Zendesk get_tickets is the first tool to opt in, replacing the manual cursor UI with fully automatic multi-page fetching.

The architecture is well-designed (adapter interface, clean key schema, TTL safety-net, scan-based cleanup mirroring the existing base64 cache pattern) and the test coverage is solid. Two issues need attention before merge:

• Stale metadata.total_returned after pagination (paginate.ts): the return value spreads lastOutput, so metadata.total_returned reflects only the last page's item count rather than the cumulative total. After hydration, tickets contains all items but metadata.total_returned remains at the last-page value — downstream blocks that read this field will see incorrect data.
• apiToken visibility violation (get_tickets.ts): the Zendesk API token is declared 'hidden', which prevents users from entering it in the block UI. The project rule requires user-provided credentials to use 'user-only'.

Confidence Score: 4/5

Not safe to merge as-is — stale metadata.total_returned produces incorrect data for downstream blocks and apiToken: 'hidden' likely breaks the get_tickets UI.

Two P1 findings: (1) lastOutput spread causes metadata.total_returned to reflect last-page count only; (2) apiToken uses 'hidden' instead of 'user-only'. Both are straightforward one-line fixes. Infrastructure is solid.

apps/sim/lib/paginated-cache/paginate.ts (stale metadata) and apps/sim/tools/zendesk/get_tickets.ts (apiToken visibility)

Important Files Changed

Filename	Overview
apps/sim/lib/paginated-cache/paginate.ts	Core auto-pagination logic; contains a P1 bug where lastOutput spreads last-page metadata (stale total_returned) and a P2 Date.now() cacheId collision risk.
apps/sim/tools/zendesk/get_tickets.ts	First tool to use auto-pagination; apiToken incorrectly uses 'hidden' visibility instead of the required 'user-only' per project rules, preventing users from entering their token in the UI.
apps/sim/lib/paginated-cache/redis-cache.ts	Redis storage adapter; clean implementation with proper error handling, TTL configuration, and atomic mget for page retrieval.
apps/sim/lib/paginated-cache/types.ts	Type definitions and type guard for paginated cache; well-structured and storage-agnostic.
apps/sim/executor/execution/block-executor.ts	Correctly hydrates cache references before block execution; minor import ordering issue.
apps/sim/app/api/workflows/[id]/execute/route.ts	Adds paginated cache cleanup at workflow end and on timeout; mirrors the existing base64 cache cleanup pattern correctly.
apps/sim/tools/index.ts	Wires auto-pagination after postProcess with correct skipPostProcess guard to prevent recursive pagination.
apps/sim/blocks/blocks/zendesk.ts	Removes manual pageAfter/perPage UI fields for get_tickets since the tool now auto-paginates; correct change.
apps/sim/lib/paginated-cache/paginate.test.ts	Well-structured unit tests for autoPaginate and hydrateCacheReferences using vi.hoisted() correctly; good coverage of error paths.
apps/sim/lib/paginated-cache/redis-cache.test.ts	Thorough tests for RedisPaginatedCache including custom TTL, missing/corrupted page handling, and scan-based delete.
packages/testing/src/mocks/redis.mock.ts	Adds mget and scan stubs to the shared Redis mock, enabling new cache tests without touching existing ones.

Sequence Diagram

sequenceDiagram
participant BE as BlockExecutor
participant TI as tools/index
participant AP as autoPaginate
participant RC as RedisPaginatedCache
participant Redis
participant HY as hydrateCacheReferences
BE->>TI: executeTool(zendesk_get_tickets, params)
TI->>TI: HTTP → page 0 result
TI->>AP: autoPaginate(initialResult, params, config)
AP->>RC: storePage(cacheId, 0, items)
RC->>Redis: SET pagcache:page:{cacheId}:0 PX ttl
loop while nextToken != null and pageIndex < maxPages
AP->>TI: executeTool(toolId, nextParams, skipPostProcess=true)
TI-->>AP: pageResult
AP->>RC: storePage(cacheId, N, items)
RC->>Redis: SET pagcache:page:{cacheId}:N PX ttl
end
AP->>RC: storeMetadata(cacheId, {totalPages, totalItems})
RC->>Redis: SET pagcache:meta:{cacheId} PX ttl
AP-->>TI: output with PaginatedCacheReference
TI-->>BE: finalResult with cache ref
Note over BE: Next block execution
BE->>HY: hydrateCacheReferences(resolvedInputs)
HY->>RC: getAllPages(cacheId, totalPages)
RC->>Redis: MGET pagcache:page:{cacheId}:0..N
Redis-->>RC: page data
RC-->>HY: CachedPage[]
HY-->>BE: inputs with tickets=[...all items]
Note over BE: Workflow end
BE->>Redis: SCAN pagcache:{executionId} then DEL

Loading

Comments Outside Diff (1)

1. apps/sim/tools/zendesk/get_tickets.ts, line 56-59 (link)

   apiToken visibility should be 'user-only', not 'hidden'

   Per the project rule: "API keys and other user-provided credentials should use 'user-only' visibility, not 'hidden'. Only framework-injected tokens (like OAuth tokens) should use 'hidden'."

   The Zendesk API token is a manually provided credential, not an OAuth/framework token. Using 'hidden' means the field is invisible in the block UI, so users cannot actually enter their token — it will always be empty unless injected by some other mechanism.

   Rule Used: API keys and other user-provided credentials shoul... (source)

   Learnt From
   simstudioai/sim#2133

_{Reviews (1): Last reviewed commit: "auto-pagination with Redis cache for pag..." | Re-trigger Greptile}

[vercel]

@abhinavDhulipala is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit dbae5f7. Configure here.}