claimDueSchedule() scales poorly with many schedules

[isimisi]

Hey! We're running about ~950 schedules in production and hit a problem where midnight cron jobs fire hours late. Traced it back to claimDueSchedule().

The current implementation calls SMEMBERS to get all schedule IDs, then loops through them client-side doing one EVAL per ID. Each of those is a Redis round-trip:

SMEMBERS schedules::index → [id1, id2, …, idN]
EVAL CLAIM_SCHEDULE_SCRIPT id1 → nil
EVAL CLAIM_SCHEDULE_SCRIPT id2 → nil
…
EVAL CLAIM_SCHEDULE_SCRIPT idK → claimed!

With ~950 schedules this gets slow fast — #dispatchDueSchedules() calls claimDueSchedule() in a loop until it returns null, so you end up with O(N) round-trips per claim × M claims.

We tested this by loading ~950 schedules into a local Redis instance, all set to become due at the same moment. Before starting the worker, we snapshotted every due schedule's target next_run_at. Then we ran the worker (concurrency 5) and waited for all schedules to be claimed. After that we read each schedule's last_run_at from Redis and compared it to the snapshotted target — giving us the actual drift per schedule and the total wall time from first to last claim.

We ran this twice: once with the current code, and once with a modified version where we replaced the per-ID client-side loop with a single Lua script that does the full SMEMBERS + HGETALL iteration server-side inside Redis — so one EVAL instead of N.

Metric	Before	After (server-side Lua)
Schedules due	927	979
Wall time (first → last claim)	21.8s	3.7s
Drift from target (min)	3.8s	839ms
Drift from target (avg)	12.8s	2.5s
Drift from target (max)	25.6s	4.6s

This is on localhost with no network latency. In production where each round-trip to Redis has real latency, the gap would be wider since the old code does N round-trips per claim vs 2.

The method signature doesn't change at all. Cron nextRunAt recalculation still happens in JS (needs cron-parser), so there's one HSET after the claim, 2 round-trips total instead of N+1.

Happy to open a PR if you're interested.

[Luis-Sejer]

Great find! Hope they fix this

[RomainLanz]

Published as 0.5.2 👌🏻

[isimisi]

Great news! Will check it out. Thanks!

[isimisi]

Hey @RomainLanz, I checked out the 0.5.2 fix.

One thing I noticed while running this in production (~950 schedules): the Lua script still does SMEMBERS to get all schedule IDs and then HGETALL on each one until it finds something due. That's O(N) work inside a single EVAL, which has two consequences

1. Every poll cycle scans all schedules, even when nothing is due. With 950 schedules, that's 950 HGETALL calls server-side before returning nil. Since Lua scripts are atomic, this blocks the Redis event loop for the entire duration. No other commands are served while the script runs.
2. The cost grows linearly with total schedule count, not with the number of due schedules. If you have 10,000 schedules but only 1 is due, the script still iterates through potentially all 10,000 before finding it (or finding nothing).

A possible next step would be adding a sorted set (e.g. schedules::due) scored by next_run_at. The Lua script would use ZRANGEBYSCORE key -inf now LIMIT 0 1 to jump directly to the first due schedule - O(log N) instead of O(N). When nothing is due, ZRANGEBYSCORE returns empty immediately without scanning anything.

The tradeoff is maintaining the ZSET on writes: ZADD in upsert/update, ZREM in delete/pause/exhaustion. These are O(log N) each but only happen on mutations, not every poll cycle.

I've been running a fork with this approach in production and it's been working well.

What do you think?

[RomainLanz]

I am not having this scale so I 100% trust you on this.

Feel free to send a PR with your changes and I will review 👍🏻

[isimisi]

PR opened for you to review: #14

Thanks