Skip to content

Wake idle workers when jobs are queued - #521

Merged
WaylandYang merged 4 commits into
deeplethe:devfrom
rksharma-owg:codex/wake-jobs-on-notify
Sep 9, 2026
Merged

WaylandYang merged 4 commits into
deeplethe:devfrom
rksharma-owg:codex/wake-jobs-on-notify

Conversation

@rksharma-owg

Copy link
Copy Markdown
Contributor

Closes #517

Summary

  • add a PostgreSQL notification channel for queued jobs
  • wait for either a job notification or the existing polling backstop when the worker is idle
  • emit notifications transactionally for normal, deduplicated, extraction, RSS, and manual requeue paths
  • add a database-backed regression test for the notification path

Validation

  • cargo fmt --all --check
  • UTOPIA_DATABASE_URL=postgres://utopia:utopia@localhost:1517/utopia UTOPIA_TEST_REQUIRE_DB=1 cargo test --workspace
  • cargo clippy --workspace --all-targets -- -D warnings
  • ./node_modules/.bin/vite build

The repository's pnpm build wrapper could not complete because this environment's pnpm policy requires explicit approval for the esbuild install script; the direct Vite build passed.

rksharma-owg and others added 4 commits September 8, 2026 16:30
Publish transactional PostgreSQL notifications for new and immediately requeued jobs so idle workers do not wait for the polling interval.

Closes deeplethe#517

Signed-off-by: RKS <rajesh.sharma@owasp.org>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit 131aefa into deeplethe:dev Sep 9, 2026
4 checks passed
@WaylandYang

Copy link
Copy Markdown
Contributor

Merged, with two changes pushed to your branch first so it could land today:

  • The idle wait sleeps when the listener fails. On a recv error the loop went straight back to claim_one; a database that accepts claims while the listener cannot reconnect (an exhausted pool is one way) would have turned that into a tight loop of one claim and one warning per iteration. The wait is now one function, jobs::wait_for_work, and an error sleeps the poll interval before the next claim.
  • The test exercises what the worker does. The original asserted that a notification arrives. The new file drives wait_for_work itself, which is what the loop calls, and pins three rules: an enqueue ends the wait with Notified inside the poll interval; a row inserted in an open transaction wakes nothing until commit, which is the ordering hazard A queued job waits two seconds for a worker that is doing nothing #517 names; with nothing queued the poll arm still fires. It stays on the wait function rather than run_worker because a live worker on a shared test database would claim other tests' jobs.

The notification sites are exactly as you wrote them, all five. One comment added: the listener keeps a pool connection for good, so the pool has one fewer to hand out. Thanks for the complete coverage; that is what decided between the two PRs for #517.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A queued job waits two seconds for a worker that is doing nothing

2 participants