Skip to content

feat: make scheduled export cancellation cooperative and resumable - #1197

Merged
greatest0fallt1me merged 1 commit into
CalloraOrg:mainfrom
p70436464-prog:fix-scheduled-export-cancellation
Aug 29, 2026
Merged

feat: make scheduled export cancellation cooperative and resumable#1197
greatest0fallt1me merged 1 commit into
CalloraOrg:mainfrom
p70436464-prog:fix-scheduled-export-cancellation

Conversation

@p70436464-prog

Copy link
Copy Markdown

Closes#1184

PR Description:

Closes #1### Summary
This PR implements cooperative and resumable cancellation for scheduled exports, ensuring that operators have a bounded behavior and a safe recovery path when dependencies or workers fail.
### Changes made1.**Cooperative Cancellation (`AbortSignal` Integration):**- Added an `AbortController` instance to `ScheduledExportsWorker` to manage lifecycle signals.
- Passed the resulting `AbortSignal` down through `ScheduledExportsService.runDueSchedules` and `runSchedule`.
- Updated the `ObjectStorageClient` interface and its mock implementation (`HmacObjectStorageClient`) to accept and respect the `AbortSignal`. - This ensures that if the worker is stopped (or crashes gracefully), any in-flight S3 network I/O or ongoing processing halts cooperatively without lingering orphaned uploads.
2.**Resumable Idempotency (Durable Recovery without Duplicate Work):**- Modified the S3 object key timestamp suffix (`stamp`) for the generated `.csv` and `.json` artifacts inside `runSchedule`. - Previously, it used `now.toISOString()`, which meant a crashed worker would generate new redundant files with new timestamps when it restarted (hidden mutation/duplicate work). - Now, it strictly uses `schedule.nextRunAt.toISOString()`. If the worker restarts and processes the same due schedule, it deterministically overwrites the original, identically-named path. This strictly preserves durable state without duplicate work.
3.**Regression Test Coverage:**- Appended a new regression test suite: `worker handles cancellation cooperatively and preserves idempotency on restart`.
- The test deliberately mocks `ObjectStorageClient.uploadObject` to inject artificial delays, triggers an explicit `worker.stop()` interrupt, validates that the operation halts gracefully via `AbortError`, and then spins the worker back up to assert the exact same S3 destination key (`nextRunAt` stamp) is used on the recovery attempt.
### Acceptance Criteria Addressed-[x] Normal, degraded, and recovery states are explicit and bounded (abort sequences introduced).
-[x] Metrics and diagnostics are actionable, cardinality-bounded, and safe (reused paths, deterministic logging identifiers).
-[x] Recovery preserves durable state without duplicate work or hidden mutation (S3 destination keys bound to logical schedule time).
-[x] Failure-injection tests cover outage, restart, and recovery.

Let me know if you need any adjustments to the description or if you would like me to tackle anything else!

ClosesCalloraOrg#1
- Adds AbortSignal support to ObjectStorageClient and ScheduledExportsWorker to allow cooperative cancellation of in-flight uploads.
- Modifies S3 upload key generation to use the schedule's \
extRunAt\ timestamp instead of the worker tick time, ensuring idempotency across restarts and avoiding duplicate work.
- Adds test coverage for worker cancellation and idempotency on restart.
@drips-wave

Copy link
Copy Markdown

@p70436464-prog Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Quality-2][High] Make scheduled export cancellation cooperative and resumable

2 participants

@p70436464-prog@greatest0fallt1me