Add startup cleanup for dangling documents

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).

[Copilot]

After a crash or redeploy, documents left mid-pipeline stay stuck in a "processing" state forever and their scans can be silently lost. This adds a startup cleanup that fails such documents and preserves their source files.

Changes

• scansynclib/scansynclib/cleanup.py (new): cleanup_dangling_documents() finds scanneddata rows with status_code BETWEEN 0 AND 4 (all in-progress/pending states), then for each:
  • Moves the original (non-OCR) file into the failed-documents dir so it stays recoverable via the web UI, and drops any leftover *_OCR.pdf working file.
  • Marks the row FAILED (status_code = -1) so it surfaces in the failed section. Missing files are tolerated — the row is still failed so nothing dangles.
• detection_service/main.py: invokes cleanup once at the top of main(). Detection is chosen because it's a single-instance container; running this in the replicated OCR/metadata services would clobber documents a sibling replica is actively processing. Import is lazy to avoid triggering sqlite_wrapper's DB init on module import (keeps test_detection side-effect free).
• tests/test_cleanup.py (new): covers move-to-failed, OCR-file removal, missing-file handling, and the empty no-op path.

Notes for reviewers

• No scheduled/periodic cleanup was added, per the issue guidance — this runs deterministically at startup only.
• RabbitMQ queues are durable, so a persisted message for a now-failed document may still be redelivered; since its source file has moved, the downstream service fails it consistently rather than resurrecting it. Worth a look if we want stricter dedup here.

[Copilot]

Pull request overview

Adds a deterministic “startup cleanup” step to prevent documents from remaining stuck in in-progress states after crashes/redeploys by failing those DB rows and preserving/moving their source scans into the failed-documents area (implemented in shared scansynclib, invoked from detection_service).

Changes:

• Introduces scansynclib.cleanup.cleanup_dangling_documents() to find scanneddata rows with status_code BETWEEN 0 AND 4, move source scans to the failed directory, remove leftover *_OCR.pdf working files, and mark rows as failed.
• Hooks the cleanup into detection_service/main.py at the start of main() via a lazy import to avoid module-import DB initialization side effects.
• Adds pytest coverage for move-to-failed, OCR file removal, missing file handling, and the no-op path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
scansynclib/scansynclib/cleanup.py	New startup cleanup logic to move dangling source files to the failed directory, delete leftover OCR working files, and fail dangling DB rows.
detection_service/main.py	Calls the cleanup once at service startup (lazy import) before scanning/publishing new files.
tests/test_cleanup.py	New unit tests validating cleanup behaviors (move, OCR deletion, missing source file, empty pending set).