Skip to content

[save-images] Add multithreaded version of save-images - #456

Merged
Breakthrough merged 4 commits into
mainfrom
issue-450-save-images
Nov 24, 2024
Merged

[save-images] Add multithreaded version of save-images#456
Breakthrough merged 4 commits into
mainfrom
issue-450-save-images

Conversation

@Breakthrough

@BreakthroughBreakthrough commented Nov 10, 2024•
edited
Loading

Copy link
Copy Markdown
Owner

This improves performance by over 50% in some cases.

Open tasks:

@Breakthrough
Breakthrough marked this pull request as draft November 10, 2024 03:23
@Breakthrough
Breakthroughforce-pushed the main branch 2 times, most recently from 85d1bdd to 2c55a55CompareNovember 13, 2024 03:03
github-advanced-security[bot]
github-advanced-securityAI found potential problems Nov 14, 2024
View reviewed changes
Comment threadscenedetect/__init__.py Fixed
Comment threadscenedetect/__init__.py Fixed
Comment threadscenedetect/scene_manager.py Fixed
github-advanced-security[bot]
github-advanced-securityAI found potential problems Nov 16, 2024
View reviewed changes
Comment threadscenedetect/__init__.py Fixed
Comment threadscenedetect/__init__.py Fixed
@Breakthrough
Breakthroughforce-pushed the issue-450-save-images branch 2 times, most recently from 921f535 to 963a5dcCompareNovember 16, 2024 23:49
github-advanced-security[bot]
github-advanced-securityAI found potential problems Nov 16, 2024
View reviewed changes
Comment threadscenedetect/__init__.py Fixed
github-advanced-security[bot]
github-advanced-securityAI found potential problems Nov 17, 2024
View reviewed changes
Comment threadscenedetect/__init__.py Fixed
Comment threadscenedetect/postprocess/images.py Fixed
Comment threadscenedetect/scene_manager.py Fixed
@Breakthrough
Breakthroughforce-pushed the issue-450-save-images branch 2 times, most recently from bb2bbff to 15e5aabCompareNovember 18, 2024 01:15
github-advanced-security[bot]
github-advanced-securityAI found potential problems Nov 18, 2024
View reviewed changes
Comment threadscenedetect/scene_manager.py
try:
return callable(*args, **kwargs)
# Errors we capture in `error_queue` will be re-raised by this thread.
except: # noqa: E722

Check notice

Code scanning / CodeQL

Except block handles 'BaseException'

Except block directly handles BaseException.
@Breakthrough
[save-images] Add multithreaded version of save-images
67913b9
This improves performance by over 50% in some cases.
This should also fix#450 since we use the Path module for files now instead
of OpenCV's imwrite.
@Breakthrough
[save-images] Add new ImageExtractor class
99eee13
The save_images function was getting quite complex and difficult to maintain,
especially with the multithreaded version. This breaks it out into an object
with smaller functions. The existing save_images function can be implemented
using this new object.
@Breakthrough
[save-images] Make all threads exception-safe
55dec7f
Ensure errors are re-raised safely from worker threads by using non-blocking
puts and monitoring a common error queue.
@BreakthroughBreakthrough modified the milestones: 0.6.6, 0.6.5Nov 24, 2024
@BreakthroughBreakthrough self-assigned this Nov 24, 2024
@Breakthrough
Breakthrough marked this pull request as ready for review November 24, 2024 21:59
@Breakthrough
Breakthrough merged commit 95091f8 into mainNov 24, 2024
@Breakthrough
Breakthrough deleted the issue-450-save-images branch November 24, 2024 22:13
@subbytesubbyte mentioned this pull request Dec 23, 2024
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wjs018wjs018Awaiting requested review from wjs018

Assignees

@BreakthroughBreakthrough

Labels

bugimprovement

Projects

None yet

Milestone

0.6.5

Development

Successfully merging this pull request may close these issues.

2 participants

@Breakthrough@github-advanced-security