Uh oh!
There was an error while loading. Please reload this page.
Add redis backend support using redis streams - #14
Conversation
4d662ab to
8091abdCompare55efa05 to
87406d6Compare87406d6 to
03232f1CompareCodecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #14 +/- ##
==========================================
+ Coverage 96.48% 97.13% +0.65%
==========================================
Files 27 28 +1 Lines 1508 1643 +135 ==========================================
+ Hits 1455 1596 +141 + Misses 53 47 -6 ☔ View full report in Codecov by Sentry. |
034c386 to
99eb021Compare| tags: hedwig-python-${{ matrix.python_version }}:latest | ||
| - name: Build docker compose image | ||
| run: make build |
There was a problem hiding this comment.
added make build (docker compose build ) as a separate step, because docker compose run doesn't allow to provide build args and it always builds docker image for default SC_PYTHON_VERSION=3.13. During make build docker SC_PYTHON_VERSION arg value is taken from SC_PYTHON_VERSION env var. Docker build image operation takes only few seconds since it uses all the layers from the cache, which is prepared in previous Build and push Docker image step.
| release: release_setup | ||
| @docker compose run --rm app ./scripts/release.sh | ||
| @docker compose run --rm \ |
There was a problem hiding this comment.
I wonder if we should run release script outside docker container to avoid passing sensitive data via env vars.
There was a problem hiding this comment.
Yeah not a bad idea. It's a specialised process only run on a few people's laptops.
| def post_process_hook_kwargs(queue_message) -> dict: | ||
| return {} | ||
| @abc.abstractmethod |
There was a problem hiding this comment.
assuming we want to require each subclass to implement all abstract methods, the base classes needs to inherit from abc.ABC or have abc.ABCMeta as a meta class for the abstractmethod decorator to raise an error on instantiation of subclasses that don't implement the method(s).
There was a problem hiding this comment.
good point, thanks
33fd3b1 to
defe3aeCompare
Add redis backend support using redis streams.
I extended the original pr: #11 and applied few minor changes:
block=500param toxreadgroup()method to block read operation for 500ms if there is no messages available in the stream to avoid making too many I/O operations in the consumer read loop. Without blocking, the network transfer between consumer and redis server is about 2-3Mbps all the time. Consumer instance cpu usage is also increased.fakeredis(https://pypi.org/project/fakeredis/), but it's not fully compatible with actual redis responses (ex:xreadgroup()- redis returns dict, fakeredis returns list of tuples,xpending()result doesn't includetimes_deliveredin fakeredis implementation).