Python background queues with an async twist

Use async functions to manage complex background task workflows, and keep using synchronous functions for everything else.

Getting Started

Install queueio:

pip install queueio

Create your routines:

# basic.pyfromtimeimportsleepfromqueueioimportgatherfromqueueioimportpausefromqueueioimportroutine@routine(name="blocking", queue="basic")defblocking():
sleep(0.1) # Regular blocking call@routine(name="yielding", queue="basic")asyncdefyielding(iterations: int):
# Do them two at a timefor_inrange(iterations//2):
awaitgather(blocking(), blocking())
awaitpause(0.2) # Release processing capacityifiterations%2==1:
awaitblocking()

Add the configuration to your pyproject.toml:

[tool.queueio]
# Configure the broker (currently supports amqp:// URIs)broker = "amqp://guest:guest@localhost:5672/"# Register the modules that the worker should load to find your routinesregister = ["basic"]

The broker configuration can be overridden with an environment variable to allow a project to be deployed in multiple environments.

QUEUEIO_BROKER='amqp://guest:guest@localhost:5672/'

Sync the queues to the broker:

queueiosync

Submit the routine to run on a worker:

fromqueueioimportactivatefrombasicimportyieldingwithactivate():
yielding(7).submit()

Then run the worker to process submitted routines:

queueio run basic=4

Monitor the status of active routine invocations:

queueio monitor

Stability

The design of the public API is under active development and is likely to change with any release. Release notes will provide upgrade instructions, but backward compatibility and deprecation warnings will not generally be implemented.