Executor

.github/agents/superjoe.agent.md

@@ -11,6 +11,7 @@ description: CodingJoe's digital clone following his coding guidelines and best
 ## Planning
 
 You MUST ALWAYS follow the `naming-things` guidelines. Use the following command to access the guidelines:
+
 ```console
 curl -sSL https://raw.githubusercontent.com/codingjoe/naming-things/refs/heads/main/README.md | cat
 ```
@@ -44,7 +45,6 @@ Avoid functions or other code inside functions.
 Avoid if-statements in favor of switch/match-statements or polymorphism.
 Do not assign names to objects which are returned in the next line.
 
-
 ## Python
 
 Follow PEP 8 guidelines for code style.

.github/copilot-instructions.md

@@ -5,3 +5,5 @@ Use the following command to access the guidelines:
 ```console
 curl -sSL https://raw.githubusercontent.com/codingjoe/naming-things/refs/heads/main/README.md | cat
 ```
+
+Read the CONTRIBUTING.md file for more information.

.github/workflows/ci.yml

@@ -27,10 +27,6 @@ jobs:
  - "3.14"
  django-version:
  - "6.0"
- extras:
- - "" # We try a run without any extras
- - "--extra sentry"
- - "--extra redis"
  services:
  redis:
  image: redis
@@ -45,7 +41,7 @@ jobs:
  - uses: astral-sh/setup-uv@v7
  with:
  python-version: ${{ matrix.python-version }}
- - run: uv run ${{ matrix.extras }} --with django~=${{ matrix.django-version }}.0 pytest
+ - run: uv run --with django~=${{ matrix.django-version }}.0 pytest -m "not benchmark"
  - uses: codecov/codecov-action@v6
  with:
  token: ${{ secrets.CODECOV_TOKEN }}
@@ -68,7 +64,7 @@ jobs:
  - uses: astral-sh/setup-uv@v7
  with:
  python-version: ${{ matrix.python-version }}
- - run: uv run --with django~=${{ matrix.django-version }}.0 pytest
  - uses: codecov/codecov-action@v6
  with:
  token: ${{ secrets.CODECOV_TOKEN }}
+ - run: uv run --with django~=${{ matrix.django-version }}.0 pytest -m "not benchmark"

.github/workflows/release.yml

@@ -3,26 +3,21 @@ on:
  release:
  types: [published]
  workflow_dispatch:
-permissions:
- id-token: write
 jobs:
-release-build:
+pypi-build:
  runs-on: ubuntu-latest
  steps:
  - uses: actions/checkout@v6
- - uses: actions/setup-python@v6
- with:
- python-version: "3.x"
- - run: python -m pip install --upgrade pip build wheel
- - run: python -m build --sdist --wheel
+ - uses: astral-sh/setup-uv@v7
+ - run: uvx --from build pyproject-build --sdist --wheel
  - uses: actions/upload-artifact@v7
  with:
  name: release-dists
  path: dist/
  pypi-publish:
  runs-on: ubuntu-latest
  needs:
- - release-build
+ - pypi-build
  permissions:
  id-token: write
  steps:

.gitignore

@@ -166,7 +166,10 @@ cython_debug/
 
 # Packaging
 
-grinder/_version.py
+threadmill/_version.py
 
 # uv
 uv.lock
+
+# pytest-benchmark
+.benchmarks/

.pre-commit-config.yaml

@@ -33,6 +33,7 @@ repos:
  - mdformat-footnote
  - mdformat-gfm
  - mdformat-gfm-alerts
+ exclude: '.github/agents/'
  - repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.15.11
  hooks:

CONTRIBUTING.md

@@ -0,0 +1,50 @@
+# Contributing
+
+When writing code, you MUST ALWAYS follow the [naming-things](https://github.com/codingjoe/naming-things/blob/main/README.md) guidelines.
+
+<!--
+Bots/Agents MUST use:
+curl -sSL https://raw.githubusercontent.com/codingjoe/naming-things/refs/heads/main/README.md
+-->
+
+## Design Principles
+
+- Consistency – We never lose data, even if someone unplugs the power or network.
+- Durability – We recover from any failures, even poorly written tasks.
+- Utilization – We keep the CPU saturated with tasks, not with idle time or waiting for locks.
+
+## Testing
+
+We have unit tests, integration tests, and benchmarks. Avoid mocking if possible.
+
+To run the tests, use the following command:
+
+```bash
+uv run pytest
+```
+
+To run only integration tests:
+
+```bash
+uv run pytest -m integration
+```
+
+To run only integration benchmarks:
+
+```bash
+uv run pytest -m "integration and benchmark"
+```
+
+Benchmarking snapshots are created automatically.
+To compare your feature branch against the main branch,
+run the test suite on main, followed by:
+
+```
+uv run pytest --benchmark-compare
+```
+
+Before your first commit, ensure that the pre-commit hooks are installed by running:
+
+```bash
+uvx prek install
+```

README.md

@@ -1,51 +1,137 @@
-# Django Grinder
+# Threadmill
 
 <p align="center">
  <picture>
- <source media="(prefers-color-scheme: dark)" srcset="https://github.com/codingjoe/django-grinder/raw/main/images/logo-dark.svg">
- <source media="(prefers-color-scheme: light)" srcset="https://github.com/codingjoe/django-grinder/raw/main/images/logo-light.svg">
- <img alt="Django Grinder: A queue agnostic worker for Django's task framework." src="https://github.com/codingjoe/django-grinder/raw/main/images/logo-light.svg">
+ <source media="(prefers-color-scheme: dark)" srcset="https://github.com/codingjoe/threadmill/raw/main/images/logo-dark.svg">
+ <source media="(prefers-color-scheme: light)" srcset="https://github.com/codingjoe/threadmill/raw/main/images/logo-light.svg">
+ <img alt="Django Grinder: A queue agnostic worker for Django's task framework." src="https://github.com/codingjoe/threadmill/raw/main/images/logo-light.svg">
  </picture>
 <br>
- <a href="https://codingjoe.dev/django-grinder/">Documentation</a> |
- <a href="https://github.com/codingjoe/django-grinder/issues/new/choose">Issues</a> |
- <a href="https://github.com/codingjoe/django-grinder/releases">Changelog</a> |
+ <a href="https://github.com/codingjoe/threadmill/">Documentation</a> |
+ <a href="https://github.com/codingjoe/threadmill/issues/new/choose">Issues</a> |
+ <a href="https://github.com/codingjoe/threadmill/releases">Changelog</a> |
  <a href="https://github.com/sponsors/codingjoe">Funding</a> 💚
 </p>
 
 **A queue agnostic worker for Django's task framework.**
 
-- self-healing workers
-- graceful shutdown
-- CPU, IO, or memory optimized workers
+## Design Principles
 
-[![PyPi Version](https://img.shields.io/pypi/v/django-grinder.svg)](https://pypi.python.org/pypi/django-grinder/)
-[![Test Coverage](https://codecov.io/gh/codingjoe/django-grinder/branch/main/graph/badge.svg)](https://codecov.io/gh/codingjoe/django-grinder)
-[![GitHub License](https://img.shields.io/github/license/codingjoe/django-grinder)](https://raw.githubusercontent.com/codingjoe/django-grinder/master/LICENSE)
+- **Durability** – We recover from any failures, even poorly written tasks.
+- **Consistency** – We never lose data, even if someone unplugs the power or network.
+- **Utilization** – We keep the CPU saturated with tasks, not with idle time or waiting for locks.
+
+> [!WARNING]
+> Threadmill requires a development version of Django and is in a preview stage.
+
+[![PyPi Version](https://img.shields.io/pypi/v/threadmill.svg)](https://pypi.python.org/pypi/threadmill/)
+[![Test Coverage](https://codecov.io/gh/codingjoe/threadmill/branch/main/graph/badge.svg)](https://codecov.io/gh/codingjoe/threadmill)
+[![GitHub License](https://img.shields.io/github/license/codingjoe/threadmill)](https://raw.githubusercontent.com/codingjoe/threadmill/master/LICENSE)
 
 ## Setup
 
 You need to have [Django's Task framework][django-tasks] setup properly.
 
 ```console
-uv add django-grinder
+uv add threadmill
 ```
 
-Add `grinder` to your `INSTALLED_APPS` in `settings.py`:
+Add `threadmill` to your `INSTALLED_APPS` in `settings.py`:
 
 ```python
 # settings.py
 INSTALLED_APPS = [
- "grinder",
+ "threadmill",
  # ...
 ]
 ```
 
-Finally, you launch the scheduler in a separate process:
+Finally, you launch the worker pool:
+
+```console
+uv run manage.py threadmill
+```
+
+## Usage
+
+The workers are inspired by Gunicorn, and the CLI is very similar.
+
+### Utilization
+
+Depending on your workload, you can tweak the number of processes and threads.
+Processes allow for parallel compute (no GIL) while threads are great for low-memory concurrent IO.
+
+```console
+uv run manage.py threadmill --processes 4 --threads 2
+```
+
+### Health
+
+If your tasks leak memory, you can recycle (restart) the workers after a certain number of tasks have been processed:
+
+```console
+uv run manage.py threadmill --max-tasks 1000 --max-tasks-jitter 100
+```
+
+This will restart the workers after 1000 tasks have been processed, with a random jitter of up to 100 tasks to avoid all workers restarting at the same time.
+
+Should a worker crash or be killed, the pool will automatically restart it.
+
+### Shutdown
+
+A graceful shutdown is possible with the `SIGTERM` or a keyboard interrupt.
+All workers will finish the tasks they acquired and publish them.
+
+You can use `--exit-empty` to exit immediately after all tasks have been processed,
+which might be useful for draining a one-off queue.
+
+### Task Backlog
+
+You can prefetch tasks from a queue to avoid IO latency bottlenecks.
+However, this will increase the memory usage of the worker pool.
 
 ```console
-uv run manage.py grinder
+uv run manage.py threadmill --prefetch 100
 ```
 
+### Task Timeouts
+
+> [!WARNING]
+> Work in progress, this feature is not yet stable.
+
+Task timeouts are important to ensure the long-term health of your pool.
+However, they need to be aligned with your queueing system's timeout settings.
+The message queue needs to requeue a task that hasn't been acknowledged within the timeout.
+
+## Integration
 
-[django-tasks]: https://docs.djangoproject.com/en/6.0/topics/tasks/
+> [!NOTE]
+> This section is for people who want to integrate Threadmill into their queueing system.
+
+Threadmill is designed to be durable and requires a queueing system to support late acknowledgement.
+
+To use Threadmill, your backend will need to inherit from `threadmill.backends.AcknowledgeableTaskBackend` and implement the following methods:
+
+```python
+class AcknowledgeableTaskBackend(BaseTaskBackend, ABC):
+ """Provide an interface for tasks queues to be processed by the executor."""
+
+ def acquire(
+ self, *queue_names: str, timeout: datetime.timedelta | None = None
+ ) -> TaskResult:
+ """
+ Return and lock the next task to be processed without removing it from the queue.
+
+ Args:
+ queue_names: The names of the queues to acquire tasks from.
+ timeout: The maximum time to wait for a task. If None, wait indefinitely.
+
+ Raises:
+ TimeoutError: If no task is available within the specified timeout.
+ """
+ raise NotImplementedError
+
+ def acknowledge(self, task_result: TaskResult) -> None:
+ """Remove the task from the queue and publish the result."""
+ raise NotImplementedError
+```