DQD - Queue daemon

DQD is a daemon that enables flows of consuming data from queue systems or pushing data, making it optimal We designed it to run as a sidecar container adjacent to an existing rest-service affectively, enabling the service to behave as a worker.

Features

• Multiple Providers support
• Multiple pipelines per instance
• Error handling
• Backpressure with dynamic concurrency
• Rate metrics
• Single binary or Docker image

Supported Providers

• AWS SQS
• Azure Queue
• Azure Service bus

Usage

Installing/Running

Using Go

run go install github.com/soluto/dqd run dqd

Using Docker

run docker run -v `pwd`:/etc/dqd soluto/dqd in a folder that conatins dqd.yaml

Examples

Consume and Process using an HTTP route

The dqd.yaml should look like this:

sources:
my-queue:
type: sqs
url: http://aws-sqs:9324/queue/my-sqs
region: us-east-1
pipe:
source: my-queue
rate:
fixed: 1
handler:
http:
endpoint: http://localhost:3000/processSqsMessages

Send to Queue Using DQD

The dqd.yaml should look like this:

sources:
my-queue:
type: sqs
url: http://aws-sqs:9324/queue/my-sqs
region: us-east-1
visibilityTimeoutInSeconds: 60

In your code, send the data to http://localhost:9999/my-sqs

Example for DQD configuration in docker-compose

image: soluto/dqd
ports:
- containerPort: 8888
protocol: TCP
env:
- name: AWS_SDK_LOAD_CONFIG
value: 'true'