ClusterGuard

Real time fault detection system for distributed operational environments. Consumes event streams from Apache Kafka, detects anomalies using a sliding-window statistical model, triggers automated resolution workflows and exposes Prometheus metrics for observability.

Built to explore end to end ownership of a Kubernetes native service from manifest authoring and Helm packaging through to debugging pod restarts with kubectl and writing a runbook for on call response.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ ┌──────────┐ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Kafka │───▶│ ClusterGuard │───▶│ Alert Manager │ │
│ │ (source) │ │ (consumer) │ │ (webhook sink) │ │
│ └──────────┘ │ │ └──────────────────┘ │
│ │ /metrics ──────────▶ Prometheus │
│ │ /healthz └──────▶ Grafana │
│ └─────────────────┘ │
│ │ │
│ ConfigMap (thresholds) │
└─────────────────────────────────────────────────────────────────┘

Data flow

1. Events arrive on the ops.events Kafka topic (JSON, ~2M/day in load tests)
2. Consumer goroutines process partitions concurrently
3. Each event is scored against a sliding window anomaly detector
4. Anomalies above threshold emit a Prometheus counter and POST to a configurable webhook
5. All decisions are written to stdout as structured JSON logs

---

Project structure

ClusterGuard/
├── .github/
│ └── workflows/
│ ├── CI.yaml
├── cmd/
│ └── main.go # entrypoint, flag parsing, signal handling
│├── deploy/
│ ├── k8s/
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ │ └── helm/
│ ├── Chart.yaml
│ ├── values.yaml
├── docs/
│ └── runbook.md
├── internal/
│ ├── detector/
│ │ ├── anomaly.go # sliding-window z-score detector
│ │ └── anomaly_test.go
│ ├── metrics/
│ │ └── prometheus.go # Prometheus counters, histograms, gauges
│ └── webhook/
│ └── alert.go # HTTP POST to configurable alert endpoint
|── README.md ├── doc.go
|── go.mod
└── go.sum

---

Getting started

Prerequisites

• Go 1.22+
• Docker
• A running Kafka broker (local: docker compose up kafka)
• kubectl configured against a cluster (local: kind create cluster)

Run locally

git clone https://github.com/RiyaJ6/ClusterGuard
cd ClusterGuard
# start a local kafka
docker run -d --name kafka -p 9092:9092 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
confluentinc/cp-kafka:latest
# build and run
go build -o ClusterGuard ./cmd/ClusterGuard
./ClusterGuard \
--brokers localhost:9092 \
--topic ops.events \
--group ClusterGuard \
--webhook-url http://localhost:8080/alerts \
--metrics-port 9090

Metrics will be available at http://localhost:9090/metrics.

Run tests

go test ./... -v -race

Deploy to Kubernetes

# raw manifests
kubectl apply -f deploy/k8s/
# or via Helm
helm install ClusterGuard deploy/helm/ClusterGuard \
--set kafka.brokers=kafka:9092 \
--set webhook.url=http://alertmanager:9093/webhook

---

Configuration

All configuration is read from environment variables (set via ConfigMap in Kubernetes).

Variable	Default	Description
KAFKA_BROKERS	localhost:9092	Comma separated broker list
KAFKA_TOPIC	ops.events	Topic to consume
KAFKA_GROUP	clusterguard	Consumer group ID
WINDOW_SIZE	100	Sliding window size for anomaly detection
ZSCORE_THRESHOLD	3.0	Z-score threshold for anomaly flag
WEBHOOK_URL	``	Alert webhook endpoint
METRICS_PORT	9090	Prometheus metrics port
LOG_LEVEL	info	Log level (debug, info, warn, error)

---

Observability

Metrics exposed at /metrics

Metric	Type	Description
clusterguard_events_processed_total	Counter	Total events consumed
clusterguard_anomalies_detected_total	Counter	Anomalies above threshold
clusterguard_processing_duration_seconds	Histogram	Per-event processing time
clusterguard_consumer_lag	Gauge	Kafka consumer lag by partition
clusterguard_webhook_errors_total	Counter	Failed alert deliveries

Structured log fields (JSON to stdout)

{
"level": "info",
"ts": "2025-03-14T09:26:00Z",
"event_id": "evt_8f3a",
"partition": 2,
"offset": 18432,
"score": 4.21,
"anomaly": true,
"processing_ms": 1.4,
"msg": "anomaly detected"
}

See docs/runbook.md for dashboard setup and on call response steps.

---

Debugging pod restarts — What I learned

During development I hit three real restart loops that were instructive to trace:

OOMKill — the consumer was buffering all in flight events in memory before processing. With a high throughput topic this blew the 128Mi memory limit within minutes. Fix: switched to a bounded channel with backpressure tuned limit to 256Mi after measuring actual RSS under load with kubectl top pod.

Failed readiness probe — the /healthz endpoint returned 200 only after the Kafka consumer group had rebalanced which takes 3 to 10 seconds on startup. The readiness probe was firing at 2 seconds. Fix: added an initialDelaySeconds: 15 to the probe.

Goroutine race — the anomaly detector's sliding window was a shared slice accessed by multiple partition goroutines without a lock. Found with go test -race. Fix: one detector instance per partition goroutine no shared state.

Each fix was validated by deploying to a local kind cluster and watching kubectl get pods -w until the restart count held at 0 for 5 minutes.

---

Terraform

The terraform/ directory provisions the AWS infrastructure used in staging: VPC, EKS node group, IAM role for the service account (IRSA), and an S3 bucket for state. See terraform/README.md for usage.

---

Self healing across a cluster of servers