FlowCore — Complete Digital Twin Platform

End-to-end Digital Twin platform for data center infrastructure management with real-time telemetry ingestion, graph-based topology, ML-powered insights, and NOC visualization.

System Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│ NOC FRONTEND (UI) │
│ http://localhost:8888/ │
├─────────────────────────────────────────────────────────────────────────────┤
│ API Gateway (:8888) │ GraphQL API │ Insights API │ Eventing API │
├─────────────────────────────────────────────────────────────────────────────┤
│ PLATFORM CONTAINER (:8080, 8001-8012, 9999) │
│ ├── Keycloak (Auth) ├── Graph Updater ├── Ingestion Services │
├─────────────────────────────────────────────────────────────────────────────┤
│ AGENTS CONTAINER (:5000, 8020, 8022) │
│ ├── MLflow UI (:5000) ├── Topology Agent (:8020) ├── Class Agent (:8022)│
├─────────────────────────────────────────────────────────────────────────────┤
│ REPLAY SERVICE (:8050) → KAFKA → All downstream consumers │
├─────────────────────────────────────────────────────────────────────────────┤
│ PERSISTENT STORES │
│ ├── Neo4j (Graph) :7474/:7687 ├── PostgreSQL :5432 ├── TimescaleDB :5433│
│ └── Kafka (Redpanda) :9092 + Kafka UI :8091 │
└─────────────────────────────────────────────────────────────────────────────┘

What's Included

Core Infrastructure (docker/docker-compose.yml)

PostgreSQL — Layer 2 reference data + Layer 4 agent outputs
TimescaleDB — Layer 3 time-series telemetry
Neo4j — Layer 1 graph topology

Streaming (docker/docker-compose.kafka.yml)

Redpanda (Kafka) — Event streaming, 12 topics
Kafka UI — Topic browser at http://localhost:8091/

Platform Services (docker/docker-compose.platform.yml)

Keycloak — Authentication at http://localhost:8080/
Graph Updater — Neo4j graph synchronization
Ingestion Services — DCIM, SNMP, BMC data ingestion

AI/ML Agents (docker/docker-compose.agents.yml)

MLflow — ML experiment tracking at http://localhost:5000/
Topology Agent — Network topology discovery (:8020)
Classification Agent — Device classification (:8022)

API + UI (docker/docker-compose.api.yml)

GraphQL API — Digital twin queries
Insights API — Anomaly detection, forecasting
Eventing API — Real-time event streaming
NOC Frontend — React UI at http://localhost:8888/

Synthetic Data (docker/docker-compose.replay.yml)

Synthetic Replay — Generates telemetry at 1000 events/sec, API at http://localhost:8050/

Prerequisites

Tool	Version	Purpose
Docker + Docker Compose	24+	Container runtime
Python 3	3.9+	Data generator (stdlib only)
8GB RAM	—	Minimum for all containers
20GB disk	—	For volumes and images

Quick Start

Option 1: One-Command Bootstrap (Recommended for first run)

# Make executable and run
chmod +x scripts/run_all.sh
./scripts/run_all.sh
# End-to-end: ~10-20 minutes (generates data + starts all containers)

Option 2: Container Manager Script (Recommended for daily use)

# Start all containers
./scripts/container.sh start
# Other commands
./scripts/container.sh stop # Stop all
./scripts/container.sh restart # Full restart
./scripts/container.sh ps # Show status
./scripts/container.sh logs # Follow logs

Option 3: Manual Docker Compose

# Create Docker network
docker network create flowcore_flowcore-net
# Start all services
docker compose \
-f docker/docker-compose.yml \
-f docker/docker-compose.kafka.yml \
-f docker/docker-compose.platform.yml \
-f docker/docker-compose.agents.yml \
-f docker/docker-compose.api.yml \
-f docker/docker-compose.replay.yml \
up -d

Service URLs After Startup

Service	URL	Description
NOC UI	http://localhost:8888/	Main dashboard, racks, devices
GraphQL	http://localhost:8888/graphql	Digital twin queries
MLflow UI	http://localhost:5000/	ML experiments, model registry
Kafka UI	http://localhost:8091/	Topic browser, consumer groups
Keycloak	http://localhost:8080/	Authentication admin
Replay API	http://localhost:8050/	Synthetic data control
Neo4j Browser	http://localhost:7474/	Graph visualization

Connection Details

Databases

Store	Host	Port	User	Password	DB
PostgreSQL	localhost	5432	flowcore	flowcore_secret	flowcore
TimescaleDB	localhost	5433	flowcore	flowcore_secret	flowcore_ts
Neo4j	localhost	7474/7687	neo4j	flowcore_secret	—

Kafka Topics

dcim.config.raw dcim.config.normalized
metrics.timeseries.raw alerts.raw
events.raw discovery.snmp.results
discovery.bmc.results graph.mutations
drift.events drift.suggestions
classification.results incidents.correlated

Project Structure

flowcore/
├── docker/
│ ├── docker-compose.yml # Core: Postgres, TimescaleDB, Neo4j
│ ├── docker-compose.kafka.yml # Redpanda + Kafka UI
│ ├── docker-compose.platform.yml # Keycloak, Graph Updater, Ingestion
│ ├── docker-compose.agents.yml # MLflow, Topology, Classification
│ ├── docker-compose.api.yml # GraphQL, Insights, Eventing, NOC UI
│ ├── docker-compose.replay.yml # Synthetic telemetry generator
│ └── dockerfiles/ # Container build definitions
├── services/
│ ├── graph-api/ # GraphQL resolvers
│ ├── insights-api/ # Analytics endpoints
│ ├── eventing-integration/ # Real-time event streaming
│ ├── synthetic-replay/ # Telemetry generator
│ ├── classification-agent/ # ML classification
│ └── topology-agent/ # Network discovery
├── noc-frontend/ # React NOC UI
├── schema/ # SQL/Cypher DDL
│ ├── 01_postgresql_schema.sql
│ ├── 02_timescaledb_schema.sql
│ └── 03_neo4j_schema.cypher
├── generators/ # Synthetic data generation
│ └── generate_all.py
├── scripts/
│ ├── run_all.sh # Master bootstrap
│ ├── container.sh # Container manager
│ └── stack.sh # Stack orchestrator
└── FIXES_SUMMARY.md # Troubleshooting guide

What Gets Created

Scale (medium staging):

2 tenants: ENEA Fusion Research (EU) + Meridian Cloud Services (US)
3 sites: Naples IT, Ashburn US, Frankfurt DE
3 data centers: CRESCO 6 HPC, Meridian Ashburn DC-1, ENEA Frankfurt DR
~1,500 infrastructure entities (49 racks, 478 devices, 98 PDUs, 850 interfaces)
9 source systems (nlyte, Sunbird, Prometheus, SNMP — mixed vendors)
40 canonical metrics × 2 tenants
30 days of hourly metric datapoints (~2.3M rows in TimescaleDB)
Full Layer 4 agent output data: drift events, correlated incidents, capacity forecasts, anomaly flags, failure probability scores, simulation scenarios/results

Troubleshooting

See FIXES_SUMMARY.md for detailed fixes including:

Docker build network timeouts
Port conflicts
UUID handling in synthetic replay
Large file git push issues
Container startup problems

Sample Queries

PostgreSQL — Layer 2 & 4

-- All tenantsSELECT name, plan_tier, status FROM tenant;
-- Open drift events by typeSELECT drift_type, severity, COUNT(*) FROM drift_event
WHERE status ='OPEN'GROUP BY drift_type, severity ORDER BY2;
-- Capacity exhaustion forecasts (90-day)SELECTcf.entity_class, cf.scope_level, cf.predicted_value, cf.unit,
ie.canonical_nameFROM capacity_forecast cf
JOIN infrastructure_entity_ref ie ONcf.entity_id=ie.entity_idWHEREcf.exhaustion_flag= TRUE ANDcf.horizon_label='90d';
-- Failure probability scores — top 10 at-risk devicesSELECTie.canonical_name, fp.probability_score, fp.contributing_factorsFROM failure_probability fp
JOIN infrastructure_entity_ref ie ONfp.entity_id=ie.entity_idORDER BYfp.probability_scoreDESCLIMIT10;
-- ITSM integration configsSELECT target_system, trigger_event_type, is_active FROM itsm_integration_config;

TimescaleDB — Layer 3

-- Average CPU utilisation per hour over last 24hSELECT time_bucket('1 hour', event_ts) AS hour,
AVG(value) AS avg_cpu
FROM metric_datapoint md
JOIN metric_catalogue mc ONmd.metric_id=mc.metric_idWHEREmc.canonical_metric_name='cpu_utilisation_pct'AND event_ts > NOW() - INTERVAL '24 hours'GROUP BY hour ORDER BY hour;
-- Alert storm detection — alerts per 15-min windowSELECT time_bucket('15 minutes', event_ts) AS window,
canonical_severity, COUNT(*) AS alert_count
FROM alert_event
WHERE event_ts > NOW() - INTERVAL '7 days'GROUP BY window, canonical_severity
HAVINGCOUNT(*) >5ORDER BY alert_count DESCLIMIT20;
-- Metric rollup — max daily power draw per entitySELECT entity_id, window_start::dateAS day, max_value AS peak_power_w
FROM metric_rollup mr
JOIN metric_catalogue mc ONmr.metric_id=mc.metric_idWHEREmc.canonical_metric_name='power_draw_w'AND window_size ='24h'ORDER BY peak_power_w DESCLIMIT20;
-- TimescaleDB chunk infoSELECT hypertable_name, num_chunks, total_bytes
FROMtimescaledb_information.hypertables;

Neo4j — Layer 1

// Count all node typesMATCH (n) RETURNlabels(n)[0] ASlabel, count(n) AScountORDER BYcountDESC;
// Rack with all devices in itMATCH (r:InfrastructureEntity{entity_class:'RACK'})<-[:IN_RACK]-(d)
RETURNr.canonical_nameASrack, collect(d.canonical_name) ASdevicesLIMIT5;
// Power chain: device → PDU → feedMATCHpath= (d:InfrastructureEntity{entity_class:'DEVICE'})
-[:POWERED_BY]->(pdu:InfrastructureEntity{entity_class:'PDU'})
-[:POWERED_BY]->(feed:InfrastructureEntity{entity_class:'FEED'})
RETURNd.canonical_name, pdu.canonical_name, feed.canonical_nameLIMIT10;
// Health score distributionMATCH (n:InfrastructureEntity)
WHEREn.entity_class='DEVICE'RETURNCASEWHENn.health_score>=0.9THEN'Healthy'WHENn.health_score>=0.7THEN'Degraded'ELSE'Critical'ENDAShealth_band, count(n) ASdevice_count;
// Blast radius: all entities reachable from a failed deviceMATCHp= (root:InfrastructureEntity{entity_class:'FEED'})
<-[:POWERED_BY*1..3]-(affected)
WHEREroot.canonical_nameCONTAINS'Feed-A'RETURNaffected.canonical_name, affected.entity_class,
length(p) AShopsORDER BYhops;
// Topology: site → DC → location → entity chainMATCH (s:Site)-[:CONTAINS]->(dc:DataCenter)
-[:ORGANISES]->(l:Location)
-[:CONTAINS]->(e:InfrastructureEntity)
WHEREe.entity_class='RACK'RETURNs.canonical_name, dc.canonical_name, l.canonical_name, e.canonical_nameLIMIT10;

Docker management

# Stop containers (data persists in volumes)
docker compose -f docker/docker-compose.yml down
# Restart
docker compose -f docker/docker-compose.yml restart
# Destroy everything including volumes (start fresh)
docker compose -f docker/docker-compose.yml down -v
# View logs
docker compose -f docker/docker-compose.yml logs -f [postgres|timescaledb|neo4j]
# Connect to PostgreSQL directly
docker exec -it flowcore-postgres psql -U flowcore -d flowcore
# Connect to TimescaleDB directly
docker exec -it flowcore-timescaledb psql -U flowcore -d flowcore_ts

Data model coverage

Layer	Store	Entities Populated
Layer 1	Neo4j	Tenant, Site, CollectionAgent, DataCenter, Location, InfrastructureEntity (Rack/Device/PDU/Feed/Interface), EntityAttribute, CapacitySpec, PlatformConfig + all relationships
Layer 2	PostgreSQL	SourceSystem, SourceRecord, FieldMapping, IdentitySignal, EntityResolutionRule, SyncLog, DeadLetterRecord, KafkaTopic, DiscoveryScanLog
Layer 3	TimescaleDB	MetricCatalogue, MetricSourceMapping, MetricDatapoint, MetricRollup, MetricBaseline, AlertCatalogue, AlertSourceMapping, AlertEvent, GraphMutationLog, StaleTelemtryEvent, LogEvent
Layer 4	PostgreSQL	DriftEvent, DriftSuggestion, ClassificationResult, CorrelatedIncident, CapacityForecast, AnomalyFlag, FailureProbability, SimulationScenario, SimulationResult, NodeQualityScore, ItsmIntegrationConfig, User, AuditLogEntry

Quick Reference

# Full startup
./scripts/container.sh start
# Or with data generation (first run)
./scripts/run_all.sh
# Check all services
curl http://localhost:8888/health # NOC UI
curl http://localhost:8050/status # Synthetic Replay
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Key Files:

FIXES_SUMMARY.md — Detailed troubleshooting guide
docker/.env — Environment configuration
scripts/container.sh — Container management
scripts/run_all.sh — Complete bootstrap

FlowCore — Digital Twin Platform for Data Center Infrastructure

Navigation Menu

Repository files navigation

FlowCore — Complete Digital Twin Platform

System Architecture

What's Included

Core Infrastructure (docker/docker-compose.yml)

Streaming (docker/docker-compose.kafka.yml)

Platform Services (docker/docker-compose.platform.yml)

AI/ML Agents (docker/docker-compose.agents.yml)

API + UI (docker/docker-compose.api.yml)

Synthetic Data (docker/docker-compose.replay.yml)

Prerequisites

Quick Start

Option 1: One-Command Bootstrap (Recommended for first run)

Option 2: Container Manager Script (Recommended for daily use)

Option 3: Manual Docker Compose

Service URLs After Startup

Connection Details

Databases

Kafka Topics

Project Structure

What Gets Created

Troubleshooting

Sample Queries

PostgreSQL — Layer 2 & 4

TimescaleDB — Layer 3

Neo4j — Layer 1

Docker management

Data model coverage

Quick Reference

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
deploy		deploy
docker		docker
docs		docs
generators		generators
noc-frontend		noc-frontend
schema		schema
scripts		scripts
services		services
.gitattributes		.gitattributes
.gitignore		.gitignore
FIXES_SUMMARY.md		FIXES_SUMMARY.md
README.md		README.md
analysis_results.md		analysis_results.md

Latest commit

History

Folders and files

Repository files navigation

FlowCore — Complete Digital Twin Platform

System Architecture

What's Included

Core Infrastructure (docker/docker-compose.yml)

Streaming (docker/docker-compose.kafka.yml)

Platform Services (docker/docker-compose.platform.yml)

AI/ML Agents (docker/docker-compose.agents.yml)

API + UI (docker/docker-compose.api.yml)

Synthetic Data (docker/docker-compose.replay.yml)

Prerequisites

Quick Start

Option 1: One-Command Bootstrap (Recommended for first run)

Option 2: Container Manager Script (Recommended for daily use)

Option 3: Manual Docker Compose

Service URLs After Startup

Connection Details

Databases

Kafka Topics

Project Structure

What Gets Created

Troubleshooting

Sample Queries

PostgreSQL — Layer 2 & 4

TimescaleDB — Layer 3

Neo4j — Layer 1

Docker management

Data model coverage

Quick Reference

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages