One problem. Six tools. The real architectural reason behind every scheduling decision.
A data engineer needs to collect data. The collection must be automated, periodic, and reliable.
Which tool do they use?
The answer depends entirely on context — company size, team maturity, infrastructure bounds, and cost profiles. This project makes those contexts concrete.
The same ingestion pipeline is implemented six times across six different scheduling layers. Each implementation starts exactly where the previous tool hits its architectural limit. Every decision is explained. Every trade-off is measurable.
Every chapter solves the exact same data transfer problem:
Fetch weather data from OpenWeatherMap API
↓
Validate constraints via Pydantic v2
↓
Commit via SQLAlchemy 2.0 ORM (Programmatic Alembic Migrations)
↓
Upsert into PostgreSQL (Storing mapped fields + Raw JSONB buffer layer)
↓
Automate & Schedule
What changes across chapters: only the automation and scheduling layer.
| # | Tool | Level | Architectural Identity | Why We Left |
|---|---|---|---|---|
| 01 | Cron | Junior DE / Single Node | Built-in OS automation, zero infrastructure overhead. Container acts as PID 1 boundary via exec cron -f. | OS-level shell isolation, zero native observability dashboards, task overlapping under volume (no backpressure control). |
| 02 | APScheduler | In-app Threading | Embedded Python scheduling threads, process-level lifecycle control, in-app max_instances and coalesce execution guards. | Single process constraint, horizontal scaling blind spot causing multi-instance task collisions, and state volatility (MemoryJobStore loss). |
| 03 | Celery Beat | Distributed Workers | Producer-broker-consumer split, low-latency asynchronous memory queueing via Redis, stateless horizontally elastic worker pools. | Pure task execution engine lacking task lineage tracking (No native DAG structures), structural visual blind spots across complex dependencies, and lack of central monitoring UI. |
| 04 | Airflow | Enterprise Orchestration | Programmatic workflow state management, metadata ledger tracking, Dual Python Runtime isolation (Control Plane vs. Data Plane). | Heavy batch-oriented framework constrained strictly by pull-based scheduling semantics and static filesystem DAG parsing loops, rendering it inefficient for event-driven, real-time serverless ingestion topologies. |
| 05 | Prefect | Modern Orchestration | Code-first dynamic tasks, low-boilerplate modern deployment hooks. | Pending execution analysis... |
| 06 | Airflow on K8s | Cloud-Native Peak | Container-native task-isolation pods. True enterprise production standard. | — The Architectural Ceiling |
- Control Variable Correctness: The data source (API JSON layout) and target destination (PostgreSQL 16 Engine) never change. Only the orchestration boundary shifts. This isolates noise and creates a pure comparison framework.
- Every Tool Earns Its Place: No tool is introduced as a default choice. Each chapter opens directly by breaking or throttling the previous layer's capacity.
- Production Standards from Day One: Even the simplest Cron implementation enforces strict Pydantic parsing, Tenacity exponential backoff retries, unified Structlog structured JSON logs, and programmatic Alembic database state updates on container startup.
- Zero Cost Isolation: Every tool runs completely locally or using free, self-hosted container images. Paid alternatives are technically evaluated but never implemented.
All standalone containers copy and reference the identical business layer directly to prevent code drift between chapters:
shared/
├── alembic/ → Schema versioning scripts tracking engine states (e.g., unique_city_timestamp)
├── alembic.ini → Local and container configuration mapper
├── extractor.py → Low-level HTTP requests using Tenacity and structured logging
├── loader.py → SQLAlchemy ORM declaration, connection factory, and PostgreSQL transactional upserts
└── models.py → Strict Pydantic parsing schemas isolating raw buffers and parsed items
- Idempotency Contract: Regulated via a database-level
UniqueConstraint('city', 'timestamp'). Duplicate runs or backward backfills overwrite mutations safely without state corruption. - The Raw JSON Buffer Layer: To survive upstream API schema evolution, the pipeline saves the complete, raw API response inside a
JSONBcolumn during the upsert phase. This allows full historical reprocessing if downstream schema fields change retrospectively. - Platform Isolation: Each chapter directory operates independently, carrying its own explicit
requirements.txt,Dockerfile, and localdocker-compose.yamllinked externally to our shared infrastructure network boundary.
de-scheduling-evolution/
├── README.md
├── docker-compose.infra.yml ← Common Infrastructure: PostgreSQL 16 + Redis 7 + pgAdmin
├── shared/ ← The immutable core layer copied during image compilation
│ ├── alembic/
│ ├── extractor.py
│ ├── loader.py
│ └── models.py
├── 01-cron/ ← Verified & Locked
├── 02-apscheduler/ ← Verified & Locked
├── 03-celery-beat/ ← Verified & Locked
├── 04-airflow/ ← Verified & Locked
├── 05-prefect/ ← CURRENT STEP
└── 06-airflow-on-kubernetes/
To guarantee resource optimization, the repository isolates data storage blocks from active compute layers. Run the primary infra layout once from the root directory:
docker-compose -f docker-compose.infra.yml up -d- A Junior Data Engineer who needs to understand why enterprise tools exist, rather than just learning syntax.
- A Mid-level DE tasked with choosing a scheduling architecture for a greenfield project.
- A Tech Lead who needs a clear, reference-grade comparison to explain architectural trade-offs to stakeholders.
- A basic tutorial on how to install tools in isolation.
- A benchmark tracking raw performance numbers.
- A vendor-driven cloud platform deployment manual.
Part of the de- series — reference-quality, open-source data engineering blueprints.Maintenaned by Oğuz Kaan Mavice