Nadzu-API

Warning

Infrastructure Notice: Due to the early expiration of DigitalOcean free credits provided by the GitHub Student Developer Pack, the project infrastructure has been temporarily destroyed.

Personal backend API built with Rust, focused on concurrency, performance, security, and long-term maintainability.

Features

Core API

CORS handling.
Rate limiting.
API versioning (v1).
Health and root endpoints.
Structured logging.
Postman v3 collection included.

Media Downloading

Multi-platform media downloading via yt-dlp.
Download acceleration via aria2c integration.
Job lifecycle management: enqueue, progress tracking, and result retrieval.
Server-Sent Events (SSE) for real-time job progress updates.
Endpoint for listing supported sites.

GitHub Contributions Graph

Fetches contribution data from the GitHub GraphQL API.
Provides an endpoint to retrieve contribution stats for a specified user.

Proxy Obfuscation

Bypasses geo-restrictions and anti-bot measures.
Dedicated container that uses the Cloudflare WARP client for outbound requests.
Uses a custom Cloudflare WARP Proxy Docker Image (1.1k pulls), maintained in its dedicated repository.

Security and Anti-Abuse

CAPTCHA verification middleware powered by reCAPTCHA.
API keys have higher rate limits and access to protected endpoints.
Caddy reverse proxy for TLS termination and IP-based access control.

Operational

CI pipelines for linting, testing, and building.
CD pipeline for Docker image builds and publishing to GitHub Container Registry, including:
- zstd compression
- zstd builder
- custom BuildKit caching for faster builds
- multi-platform Docker image support

Architecture Diagrams

Core system diagram

flowchart TB
%% ==========================================
%% BRAND COLORS & STYLING CLASSES
%% ==========================================
classDef extClient fill:#2EA043,color:#ffffff,stroke:#1e6a2c,stroke-width:2px;
classDef proxy fill:#00ADD8,color:#ffffff,stroke:#007a99,stroke-width:2px;
classDef rustApp fill:#111111,color:#DEA584,stroke:#DEA584,stroke-width:2px;
classDef module fill:#1a1a1a,color:#DDDDDD,stroke:#444444,stroke-width:1px,rx:4px;
classDef pg fill:#336791,color:#ffffff,stroke:#234a6a,stroke-width:2px;
classDef redis fill:#DC382D,color:#ffffff,stroke:#9e2820,stroke-width:2px;
classDef extAPI fill:#444444,color:#ffffff,stroke:#222222,stroke-width:2px;
classDef warp fill:#F38020,color:#ffffff,stroke:#b35c00,stroke-width:2px;
%% ==========================================
%% INGRESS
%% ==========================================
Client["Web Client / API Consumer"]:::extClient
Caddy{"Caddy Reverse Proxy"}:::proxy
Client ==>|"HTTPS Requests"| Caddy
%% ==========================================
%% CORE RUST BACKEND
%% ==========================================
subgraph CoreBackend ["Rust Backend Application Core"]
direction TB
MW["Middleware Layer\n(Auth, CORS, RateLimit, Captcha)"]:::module
Router["Routing Layer\n(/api/v1/*)"]:::module
subgraph Controllers ["Controllers"]
direction LR
RootCtrl["Root / Health"]:::module
YtCtrl["YTDLP Controller"]:::module
ContribCtrl["Contributions"]:::module
ValidCtrl["Validation"]:::module
end
subgraph Services ["Business Logic Services"]
direction LR
YtSvc["YT-DLP Service"]:::module
ContribSvc["Contributions Service"]:::module
end
subgraph DataState ["Data & State Management"]
direction LR
PgConn["Postgres Pool (sqlx)"]:::module
RdConn["Redis Multiplex (bb8)"]:::module
AppState["Shared App State (DashMap)"]:::module
end
%% Internal Flow
MW --> Router
Router --> RootCtrl & YtCtrl & ContribCtrl & ValidCtrl
YtCtrl -->|"Calls"| YtSvc
ContribCtrl -->|"Calls"| ContribSvc
YtSvc --> AppState
ContribSvc --> AppState
AppState --> PgConn & RdConn
end
%% ==========================================
%% DATA & EXTERNAL WORKERS
%% ==========================================
subgraph DataLayer ["Data Layer"]
PG[(PostgreSQL 15)]:::pg
Redis[(Redis Alpine)]:::redis
end
subgraph WorkerLayer ["Media Processing Layer"]
Worker["yt-dlp / aria2c Process"]:::rustApp
WARP["WARP SOCKS5 Proxy"]:::warp
end
subgraph ExternalLayer ["External Services"]
GH["GitHub GraphQL API"]:::extAPI
YT["External Media Hosts"]:::extAPI
end
%% ==========================================
%% CROSS-BOUNDARY CONNECTIONS
%% ==========================================
Caddy ==>|"Proxies API Traffic"| MW
PgConn <==>|"sqlx queries"| PG
RdConn <==>|"bb8 multiplexing"| Redis
YtSvc -.->|"tokio::process::Command"| Worker
Worker ==>|"SOCKS5"| WARP
WARP ==>|"Obfuscated Download"| YT
ContribSvc ==>|"Anti-Corruption Layer"| GH
%% ==========================================
%% STYLING FIXES (GitHub Compatible)
%% ==========================================
style CoreBackend fill:none,stroke:#FF5252,stroke-width:2px
style Controllers fill:none,stroke:none
style Services fill:none,stroke:none
style DataState fill:none,stroke:none
style DataLayer fill:none,stroke:none
style WorkerLayer fill:none,stroke:none
style ExternalLayer fill:none,stroke:none

Infrastructure diagram

flowchart TB
%% BRAND COLORS & STYLING CLASSES
classDef extAdmin fill:#2EA043,color:#ffffff,stroke:#1e6a2c,stroke-width:2px;
classDef extPublic fill:#444444,color:#ffffff,stroke:#222222,stroke-width:2px;
classDef cloudflare fill:#F38020,color:#ffffff,stroke:#b35c00,stroke-width:2px;
classDef digitalocean fill:#0069FF,color:#ffffff,stroke:#004bbf,stroke-width:2px;
classDef firewall fill:#D93F0B,color:#ffffff,stroke:#8c2907,stroke-width:2px;
classDef github fill:#24292E,color:#ffffff,stroke:#111417,stroke-width:2px;
classDef docker fill:#2496ED,color:#ffffff,stroke:#1868a6,stroke-width:2px;
classDef rust fill:#000000,color:#DEA584,stroke:#DEA584,stroke-width:2px;
classDef caddy fill:#00ADD8,color:#ffffff,stroke:#007a99,stroke-width:2px;
classDef pg fill:#336791,color:#ffffff,stroke:#234a6a,stroke-width:2px;
classDef redis fill:#DC382D,color:#ffffff,stroke:#9e2820,stroke-width:2px;
%% EXTERNAL TRAFFIC
Admin["Admin (Trusted VPN IP)"]:::extAdmin
Public["Public Internet"]:::extPublic
%% CLOUDFLARE
DNS["Cloudflare DNS & Proxy\n(api.nadzu.me)"]:::cloudflare
%% DIGITALOCEAN
subgraph DigitalOcean ["DigitalOcean Infrastructure"]
subgraph DO_Firewall ["DO Cloud Firewall"]
direction LR
FW_SSH{"Port 22 (SSH)\nALLOW: Admin"}:::firewall
FW_Web{"Ports 80/443 (HTTP/S)\nALLOW: Cloudflare"}:::firewall
end
subgraph Droplet ["Ubuntu Droplet Runtime"]
Caddy{"Caddy Reverse Proxy"}:::caddy
Volume[("DO Block Storage\nMounted at /downloads")]:::digitalocean
subgraph Docker_Compose ["Docker Compose Environment"]
App["Rust Backend API"]:::rust
Worker["YT-DLP / Aria2c Process"]:::rust
WARP["WARP SOCKS5 Proxy"]:::cloudflare
subgraph DBs ["Data Layer"]
direction LR
PG[(PostgreSQL)]:::pg
Redis[(Redis)]:::redis
end
end
end
end
ExternalInternet(("External Internet\n(YouTube, GitHub)")):::extPublic
%% INDEPENDENT REGISTRIES (Placed outside subgraphs to fix layout bugs)
GHCR[("GitHub Container Registry\n(Private)")]:::github
DockerHub[("Docker Hub\n(Public)")]:::docker
%% ----------------------------------------------------
%% TRAFFIC FLOW (Rank 1 to 4)
%% ----------------------------------------------------
Admin -->|"SSH / Deploy"| FW_SSH
Admin -->|"API & Files"| DNS
Public -->|"API Only"| DNS
DNS ==>|"Proxied Web Traffic"| FW_Web
FW_SSH -.->|"Access Granted"| Droplet
FW_Web ==>|"Forwards to Web Server"| Caddy
%% ----------------------------------------------------
%% CADDY ROUTING & VOLUME TRICK (Rank 5)
%% ----------------------------------------------------
Caddy ==>|"Public API (/api/v1/*)"| App
Caddy -.->|"SECURE: Admin IP Only\n(/nadun/fs/*)"| Volume
%% ----------------------------------------------------
%% APP LOGIC & VOLUME (Rank 6)
%% ----------------------------------------------------
Volume -.->|"Mounted into"| Worker
App ==>|"Spawns DL Tasks"| Worker
App -->|"Persists Data"| PG
App -->|"Caches State"| Redis
%% ----------------------------------------------------
%% OBFUSCATION (Rank 7 & 8)
%% ----------------------------------------------------
Worker ==>|"Routes via SOCKS5"| WARP
WARP ==>|"Obfuscated Outbound"| ExternalInternet
%% ----------------------------------------------------
%% REGISTRY PULLS (Auto-aligns cleanly)
%% ----------------------------------------------------
GHCR -.->|"Pulls App Image"| App
DockerHub -.->|"Pulls WARP Image"| WARP
%% STYLES
style DigitalOcean fill:none,stroke:#0069FF,stroke-width:2px,stroke-dasharray: 5 5
style DO_Firewall fill:none,stroke:#D93F0B,stroke-width:2px
style Droplet fill:none,stroke:#0069FF,stroke-width:1px
style Docker_Compose fill:none,stroke:#2496ED,stroke-width:2px
style DBs fill:none,stroke:none

Engineering Design

Clean layered architecture (controllers -> services -> models).
Memory management with DashMap sharding and weak references for lifecycle control.
Concurrency control using Tokio semaphores.

Infrastructure

Provisioned with Terraform using Infrastructure as Code principles.

DigitalOcean Provider: Droplet provisioning with cloud-init, block volume management, and firewall configuration.
Cloudflare Provider: An R2 bucket used for Terraform remote state and DNS record management for full HTTPS support.

Project Structure

Application Directory Structure

.
|-- Caddyfile
|-- Caddyfile.local
|-- Cargo.lock
|-- Cargo.toml
|-- Dockerfile
|-- Dockerfile.dev
|-- GEMINI.md
|-- LICENSE
|-- Makefile
|-- README.md
|-- docker-compose.dev.yml
|-- docker-compose.yml
|-- docker-entrypoint.sh
|-- downloads
|-- infra
| |-- common
| | |-- browse.html
| | |-- certificates
| | | |-- api.nadzu.me.key
| | | `-- api.nadzu.me.pem
| | `-- cloud-init.template
| `-- digitalocean
| |-- accounts
| | `-- naduns-team
| | |-- backend.tf
| | |-- main.tf
| | |-- outputs.tf
| | |-- terraform.tfvars
| | |-- terraform.tfvars.example
| | `-- variables.tf
| `-- components
| |-- c-cloudflare-record.tf
| |-- c-cloudflare-zone.tf
| |-- c-digitalocean-firewall.tf
| |-- data.tf
| |-- locals.tf
| |-- outputs.tf
| |-- provider.tf
| |-- r-digitalocean_droplet.tf
| |-- r-digitalocean_volume.tf
| |-- r-digitalocean_volume_attachment.tf
| |-- variables.tf
| `-- versions.tf
|-- nadunssh
|-- postman
| |-- collections
| | `-- Nadzu API
| | |-- Contributions (Default User).request.yaml
| | |-- Contributions (Specific User).request.yaml
| | |-- Health.request.yaml
| | |-- Root.request.yaml
| | |-- Validate User.request.yaml
| | |-- YT-DLP Download File.request.yaml
| | |-- YT-DLP Enqueue.request.yaml
| | |-- YT-DLP Get Job By ID.request.yaml
| | |-- YT-DLP List Jobs.request.yaml
| | |-- YT-DLP Stream Job Progress.request.yaml
| | `-- supported sites.request.yaml
| |-- environments
| | |-- Nadzu Local.environment.yaml
| | |-- dev.environment.yaml
| | `-- nadzu prod.environment.yaml
|-- rustfmt.toml
|-- src
| |-- app.rs
| |-- bin
| | `-- config_exit.rs
| |-- config.rs
| |-- controllers
| | |-- api
| | | |-- mod.rs
| | | `-- v1
| | | |-- contributions_controller.rs
| | | |-- mod.rs
| | | `-- ytdlp_controller.rs
| | |-- error_controller.rs
| | |-- health_controller.rs
| | |-- mod.rs
| | |-- root_controller.rs
| | `-- validation_controller.rs
| |-- db
| | |-- mod.rs
| | |-- postgres.rs
| | `-- redis.rs
| |-- error.rs
| |-- extractors
| | |-- mod.rs
| | `-- validated_json.rs
| |-- lib.rs
| |-- main.rs
| |-- middleware
| | |-- api_key.rs
| | |-- auth.rs
| | |-- captcha.rs
| | |-- cors.rs
| | |-- mod.rs
| | `-- rate_limit.rs
| |-- models
| | |-- contributions.rs
| | |-- github_dto.rs
| | |-- health.rs
| | |-- mod.rs
| | |-- validation.rs
| | |-- ytdlp.rs
| | `-- ytdlp_dto.rs
| |-- routes
| | |-- api
| | | |-- mod.rs
| | | `-- v1
| | | |-- contributions_routes.rs
| | | |-- mod.rs
| | | `-- ytdlp_routes.rs
| | |-- health_routes.rs
| | |-- mod.rs
| | `-- validation_routes.rs
| |-- services
| | |-- contributions.rs
| | |-- mod.rs
| | `-- ytdlp
| | `-- mod.rs
| `-- state.rs
|-- tests
| |-- api
| | |-- auth_tests.rs
| | |-- captcha_tests.rs
| | |-- common.rs
| | |-- contributions_tests.rs
| | |-- cors_tests.rs
| | |-- health_tests.rs
| | |-- rate_limit_tests.rs
| | |-- root_tests.rs
| | |-- routing_tests.rs
| | |-- validation_tests.rs
| | `-- ytdlp_tests.rs
| |-- api_tests.rs
| `-- layer_unit_tests.rs

Terraform Directory Structure

infra/
├── common/
│ └── cloud-init.template # Bootstraps the VM (Docker, secrets, runs container)
└── digitalocean/
├── accounts/<account-name>/ # Root module per account/environment
│ ├── backend.tf # Cloudflare R2 remote state setup
│ ├── main.tf # Calls the components module
│ ├── outputs.tf # Exposed outputs (Droplet IP, etc.)
│ ├── terraform.tfvars # Secret & environment bindings (gitignored)
│ └── variables.tf # Root variable definitions
└── components/ # The reusable DigitalOcean module
├── locals.tf # Local variables; renders cloud-init
├── outputs.tf # Module outputs
├── provider.tf # DigitalOcean Terraform provider configuration
├── r-digitalocean_droplet.tf # VM resource definitions
├── r-digitalocean_volume*.tf # Block storage resource & attachment
├── variables.tf # Component variable definitions
└── versions.tf # Terraform & provider dependencies

Development Workflow

Iterative development flow designed for fast delivery.
Makefile-first approach for task automation and consistency.
CI pipeline using GitHub Actions for linting (cargo clippy), testing (cargo test), and building.
Unit and integration test coverage.
Production-like local development environment using Docker Compose and Caddy with self-signed TLS.
Active Public Changelog including release notes.

Packaging and Deployment

Dockerized using multi-stage and multi-platform builds (5-stage build with tini).
Optimized Rust builds using Cargo-Chef.
Custom Docker builder implementing ZSTD compression.
Pre-installed dependencies: yt-dlp Python packages, FFmpeg, and FFprobe for media validation.
Published automatically to a private GitHub Container Registry.

Things I Learned

Rust: The initial learning curve is steep, but the long-term benefits in performance, safety, and low-level control are worth it. Very low RAM usage.
Terraform: cloud-init is excellent for bootstrapping a server, but it has provider-specific size limits.
Terraform: The Cloudflare provider only supports R2 buckets; use the AWS Terraform provider for object uploads to R2.
Terraform: It's challenging to maintain local > S3 > cloud-init userdata passing when the project becomes complex over time.

Acknowledgements

yt-dlp

Navigation Menu

Repository files navigation

Nadzu-API

Table of Contents

Features

Core API

Media Downloading

GitHub Contributions Graph

Proxy Obfuscation

Security and Anti-Abuse

Operational

Architecture Diagrams

Engineering Design

Infrastructure

Project Structure

Development Workflow

Packaging and Deployment

Things I Learned

Acknowledgements

About

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 131 Commits
.gemini		.gemini
.github		.github
infra		infra
postman		postman
src		src
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Caddyfile		Caddyfile
Caddyfile.local		Caddyfile.local
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
Dockerfile.dev		Dockerfile.dev
GEMINI.md		GEMINI.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.yml		docker-compose.yml
docker-entrypoint.sh		docker-entrypoint.sh
rustfmt.toml		rustfmt.toml

Folders and files

Latest commit

History

Repository files navigation

Nadzu-API

Table of Contents

Features

Core API

Media Downloading

GitHub Contributions Graph

Proxy Obfuscation

Security and Anti-Abuse

Operational

Architecture Diagrams

Engineering Design

Infrastructure

Project Structure

Development Workflow

Packaging and Deployment

Things I Learned

Acknowledgements

About

Topics

Resources

Stars

Watchers

Forks

Packages

Contributors

Languages