Distributed File System (DFS)

A simplified Distributed File System inspired by GFS and HDFS. Features a master node for metadata management and multiple chunk servers for distributed data storage with automatic replication.

Quick Start

# Build all binaries
go build -o bin/dfs-master ./cmd/master
go build -o bin/dfs-chunkserver ./cmd/chunkserver
go build -o bin/dfs-client ./cmd/client
# Start the master server
./bin/dfs-master
# Start chunk servers (in separate terminals)
./bin/dfs-chunkserver -port :9001 -master localhost:8000 -dir ./data/cs1
./bin/dfs-chunkserver -port :9002 -master localhost:8000 -dir ./data/cs2
# Upload a file
./bin/dfs-client put /path/to/local/file.txt remote-name.txt
# Download a file
./bin/dfs-client get remote-name.txt /path/to/download/dir

---

Features

✅ Implemented

Feature	Description
File Upload	Chunked upload with automatic allocation
File Download	Reassembly from distributed chunks
Chunk Replication	Configurable replication factor (default: 2)
Heartbeat System	Bidirectional streaming for health monitoring
Dead Server Detection	Automatic detection and replication triggering
Replica Placement	Storage-aware server selection
Structured Logging	slog-based logging with levels
Client SDK	High-level API for file operations
CLI Tool	Command-line interface for put/get operations

🔜 Planned

• Write pipeline (chain replication)
• Lease management for primary writes
• Checksum verification
• Graceful shutdown handling

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ Client │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ DFSClient SDK │ │
│ │ ┌───────────┐ ┌────────────┐ ┌────────────────────┐ │ │
│ │ │ Uploader │ │ Downloader │ │ MasterClient │ │ │
│ │ └───────────┘ └────────────┘ └────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
│ gRPC
▼
┌─────────────────────────────────────────────────────────────────┐
│ Master Server │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ Metadata │ │ Heartbeat │ │ Chunk Placement │ │
│ │ Manager │ │ Manager │ │ Manager │ │
│ └─────────────┘ └──────────────┘ └────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
│ gRPC (Heartbeat + Tasks)
▼
┌─────────────────────────────────────────────────────────────────┐
│ Chunk Servers │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ ChunkServer 1 │ │ ChunkServer 2 │ │ ChunkServer N │ │
│ │ :9001 │ │ :9002 │ │ :900N │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Component Diagram

flowchart TB
classDef client fill:#fef08a,stroke:#eab308,color:#1c1917
classDef master fill:#c4b5fd,stroke:#8b5cf6,color:#1c1917
classDef chunk fill:#93c5fd,stroke:#3b82f6,color:#1c1917
Client["Client<br/><br/>CLI / SDK<br/>File operations"]:::client --> |gRPC| Master
subgraph Master["Master Server"]
MM["Metadata Manager<br/>File → Chunk mapping"]:::master
HM["Heartbeat Manager<br/>Monitor ChunkServers"]:::master
CM["Chunk Placement<br/>Replica selection"]:::master
end
subgraph CSGroup["ChunkServers"]
CS1["ChunkServer 1<br/>:9001"]:::chunk
CS2["ChunkServer 2<br/>:9002"]:::chunk
CSN["ChunkServer N<br/>:900N"]:::chunk
end
Master --> |Heartbeat| CS1
Master --> |Heartbeat| CS2
Master --> |Heartbeat| CSN
Client --> |Read/Write| CS1
Client --> |Read/Write| CS2
Client --> |Read/Write| CSN

Loading

Write Sequence

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#c4b5fd', 'secondaryColor': '#93c5fd', 'tertiaryColor': '#fef08a', 'primaryTextColor': '#1c1917', 'lineColor': '#8b5cf6'}}}%%
sequenceDiagram
participant C as Client
participant M as Master
participant CS1 as ChunkServer 1
participant CS2 as ChunkServer 2
C->>M: AllocateChunk(filename, index)
M-->>C: ChunkID + ReplicaServers
C->>CS1: UploadChunk(stream)
CS1-->>C: Success
Note over M,CS2: Replication via heartbeat
M->>CS1: ReplicationTask
CS1->>CS2: ReplicateChunk
CS2-->>CS1: Ack

Loading

Read Sequence

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#c4b5fd', 'secondaryColor': '#93c5fd', 'tertiaryColor': '#fef08a', 'primaryTextColor': '#1c1917', 'lineColor': '#8b5cf6'}}}%%
sequenceDiagram
participant C as Client
participant M as Master
participant CS as ChunkServer
C->>M: GetFileInfo(filename)
M-->>C: ChunkIDs + Locations
C->>CS: DownloadChunk(chunkID)
CS-->>C: ChunkData stream

Loading

---

Project Structure

dfs/
├── cmd/ # Entry points
│ ├── master/main.go # Master server binary
│ ├── chunkserver/main.go # Chunk server binary
│ └── client/main.go # CLI client binary
│
├── internal/ # Private implementation
│ ├── master/ # Master server logic
│ ├── chunkserver/ # Chunk server logic
│ └── client/ # Client SDK
│ ├── dfsclient.go # High-level SDK
│ ├── uploader/ # Upload handling
│ ├── downloader/ # Download handling
│ └── masterclient/ # Master communication
│
├── pkg/ # Shared packages
│ └── logger/ # Structured logging
│
├── dfs/ # Generated protobuf code
│ ├── masterpb/
│ └── chunkpb/
│
└── proto/ # Protobuf definitions

---

CLI Usage

# Upload a file
./bin/dfs-client put <local-file><remote-name># Download a file
./bin/dfs-client get <remote-name><local-directory># Specify custom master address
./bin/dfs-client -master localhost:9000 put file.txt myfile.txt

Chunk Server Flags

./bin/dfs-chunkserver [flags]
Flags:
-port string Chunk server address (default ":9001")
-master string Master server address (default ":8000")
-dir string Storage directory (default "./data")

---

Configuration

Parameter	Default	Description
REPLICATION_FACTOR	2	Number of replicas per chunk
CHUNK_SIZE	64 MB	Size of each chunk
LIVE_THRESHOLD	30s	Server considered dead after this
Heartbeat Interval	5s	Chunk server heartbeat frequency

---

Data Flow

Write Path

1. Client requests chunk allocation from Master
2. Master returns ChunkID and replica servers
3. Client uploads directly to chunk server
4. Master triggers replication via heartbeat

Read Path

1. Client requests file info from Master
2. Master returns chunk locations
3. Client downloads chunks directly from chunk servers
4. Client reassembles file from chunks

---

Heartbeat System

Chunk servers maintain a bidirectional gRPC stream with the master:

ChunkServer → Master:

• Server address
• Free storage (MB)
• List of stored chunks

Master → ChunkServer:

• Replication tasks
• Delete tasks

---

Development

# Build all
go build ./...
# Run tests
go test ./...
# Lint
go vet ./...

---

License

MIT