pg_fast_data_transfer 🚀

pg_fast_data_transfer is a high-performance PostgreSQL data transfer tool written in Go. It is designed to efficiently move data between databases—even across different servers—handling large datasets with ease using streaming COPY protocols and batch processing.

Key features include:

• ⚡ High Performance: Uses PostgreSQL COPY protocol for maximum throughput.
• 🔄 Streaming: Low memory footprint by streaming data directly between connections.
• 📦 Batch Processing: Configurable batch sizes and error handling.
• 🛠️ Flexible Configuration: Configure via .env file or CLI arguments.
• 🛡️ Type Safe: Supports all PostgreSQL data types (JSONB, Arrays, Vectors, etc.).

---

📋 Table of Contents

• Prerequisites
• Installation
• Configuration
• Usage
• Project Structure

🛠 Prerequisites

• Go 1.20 or higher
• PostgreSQL (Source and Destination databases)

📦 Installation

1. Clone the repository:

   git clone https://github.com/username/pg_fast_data_transfer.git
   cd pg_fast_data_transfer

2. Build the binary:

   go build -o pg_fast_data_transfer.exe

⚙ Configuration

The application uses a .env file for configuration. Copy the example file to get started:

cp .env.example .env

Edit the .env file with your database credentials and transfer settings:

# Source DatabaseSOURCE_DB_HOST=localhostSOURCE_DB_PORT=5432SOURCE_DB_USER=postgresSOURCE_DB_PASSWORD=secretSOURCE_DB_NAME=source_dbSOURCE_DB_SCHEMA=public# Destination DatabaseDEST_DB_HOST=localhostDEST_DB_PORT=5432DEST_DB_USER=postgresDEST_DB_PASSWORD=secretDEST_DB_NAME=dest_dbDEST_DB_SCHEMA=public# Tables to Transfer (comma-separated lists must match order)SOURCE_TABLES=users,ordersDEST_TABLES=users_backup,orders_archive# Performance TuningBATCH_SIZE=10000TRUNCATE_BEFORE_TRANSFER=false

🚀 Usage

Basic Usage

Run the tool using the configuration from your .env file:

./pg_fast_data_transfer.exe

CLI Arguments

You can override .env settings or run ad-hoc transfers using command-line flags:

Flag	Description	Example
--source-table	Specific source table (schema.table)	--source-table=public.users
--dest-table	Specific destination table	--source-table=backup.users
--truncate	Truncate destination before transfer	--truncate
--help	Show help message	--help

Example: Single Table Transfer

./pg_fast_data_transfer.exe --source-table=users --dest-table=users_backup --truncate

📂 Project Structure

├── config/ # Configuration loading and parsing
├── database/ # Database connection logic
├── transfer/ # Core data transfer and streaming logic
├── .env # Environment configuration
└── main.go # Entry point and CLI handling