pyproc

Run Python like a local function from Go — no CGO, no microservices.

🎯 Purpose & Problem Solved

The Challenge

Go excels at building high-performance web services, but sometimes you need Python:

• Machine Learning Models: Your models are trained in PyTorch/TensorFlow
• Data Science Libraries: You need pandas, numpy, scikit-learn
• Legacy Code: Existing Python code that's too costly to rewrite
• Python-Only Libraries: Some libraries only exist in Python ecosystem

Traditional solutions all have major drawbacks:

Solution	Problems
CGO + Python C API	Complex setup, crashes can take down entire Go service, GIL still limits performance
REST/gRPC Microservice	Network latency, deployment complexity, service discovery, more infrastructure
Shell exec	High startup cost (100ms+), no connection pooling, process management nightmare
Embedded Python	GIL bottleneck, memory leaks, difficult debugging

The Solution: pyproc

pyproc lets you call Python functions from Go as if they were local functions, with:

• Zero network overhead - Uses Unix Domain Sockets for IPC
• Process isolation - Python crashes don't affect your Go service
• True parallelism - Multiple Python processes bypass the GIL
• Simple deployment - Just your Go binary + Python scripts
• Connection pooling - Reuse connections for high throughput

🎯 Target Audience & Use Cases

Perfect for teams who need to:

• Integrate existing Python ML models (PyTorch, TensorFlow, scikit-learn) into Go services
• Process data with Python libraries (pandas, numpy) from Go applications
• Handle 1-5k RPS with JSON payloads under 100KB
• Deploy on the same host/pod without network complexity
• Migrate gradually from Python microservices to Go while preserving Python logic

Ideal deployment scenarios:

• Kubernetes same-pod deployments with shared volume for UDS
• Docker containers with shared socket volumes
• Traditional server deployments on Linux/macOS

❌ Non-Goals

pyproc is NOT designed for:

• Cross-host communication - Use gRPC/REST APIs for distributed systems
• Windows UDS support - Windows named pipes are not supported
• GPU management - Use dedicated ML serving frameworks (TensorRT, Triton)
• Large-scale ML serving - Consider Ray Serve, MLflow, or KServe for enterprise ML
• Real-time streaming - Use Apache Kafka or similar for high-throughput streams
• Database operations - Use native Go database drivers directly

🔄 Alternatives & Comparison

pyproc is a dedicated IPC engine for integrating Python ML/DS code into Go services on the same host. It differs from general-purpose plugin systems and embedded runtimes in design philosophy.

Solution	Pros	Cons	Best For
go-embed-python	✅ Python runtime embedded / No Python installation required on host	❌ Increased binary size / Python operations are DIY	Tools distributed as a single binary
go-plugin (HashiCorp)	✅ Multi-language plugin support / Proven in Terraform, Vault	❌ Requires gRPC proto definitions / Not optimized for Python	Language-agnostic plugin architecture
pyproc	✅ Optimized for ML/DS workloads / Built-in worker pool, health checks, auto-restart / Ultra-low latency (~45µs p50)	❌ Python-only / Same-host only	Integrating Python ML/DS into Go services

When to Choose What

Choose go-embed-python if:

• You want to distribute a single binary (no Python required on host)
• Increased binary size is acceptable

Choose go-plugin if:

• You need multi-language support (Rust, Ruby, etc.) beyond Python
• You're integrating with HashiCorp ecosystem

Choose pyproc if:

• You're calling Python ML models (PyTorch, TensorFlow) or DS libraries (pandas, NumPy) from Go
• You need low latency (<100µs) on the same host
• You want built-in worker pool management, health checks, and auto-restart

Non-Goals (Recap)

pyproc is NOT designed for:

• ❌ General-purpose plugin system → Use go-plugin
• ❌ Embedded Python runtime → Consider go-embed-python
• ❌ Cross-host communication → Use gRPC/REST microservices
• ❌ GPU cluster management → Use Ray Serve, Triton

🔐 Trust Model & Security Considerations

pyproc is designed for trusted code execution

pyproc is NOT a sandbox environment. It operates under the following assumptions:

• ✅ Target: Python code developed and managed by your organization (ML models, data processing logic)
• ✅ Process isolation: Python crashes do not affect the Go service
• ❌ No security isolation: Python workers can access the same filesystem and network as the parent Go process

Intended Use Cases

✅ Recommended:

• Running your own trained PyTorch/TensorFlow models for inference
• Data transformation pipelines using pandas/NumPy
• Integrating scikit-learn models into Go recommendation engines

❌ Not Recommended:

• Executing arbitrary user-submitted Python scripts
• Dynamically loading third-party plugins
• Running untrusted code

Security Details

For detailed threat model, security architecture, and best practices, see SECURITY.md.

Key Guarantees:

• OS-level access control via Unix Domain Socket filesystem permissions
• Fault tolerance through process isolation
• Configurable resource limits (memory, CPU)

Limitations:

• Inter-process communication on the same host only (cross-host is out of scope)
• Does not provide sandbox environment (use gVisor, Firecracker if needed)

📋 Compatibility Matrix

Component	Requirements
Operating System	Linux, macOS (Unix Domain Sockets required)
Go Version	1.22+
Python Version	3.9+ (3.12 recommended)
Deployment	Same host/pod only
Container Runtime	Docker, containerd, any OCI-compatible
Orchestration	Kubernetes (same-pod), Docker Compose, systemd
Architecture	amd64, arm64

✨ Features

• No CGO Required - Pure Go implementation using Unix Domain Sockets
• Bypass Python GIL - Run multiple Python processes in parallel
• Type-Safe API - Call Python with compile-time type checking using Go generics (zero overhead)
• Minimal Overhead - 45μs p50 latency, 200,000+ req/s with 8 workers
• Production Ready - Health checks, graceful shutdown, automatic restarts
• Easy Deployment - Single binary + Python scripts, no service mesh needed
• Full Observability - OpenTelemetry tracing, Prometheus metrics, structured logging (v0.7.1+)

🚀 Quick Start (5 minutes)

1. Install

Go side:

go get github.com/YuminosukeSato/pyproc@latest

Python side:

pip install pyproc-worker

2. Create a Python Worker

# worker.pyfrompyproc_workerimportexpose, run_worker@exposedefpredict(req):
"""Your ML model or Python logic here"""return {"result": req["value"] *2}
if__name__=="__main__":
run_worker()

3. Call from Go (Type-Safe API - Recommended)

package main
import (
"context""fmt""log""github.com/YuminosukeSato/pyproc/pkg/pyproc"
)
// Define request/response types (compile-time type safety)typePredictRequeststruct {
Valuefloat64`json:"value"`
}
typePredictResponsestruct {
Resultfloat64`json:"result"`
}
funcmain() {
// Create a pool of Python workerspool, err:=pyproc.NewPool(pyproc.PoolOptions{
Config: pyproc.PoolConfig{
Workers: 4, // Run 4 Python processesMaxInFlight: 10, // Global concurrent requestsMaxInFlightPerWorker: 1, // Per-worker in-flight cap
},
WorkerConfig: pyproc.WorkerConfig{
SocketPath: "/tmp/pyproc.sock",
PythonExec: "python3",
WorkerScript: "worker.py",
},
}, nil)
iferr!=nil {
log.Fatal(err)
}
// Start all workersctx:=context.Background()
iferr:=pool.Start(ctx); err!=nil {
log.Fatal(err)
}
deferpool.Shutdown(ctx)
// Call Python function with type safety (automatically load-balanced)result, err:=pyproc.CallTyped[PredictRequest, PredictResponse](
ctx, pool, "predict", PredictRequest{Value: 42},
)
iferr!=nil {
log.Fatal(err)
}
fmt.Printf("Result: %v\n", result.Result) // Result: 84 (type-safe!)
}

4. Run

go run main.go

That's it! You're now calling Python from Go without CGO or microservices.

Try the demo in this repo

If you cloned this repository, you can run a working end to end example without installing a Python package by using the bundled worker module.

make demo

This starts a Python worker from examples/basic/worker.py and calls it from Go. The example adjusts PYTHONPATH to import the local worker/python/pyproc_worker package.

📊 Observability (v0.7.1+)

pyproc includes built-in support for distributed tracing, metrics, and structured logging.

Distributed Tracing with OpenTelemetry

import (
"context""github.com/YuminosukeSato/pyproc/pkg/pyproc""github.com/YuminosukeSato/pyproc/pkg/pyproc/telemetry"
)
funcmain() {
// Initialize telemetry providerprovider, shutdown:=telemetry.NewProvider(telemetry.Config{
ServiceName: "my-service",
Enabled: true,
SamplingRate: 0.01, // 1% samplingExporterType: "stdout", // or "otlp" for production
})
defershutdown(context.Background())
// Create poolpool, _:=pyproc.NewPool(poolOpts, logger)
// Attach tracer (opt-in)pool.WithTracer(provider.Tracer("my-service"))
// All calls are now traced automaticallyctx:=context.Background()
result, _:=pyproc.CallTyped[Req, Resp](ctx, pool, "predict", request)
}

Key features:

• ✅ Automatic span creation for all Pool.Call() invocations
• ✅ W3C Trace Context propagation over Unix Domain Sockets
• ✅ <1% overhead with 1% sampling (production target)
• ✅ Zero overhead when disabled (no-op mode)
• ✅ Fully backward compatible (opt-in via WithTracer())

Metrics

Built-in Prometheus metrics:

// Expose metrics endpointhttp.Handle("/metrics", promhttp.Handler())
// Metrics automatically collected:// - pyproc_pool_calls_total// - pyproc_pool_call_duration_seconds// - pyproc_pool_errors_total// - pyproc_worker_active_connections

Structured Logging

import"log/slog"logger:=slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
pool, _:=pyproc.NewPool(poolOpts, logger)

For comprehensive observability documentation, see docs/observability.md.

📚 Detailed Usage Guide

Installation

Go Application

go get github.com/YuminosukeSato/pyproc@latest

Python Worker

# Install from PyPI
pip install pyproc-worker
# Or install from sourcecd worker/python
pip install -e .

Configuration

Basic Configuration

cfg:= pyproc.WorkerConfig{
ID: "worker-1",
SocketPath: "/tmp/pyproc.sock",
PythonExec: "python3", // or path to virtual envWorkerScript: "path/to/worker.py",
StartTimeout: 30*time.Second,
Env: map[string]string{
"PYTHONUNBUFFERED": "1",
"MODEL_PATH": "/models/latest",
},
}

Pool Configuration

poolCfg:= pyproc.PoolConfig{
Workers: 4, // Number of Python processesMaxInFlight: 10, // Global concurrent requestsMaxInFlightPerWorker: 1, // Per-worker in-flight capHealthInterval: 30*time.Second, // Health check frequency
}

Python Worker Development

Basic Worker

frompyproc_workerimportexpose, run_worker@exposedefadd(req):
"""Simple addition function"""return {"result": req["a"] +req["b"]}
@exposedefmultiply(req):
"""Simple multiplication"""return {"result": req["x"] *req["y"]}
if__name__=="__main__":
run_worker()

ML Model Worker

importpicklefrompyproc_workerimportexpose, run_worker# Load model once at startupwithopen("model.pkl", "rb") asf:
model=pickle.load(f)
@exposedefpredict(req):
"""Run inference on the model"""features=req["features"]
prediction=model.predict([features])[0]
confidence=model.predict_proba([features])[0].max()
return {
"prediction": int(prediction),
"confidence": float(confidence)
}
@exposedefbatch_predict(req):
"""Batch prediction for efficiency"""features_list=req["batch"]
predictions=model.predict(features_list)
return {
"predictions": predictions.tolist()
}
if__name__=="__main__":
run_worker()

Data Processing Worker

importpandasaspdfrompyproc_workerimportexpose, run_worker@exposedefanalyze_csv(req):
"""Analyze CSV data using pandas"""df=pd.DataFrame(req["data"])
return {
"mean": df.mean().to_dict(),
"std": df.std().to_dict(),
"correlation": df.corr().to_dict(),
"summary": df.describe().to_dict()
}
@exposedefaggregate_timeseries(req):
"""Aggregate time series data"""df=pd.DataFrame(req["data"])
df['timestamp'] =pd.to_datetime(df['timestamp'])
df.set_index('timestamp', inplace=True)
# Resample to hourlyhourly=df.resample('H').agg({
'value': ['mean', 'max', 'min'],
'count': 'sum'
})
returnhourly.to_dict()
if__name__=="__main__":
run_worker()

Go Integration Patterns

Simple Request-Response

funccallPythonFunction(pool*pyproc.Pool) error {
input:=map[string]interface{}{
"a": 10,
"b": 20,
}
varoutputmap[string]interface{}
iferr:=pool.Call(context.Background(), "add", input, &output); err!=nil {
returnfmt.Errorf("failed to call Python: %w", err)
}
fmt.Printf("Result: %v\n", output["result"])
returnnil
}

With Timeout

funccallWithTimeout(pool*pyproc.Pool) error {
ctx, cancel:=context.WithTimeout(context.Background(), 5*time.Second)
defercancel()
input:=map[string]interface{}{"value": 42}
varoutputmap[string]interface{}
iferr:=pool.Call(ctx, "slow_process", input, &output); err!=nil {
iferr==context.DeadlineExceeded {
returnfmt.Errorf("Python function timed out")
}
returnerr
}
returnnil
}

Batch Processing

funcprocessBatch(pool*pyproc.Pool, items []Item) ([]Result, error) {
input:=map[string]interface{}{
"batch": items,
}
varoutputstruct {
Predictions []float64`json:"predictions"`
}
iferr:=pool.Call(context.Background(), "batch_predict", input, &output); err!=nil {
returnnil, err
}
results:=make([]Result, len(output.Predictions))
fori, pred:=rangeoutput.Predictions {
results[i] =Result{Value: pred}
}
returnresults, nil
}

Error Handling

funcrobustCall(pool*pyproc.Pool) {
forretries:=0; retries<3; retries++ {
varoutputmap[string]interface{}
err:=pool.Call(context.Background(), "predict", input, &output)
iferr==nil {
// Successreturn
}
// Check if it's a Python errorifstrings.Contains(err.Error(), "ValueError") {
// Invalid input, don't retrylog.Printf("Invalid input: %v", err)
return
}
// Transient error, retry with backofftime.Sleep(time.Duration(retries+1) *time.Second)
}
}

Deployment

Docker

FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .
FROM python:3.11-slim
RUN pip install pyproc-worker numpy pandas scikit-learn
COPY --from=builder /app/myapp /app/myapp
COPY worker.py /app/
WORKDIR /app
CMD ["./myapp"]

Kubernetes

apiVersion: apps/v1kind: Deploymentmetadata:
name: myappspec:
replicas: 3template:
spec:
containers:
- name: appimage: myapp:latestenv:
- name: PYPROC_POOL_WORKERSvalue: "4"
- name: PYPROC_SOCKET_DIRvalue: "/var/run/pyproc"volumeMounts:
- name: socketsmountPath: /var/run/pyprocvolumes:
- name: socketsemptyDir: {}

Monitoring & Debugging

Enable Debug Logging

logger:=pyproc.NewLogger(pyproc.LoggingConfig{
Level: "debug",
Format: "json",
})
pool, _:=pyproc.NewPool(opts, logger)

Health Checks

health:=pool.Health()
fmt.Printf("Workers: %d healthy, %d total\n", health.HealthyWorkers, health.TotalWorkers)

Metrics Collection

// Expose Prometheus metricshttp.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":9090", nil)

Common Issues & Solutions

Issue: Worker won't start

# Check Python dependencies
python3 -c "from pyproc_worker import run_worker"# Check socket permissions
ls -la /tmp/pyproc.sock
# Enable debug loggingexport PYPROC_LOG_LEVEL=debug

Issue: High latency

// Increase worker countpoolCfg.Workers=runtime.NumCPU() *2// Pre-warm connectionspool.Start(ctx)
time.Sleep(1*time.Second) // Let workers stabilize

Issue: Memory growth

# Add memory profiling to workerimporttracemalloctracemalloc.start()
@exposedefget_memory_usage(req):
current, peak=tracemalloc.get_traced_memory()
return {
"current_mb": current/1024/1024,
"peak_mb": peak/1024/1024
}

Use Cases

Machine Learning Inference

@exposedefpredict(req):
model=load_model() # Cached after first loadfeatures=req["features"]
return {"prediction": model.predict(features)}

Data Processing

@exposedefprocess_dataframe(req):
importpandasaspddf=pd.DataFrame(req["data"])
result=df.groupby("category").sum()
returnresult.to_dict()

Document Processing

@exposedefextract_pdf_text(req):
importPyPDF2# Process PDF and return textreturn {"text": extracted_text}

Architecture

┌─────────────┐ UDS ┌──────────────┐
│ Go App │ ◄──────────────────────► │ Python Worker│
│ │ Low-latency IPC │ │
│ - HTTP API │ │ - Models │
│ - Business │ │ - Libraries │
│ - Logic │ │ - Data Proc │
└─────────────┘ └──────────────┘
▲ ▲
│ │
└──────────── Same Host/Pod ────────────────┘

Benchmarks

Run benchmarks locally:

# Quick benchmark
make bench
# Full benchmark suite with memory profiling
make bench-full

Example results on M1 MacBook Pro:

BenchmarkPool/workers=1-10 10 235µs/op 4255 req/s
BenchmarkPool/workers=2-10 10 124µs/op 8065 req/s BenchmarkPool/workers=4-10 10 68µs/op 14706 req/s
BenchmarkPool/workers=8-10 10 45µs/op 22222 req/s
BenchmarkPoolParallel/workers=2-10 100 18µs/op 55556 req/s
BenchmarkPoolParallel/workers=4-10 100 9µs/op 111111 req/s
BenchmarkPoolParallel/workers=8-10 100 5µs/op 200000 req/s
BenchmarkPoolLatency-10 100 p50: 45µs p95: 89µs p99: 125µs

The benchmarks show near-linear scaling with worker count, demonstrating the effectiveness of bypassing Python's GIL through process-based parallelism.

Advanced Features

Worker Pool

pool, _:=pyproc.NewPool(pyproc.PoolOptions{
Config: pyproc.PoolConfig{
Workers: 4,
MaxInFlight: 10,
MaxInFlightPerWorker: 1,
},
WorkerConfig: pyproc.WorkerConfig{
SocketPath: "/tmp/pyproc.sock",
PythonExec: "python3",
WorkerScript: "worker.py",
},
}, nil)
ctx:=context.Background()
pool.Start(ctx)
deferpool.Shutdown(ctx)
varresultmap[string]interface{}
pool.Call(ctx, "predict", input, &result)

gRPC Mode (coming in v0.4)

pool, _:=pyproc.NewPool(ctx, pyproc.PoolOptions{
Protocol: pyproc.ProtocolGRPC(),
// Unix domain socket with gRPC
})

Arrow IPC for Large Data (coming in v0.5)

pool, _:=pyproc.NewPool(ctx, pyproc.PoolOptions{
Protocol: pyproc.ProtocolArrow(),
// Zero-copy data transfer
})

🚀 Operational Standards

Performance Targets

Metric	Target	Notes
Latency (p50)	< 100μs	Simple function calls
Latency (p99)	< 500μs	Including GC and process overhead
Throughput	1-5k RPS	Per service instance
Payload Size	< 100KB	JSON request/response
Worker Count	2-8 per CPU core	Based on workload type

Health & Monitoring

Required Metrics:

• Request latency (p50, p95, p99)
• Request rate and error rate
• Worker health status
• Connection pool utilization
• Python process memory usage

Health Check Endpoints:

// Built-in health checkhealth:=pool.Health()
ifhealth.HealthyWorkers<health.TotalWorkers/2 {
log.Warn("majority of workers unhealthy")
}

Alerting Thresholds:

• Worker failure rate > 5%
• p99 latency > 1s
• Memory growth > 500MB/hour
• Connection pool exhaustion

Deployment Best Practices

Resource Limits:

resources:
requests:
memory: "256Mi"cpu: "200m"limits:
memory: "1Gi"cpu: "1000m"

Restart Policies:

• Python worker restart on OOM or crash
• Exponential backoff for failed restarts
• Maximum 3 restart attempts per minute
• Circuit breaker after 10 consecutive failures

Socket Management:

• Use /tmp/sockets/ or shared volume in K8s
• Set socket permissions 0660
• Clean up sockets on graceful shutdown
• Monitor socket file descriptors

Production Checklist

• Set appropriate worker count based on CPU cores
• Configure health checks and alerting
• Set up monitoring (metrics exposed at :9090/metrics)
• Configure restart policies and circuit breakers
• Set resource limits (memory, CPU)
• Handle worker failures gracefully
• Test failover scenarios
• Configure socket permissions and cleanup
• Set up log aggregation for Python workers
• Document runbook for common issues

Documentation

• Design Document
• Operations Guide
• Security Guide
• API Reference

Contributing

We welcome contributions! Check out our "help wanted" issues to get started. Issues and PRs receive an initial response within 14 days; stable releases keep open bug reports under 6 months. PR descriptions must include links to pkg.go.dev, Go Report Card, and Coverage.

License

Apache 2.0 - See LICENSE for details.

References

• Python GIL Documentation
• Unix Domain Sockets
• gRPC Unix Sockets
• Apache Arrow IPC