Feedforward Neural Network (FFNN) from Scratch

A pure NumPy implementation of a feedforward neural network with support for multiple activation functions, designed for educational purposes and deep learning fundamentals.

Features

• Pure NumPy Implementation: Built from scratch without high-level ML frameworks
• Multiple Activation Functions: Supports ReLU, Sigmoid, Tanh, and Softmax
• Flexible Architecture: Configurable number of layers and neurons
• MNIST Ready: Includes data preprocessing and visualization utilities
• Training Metrics: Tracks loss and accuracy during training
• Mini-batch Training: Efficient batch processing with shuffling

Quick Start

# Load and preprocess MNIST dataX_train, Y_train, X_test, Y_test=load_mnist_data()
# Create network: 784 inputs → 128 hidden (ReLU) → 64 hidden (ReLU) → 10 outputsnetwork=FFNN(
layer_sizes=[784, 128, 64, 10],
activations=["relu", "relu"] # Hidden layer activations only
)
# Train the networkhistory=network.train(
X_train, Y_train, X_test, Y_test,
learning_rate=0.01,
num_epochs=50,
batch_size=32
)
# Visualize training progressplot_training_history(history)

Installation

pip install numpy matplotlib tensorflow seaborn

Architecture

FFNN Class

The main neural network class with the following key methods:

__init__(layer_sizes, activations)

• layer_sizes: List of integers defining network architecture (e.g., [784, 128, 10])
• activations: List of activation functions for hidden layers (output uses softmax)

train(X_train, Y_train, X_test, Y_test, learning_rate, num_epochs, batch_size)

Trains the network using mini-batch gradient descent with the following parameters:

• learning_rate: Step size for parameter updates (typically 0.001-0.1)
• num_epochs: Number of complete passes through training data
• batch_size: Number of samples per mini-batch (default: 32)

Supported Activation Functions

Function	Use Case	Formula
ReLU	Hidden layers	max(0, x)
Sigmoid	Hidden layers	1/(1 + e^(-x))
Tanh	Hidden layers	tanh(x)
Softmax	Output layer (automatic)	e^x / Σe^x

Implementation Details

Weight Initialization

Uses He initialization for optimal gradient flow:

W=np.random.randn(n_out, n_in) *sqrt(2.0/n_in)

Loss Function

Cross-entropy loss with numerical stability:

loss=-Σ(y_true*log(y_pred+ε)) /m

Gradient Computation

Implements backpropagation with activation-specific derivatives:

• ReLU: dZ = dA * (Z > 0)
• Sigmoid: dZ = dA * A * (1 - A)
• Tanh: dZ = dA * (1 - A²)

Example Usage

Basic Classification

# Simple 3-layer networknetwork=FFNN([784, 64, 10], ["relu"])
history=network.train(
X_train, Y_train, X_test, Y_test,
learning_rate=0.01,
num_epochs=30
)

Deep Network

# Deeper network with mixed activationsnetwork=FFNN(
layer_sizes=[784, 256, 128, 64, 10],
activations=["relu", "tanh", "relu"]
)

Making Predictions

# Forward pass for predictionspredictions, _=network.forward_propagation(X_test)
predicted_classes=np.argmax(predictions, axis=0)

Performance

Typical results on MNIST (784→128→64→10 architecture):

• Training Accuracy: ~98-99%
• Test Accuracy: ~96-97%
• Training Time: ~2-3 minutes (50 epochs)

Data Format

Input Requirements

• X: Shape (n_features, n_samples) - Features in rows, samples in columns
• Y: Shape (n_classes, n_samples) - One-hot encoded labels

MNIST Preprocessing

The load_mnist_data() function automatically handles:

• Reshaping images from 28×28 to 784×1 vectors
• Normalization to [0,1] range
• One-hot encoding of labels
• Proper matrix transposition

Visualization

The plot_training_history() function creates dual plots showing:

1. Training Loss: Cross-entropy loss over epochs
2. Accuracy Comparison: Training vs test accuracy curves

File Structure

ffnn.py # Main implementation
├── FFNN # Neural network class
├── load_mnist_data() # Data preprocessing
└── plot_training_history() # Visualization utilities

Educational Value

This implementation is ideal for understanding:

• Forward Propagation: How data flows through network layers
• Backpropagation: Gradient computation and error propagation
• Activation Functions: Impact of different non-linearities
• Weight Initialization: Importance of proper parameter setup
• Mini-batch Training: Efficient stochastic gradient descent

Limitations

• Performance: Slower than optimized frameworks (TensorFlow, PyTorch)
• GPU Support: CPU-only implementation
• Advanced Features: No regularization, dropout, or adaptive optimizers
• Memory: Stores all intermediate values for gradient computation

Contributing

This is an educational implementation. For production use, consider:

• Adding regularization techniques (L1/L2, dropout)
• Implementing adaptive optimizers (Adam, RMSprop)
• Adding batch normalization
• GPU acceleration with CuPy or similar

License

Open source - feel free to use for educational purposes.

---

Built for learning deep learning fundamentals through hands-on implementation.