DeepLearnToolbox

A Matlab toolbox for Deep Learning.

Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data. It is inspired by the human brain's apparent deep (layered, hierarchical) architecture. A good overview of the theory of Deep Learning theory is Learning Deep Architectures for AI

For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.

• The Next Generation of Neural Networks (Hinton, 2007)
• Recent Developments in Deep Learning (Hinton, 2010)
• Unsupervised Feature Learning and Deep Learning (Ng, 2011)

If you use this toolbox in your research please cite Prediction as a candidate for learning deep hierarchical models of data

@MASTERSTHESIS\{IMM2012-06284,
author = "R. B. Palm",
title = "Prediction as a candidate for learning deep hierarchical models of data",
year = "2012",
}

Contact: rasmusbergpalm at gmail dot com

Directories included in the toolbox

NN/ - A library for Feedforward Backpropagation Neural Networks

CNN/ - A library for Convolutional Neural Networks

DBN/ - A library for Deep Belief Networks

SAE/ - A library for Stacked Auto-Encoders

CAE/ - A library for Convolutional Auto-Encoders

util/ - Utility functions used by the libraries

data/ - Data used by the examples

tests/ - unit tests to verify toolbox is working

For references on each library check REFS.md

Setup

1. Download.
2. addpath(genpath('DeepLearnToolbox'));

Everything is work in progress

Example: Deep Belief Network

functiontest_example_DBNloadmnist_uint8;
train_x = double(train_x) /255;
test_x = double(test_x) /255;
train_y = double(train_y);
test_y = double(test_y);
%%ex1 train a 100 hidden unit RBM and visualize its weights
rng(0);
dbn.sizes = [100];
opts.numepochs =1;
opts.batchsize =100;
opts.momentum =0;
opts.alpha =1;
dbn = dbnsetup(dbn, train_x, opts);
dbn = dbntrain(dbn, train_x, opts);
figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%%ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NN
rng(0);
%train dbn
dbn.sizes = [100100];
opts.numepochs =1;
opts.batchsize =100;
opts.momentum =0;
opts.alpha =1;
dbn = dbnsetup(dbn, train_x, opts);
dbn = dbntrain(dbn, train_x, opts);
%unfold dbn to nn
nn = dbnunfoldtonn(dbn, 10);
nn.activation_function ='sigm';
%train nn
opts.numepochs =1;
opts.batchsize =100;
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.10, 'Too big error');

Example: Stacked Auto-Encoders

functiontest_example_SAEloadmnist_uint8;
train_x = double(train_x)/255;
test_x = double(test_x)/255;
train_y = double(train_y);
test_y = double(test_y);
%%ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)
rng(0);
sae = saesetup([784100]);
sae.ae{1}.activation_function ='sigm';
sae.ae{1}.learningRate =1;
sae.ae{1}.inputZeroMaskedFraction =0.5;
opts.numepochs =1;
opts.batchsize =100;
sae = saetrain(sae, train_x, opts);
visualize(sae.ae{1}.W{1}(:,2:end)')
% Use the SDAE to initialize a FFNN
nn = nnsetup([78410010]);
nn.activation_function ='sigm';
nn.learningRate =1;
nn.W{1} = sae.ae{1}.W{1};
% Train the FFNN
opts.numepochs =1;
opts.batchsize =100;
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.16, 'Too big error');
%%ex2 train a 100-100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)
rng(0);
sae = saesetup([784100100]);
sae.ae{1}.activation_function ='sigm';
sae.ae{1}.learningRate =1;
sae.ae{1}.inputZeroMaskedFraction =0.5;
sae.ae{2}.activation_function ='sigm';
sae.ae{2}.learningRate =1;
sae.ae{2}.inputZeroMaskedFraction =0.5;
opts.numepochs =1;
opts.batchsize =100;
sae = saetrain(sae, train_x, opts);
visualize(sae.ae{1}.W{1}(:,2:end)')
% Use the SDAE to initialize a FFNN
nn = nnsetup([78410010010]);
nn.activation_function ='sigm';
nn.learningRate =1;
%add pretrained weights
nn.W{1} = sae.ae{1}.W{1};
nn.W{2} = sae.ae{2}.W{1};
% Train the FFNN
opts.numepochs =1;
opts.batchsize =100;
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.1, 'Too big error');

Example: Convolutional Neural Nets

functiontest_example_CNNloadmnist_uint8;
train_x = double(reshape(train_x',28,28,60000))/255;
test_x = double(reshape(test_x',28,28,10000))/255;
train_y = double(train_y');
test_y = double(test_y');
%%ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% error
rng(0)
cnn.layers = {
struct('type', 'i') %input layer
struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer
struct('type', 's', 'scale', 2) %sub sampling layer
struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer
struct('type', 's', 'scale', 2) %subsampling layer
};
cnn = cnnsetup(cnn, train_x, train_y);
opts.alpha =1;
opts.batchsize =50;
opts.numepochs =1;
cnn = cnntrain(cnn, train_x, train_y, opts);
[er, bad] = cnntest(cnn, test_x, test_y);
%plot mean squared errorfigure; plot(cnn.rL);
assert(er<0.12, 'Too big error');

Example: Neural Networks

functiontest_example_NNloadmnist_uint8;
train_x = double(train_x) /255;
test_x = double(test_x) /255;
train_y = double(train_y);
test_y = double(test_y);
% normalize
[train_x, mu, sigma] = zscore(train_x);
test_x = normalize(test_x, mu, sigma);
%%ex1 vanilla neural net
rng(0);
nn = nnsetup([78410010]);
opts.numepochs =1; % Number of full sweeps through data
opts.batchsize =100; % Take a mean gradient step over this many samples
[nn, L] = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.08, 'Too big error');
% Make an artificial one and verify that we can predict it
x = zeros(1,28,28);
x(:, 14:15, 6:22) =1;
x = reshape(x,1,28^2);
figure; visualize(x');
predicted = nnpredict(nn,x)-1;
assert(predicted==1);
%%ex2 neural net with L2 weight decay
rng(0);
nn = nnsetup([78410010]);
nn.weightPenaltyL2 =1e-4; % L2 weight decay
opts.numepochs =1; % Number of full sweeps through data
opts.batchsize =100; % Take a mean gradient step over this many samples
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.1, 'Too big error');
%%ex3 neural net with dropout
rng(0);
nn = nnsetup([78410010]);
nn.dropoutFraction =0.5; % Dropout fraction 
opts.numepochs =1; % Number of full sweeps through data
opts.batchsize =100; % Take a mean gradient step over this many samples
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.1, 'Too big error');
%%ex4 neural net with sigmoid activation function
rng(0);
nn = nnsetup([78410010]);
nn.activation_function ='sigm'; % Sigmoid activation function
nn.learningRate =1; % Sigm require a lower learning rate
opts.numepochs =1; % Number of full sweeps through data
opts.batchsize =100; % Take a mean gradient step over this many samples
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.1, 'Too big error');
%%ex5 plotting functionality
rng(0);
nn = nnsetup([7842010]);
opts.numepochs =5; % Number of full sweeps through data
nn.output ='softmax'; % use softmax output
opts.batchsize =1000; % Take a mean gradient step over this many samples
opts.plot =1; % enable plotting
nn = nntrain(nn, train_x, train_y, opts);
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.1, 'Too big error');
%%ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation data
vx = train_x(1:10000,:);
tx = train_x(10001:end,:);
vy = train_y(1:10000,:);
ty = train_y(10001:end,:);
rng(0);
nn = nnsetup([7842010]); nn.output ='softmax'; % use softmax output
opts.numepochs =5; % Number of full sweeps through data
opts.batchsize =1000; % Take a mean gradient step over this many samples
opts.plot =1; % enable plotting
nn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)
[er, bad] = nntest(nn, test_x, test_y);
assert(er<0.1, 'Too big error');