cTensor

A lightweight neural network library written in C11 for embedded systems.

Overview

cTensor is a compact tensor computation library designed for small client-side devices, such as mobile phones and microcontrollers. The library implements automatic differentiation and dynamic compute graph functionality, allowing for efficient training and deployment of neural networks on resource-constrained devices.

This library was developed as part of GSoC 2025 and has been successfully validated on ARM Cortex-M3 microcontrollers, achieving 90% classification accuracy on the Iris dataset in a bare-metal environment.

Features

Core Infrastructure

• Lightweight C11 Implementation: Minimal dependencies for wide compatibility
• Automatic Differentiation Framework: Complete gradient computation with backward pass
• Dynamic Compute Graph: Efficient computation flow with gradient tracking
• Pool-based Memory Management: Efficient memory allocation system for embedded devices

Tensor Operations

• Basic Arithmetic: add, subtract, multiply, divide, power (both tensor-tensor and tensor-scalar)
• Unary Operations: negation, absolute value, square, reciprocal
• Matrix Operations: matrix multiplication, transpose
• Mathematical Functions: logarithm, exponential, sine, cosine, tangent
• Shape Operations: unsqueeze, detach
• Broadcasting: Element-wise broadcasting for operations on tensors with different shapes

Reduction Operations

• Sum: All elements or along specific dimension
• Mean: All elements or along specific dimension
• Max/Min: All elements or along dimension with indices
• Argmax: Find indices of maximum values

Neural Network Components

• Layers: Linear (fully connected) layer
• Activation Functions: ReLU, Sigmoid, Tanh, ELU, SELU, Softmax
• Loss Functions: Cross-entropy, Softmax Cross-entropy, MSE, MAE, Huber Loss
• Weight Initialization: Glorot/Xavier initialization

Optimizers

• SGD: Stochastic Gradient Descent with momentum
• Adam: Adaptive moment estimation
• RMSProp: Root Mean Square Propagation
• AdaGrad: Adaptive Gradient Algorithm
• Features: Weight decay support for all optimizers

Training Utilities

• Gradient Clipping: By norm, value, range, positive/negative values
• Evaluation Mode: Disable gradient computation for inference
• Dataset Utilities: Normalization, shuffling

Validation

cTensor has been successfully deployed and tested on:

• ARM Cortex-M3 (STM32F103ZE) using Keil MDK simulation
• Task: Neural network classification on Iris dataset
• Result: 90% accuracy matching desktop performance
• Complete validation project:cTensor_Cortex_SIM

Getting Started

Prerequisites

• C Compiler with C11 support (GCC, Clang)
• CMake (3.10+) for build configuration
• Math library (automatically linked on non-Windows systems)

Building with CMake

On Windows:

build.bat

On Linux/macOS:

mkdir -p build &&cd build
cmake ..
cmake --build .cd ..

Building with Direct Compilation

On Linux/macOS:

./build_g.sh

On Windows with GCC:

gcc -std=c11 -Iinclude -O0 -Wfatal-errors -g -DDEBUG -lm src/nn.c src/operator.c src/basic.c src/iris_dataset.c src/context.c src/pool.c src/utils.c src/common/vector.c src/optimizer/sgd.c src2/main.c -o main

and run main.exe from root directory

Testing the Library

cTensor uses a custom test framework. To run the tests:

# Build the test executable with CMake
mkdir -p build &&cd build
cmake ..
cmake --build .# Run the tests
./cten_exe

For detailed testing information, refer to Testing Documentation.

Usage Example

Here's a complete example of training a neural network to predict sine wave values with noise:

#include"cten.h"#include<stdio.h>#include<stdlib.h>#include<math.h>// Define memory poolsenumMemoryPoolIds {
PoolId_Default=0,
PoolId_Model=1,
PoolId_Optimizer=2,
};
// Define the model structuretypedefstruct {
Tensorw1, b1;
Tensorw2, b2;
Tensorw3, b3;
} Model;
// Forward pass for the modelTensorModel_forward(Model*model, Tensorx) {
x=nn_linear(x, model->w1, model->b1);
x=nn_elu(x, 1.0f);
x=nn_linear(x, model->w2, model->b2);
x=nn_elu(x, 1.0f);
x=nn_linear(x, model->w3, model->b3);
returnx;
}
intmain() {
cten_initilize();
// Generate sine wave dataintn_samples=2048;
float*x_data=malloc(n_samples*sizeof(float));
float*y_data=malloc(n_samples*sizeof(float));
// ... (data generation logic) ...// Create model and allocate in its own memory poolModelmodel;
cten_begin_malloc(PoolId_Model);
model.w1=Glorot_init((TensorShape){1, 64}, true);
model.b1=Tensor_zeros((TensorShape){1, 64}, true);
model.w2=Glorot_init((TensorShape){64, 32}, true);
model.b2=Tensor_zeros((TensorShape){1, 32}, true);
model.w3=Glorot_init((TensorShape){32, 1}, true);
model.b3=Tensor_zeros((TensorShape){1, 1}, true);
cten_end_malloc();
// Create optimizerfloatlearning_rate=0.01f;
cten_begin_malloc(PoolId_Optimizer);
optim_adam*optimizer=optim_adam_new(6, (Tensor*)&model, learning_rate, 0.9f, 0.999f, 1e-8f, 0.0f);
cten_end_malloc();
// Training loopintbatch_size=64;
for (intepoch=0; epoch<200; epoch++) {
// ... (training logic with batching, loss calculation, backpropagation) ...cten_begin_malloc(PoolId_Default); // for temporary tensors in each step// ... create input and y_true tensors ...optim_adam_zerograd(optimizer);
Tensory_pred=Model_forward(&model, input);
// Combined LossTensorhuber=nn_huber_loss(y_true, y_pred, 1.0f);
Tensormae=nn_mae_loss(y_true, y_pred);
Tensorloss=Tensor_add(huber, Tensor_mulf(mae, 0.3f));
Tensor_backward(loss, Tensor_ones((TensorShape){1}, false));
// Gradient Clippingcten_clip_grad_norm((Tensor*)&model, 6, 5.0f);
optim_adam_step(optimizer);
cten_end_malloc();
cten_free(PoolId_Default); // free temporary tensors
}
// Evaluate modelcten_begin_eval();
// ... (evaluation logic) ...cten_end_eval();
// Free memory poolscten_free(PoolId_Optimizer);
cten_free(PoolId_Model); cten_finalize();
return0;
}

API Overview

Tensor Creation and Management

// Basic tensor creationTensorTensor_new(TensorShapeshape, boolrequires_grad);
TensorTensor_zeros(TensorShapeshape, boolrequires_grad);
TensorTensor_ones(TensorShapeshape, boolrequires_grad);
// Tensor manipulationTensorTensor_transpose(Tensorself);
TensorTensor_detach(Tensorself);
TensorTensor_unsqueeze(Tensorself, intdim);
// Element accessfloatTensor_get(Tensorself, inti, intj, intk, intl);
voidTensor_set(Tensorself, inti, intj, intk, intl, floatvalue);
// BackpropagationvoidTensor_backward(Tensorself, Tensorgrad);

Basic Operations

// Element-wise operations with tensorsTensorTensor_add(Tensorself, Tensorother);
TensorTensor_sub(Tensorself, Tensorother);
TensorTensor_mul(Tensorself, Tensorother);
TensorTensor_div(Tensorself, Tensorother);
TensorTensor_pow(Tensorself, Tensorother);
// Element-wise operations with scalarsTensorTensor_addf(Tensorself, floatother);
TensorTensor_subf(Tensorself, floatother);
TensorTensor_mulf(Tensorself, floatother);
TensorTensor_divf(Tensorself, floatother);
TensorTensor_powf(Tensorself, floatother);
// Matrix operationsTensorTensor_matmul(Tensorself, Tensorother);
// Unary operationsTensorTensor_neg(Tensorself);
TensorTensor_abs(Tensorself);
TensorTensor_square(Tensorself);
TensorTensor_reciprocal(Tensorself);

Mathematical Functions

// Logarithmic and exponentialTensornn_log(Tensorself);
Tensornn_exp(Tensorself);
// Trigonometric functionsTensornn_sin(Tensorself);
Tensornn_cos(Tensorself);
Tensornn_tan(Tensorself);

Reduction Operations

// Reduction operations (with macro dispatch)TensorTensor_sum(Tensorself); // Sum all elementsTensorTensor_sum(Tensorself, intdim); // Sum along dimensionTensorTensor_mean(Tensorself); // Mean of all elementsTensorTensor_mean(Tensorself, intdim); // Mean along dimensionTensorTensor_max(Tensorself); // Max of all elementsTensorMaxMinResultTensor_max(Tensorself, intdim); // Max along dimensionTensorTensor_min(Tensorself); // Min of all elementsTensorMaxMinResultTensor_min(Tensorself, intdim); // Min along dimension// Argmax operationvoidTensor_argmax(Tensorself, int*out);

Neural Network Functions

// Neural network layersTensornn_linear(Tensorinput, Tensorweight, Tensorbias);
// Activation functionsTensornn_relu(Tensorinput);
Tensornn_sigmoid(Tensorinput);
Tensornn_tanh(Tensorinput);
Tensornn_elu(Tensorself, floatalpha);
Tensornn_selu(Tensorself);
Tensornn_softmax(Tensorinput, intdim);
// Loss functionsTensornn_crossentropy(Tensory_true, Tensory_pred);
Tensornn_softmax_crossentropy(Tensory_true, Tensorlogits);
Tensornn_mse_loss(Tensory_true, Tensory_pred);
Tensornn_mae_loss(Tensory_true, Tensory_pred);
Tensornn_huber_loss(Tensory_true, Tensory_pred, floatdelta);
// Weight initializationTensorGlorot_init(TensorShapeshape, boolrequires_grad);

Optimizers

// SGD Optimizeroptim_sgd*optim_sgd_new(intn_params, Tensor*params, floatweight_decay);
voidoptim_sgd_config(optim_sgd*self, floatlr, floatmomentum);
voidoptim_sgd_zerograd(optim_sgd*self);
voidoptim_sgd_step(optim_sgd*self);
// Adam Optimizeroptim_adam*optim_adam_new(intn_params, Tensor*params, floatlr, floatβ1, floatβ2, floatε, floatweight_decay);
voidoptim_adam_zerograd(optim_adam*self);
voidoptim_adam_step(optim_adam*self);
// RMSProp Optimizeroptim_rmsprop*optim_rmsprop_new(intn_params, Tensor*params, floatlr, floatβ, floatε, floatweight_decay);
voidoptim_rmsprop_zerograd(optim_rmsprop*self);
voidoptim_rmsprop_step(optim_rmsprop*self);
// AdaGrad Optimizeroptim_adagrad*optim_adagrad_new(intn_params, Tensor*params, floatlr, floatε, floatweight_decay);
voidoptim_adagrad_zerograd(optim_adagrad*self);
voidoptim_adagrad_step(optim_adagrad*self);

Gradient Clipping

// Gradient clipping functionsvoidcten_clip_grad_norm(Tensor*params, intn_params, floatmax_norm);
voidcten_clip_grad_value(Tensor*params, intn_params, floatmax_value);
voidcten_clip_grad_value_range(Tensor*params, intn_params, floatmin_value, floatmax_value);
voidcten_clip_grad_positive(Tensor*params, intn_params, floatmax_value);
voidcten_clip_grad_negative(Tensor*params, intn_params, floatmin_value);

Utility Functions

// TensorShape utilitiesintTensorShape_numel(TensorShapeshape);
intTensorShape_dim(TensorShapeshape);
intTensorShape_asdim(TensorShapeshape, intdim);
intTensorShape_tostring(TensorShapeshape, char*buf, intsize);
// Dataset utilitiesintload_iris_dataset(constfloat (**X)[4], constint**y);
voidTensor_normalize_dataset(constfloat (*X)[4], float (*X_norm)[4], intn_samples, intn_train_samples, intn_features);
voidTensor_shuffle_dataset(constfloat (*X)[4], constint*y, float (*X_shuffled)[4], int*y_shuffled, intn_samples, intn_features);
// Evaluation modevoidcten_begin_eval();
boolcten_is_eval();
voidcten_end_eval();
// Broadcastingboolcten_elemwise_broadcast(Tensor*a, Tensor*b);
Tensorreduce_gradient_for_broadcasting(Tensorgrad, TensorShapeoriginal_shape, TensorShapebroadcasted_shape);

Memory Management

cTensor uses a pool-based memory management system to efficiently handle tensor allocations:

voidcten_initilize();
voidcten_finalize();
voidcten_begin_malloc(PoolIdid);
voidcten_end_malloc();
voidcten_free(PoolIdid);

Project Structure

cTensor/
├── include/ # Header files defining the API
│ └── cten.h # Complete API header
├── src/ # Core implementation files
│ ├── basic.c # Basic tensor operations
│ ├── nn.c # Neural network primitives
│ ├── operator.c # Mathematical operators
│ ├── context.c # Memory management
│ ├── utils.c # Utility functions
│ ├── optimizer/ # Optimizer implementations
│ └── ...
├── src2/ # Example applications
│ └── main.c # Sine regression example
└── tests/ # Test suite

Implemented Features Summary

Category	Components	Status
Core Structs	Tensor, GradNode, TensorMaxMinResult	✅
Autograd	Tensor_backward, requires_grad, detach	✅
Tensor Creation	Tensor_new, zeros, ones, Glorot_init	✅
Binary Operations	add, sub, mul, div, pow, matmul	✅
Unary Operations	neg, abs, square, reciprocal	✅
Math Functions	log, exp, sin, cos, tan	✅
Aggregations	sum, mean, max, min (with indices)	✅
Search/Sort	argmax	✅
Shape Operations	transpose, unsqueeze	✅
NN Layers	nn_linear	✅
Activations	ReLU, Sigmoid, Tanh, ELU, SELU, Softmax	✅
Loss Functions	CrossEntropy, MSE, MAE, Huber	✅
Optimizers	SGD, Adam, RMSProp, AdaGrad	✅
Training Utils	Gradient Clipping, Evaluation Mode, Weight Decay	✅

Contributing

Contributions to cTensor are welcome! Key areas for contribution include:

1. Performance Optimization: Benchmarking and SIMD implementations
2. Advanced Layers: Convolutional and recurrent neural network layers
3. Documentation: Examples, tutorials, and API documentation improvements
4. Testing: Expanding test coverage and validation on different platforms

GSoC 2025 Acknowledgments

This project was developed during Google Summer of Code 2025 by Advait Gaur under the mentorship of PrimedErwin, Anurag Bhat, and blueloveTH. The project successfully transformed cTensor from a basic prototype into a functional deep learning framework suitable for embedded applications.

License

This project is licensed under the MIT License - see the LICENSE file for details.