Counting push-ups from video.
Given an .mp4 of a single person doing push-ups, output an integer: how many complete
reps were performed. A rep is defined (per the assignment brief) as the motion starting
from full arm extension and ending at the next full arm extension.
The approach here does not look at the video as a video. Every clip is collapsed into a temporal self-similarity matrix (TSM) — a square image encoding how similar each frame is to every other frame — and a small 2D CNN, trained from scratch, classifies that image into a rep count between 1 and 10.
All code lives in Temporal_self_matrix_solution.ipynb. The task specification is in MV_PushUpsLab.pdf.
Counting repetitions is fundamentally a question about periodicity, not about appearance. Who is in the frame, what they're wearing, the lighting, the camera angle — none of it matters. What matters is that the body returns to the same configuration once per rep.
Cosine similarity between every pair of frames captures exactly that. When the subject returns to the top of a push-up, that frame is near-identical to the previous top, so the matrix lights up off the diagonal. Repeat the motion N times and you get N bright blobs along the diagonal with a regular checkerboard of off-diagonal stripes between them. Rep counting becomes a texture-classification problem on a single grayscale image.
Two useful consequences fall out of this:-
- Appearance is discarded for free. The network never sees a person, so it can't overfit to backgrounds, clothing, or gym equipment. It only sees the shape of the motion.
- Variable-length video becomes fixed-size input. An N-frame video produces an N×N matrix, which is bilinearly resized to a fixed 128×128. A 4-second clip and a 40-second clip both arrive at the network as the same tensor shape — no padding, no attention masks, no truncation. Resizing changes the scale of the stripe pattern but not its count, which is the quantity being predicted.
| Path | What it is |
|---|---|
Temporal_self_matrix_solution.ipynb |
Everything: data pipeline, model, training loop, evaluation, interpretability, HF upload |
models/mv-final-assignment.pt |
Same weights under the name used for the HuggingFace upload |
outputs/checkpoints/ |
Training-dynamics, layer-gradient and weight-histogram plots, written every 2 epochs during training |
outputs/cnn_feature_outputs_<video>/ |
Per-layer interpretability output for two example videos (feature maps, filters, Grad-CAM, guided backprop, SmoothGrad) |
MV_PushUpsLab.pdf |
Assignment brief |
video-data/ |
Training videos — not committed |
Note that the notebook itself still writes to ./checkpoints/ and ./cnn_feature_outputs_*/
at the repo root (CHECKPOINT_DIR in cell 0, save_dir in the visualisation cell); the
committed copies were moved under outputs/ afterwards.
Public model: https://huggingface.co/MannSingh/mv-final-assignment
Dependencies
pip3 install torch torchvision opencv-python scikit-learn numpy matplotlib seaborn boto3 kornia torchview huggingface_hubData
The notebook expects a ./video-data/ directory alongside it; there is currently no download cell
Training videos (Google Drive)
Labels are read from the filename prefix, so the naming convention matters:
3_video_20251206_....mp4 means three reps, which maps to class index 2.
For one video, end to end:
- Decode frames with OpenCV —
(N, H, W, 3). - Grayscale + resize each frame to 64×64. Full resolution buys nothing here; global similarity is what's wanted, and small frames keep the pairwise computation cheap.
- Flatten each frame to a 4096-vector, stack into
(N, 4096). - Normalise to
[0, 1]and add1e-6. The epsilon matters: an all-black frame has zero norm, and cosine similarity would divide by it. - Cosine similarity of the stack against itself →
(N, N)self-similarity matrix. - Resize to 128×128 and clip to
[-1, 1]to absorb float error. - Augment (training split only) — see below.
- To tensor, add a channel dimension →
(1, 128, 128). - CNN forward → 10 logits.
argmax + 1→ the predicted rep count.
Steps 1–6 live in SimilarityMatrixDataset._compute_similarity_matrix; the same logic is
available standalone as generate_video_similarity_matrix for producing heatmap figures,
and inline in predict_video_count for single-video inference.
RepetitionCounterNet — 391,370 parameters, trained from scratch. No pretrained backbone,
because the input is a similarity matrix, not a natural image; ImageNet features have
nothing useful to say about it.
| Block | Layers | Output | Intent |
|---|---|---|---|
| 1 | Conv 1→32, BN, ReLU, MaxPool | 32 × 64 × 64 | Edges — the boundaries of the bright similarity blobs |
| 2 | Conv 32→64, BN, ReLU, MaxPool | 64 × 32 × 32 | Blob and grid structure |
| 3 | Conv 64→128, BN, ReLU, MaxPool | 128 × 16 × 16 | Density of the repeating pattern, i.e. its frequency |
| 4 | Conv 128→256, BN, ReLU | 256 × 16 × 16 | Deeper features; no pooling, to keep spatial detail before averaging |
| Head | AdaptiveAvgPool → Linear 256→10 | 10 | Class logits |
The head is global average pooling rather than flatten-and-FC. Flattening 256 × 16 × 16
would need ~650k parameters in the first FC layer alone — more than the entire rest of the
network, on a dataset of 77 videos. GAP also fits the problem: averaging each filter's
response across the whole matrix answers "how much repetition texture is present overall",
which is close to the quantity being counted.
EMDLoss computes the squared L2 distance between the CDF of the predicted softmax and the
CDF of the one-hot target. Cross-entropy treats all mistakes as equally wrong; predicting 9
when the answer is 5 costs the same as predicting 4. But the classes here are ordinal —
they're counts. EMD penalises by how far the prediction is along the class axis, so the
network is pushed toward being off-by-one rather than off-by-four, which is exactly the
gradient signal a counting task wants.
The brief allows augmenting the provided videos but not adding new ones. MatrixAugmenter
augments in matrix space rather than pixel space, which is far cheaper (no re-decoding, no
recomputing similarity) and maps more directly onto the properties that matter:
- Gaussian noise (σ=0.05) — simulates a noisier similarity estimate, e.g. from compression artefacts or camera shake.
- Time-warp zoom/crop (0.8×–1.2×) — rescales the matrix, which is equivalent to the subject performing the same reps faster or slower.
- Random cutout (up to 20×20) — blanks a patch, standing in for occlusion or dropped frames, and forces the network not to rely on any single stretch of the diagonal.
Each is applied with p=0.5, and only to the training split; validation runs on clean matrices.
Measured on Apple Silicon (MPS), 50 epochs.
| Metric | Value |
|---|---|
| Parameters | 391,370 |
| Checkpoint size | 1.5 MB |
| Validation accuracy | 68.75% (11/16) |
| Full-set accuracy | 80.52% (62/77) |
| Forward pass | 1.6–2.1 ms per video |
| End-to-end (decode → prediction) | 0.18–0.33 s per video |
| End-to-end throughput | ~460–800 frames/s |
The gap between the two accuracy figures is the expected one: the full-set number includes the 61 training videos the model has already seen, so 68.75% on held-out data is the honest figure. With only 16 validation samples, that's 11 correct — a single video moves it by 6 percentage points, so it should be read as a rough indicator, not a precise measurement.
Latency is dominated by preprocessing, not the network. The forward pass is ~2 ms; decoding the video and building the similarity matrix is ~100× that.
| Constraint | Status |
|---|---|
| Inference footprint ≤ 12 GB | 1.5 MB of weights; the largest intermediate is the N×N matrix, ~4 MB at the 1,000-frame limit |
| Training footprint ≤ 20 GB | Batch of 8 × 128×128 single-channel inputs — negligible |
| 100 frames in < 60 s | ~0.15 s at the measured rate |
| Training in < 5 h on an L4 | Minutes, not hours — preprocessing 77 short clips plus 50 epochs over a 391k-parameter net |
| No models consumed via API | Nothing pretrained, nothing remote; the network is trained from scratch in the notebook |
| Public model on HuggingFace | https://huggingface.co/MannSingh/mv-final-assignment |
The timing figures above were measured on MPS, not on an L4. The margins are wide enough that the conclusions should hold, but they have not been verified on the grading hardware.
The assignment brief (§1.1) actively encourages the use of AI tools for this coursework, with one explicit caveat (§1.1.1): they may be used to brainstorm the approach and to help write code, but not to write the Project Report.