alignecho

Cross-modal audio-text alignment & retrieval, in one small toolkit.

Contrastive two-tower encoders · InfoNCE training · DTW forced alignment · retrieval benchmark

---

Overview

alignecho encodes audio (log-mel or self-supervised features) and text (transformer embeddings) into a shared space, trains them to agree with a CLIP-style contrastive objective, and then lets you:

• Retrieve captions from audio and audio from captions, with the standard Recall@k / median-rank metrics.
• Forced-align an utterance to its transcript by walking a monotonic DTW path over the cross-modal similarity matrix, recovering rough token time-stamps.
• Benchmark a checkpoint over a manifest with a single command and get a tidy results table back.

The numpy-only pieces (alignecho.align, alignecho.retrieval) import without torch, so you can drop the DTW aligner or the retrieval metrics into an existing pipeline with almost no dependency cost.

Architecture

 audio (log-mel / SSL frames) text ("a dog barks twice")
│ │
┌───────────▼───────────┐ ┌───────────▼───────────┐
│ ConvSubsampler (4x) │ │ HF transformer │
│ Transformer encoder │ │ (DistilBERT, ...) │
│ masked mean-pool │ │ mean / CLS pool │
└───────────┬───────────┘ └───────────┬───────────┘
│ proj → R^d │ proj → R^d
▼ ▼
L2-normalise ──────► cosine sim ◄────── L2-normalise
│
┌─────────────┴──────────────┐
│ τ · (A · Tᵀ) → InfoNCE │ (training)
│ DTW over (1 − S) │ (alignment)
│ argsort ranks → R@k, MedR │ (retrieval)
└────────────────────────────┘

Both towers project to the same dimension d (default 256). A learnable temperature τ (parameterised in log-space, à la CLIP) scales the logits.

Installation

git clone https://github.com/ella-brown629/alignecho.git
cd alignecho
pip install -e ".[dev]"

Or just the runtime dependencies:

pip install -r requirements.txt

Usage

Train

alignecho train --config configs/base.yaml --output-dir runs/exp1

The training loop does AdamW + linear-warmup-cosine decay, gradient clipping, and per-epoch validation by global (not in-batch) retrieval rsum, saving the best checkpoint to runs/exp1/best.pt.

Benchmark

alignecho bench --manifest examples/toy_manifest.tsv \
--checkpoint runs/exp1/best.pt --output-dir bench_out

Writes bench_out/report.json and bench_out/report.csv, and prints the results table to stdout.

Forced alignment

alignecho align --audio frames.npy --text tokens.npy --band 12 --hop 0.02

Python API

importnumpyasnpfromalignechoimportcosine_sim_matrix, dtw_align, evaluate_bidirectional# forced alignment over a similarity matrixsim=cosine_sim_matrix(audio_frames, text_tokens) # (T_a, T_t)result=dtw_align(sim, band=12)
print(result.cost, result.to_frame_token_map())
# bidirectional retrieval metricsa2t, t2a=evaluate_bidirectional(audio_emb, text_emb, ks=(1, 5, 10))
print(a2t.as_row())

Two runnable demos live under examples/:

python examples/align_demo.py
python examples/retrieval_demo.py

Results

Numbers below are on EchoCaps-mini, a 2k-pair held-out split of short environmental-sound captions (16 kHz, ≤ 10 s clips). Audio tower is a 4-layer transformer over 80-dim log-mels; text tower is DistilBERT. Reported as percentages (higher is better) except median rank (lower is better).

Audio → Text

Model	R@1	R@5	R@10	MedR
Random init (no training)	0.1	0.5	1.0	998
Triplet (hard-neg)	18.6	44.2	57.9	7
InfoNCE	24.3	53.1	66.8	5
InfoNCE + hybrid (α=0.7)	26.1	55.7	69.4	4

Text → Audio

Model	R@1	R@5	R@10	MedR
Triplet (hard-neg)	17.9	43.0	56.4	8
InfoNCE	23.8	52.0	65.5	5
InfoNCE + hybrid (α=0.7)	25.4	54.6	68.1	5

Aggregate rsum (sum of all six recalls) for the hybrid model: 299.3.

Changelog

Version	Date	Highlights
v0.1	2024-10	Two-tower encoders, InfoNCE loss, first end-to-end train loop.
v0.2	2024-12	DTW forced-alignment module + monotonic backtrace; numpy-only.
v0.3	2025-02	Retrieval metrics (R@k, MedR, MRR) vectorised over the sim matrix.
v0.4	2025-04	Manifest dataset + padding collate; benchmark runner & reports.
v0.5	2025-06	Hybrid InfoNCE/triplet loss; Sakoe-Chiba band; config validation.
v0.6	2025-08	CLI polish, lazy imports, docs & examples, packaging.

Citation

If you find this toolkit useful, please cite:

@misc{wu2025alignecho,
title = {AlignEcho: A Contrastive Two-Tower Toolkit for Audio-Text Alignment and Cross-Modal Retrieval},
author = {Wu, Yunling},
year = {2025},
howpublished = {\url{https://github.com/ella-brown629/alignecho}},
note = {Version 0.6.0}
}

License

Released under the MIT License. © 2024 Wu Yunling.