Mul-Net

Paper: "Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads" (under review)

Overview

A multiplicative dual-encoder head predicts an interaction through an inner product of two learned encoders,

F(u, v) ≈ ⟨ f_θ(u), g_φ(v) ⟩,

a form shared by two-tower retrieval, CLIP, DeepONet, UVFA/BVN value functions, factorization machines, DistMult/ComplEx, and linear attention. The paper studies this whole family as a single object: an adaptive rank-d approximator of the low-interaction-rank function classM_d, where the interaction rank is the number of nonzero singular values of the interaction operator T_F (its interaction spectrum{σ_k}). Three questions are answered on that one object:

1. Approximation / unification — one class and one spectrum unify the family; the best achievable error is the spectral tail Σ_{k>d} σ_k².
2. Identifiability as gauge-fixing — the head has a GL_d gauge symmetry; normalizations are sections of that gauge orbit. Whitening plus a spectral gap pins the interaction modes uniquely up to permutation and sign.
3. Sample complexity and when not to use — excess risk scales with the marginal encoder complexities, not the product space; a flat spectrum makes the head provably lose to early interaction.

This repository reproduces every experiment in the paper. Each experiment lives in its own self-contained folder under experiments/, with its own README, its own results/ (precomputed JSON/CSV) and figures/ (precomputed PNGs) so you can inspect the numbers without running anything, then re-run to regenerate them.

Repository layout

.
├── README.md (this file)
├── LICENSE (MIT)
├── requirements.txt
├── common/ shared numpy utilities used by the synthetic studies
│ ├── spectral.py known-singular-system targets (shifted Legendre) + spectra
│ ├── numpy_train.py LinearDualHead + Adam training with analytic gradients
│ ├── normalizations.py the seven-normalization "zoo" (gauge-orbit sections)
│ ├── metrics.py mode alignment, gauge conditioning, cross-seed distance
│ └── models.py torch DualEncoder / JointMLP / CrossAttn heads (tier-2)
└── experiments/
├── 01_normalization_taxonomy/ gauge taxonomy (the centerpiece)
├── 02_gap_law/ the Δ·√n unifying law
├── 03_sample_complexity/ the σ²d(p+q)/n minimax rate
├── 04_negative_result/ flat-spectrum separation (when NOT to use)
├── 05_deeponet/ operator learning, whitening recovers eigenbasis
├── 06_flat_spectrum_anchor/ trained flat-spectrum usability anchor
└── 07_clip/ CLIP multimodal interpretability

Install

python -m venv .venv &&source .venv/bin/activate # or your preferred env
pip install -r requirements.txt

Experiments 01–04, 06, and the sandbox math-cores/figures of 05 and 07 need only numpy + matplotlib. The two real-data experiments need extra packages:

Experiment	Extra dependency	Why
05_deeponet/tier2_deeponet.py	torch (scipy for the 2D Darcy operator)	trains real DeepONets
07_clip/tier2B_clip.py	torch, open_clip_torch, torchvision	pulls real CLIP embeddings

Everything else in those two folders (the numpy math-core and the figure scripts that read the committed JSON) runs without torch.

Quick start

Each folder is independent. From the repo root, for example:

cd experiments/01_normalization_taxonomy && python exp1_taxonomy.py

See each experiment's own README.md for the exact commands, expected numbers, and which paper claim it validates.

Experiment map

Folder	Claim validated
01_normalization_taxonomy	Only whitening pins modes: alignment 1.000, cross-seed drift 0.000; looser gauges stall at 0.5–0.65
02_gap_law	Mode error ∝ 1/Δ and ∝ n^{−1/2}; the two axes collapse to err·Δ·√n ≈ 2.48 (CV 0.12)
03_sample_complexity	Excess risk ≍ σ²d(p+q)/n: log-log slopes −1.03 / +0.89 / +1.24 vs theory −1 / +1 / +1 (Marchenko–Pastur finite-n correction)
04_negative_result	Boolean equality needs d ≈ 0.9·2^m (late) vs 2m+1 params (early); narrowband d_ε ∝ 1/h
05_deeponet	Post-hoc whitening recovers the analytic Fourier trunk basis (0.5 → 0.92); rank selection tracks exp vs poly decay
06_flat_spectrum_anchor	Trained head hugs the 1 − d/m floor on a flat spectrum; equal-budget MLP escapes to ~0.013
07_clip	Interaction spectrum is the shared invariant across backbones (corr 0.99); whitened modes read as concept axes

Reproducing from committed outputs

Every folder ships its results/*.json|csv and figures/*.png already populated. To regenerate a figure from committed results without re-running the experiment, run that folder's make_*.py script (where present). To regenerate the numbers themselves, run the main experiment script.

Citation