← sources

I took EmbeddingGemma-2 (577M) from Rank #26 to #1 on the MTEB Audio leaderboard (MAEB), beating 9B models by adding a 2.4 MB PCA head with zero training

source: reddit (reddit:LocalLLaMA)published 2026-10-28 13:23ingested 2026-10-29 02:42

retained text

Model & weights: [BrunoHays/embeddinggemma-2-nullspace-audio](https://huggingface.co/BrunoHays/embeddinggemma-2-nullspace-audio)


MAEB (Massive Audio Embedding Benchmark) is the main [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) benchmark for audio and text-to-audio embeddings, currently dominated by 8B-9B multimodal models.


I took Google's edge-sized `google/embeddinggemma-2` (577M params, 768d vectors) and boosted its MAEB score from **49.3% to 54.4% (+5.1 points)**, jumping from **Rank #26 to #1** on `MAEB audio-only` (and **#16 to #1** on full `MAEB`) without training a single weight. The whole addition is a 2.37 MB safetensors file of frozen PCA matrices.


### Why this matters for local / edge setups


At 577M parameters and 768d vectors (15x smaller than 9B Omni models), it runs easily on a laptop or edge device.


Standard text-aligned audio models are great at matching spoken words to text, but terrible at grouping audio by voice, emotion, or sound similarity because the final layers keep *what words were said* and drop *how the sound was made*.


By pulling features from earlier layers of the frozen audio encoder, this model keeps the base model's 100+ language text-to-audio search while fixing audio-to-audio tasks:


* Unsupervised voice/gender clustering: **0.05% -> 63.1%**

* Artist / song audio-to-audio search: **55.2% -> 93.8%**

* Vocal emotion classification: **19.7% -> 40.2%**


### How it works


* **Early layers still know acoustics:** Speech representation work ([Pasad et al.](https://arxiv.org/abs/2107.04734); [Bouziane, Mdhaffar & Estève](https://arxiv.org/abs/2610.21941)) shows that early/middle layers keep speaker, pitch, and emotion traits that disappear in the final layers. In EmbeddingGemma-2, Layer 2 and Layer 6 still hold those signals.

* **256 empty directions in the 768d vector:** EmbeddingGemma-2's text backbone is actually 512d, projected up to 768d at the end by a `768 x 512` linear matrix. That leaves **256 perpendicular directions (the null space)** that text vectors never touch.

* **Filling the null space with PCA:** I compressed Layer 2 and Layer 6 down to 256 dimensions with PCA (fitted on 1,400 external audio clips) and injected them into those 256 unused directions.

* **Caveats:** Because there is no retraining, audio embeddings need all 768 dimensions (no Matryoshka truncation to 512/256/128), and adding unweighted acoustic features hurts some purely spoken-intent tasks (e.g. `MInDS14` drops `97.6% -> 88.3%`).


It shows how much untapped signal sits inside today's overtrained models, even with zero training.


Full breakdown and usage in the [Hugging Face model card](https://huggingface.co/BrunoHays/embeddinggemma-2-nullspace-audio).

document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhny7x/i_took_embeddinggemma2_577m_from_rank_26_to_1_on