loading
loading
Idiap's Abstract Compression Halves Audio Tokens Without Losing Conversation Context Idiap and Uniphore researchers show that squeezing prior conversational audio into a handful of latent tokens recovers most context-aware ASR gains while slashing prompt length. - New paper proposes Abstract Compression for conversational context in LLM-based ASR. - Prior-turn audio is distilled into K learned latent tokens; transcripts kept as text. - Built on Phi-4-Multimodal with Perceiver-style cross-attention and two-stage training. - On WoW out-of-domain set, Bias-WER drops from 25.6 to 24.5 vs 23.3 for raw context. - Benefits saturate around five context turns; K=16 latents work best. - Accepted to IEEE SLT 2026; evaluated only on Phi-4-Multimodal backbone. Speech recognizers often process each utterance without the clues established earlier in a conversation, making uncommon names, places, and domain terms easier to miss. A new paper from Idiap Research Institute, EPFL, Uniphore, and collaborators presents Abstract Compression, a method that preserves part of the accuracy gain from conversational context while reducing the number of audio tokens sent to a multimodal large language model. Why conversation history bloats the cache Models such as Phi-4-Multimodal convert audio into long sequences of token embeddings and pass them to a text decoder. Adding earlier turns gives the decoder lexical and acoustic context, but every extra token expands the key-value cache, which stores attention state during generation. Longer sequences therefore raise memory use and computational cost. Naive multi-turn prompting at inference time performed worse than the single-turn baseline. The model began using conversational history effectively after supervised fine-tuning on multi-turn examples, with the largest gains appearing in Bias-WER. This metric calculates word error rate within annotated entity spans, including names and locations. K tokens replace each past audio stream Abstract Compression places a learned cross-attention resampler between previous audio turns and the language model. Learned query matrices associated with each relative turn index compress a variable-length sequence of audio embeddings into exactly This story is for Pro members You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
document fetched 2026-10-23 22:18