loading
loading
Tencent's Hy-SimulMT Translates Chinese and English Before Speakers Finish Talking Tencent open-sourced a 30B MoE model for live Chinese-English speech interpretation, hitting roughly 3-second latency with real-time voice cloning. - Tencent open-sourced Hy-SimulMT, a real-time Chinese-English speech-to-speech interpretation model with voice cloning. - Hits ~3 second StreamLAAL on long-form audio, lowest among compared open and commercial systems. - GEMBA quality beats four of five commercial APIs plus every open-source baseline tested. - Six variants on Hugging Face: 30B MoE with 3B active, in FP8 and GGUF formats. - Ships SimulInfer inference server with WebSocket, web UI, and optional TTS, plus SimulTrain for fine-tuning. - Caveats: Chinese-English only, external TTS needed for voice, llama.cpp support not yet merged upstream. Tencent Hunyuan has released Hy-SimulMT, a simultaneous translation model for live Chinese-English speech. The public release includes model weights, quantized variants, an inference server, a web interface, and continued-training scripts. Hy-SimulMT processes incoming audio continuously and emits translated text before the speaker finishes a sentence. Its server can also send that text to an external text-to-speech service for synthesized speech and voice cloning. That separation matters for deployment: the model handles streaming translation, while a separately configured TTS system produces the target-language voice. Translation starts before the sentence ends Simultaneous translation balances context against delay. Waiting provides more linguistic context, while earlier output increases the chance of revisions when later words change a clause’s meaning. Tencent evaluates that balance with StreamLAAL, where lower values indicate less streaming lag, and GEMBA, where higher scores indicate better translation quality. Tencent also reports a matched evaluation using 18 Chinese and English recordings, each about six minutes long, and six experienced interpreters. Three interpreters held CATTI Level 1 certificates and three held Level 2 certificates. The company says Hy-SimulMT reached comparable quality and usually responded faster. The human comparison remains narrow and vendor-run, with no confidence intervals or detailed per-domain results in the supplied summary. Tencent dates the collected recordings from September 2026 onward; that date should be checked against the release timeline. The study does not establish equivalent performance across medical, legal, technical, or noisy conversational speech. Six checkpoints cover two output modes The model collection contains six variants of a 30-billion-parameter mixture-of-experts model. About 3 billion parameters are active for each token, reducing compute relative to a dense 30B model, although serving still requires enough memory for the selected weight format. Official llama.cpp support has yet to merge, so the GGUF packages require the project’s current integration path or a compatible build. Actual VRAM use, throughput, and real-time factor will depend on quantization, context length, audio duration, concurrency, and GPU architecture. The repository also provides SimulInfer, a web and WebSocket inference service with optional TTS integration, and SimulTrain, scripts for continued training from an existing checkpoint. The web interface accepts microphone audio and uploaded files, while the API supports streaming translation clients. The bundled deployment script starts vLLM and the web/API service on port 8080: git clone https://github.com/Tencent-Hunyuan/Hy-SimulMT.git cd Hy-SimulMT/SimulInfer bash deploy.sh --install-server --model tencent/Hy-SimulMT-30B-A3B-Interleave Once running, clients can stream files through a curl endpoint or use the supplied Python client. When TTS is configured, the client can write synthesized speech to a WAV file. Internet-facing deployments should add authentication, TLS, request limits, logging controls, and isolation appropriate to the audio being processed. Cloned speech runs through a separate service Hy-SimulMT’s application layer can preserve a speaker’s vocal characteristics in translated speech, which helps listeners distinguish participants in interviews, meetings, and panel discussions. Tencent’s English-to-Chinese demo shows three speakers retaining distinct voices through translation. The release summary provides no objective speaker-similarity score, and voice quality will depend on the chosen TTS service. That service also adds latency, infrastructure cost, and another failure point. Deployments that clone voices should obtain speaker consent and define retention policies for source audio, reference samples, transcripts, and synthesized output. Early output brings visible revisions Wang Weiwei, identified as a vice dean of an interpreting school, described the system as beginning speech early to control latency and correcting itself afterward. The resulting backtracking resembles an interpreter’s spoken self-correction when later context changes the intended meaning. Client applications should treat partial transcripts as mutable, separate interim output from finalized segments, and preserve correction events for debugging. Spoken output requires additional care because previously synthesized audio cannot be withdrawn; the TTS layer may need to voice a correction explicitly or delay synthesis until a segment reaches an acceptable confidence threshold. Production fit depends on language, risk, and hardware Likely uses include internal bilingual meetings, livestreams, conference talks, interviews, lectures, and accessibility tools where roughly three seconds of translation lag is acceptable. Legal testimony, medical consent, emergency response, and other high-consequence settings still require qualified interpreters and accountable review procedures. - Language coverage: The released model supports Chinese and English only. - Hardware: FP8 and GGUF reduce memory requirements, but the 30B MoE architecture still needs benchmarking on the target machine. - Runtime support: Official llama.cpp integration remains pending. - Speech generation: TTS and voice cloning require an external service. - Quality variation: CoVoST2 results show that Hy-SimulMT does not lead every short-form benchmark. - Licensing: Commercial users should review the repository, model-card, dependency, and TTS-provider terms before deployment. Teams replacing an ASR, machine-translation, and TTS cascade should test Hy-SimulMT on representative audio from their own domain. A useful evaluation should measure first-output delay, StreamLAAL, final translation quality, correction frequency, TTS latency, speaker similarity, GPU memory, throughput, and cost under realistic concurrency.
document fetched 2026-10-27 02:55