# The Computer Science Behind Translating Speech in Over 100 Languages
In a world that is becoming progressively interconnected, the capability to communicate effortlessly across linguistic divides is increasingly vital. The aspiration for a “universal translator”—a device that can translate speech in real-time among various languages—has long been a mainstay of science fiction, as seen in *Star Trek* and *The Hitchhiker’s Guide to the Galaxy*. Today, developments in artificial intelligence (AI) are moving this dream toward fruition. One of the most ambitious initiatives in this domain is Meta’s Seamless, a cutting-edge translation system with the potential to transform how we comprehend and engage with each other.
This article explores the computer science involved in translating speech from 100 source languages, the obstacles encountered in constructing such a system, and the creative solutions enabling its realization.
—
## The Vision: Instant, Expressive Translation
In 2023, researchers at Meta spoke with native Spanish and Mandarin speakers residing in the United States who relied heavily on translation tools in everyday situations. These individuals expressed a wish for an AI-driven translator that could not only offer real-time speech translation but also maintain the speaker’s voice, tone, and emotional nuances. This input motivated Meta to gather a team of over 50 specialists to create Seamless, an innovative translation framework that can interpret speech across 36 languages and process text in even more.
—
## The Challenges: Lack of Data and Complexity
### 1. **The Issue of Language Data**
Currently, AI translation systems are mainly text-centric, owing to the wealth of digitized text available on the internet. Organizations such as the United Nations and the European Parliament have developed extensive databases of aligned texts—documents translated into multiple languages by professional translators. These repositories have played a crucial role in training AI models for text-to-text translation.
Nevertheless, two major challenges emerge when transitioning from text to speech translation:
– **Formality Bias**: Text-based training often includes formal documents, which can result in translations that sound overly formal or robotic, even when translating informal or humorous speech.
– **Lack of Aligned Audio Data**: While aligned text corpora are plentiful, aligned audio data—spoken sentences accompanied by their translations—are significantly harder to acquire, particularly for low-resource languages like Zulu or Icelandic.
### 2. **Cascading Errors in Speech Translation**
Conventional speech translation systems frequently utilize a cascading method: speech is initially converted to text (speech-to-text), then translated into the desired language (text-to-text), and finally synthesized back into speech (text-to-speech). Each transition adds errors, compromising overall effectiveness and rendering real-time translation nearly unfeasible.
—
## The Breakthrough: A Universal Language in Vector Space
To tackle these difficulties, the Seamless team adopted an innovative idea first suggested by mathematician Warren Weaver in 1949: the notion of a “universal language” that underpins all human interaction. In the realm of AI, this universal language is not linguistic but mathematical—multidimensional vectors.
### **How Vectorization Functions**
Machines do not “grasp” words in the same way humans do. Instead, they depict words as numerical vectors, referred to as word embeddings. These embeddings represent points in a high-dimensional space, where words with akin meanings (e.g., “tea” and “coffee”) are positioned closely together. When aligned text in two languages undergoes vectorization, the resulting vector spaces can be correspondingly mapped, facilitating translation.
### **SONAR: A Multimodal Embedding Space**
The Seamless team advanced this idea by developing SONAR (Sentence-level Multimodal and Language-Agnostic Representations), a consolidated embedding space for both text and speech. Here’s the mechanism:
1. **Unified Vectorization**: Text and speech data from various languages are vectorized as if they belong to a singular language, clustering into a common embedding space.
2. **Automatic Alignment**: Sentences with analogous meanings—whether in text or speech—naturally group within the vector space. This circumvents the necessity for manually aligned data, allowing for the integration of low-resource languages.
3. **Metadata Encoding**: Each embedding incorporates metadata that identifies its source language and modality (text or speech).
This methodology enabled the team to produce extensive amounts of automatically aligned data, encompassing millions of text pairs and thousands of hours of transcribed audio, even for languages with minimal resources.
—
## The Result: SEAMLESSM4T
The data generated via SONAR was utilized to train SEAMLESSM4T, the most sophisticated model within the Seamless framework. Here’s what it is capable of:
– **Speech-to-Speech Translation**: Convert speech from 101 source languages into 36 target languages.
– **Text-to-Text Translation**: Translate written text among over 100 languages.
– **Speech-to-Text Translation**: Transform spoken language into written text in 96 languages.
– **Text
