Real-time Hindi → English Speech-to-Speech Translation
| In one line |
Build the training data, then fine-tune a model that listens to live Hindi speech and speaks the English translation while the speaker is still talking, with meaning, numbers and names intact. |
| Source |
Hindi speech, freely mixed with English |
| Target |
English speech |
| Resources |
Each team gets USD 1,000 of Modal credits (H100 GPUs) and USD 1,000 of Sarvam API credits, usable on all Sarvam APIs. |
1. About Sarvam
Sarvam AI builds language and speech models for India, designed for Indian languages as people speak them, often mixed with English.
2. The problem
A caller speaks Hindi. The listener on the other end understands only English and needs to hear the translation while the caller is still speaking. Four things make this hard:
- Word order. Hindi usually puts the verb last, while English needs it early.
- In “मैंने कल अपना पुराना बैंक खाता बंद करवा दिया” (“Yesterday I got my old bank account closed”), the verb phrase “बंद करवा दिया” (got closed) comes last.
- Until it is heard, the system cannot tell whether the account was opened, closed or only checked.
- No take-backs. Speech that has been played cannot be withdrawn, so the system must decide when it has heard enough to speak.
- Details. Amounts in lakh and crore, account digits, names, dates, negations and self-corrections must be translated correctly.
- Little data. Public Hindi–English data with speech on both sides is scarce, so you will build your own.
3. What you must build
There are two phases:
- Before the mid-term, you build the data.
- After the mid-term, Sarvam gives you a base speech-to-speech checkpoint with inference code. You fine-tune it on your data and submit it as a streaming service.
Part A (before the mid-term): the data engine
A speech-to-speech model takes Hindi audio as input and produces English audio, both mono at 24 kHz. Build a pipeline that produces the Hindi side of the training data and its translations, and deliver the resulting dataset (the “Part A dataset”).
Each record contains the following.
- Hindi audio
- Mono, 16-bit, 24 kHz WAV.
- Real recordings, synthetic speech, or both.
- Real recordings only from datasets whose licence allows redistribution.
- One speaker per clip.
- Licence and consent compliance is your responsibility.
- A verbatim Hindi transcript
- Exactly what was said, following IndicVoices Level 1 (“transcribed as spoken”). Disfluencies, repetitions and fillers are optional and not scored. If you leave them out, the remaining words must still be aligned correctly.
- English words are written in Devanagari as spoken, without bracketed Latin forms.
- Numbers are written as spoken words.
- Every sentence ends with sentence-final punctuation.
- Noise tags are optional.
- Manual correction is allowed.