Real-time Hindi → English Speech-to-Speech Translation

In one line Build the training data, then fine-tune a model that listens to live Hindi speech and speaks the English translation while the speaker is still talking, with meaning, numbers and names intact.
Source Hindi speech, freely mixed with English
Target English speech
Resources Each team gets USD 1,000 of Modal credits (H100 GPUs) and USD 1,000 of Sarvam API credits, usable on all Sarvam APIs.

1. About Sarvam

Sarvam AI builds language and speech models for India, designed for Indian languages as people speak them, often mixed with English.

2. The problem

A caller speaks Hindi. The listener on the other end understands only English and needs to hear the translation while the caller is still speaking. Four things make this hard:

3. What you must build

There are two phases:

  1. Before the mid-term, you build the data.
  2. After the mid-term, Sarvam gives you a base speech-to-speech checkpoint with inference code. You fine-tune it on your data and submit it as a streaming service.

Part A (before the mid-term): the data engine

A speech-to-speech model takes Hindi audio as input and produces English audio, both mono at 24 kHz. Build a pipeline that produces the Hindi side of the training data and its translations, and deliver the resulting dataset (the “Part A dataset”).

Each record contains the following.