The Vision & Engineering Challenge
Translating and dubbing video content across global languages has traditionally been a grueling manual process involving multiple translators, voice actors, and video editors.
DubNour AI was engineered as a comprehensive desktop studio that transforms raw video in one language into natural, lip-synced dubbed video in over 20 languages.
System Architecture & 4-Stage Pipeline
1. Speech-to-Text & Diarization
Audio stream extraction using FFmpeg, split into frequency-normalized vocal stems with precise timecoded phoneme timestamps.
2. Contextual LLM Translation
Google Gemini 1.5 Pro translates spoken dialogue while strictly respecting speech duration constraints.
3. Zero-Shot Neural Voice Cloning
ElevenLabs Multilingual v2 generates natural speech matching original vocal emotion, pitch, and inflection.
4. Frame-Accurate Video Remuxing
FFmpeg pipeline dynamically adjusts audio tempo without pitch shift and burns bilingual subtitles.
💡 Technical Advantage: Gemini 1.5 Pro applies length-constrained semantic translation to match source mouth movement and cadence accurately.
Core Audio-Video Sync Pipeline (Python)
# DubNour AI Core Sync Controller (Python 3.11)
import subprocess, os
import google.generativeai as genai
from elevenlabs.client import ElevenLabs
class DubNourPipeline:
def __init__(self, gemini_key, eleven_key):
genai.configure(api_key=gemini_key)
self.model = genai.GenerativeModel('gemini-1.5-pro')
self.eleven = ElevenLabs(api_key=eleven_key)
def remux_dubbed_video(self, original_video, new_audio, output_path):
cmd = [
'ffmpeg', '-y', '-i', original_video, '-i', new_audio,
'-c:v', 'copy', '-c:a', 'aac', '-map', '0:v:0', '-map', '1:a:0',
'-shortest', output_path
]
subprocess.run(cmd, check=True)
Key Achievements & Real-World Impact
DubNour AI demonstrates the power of combining modern Generative AI with robust systems programming in Python.