devcon8 / voxbento when 2 seconds matter building real time ai captions and live interpretations
- YouTube
- Details
Voxbento: When 2 Seconds Matter — Building Real-Time AI Captions and Live Interpretations
Duration: NaN:NaN:NaN
Speaker: Arnav Angarkar
Type: CLS - Talk (Open Tech Summit)
Expertise:
Event: Devcon
Date: Nov 2026
**When 2 Seconds Matter: Building Real-Time AI Captions for Live Events**
Live events need captions and translations that feel instantaneous, but building real-time AI streaming is very different from processing uploaded audio.
In this talk, I’ll share what I learned building **VoxBento**, an open source real-time multilingual streaming infrastructure using FastAPI, WebRTC, MediaMTX, and pluggable transcription providers.
I’ll walk through the architecture from live audio ingestion to streaming transcription and caption delivery, and the engineering challenges we encountered along the way. In particular, I’ll cover why we moved from HLS to WHEP to reduce playback handoff latency, how we integrated both cloud and local transcription services, and why naïve chunk-based Whisper inference can cause context and caption consistency problems.
I’ll also discuss practical approaches to handling overloaded transcription pipelines, including bounded queues, dropping stale audio, voice activity detection, and caption aggregation.
The goal is to give attendees a practical look at the tradeoffs involved in building low-latency, extensible AI infrastructure in open source, along with lessons that can be applied to other real-time AI applications.
Categories