Video Summarization Pipeline

Whisper large-v3 is free to run, handles 99 languages, and is competitive with paid APIs on WER. Deepgram Nova-3 has better speaker diarization and lower latency but costs $0.0043/min. Use Whisper for batch, Deepgram for realtime.

Alternatives: Deepgram Nova-3, AssemblyAI Universal-2, Google STT v2

Pyannote 3.1 is the strongest open-source diarizer and pairs cleanly with Whisper. Managed services (Deepgram, AssemblyAI) bundle diarization with transcription and are easier to run end-to-end if you don't want to manage GPU infra.

Alternatives: Deepgram diarization, AssemblyAI speaker labels

Gemini 2.5 Pro gives the best frame descriptions at $1.25/$5 per MTok and handles slides/whiteboards well. Gemini 2.0 Flash is 10x cheaper and fine for simple scene descriptions. Use Flash for talking-head footage, Pro for screen-share-heavy content.

Alternatives: GPT-4o vision, Claude Sonnet 4 vision, Gemini 2.0 Flash

Gemini 2.5 Pro can ingest up to 1 hour of video natively in one call, mixing visual and audio understanding. For videos under 30 min, this is simpler than a pipeline. Over 1 hour, you still need sampling and fusion.

Alternatives: GPT-4o with frame sampling, Claude Sonnet 4 with frames

Sonnet 4 follows chapter-structure prompts reliably and produces consistent summary quality across hours-long inputs. Gemini 2.5 Pro shines when you need 1M+ token context for very long transcripts (multi-hour podcasts).

Alternatives: GPT-4o, Gemini 2.5 Pro

Mux and Cloudflare Stream handle ingest, transcoding, delivery, and analytics in one service. Roll-your-own with S3+CloudFront is cheaper at huge scale but takes months to get right.

Alternatives: AWS IVS, Vimeo OTT, Self-hosted + Bunny CDN

Native multimodal vs pipelined

Gemini 2.5 Pro Video can process up to 1 hour natively in a single API call - simpler code, slightly better multimodal reasoning. Downside: pricing for native video is 3-5x more per minute than a pipelined Whisper + frame-sampling approach, and you hit context limits past 1 hour. Pipeline for scale; native for short-form prototypes.

Frame sampling rate

Sampling every scene change is adequate for most content. Sampling every 5 seconds gives more visual detail but triples vision cost. For talking-head content (podcasts, interviews), 1 frame per 30s is enough. For tutorials with screen-share, sample every 5-10s to catch code/slide changes.

Whisper vs managed STT

Whisper large-v3 self-hosted is free per minute but requires GPU infra. Deepgram Nova-3 at $0.0043/min gives better diarization and zero ops burden. Break-even is around 5k hours/month - above that, self-hosting wins on unit economics if you have ML ops capacity.

Speaker diarization swaps two speakers mid-conversation

Mitigation: Enforce a minimum speaker-segment length (3-5 seconds) to reduce false splits. When diarization confidence is low, mark the segment 'speaker unknown' rather than guessing. Post-process with Pyannote re-clustering on the full audio embedding to catch swaps.

Vision model misreads slide/whiteboard text

Mitigation: Run an OCR pass (Tesseract or Gemini) on detected slide frames and feed the raw OCR text alongside the vision description. Disagreements between OCR text and vision caption indicate a problem - flag for review or request a clearer frame.

Non-English audio transcribed incorrectly or mixed with English hallucinations

Mitigation: Always run language detection on a 60s audio sample before transcription. Pass the detected language explicitly to Whisper. Whisper is prone to English hallucinations on quiet/non-speech audio - filter out segments where WER confidence drops below 0.5.

Long video exceeds context window for summary generation

Mitigation: Use a map-reduce approach: summarize 10-15 min chunks individually, then produce a final summary from the chunk summaries. Gemini 2.5 Pro's 1M+ token context handles 5-8 hours of transcript directly - use it for podcasts, Sonnet 4 for shorter content with better quality.

Chapters generated do not match actual video structure

Mitigation: Eval chapter quality against human-labeled ground truth on 50-100 sample videos. Require chapters to have a clear topic change at the boundary. If the model splits mid-topic or misses an obvious chapter, adjust the prompt to use the visual scene changes as stronger hints.