Audio-Visual Target-Speaker ASR for Meetings
Abstract.Research on transcribing one chosen speaker in noisy multi-party meetings, using room audio and that person's lip video. The work is unpublished, so this page covers the problem, my role and the engineering, and leaves out the model design and the detailed results.
- Role
- Senior AI Research Engineer
- Period
- Aug 2026 to present
- Context
- Automotive research institute (on site via Creative Synergies Group)
- Team
- About 5
| 0.9B | Parameters in the published system I reproduced, matching the released model |
| 114 | Models trained and evaluated on 8 NVIDIA H100 GPUs |
| 600 h | Simulated multi-speaker audio-visual training data built |
01From problem to approach
The problem
In real meetings, people talk over each other. Audio-only ASR breaks down when speech overlaps, and it has no way of knowing which speaker you actually want transcribed.
The approach
Start from a published audio-visual system and reproduce it faithfully, so every later change is measured against a baseline I can trust. Then improve how the model copes when people talk over each other, and let careful evaluation on real meeting recordings decide what stays. The model design is unpublished, so it is not described here.
02What I built
- Reproduced a published audio-visual target-speaker ASR system (0.9B parameters) in NVIDIA NeMo and matched the released model within statistical noise.
- Designed, trained and tested new model components for overlapping speech. Their design is not public yet.
- Rebuilt the evaluation on real, leak-checked meeting recordings, and found and fixed scoring and labelling bugs that had been distorting the results.
- Used word-level error analysis to trace a large share of the errors to the way the training data was prepared, then fixed it and retrained.
- Ported a second published base model (1.0B parameters) from another framework and verified its output token for token against the published weights.
- Trained and evaluated 114 models on 8 NVIDIA H100 GPUs (multi-GPU DDP, bf16, Docker) with job queues, a GPU scheduler and automatic evaluation, and judged every result with paired tests, bootstrap confidence intervals and replicate runs.
- Built 600 hours of simulated multi-speaker audio-visual training data and meeting-style mixtures, behind automated quality checks and leak-free dev splits.
- Built LLM-based conversation clustering that groups speakers into conversations from their transcripts.
03Technologies
- PyTorch
- PyTorch Lightning
- NVIDIA NeMo
- ESPnet
- Lhotse
- Hydra
- MeetEval
- vLLM
- Docker
- Weights & Biases