Back to projects

Current research Speech · Multimodal Aug 2026 to present

Audio-Visual Target-Speaker ASR for Meetings

Abstract.Research on transcribing one chosen speaker in noisy multi-party meetings, using room audio and that person's lip video. The work is unpublished, so this page covers the problem, my role and the engineering, and leaves out the model design and the detailed results.

Role
Senior AI Research Engineer
Period
Aug 2026 to present
Context
Automotive research institute (on site via Creative Synergies Group)
Team
About 5
Table 1Key facts.
0.9B Parameters in the published system I reproduced, matching the released model
114 Models trained and evaluated on 8 NVIDIA H100 GPUs
600 h Simulated multi-speaker audio-visual training data built

01From problem to approach

The problem

In real meetings, people talk over each other. Audio-only ASR breaks down when speech overlaps, and it has no way of knowing which speaker you actually want transcribed.

The approach

Start from a published audio-visual system and reproduce it faithfully, so every later change is measured against a baseline I can trust. Then improve how the model copes when people talk over each other, and let careful evaluation on real meeting recordings decide what stays. The model design is unpublished, so it is not described here.

02What I built

  1. Reproduced a published audio-visual target-speaker ASR system (0.9B parameters) in NVIDIA NeMo and matched the released model within statistical noise.
  2. Designed, trained and tested new model components for overlapping speech. Their design is not public yet.
  3. Rebuilt the evaluation on real, leak-checked meeting recordings, and found and fixed scoring and labelling bugs that had been distorting the results.
  4. Used word-level error analysis to trace a large share of the errors to the way the training data was prepared, then fixed it and retrained.
  5. Ported a second published base model (1.0B parameters) from another framework and verified its output token for token against the published weights.
  6. Trained and evaluated 114 models on 8 NVIDIA H100 GPUs (multi-GPU DDP, bf16, Docker) with job queues, a GPU scheduler and automatic evaluation, and judged every result with paired tests, bootstrap confidence intervals and replicate runs.
  7. Built 600 hours of simulated multi-speaker audio-visual training data and meeting-style mixtures, behind automated quality checks and leak-free dev splits.
  8. Built LLM-based conversation clustering that groups speakers into conversations from their transcripts.

03Technologies

  • PyTorch
  • PyTorch Lightning
  • NVIDIA NeMo
  • ESPnet
  • Lhotse
  • Hydra
  • MeetEval
  • vLLM
  • Docker
  • Weights & Biases

On request

Request my resume

I share my resume on request. Tell me a little about yourself, and I'll email it to you once I've reviewed your request.

I am a…

Which document?

0 / 100 words

Your details are used only to handle this request and are never shared.