Audio-Visual Target-Speaker ASR for Meetings
Teaching a recogniser to transcribe one chosen speaker in a multi-party meeting, using the room audio and that person's lip video.
View case studySenior AI Research Engineer · Tokyo
Satyam Tiwari
Abstract.I take speech, multimodal and LLM systems from research to production. Currently researching audio-visual speech recognition for multi-party meetings at an automotive research institute; previously shipped an LLM agent, offline RAG and real-time speech systems for Japanese enterprises.
Keywords:ASR, audio-visual speech recognition, LLM agents, offline RAG, rigorous evaluation.
| Measure | Value | Context |
|---|---|---|
| Agreement with expert reviewers Autonomous LLM agent for network design review | 18/19 | Autonomous LLM agent for network design review |
| LLM calls per review job Fully offline hybrid RAG on air-gapped, CPU-only servers | 25 → 10 | Fully offline hybrid RAG on air-gapped, CPU-only servers |
| Models trained and evaluated Current research · 8 NVIDIA H100 GPUs · DDP · bf16 · paired tests and bootstrap CIs | 114 | Current research · 8 NVIDIA H100 GPUs · DDP · bf16 · paired tests and bootstrap CIs |
| Audio-visual training data built Simulated multi-speaker meetings, behind automated quality checks | 600 h | Simulated multi-speaker meetings, behind automated quality checks |
| Years building AI systems Research → production, in Japan and India | 3+ | Research → production, in Japan and India |
01About
Research depth, production delivery
I'm an AI research engineer with 3+ years of building and shipping AI systems, from research prototypes to production services. My work centres on speech and multimodal AI (ASR, TTS and now audio-visual speech recognition), together with LLM agents and retrieval-augmented generation.
Today I research audio-visual target-speaker speech recognition for multi-party meetings at an automotive research institute. I reproduced a published 0.9B-parameter system and now work on making recognition hold up when people talk over each other, training and evaluating 114 models on 8 NVIDIA H100 GPUs and judging every result with paired significance tests. The model design is unpublished, so I keep its details off this site. Before that I delivered an autonomous LLM agent, offline RAG over confidential data on air-gapped servers, medical similarity search and real-time ASR / TTS for Japanese enterprises, working in Japanese with client teams and management.
- 1.1
Speech & multimodal AI
ASR, TTS and audio-visual target-speaker ASR for meetings, from 600-hour data pipelines to large multi-GPU training and evaluation runs.
- 1.2
LLM agents & RAG
Agents that plan and call tools, with code-enforced guardrails, deterministic grounding and offline hybrid retrieval.
- 1.3
Rigorous evaluation
Public benchmarks, ablations, replicates and paired significance tests, with every metric traceable to raw results.
- 1.4
Bilingual delivery
Requirements to operation with Japanese clients, presenting to management in Japanese.
02Projects
Selected projects
Research and production AI built for Japanese enterprises. Open a project to see how it was built.
- 2.2
Autonomous AI Agent for Network Design Review
An LLM planner that reviews network design documents against live device configs, fully offline on air-gapped servers.
Pipeline: Design docs + configs → Extract & selective OCR → Offline hybrid RAG → Agent plans & acts → Guardrails & grounding → Review report
18/19 Agreement with reviewer-judged findings; recurring false-positive classes eliminated
View case study - 2.3
Medical Document Similarity & EMR System
Similarity search over confidential clinical records, with embedding models chosen by AUROC / F1.
Pipeline: Ingest EMR (JSON) → Preprocess → Embed → Similarity & ranking → Benchmark → Review in Streamlit
4 Embedding APIs benchmarked
View case study - 2.4
Real-Time Speech AI (ASR / TTS)
Hands-free voice data entry: Faster-Whisper on int8 CTranslate2, with VITS read-back.
Pipeline: Voice input → ASR · Faster-Whisper → Intent matching → Data action → TTS · VITS → Audio response
int8 CTranslate2 inference for low latency
View case study - 2.5
Voice-Based Email for Visually Impaired Users
Compose, send and read email by voice alone, with about 90% ASR accuracy and 50% less task time.
Pipeline: Voice login → Speech → text → Choose an action → Gmail API → Audio feedback
~90% Speech recognition accuracy
View case study - 2.6
Real-Time Object Recognition
Recognizes everyday objects from a live camera and announces them by audio.
Pipeline: Camera feed → Detect (TensorFlow) → Annotate (OpenCV) → Announce by audio
~1,000 Self-collected training images
View case study
2.7Personal projects
- AI Blog Generator (LLaMA 2) A local LLM app that writes a blog post to a chosen topic, style and length.
- Sales Performance Dashboard A Power BI dashboard with DAX KPIs for revenue, quantity and repeat rate.
- Facial Emotion Detection Emotion classification on about 16,000 face images, comparing four classical models.
- Multi-Disease Prediction System Risk prediction for heart disease, diabetes and Parkinson's, comparing ML and deep-learning models.
03Experience
Work experience
From corporate R&D labs to production systems, in Japan and India.
-
3.1 Aug 2026 to present
Senior AI Research Engineer Current
Creative Synergies Group · Saitama, Japan
On site at an automotive research institute, researching audio-visual target-speaker ASR for multi-party meetings: reproducing a published system, new modelling for overlapping speech, evaluation on real meetings and multi-GPU H100 training infrastructure.
-
3.2 Mar 2024 to Aug 2026
AI Engineer / Data Scientist
Human Resocia Co., Ltd. · Tokyo, Japan
Placed at client R&D labs and system integrators as the AI engineer, owning projects end to end.
-
3.3 Jun 2022 to Jun 2023
AI Developer Intern (two terms)
Digipodium, Inc. · Lucknow, India
Built assistive speech and vision apps for visually impaired users, and wrote their design documents and test specifications.
04Skills
Skills & background
| Area | Tools and methods |
|---|---|
| Programming |
|
| ML frameworks |
|
| Speech & multimodal |
|
| LLMs & generative AI |
|
| Research & evaluation |
|
| Infrastructure & backend |
|
4.1Education
- 2021-2023 Master of Computer Applications (IT) · GPA 8.45 Amity University, Lucknow · India
- 2018-2021 Bachelor of Science RMLAU University · India
- 2023-2024 Japanese language program Human Academy Japanese Language School · Japan
4.2Certifications
- 2024 NAT-TEST Japanese, Level 2Q (JLPT N2 equivalent)
- 2023 IBM · Python for Data Science (Coursera)
- 2023 IBM · Data Analysis Using Python
- 2022 Cisco · CCNAv7: Switching, Routing and Wireless Essentials
- 2022 Oracle Academy · Database Foundations
05Contact
Let's work together
Open to research roles, collaborations and conversations about speech, multimodal and LLM systems. Based in Tokyo, open to India.
| itssatyam.345@gmail.com | |
| Location | Tokyo, Japan |