Noise cancellation in automatic lip sync by speech signal analysis

scientific master's theses

• Title (croatian): Poništavanje šuma pri automatskoj sinkronizaciji usana pomoću analize govornog signala
• Author: Tomislav Košutić
• Mentor: Igor Sunday Pandžić (mentor)
• Committee member: Maja Matijašević (predsjednik povjerenstva)
• Committee member: Igor Sunday Pandžić (član povjerenstva)
• Committee member: Krešimir Matković (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; (Department of Telecommunications); Zagreb
• Defense date and country: 2011-11-04, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing
• Universal decimal classification (UDC): 004 - Computer science and technology. Computing. Data processing

Abstract

This master thesis investigates noise cancellation techniques in order to enhance the performance of existing automatic lip synchronization system in noisy environments. Automatic lip synchronization is a method for generating virtual character face animation where animation is driven by the speech signal. The method consists of speech signal analysis, audio to visual mapping and face animation synthesis. Speech signal is mapped to visual representations of speech, visemes, by speech feature recognition. The presence of background noise in the speech signal degrades the speech feature recognition and results in poor lip sync animation quality. The thesis investigates different speech signal enhancement methods, proposes and implements algorithms for noise cancellation as part of the enhanced automatic lip synchronization system. Proposed method based on spectral subtraction effectively tracks and cancels the stationary and nonstationary noise in real time. Additional filters applied on speech features improve the recognition accuracy of the speech signal. Existing speech recognition system performance is improved with the use of dynamic neural networks, resulting in overall better lip sync animation quality. Implemented system is validated with several evaluation methods, showing robustness to different noise environments. Potential applications of the system are discussed in details. Enhanced lip synchronization system is speaker independent and suitable for real-time and offline applications.

Abstract (croatian)

Ovaj rad istražuje tehnike poništavanja šuma s ciljem poboljšanja performansi postojećeg sustava za automatsku sinkronizaciju usana u bučnom okruženju. Automatska sinkronizacija usana je metoda generiranja animacije govora virtualnog lika na temelju zvučnog signala govora. Postupak se sastoji od analize signala govora, mapiranja govora u vizualni skup parametara i sinteze animacije virtualnog lika. Mapiranje signala govora u skup vizualnih parametara ostvaruje se podsustavom za prepoznavanje kodiranog govornog signala. Pozadinska buka u zučnom signalu degradira postupak prepoznavanja kodiranog govornog signala, što rezultira lošijom kvalitetom animacije govora virtualnog lika. U radu su istraženi različiti postupci za poboljšanje zašumljenog zvučnog signala govora. Na temelju istraživanja predložen je odgovarajući algoritam za poništavanje šuma u zvučnom signalu, zasnovan na spektralnoj subtrakciji. Algoritam je implementiran kao nadogradnja postojećeg sustava za automatsku sinkronizaciju usana. Algoritam spektralne subtrakcije efikasno prepoznaje i uspješno poništava šum u stvarnom vremenu. Uz poništavanje šuma dodatnim filtriranjem kodiranog govornog signala pospješuje se prepoznavanje govora u zvučnom signalu. Postojeći podsustav za prepoznavanje kodiranog govornog signala unaprijeđen je korištenjem dinamičkih neuronskih mreža. Rezultat jest kvalitetnija animacija govora virtualnog lika. Implementirani sustav validiran je pomoću različitih metoda za testiranje koje pokazuju otpornost sustava prema različitim razinama buke u zvučnom signalu. Potencijalne primjene sustava za automatsku sinkronizaciju usana izložene su do detalja. Poboljšani sustav za automatsku sinkronizaciju usana neovisan je o govorniku i prikladan za primjenu u stvarnom vremenu.

• Keywords: virtual characters; lip synchronization; facial animation; noise cancellation; spectral subtraction; speech signal enhancement; dynamic neural networks
• Keywords (croatian): virtualni likovi; automatska sinkronizacija usana; animacija lica; poništavanje šuma u signalu; spektralna subtrakcija; dinamičke neuronske mreže
• Language: english
• URN:NBN: urn:nbn:hr:168:877156
• Project: Number: 036-0362027-2028; Title: Utjelovljeni razgovorni agenti za usluge u umreženim i pokretljivim sustavima; Leader: Igor Sunday Pandžić; Jurisdiction: Croatia; Funder: MZOS; Funding stream: ZP
• Study programme: Title: Postgraduate master programe in computing; Study programme type: university; Study level: postgraduate; Academic / professional title: magistar inženjer računarstva (magistar inženjer računarstva)
• Catalog URL: http://lib.fer.hr/cgi-bin/koha/opac-detail.pl?biblionumber=36911
• Type of resource: Text
• Extent: 83 str ; 30 cm
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2020-08-25 08:42:48