A hybrid approach to real-time speech driven facial gesturing of virtual characters

doctoral thesis

• Title (croatian): Hibridni pristup stvaranju gesti lica virtualnog lika govornim signalom u stvarnom vremenu
• Author: Goranka Zorić
• Mentor: Igor Sunday Pandžić (mentor)
• Committee member: Maja Matijašević (predsjednik povjerenstva)
• Committee member: Dragan Jevtić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2010-07-09, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing
• Universal decimal classification (UDC): 004 - Computer science and technology. Computing. Data processing

Abstract

This thesis investigates automatic facial gesturing of virtual characters based only on a speech input. Facial gestures that have been included are various nods and head movements, eye blinks, eyebrow gestures and gaze, i.e. facial and head movements used in nonverbal communication that are connected to the syntactic and prosodic structure of the underlying speech rather than to semantic or emotions. The main issue in speech driven facial gesturing is an audio to visual mapping. In this thesis, a hybrid method for the real-time correlation of the speech signal with facial gestures has been proposed. An idea behind the proposed method is to include different issues that are considered important for visual prosody through three different modules. In a data-driven module facial gestures are correlated with prominent parts of the speech. A rule-based module consists of a set of rules, which among others include rules for punctuation and prolonged pauses. Additionally, in a statistics module, results of first two modules are fine-tuned and further shaped. In order to check how believable virtual characters are if animated using facial gestures obtained by the proposed method, a system for speech driven facial gesturing has been implemented, and used for a perceptual evaluation. Testing subjects generally considered facial gestures consistent and time aligned with the underlying speech, and a virtual human applying them is perceived as a natural. Moreover, the set up hypothesis has been proved: adding facial gestures driven by the speech signal to the animation of virtual characters using the proposed hybrid approach in real-time will result in more coherent and appropriate facial movements than random or no movements. Further more, the system for speech driven facial gesturing of virtual characters has been put in a context of virtual news presenters used in networked environments, since that is where the future research of this system strives next.

Abstract (croatian)

Ova doktorska disertacija istražuje automatsko stvaranje gesti virtualnog lika govornim signalom u stvarnom vremenu. Pokreti lica uključeni u ovaj rad su različiti pokreti glave i obrva, treptanje i pogled, tj. svi pokreti lica i glave koji se koriste u neverbalnoj komunikaciji povezani sa sintaksom i prozodijom govora, a ne s emocijama ili značenjem riječi. Kod generiranja animacije lica iz govora potrebno je naći korelaciju između govornog signala i pokretanja lica. Ta korelacija nije direktna, već je posljedica više elemenata, te postoje individualne i kulturološke specifičnosti. U disertaciji su istražene različite metode za preslikavanja iz audio informacije u vizualnu. Odabrano rješenje temelji se na hibridnom pristupu, pri čemu su statistički ispravni pokreti lica i glave usklađeni sa prozodijom govora u stvarnom vremenu dobiveni tehnikom strojnog učenja, korištenjem pravila te statistikom dobivenom iz baze podataka. Tehnikom strojnog učenja dobiveni su pokreti lica koji su povezani s naglašenim dijelovima govora, dok su pomoću pravila, između ostalog, dobiveni pokreti lica povezani s različitim pauzama u govoru. Ovi rezultati su dodatno oblikovani statistikom dobivenom iz baze podataka. Kako bi provjerili pokrete lica virtualnog lika, izgrađen je sustav za automatsko pokretanje lica govornim signalom zasnovan na predloženoj hibridnoj metodi, te je iskorišten za subjektivnu evaluaciju. Sudionici testiranja su doživjeli pokrete lice uglavnom kao vremenski usklađene, te konzistentne s govorom, dok je općeniti dojam da virtualni lik djeluje prirodno. Također, potvrđena je postavljena hipoteza: pokreti lica iz govora dodani animaciji virtualnog lika korištenjem predložene hibridne metode u realnom vremenu daju bolje rezultate nego slučajni pokreti ili lice bez pokreta. Tako izgrađen sustav stavljen je u kontekst virtualnih prezentatora vijesti koji se koriste u umreženim okruženjima, budući da daljnje istraživanje ove problematike ide u tom smjeru.

• Keywords: Facial gestures; Visual prosody; Virtual characters; Embodied Conversational Agents; Speech signal; Facial animation; Nonverbal communication; Audio-visual speech processing; Human-Computer interaction; HMMs
• Keywords (croatian): Geste lica; Vizualna prozodija; Virtualni likovi; Utjelovljeni razgovorni agent; Govorni signal; Animacija lica; Neverbalna komunikacija; Audio-vizualna obrada govora; Interakcija čovjeka i računala; HMM
• Language: english
• URN:NBN: urn:nbn:hr:168:380931
• Promotion: 2010
• Study programme: Title: Computer Science; Study programme type: university; Study level: postgraduate; Academic / professional title: Doktor znanosti (Doktor znanosti)
• Type of resource: Text
• Extent: 85 str. : graf. prikazi
• File origin: Born digital
• Access conditions: Open access
• Created on: 2024-03-20 10:26:13