Konvolucijski modeli za raspoznavanje događaja u slijedu slika

master's thesis

• Title (english): Convolutional models for event recognition in video
• Author: Igor Farszky
• Mentor: Siniša Šegvić (mentor)
• Committee member: Siniša Šegvić (predsjednik povjerenstva)
• Committee member: Zoran Kalafatić (član povjerenstva)
• Committee member: Slobodan Ribarić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2018-07-18, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Ovaj rad opisuje konvolucijske modele za analizu događaja u slijedu slika. Opisuju se razne arhitekture koje su pogodne za prepoznavanje akcija i događaja. Specifično u ovom radu se radilo istraživanje o problemu čitanja s usana, gdje su se sljedovi slika pomaka ljudskih usana prosljeđivali kroz razne konvolucijske modele na razne načine kako bi se istražila najpogodnija metoda za takvu vrstu problema koja bi se mogla primijeniti u mnogim područjima ljudskog rada, kao što su pomoć osobama sa slušnim poteškoćama, analiza video snimaka u sigurnosnim mjerama, kreiranjem zvukovnih izgovora i još mnogo toga. Podaci su filtrirane video snimke reportera. Predložene su dvije glavne arhitekture na kojima se baziraju svi modeli, VGG-M i VGG-16. Nad tim arhitekturama su modelirani specifični modeli EF i EF-3 koji se temelje na ranom stapanju sekvenci slika, te MT i MT-3 koji s druge strane prvo uče pojedinačne slike, gdje svaka slika prolazi kroz jedan toranj, a nakon toga se tek vrši stapanje sekvence namijenjeno za prepoznavanje događaja u slikama. Ukratko su prikazani rezultati modela te je donesen zaključak kako metoda tornjeva radi bolje od ranog stapanja. Na samom kraju, opisano je i kako je moguće riješiti problem čitanja s usana s predloženim konvolucijskim modelima i kako takav sustav radi sa solidnom preciznošću.

Abstract (english)

This paper describes convolutional models for event analysis in the image sequences. Work describes various architectures suitable for recognizing actions and events. Particularly in this paper is shown a way of resolving a problem of reading from the lips, where the images of human faces with focus on the lips, are transmitted through various convoluational models in various forms to explore the most appropriate method for such a problem that could be applied in many areas of human work as helping people with auditory difficulties, analyzing video footage in security measures, creating audio pronunciation and more. The data for learning such models are videos of television reporters speaking. All models are based on two main architectures, VGG-16 VGG-16. There is also four main models based on those architecutres, EF and EF-3 are early fusion models where point is fuse sequence of images on the beggining and then learn the model, and MT and MT-3 models which first learning individual images passing each image trough a separate tower, and then fusing them and learning their actions. The results of the models were presented and a conclusion was drawn that the method of the multiple towers was better then the early fusion. For the conclusion, this work describes how it is possible to solve the problem of reading from the lips and how those models and systems works with firm precision.

• Keywords: Konvolucijski modeli; sekvence slika; čitanje s usana; nadzirano učenje; klasifikacija
• Keywords (english): Convolutional models; image sequences; lip reading; supervised learning; classification
• Language: croatian
• URN:NBN: urn:nbn:hr:168:244410
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2019-03-13 20:11:49