Predviđanje višemodalne semantičke budućnosti u videu

master's thesis

• Title (english): Multimodal semantic forecasting in video
• Author: Kristijan Fugošić
• Mentor: Siniša Šegvić (mentor)
• Committee member: Siniša Šegvić (predsjednik povjerenstva)
• Committee member: Zoran Kalafatić (član povjerenstva)
• Committee member: Tomislav Hrkać (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2020-07-14, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

U ovom radu razmatramo semantičko predviđanje višemodalne budućnosti. Glavni problem u tom kontekstu predstavljaju novootkriveni prostori i artikulirani objekti kod kojih se i u kratkoročnom slučaju suočavamo s velikom nesigurnošću predviđanja. Ideja ovog rada je dopustiti modelu nešto više slobode u smislu mogućnosti predviđanja više različitih budućnosti. To činimo pretvaranjem osnovnog regresijskog modela u uvjetni generativni model temeljen na suparničkom učenju primjenom gubitaka rekonstrukcije momenata. U slučaju kratkoročnog predviđanja dobivena je veća točnost modela, dok kod srednjoročnog predviđanja imamo nešto nižu točnost, ali rezultati su raznoliki i vizualno zanimljiviji. Pokazalo se da učenje novog modela traje 6 puta duže, a vrijeme zaključivanja pribiližno je osnovnom modelu.

Abstract (english)

In this paper we address the multimodal semantic forecasting in road driving scenarios. The main problem in this context is great uncertainty in forecasting of articulated objects and unoccluded spaces, even in the case of short-term predictions. The idea of this paper is to give the model somewhat more freedom by allowing it to predict multiple futures. We do this by converting the base regression model into a conditional generative adversarial model and by using moment reconstruction losses. In the case of short-term prediction, higher accuracy of the model was obtained, while in the case of long-term prediction, we observe slightly lower accuracy, but the predictions are diverse and visually more interesting. It takes about 6 times longer to train the model, while the evaluation time is similar to the base model.

• Keywords: duboko učenje; računalni vid; generativni suparnički modeli; GAN; multimodalno generiranje slika; MR-GAN; predviđanje semantičke budućnosti; predviđanje iz značajki u značajke; F2F; semantička segmentacija; segmentacija instanci
• Keywords (english): deep learning; computer vision; generative adversarial models; GAN; multimodal image generation; MR-GAN; semantic future prediction; feature-to-feature; F2F; semantic segmentation; instance segmentation
• Language: croatian
• URN:NBN: urn:nbn:hr:168:467493
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 12:16:07