Združeno prognoziranje značajki i njihova pomaka za predviđanje semantičke budućnosti u videu

doctoral thesis

• Title (english): Joint forecasting of features and feature motion for semantic future prediction in video
• Author: Josip Šarić
• Mentor: Siniša Šegvić (mentor)
• Committee member: Zoran Kalafatić (predsjednik povjerenstva)
• Committee member: Ivan Marković (član povjerenstva)
• Committee member: Tonći Antunović (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; (Department of Electronics, Microelectronics, Computer and Intelligent Systems); Zagreb
• Defense date and country: 2022-07-14, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing
• Universal decimal classification (UDC): 004 - Computer science and technology. Computing. Data processing

Abstract

Gusto semantičko prognoziranje anticipira događaje u videu predviđanjem semantike na razini piksela u budućem neopaženom slikovnom okviru. Ova disertacija posebnu pažnju posvećuje metodama gustog prognoziranja na razini apstraktnih značajki dubokog modela. Predložena prognostička metoda može se primijeniti na različite oblike gustog raspoznavanja u jednoj slici. Metoda prognozira značajke uz pomoć dva komplementarna modula. Modul F2M (eng. features-to-motion} prognozira gusto polje pomaka koji pomiču značajke iz prošlosti prema njihovim budućim položajima. Modul F2F (eng. features-to-features) izravno regresira buduće značajke i zbog toga ima mogućnost zamišljanja novootkrivenih dijelova scene. Združeni model F2MF kombinira prognoze dvaju modula u skladu s predviđenim poljem težina. Posljedično, model ima mogućnost prepoznavanja novootkrivenih dijelova scene jer u njima težine preferiraju izravno prognoziranje modulom F2F. Prognoziranje defomiranjem dominantno je u dijelovima scene gdje se lako može uspostaviti korespondencija. Predložena metoda je učinkovita jer prognozira najsažetiju i najapstraktniju moguću reprezentaciju modela za gusto raspoznavanje. Prognostičku točnost modela poboljšavaju izvedbeni detalji kao što su deformabilne konvolucije i reprezentacija obogaćena prostorno-vremenskim korelacijskim koeficijentima. Eksperimentalno vrednovanje razmatra prognoziranje semantičke segmentacije, segmentacije instanci i panoptičke segmentacije. Rezultati su pokazali da predložena metoda ostvaruje kompetitivnu točnost na podatkovnom skupu Cityscapes.

Abstract (english)

Dense semantic forecasting anticipates events in video by predicting pixel-level semantics in a future unobserved frame. This dissertation focuses on dense semantic forecasting methods which operate at the level of abstract features of a deep model. The proposed forecasting method can be applied to different single-frame dense recognition models. The method forecasts features through two complementary modules. The F2M (features-to-motion) module forecasts a dense deformation field which warps past features to their future positions. The F2F (features-to-features) module directly regresses future features and therefore has the ability to imagine novel parts of the scene. The joint F2MF model combines the forecasts of the two modules according to the predicted weights. Consequently, the model has the ability to recognize novel parts of the scene because there the weights favor direct forecasting by the F2F module. F2M forecasting is dominant in the parts of the scene where correspondence can be easily established. The proposed method is efficient because it forecasts the best possible concise and abstract representation of the dense recognition model. The forecasting accuracy of the model is improved by implementation details such as deformable convolutions and the representation enriched with spatio-temporal correlation coefficients. Experimental evaluation considers the forecasting of semantic segmentation, instance segmentation and panoptic segmentation. The results show that the proposed method achieves competitive accuracy on the Cityscapes dataset.

• Keywords: računalni vid; duboko učenje; gusto semantičko prognoziranje; predviđanje budućnosti
• Keywords (english): computer vision; deep learning; dense semantic forecasting; future prediction
• Language: croatian
• URN:NBN: urn:nbn:hr:168:127487
• Promotion: 2022
• Study programme: Title: Computer Science; Study programme type: university; Study level: postgraduate; Academic / professional title: Doktor znanosti (Doktor znanosti)
• Type of resource: Text
• Extent: x, 86 str. : graf. prikazi ; 30 cm. + CD-ROM.
• File origin: Born digital
• Access conditions: Open access
• Created on: 2022-11-28 09:18:25