Raspoznavanje prednaučenim samonadziranim i multimodalnim modelima

master's thesis

• Title (english): Recognition with pre-trained self-supervised and multimodal models
• Author: Jurica Kuleš
• Mentor: Siniša Šegvić (mentor)
• Committee member: Siniša Šegvić (predsjednik povjerenstva)
• Committee member: Marin Oršić (član povjerenstva)
• Committee member: Ivan Grubišić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2025-02-28, Croatia

Abstract

Raspoznavanje slika predstavlja važan izazov u području računalnog vida, s mnogim praktičnim primjenama. Na početku ovog rada dan je uvid u postojeće duboke arhitekture za raspoznavanje slika, s posebnim naglaskom na modele temeljene na mehanizmu pažnje. Središnja tema rada bila je raspoznavanje slika pomoću predučenih samonadziranih i multimodalnih modela. U tu smo svrhu pomoću predučenih okosnica modela OpenCLIP i DINOv2 proveli klasifikaciju nad skupom podataka TACO. Došli smo do zaključka... More da korištenjem navedenih modela za klasifikaciju bez dodatnog ugađanja okosnice, postižemo znatno lošije rezultate nego ako povrh njih dodamo čak i samo jedan potpuno povezani sloj. Nadalje, uočili smo da znatnu ulogu u gubitku točnosti igra i sama priroda skupa podataka TACO. Također smo pokazali da ansambliranjem okosnica ovih dvaju modela postižemo najbolje performanse, što nas je navelo na zaključak da OpenCLIP i DINOv2 ekstrahiraju komplementarne značajke. Less

Abstract (english)

Image recognition represents a significant challenge in the field of computer vision, with numerous practical applications. At the beginning of this work, an overview of existing deep architectures for image recognition is provided, with a particular emphasis on attention-based models. The central theme of this research is image recognition using pretrained self-supervised and multimodal models. To this end, we performed classification on the TACO dataset using the pretrained backbones of the... More OpenCLIP and DINOv2 models.
We concluded that using these models in a zero-shot setting results in significantly poorer performance compared to the event of an addition of even a single fully connected layer on top of the model backbones. Furthermore, we observed that the nature of the TACO dataset itself plays a significant role in the observed loss of accuracy. Additionally, we demonstrated that ensemble learning—combining the backbones of both models—yields the best performance, leading us to conclude that OpenCLIP and DINOv2 extract complementary features. Less

• Keywords: duboko učenje; računalni vid; raspoznavanje slika; predučenje; samo- nadzirano učenje; multimodalni modeli
• Keywords (english): deep learning; computer vision; image recognition; pretraining; self- supervised learning; multimodal models
• Language: croatian
• URN:NBN: urn:nbn:hr:168:776667
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2025-03-12 11:11:20