Sustav za generiranje opisa slika

master's thesis

• Title (english): System for Image Captioning
• Author: Mateo Imbrišak
• Mentor: Tomislav Hrkać (mentor)
• Committee member: Tomislav Hrkać (predsjednik povjerenstva)
• Committee member: Dejan Škvorc (član povjerenstva)
• Committee member: Zoran Kalafatić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2022-07-04, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Automatsko generiranje opisa slika problem je koji spaja računalni vid i obradu prirodnog jezika. Automatsko generiranje opisa primjenjivo je na društvenim mrežama kao i u sustavim za sintezu govora za pomoć osobama slabijeg vida. U ovom radu ostvarena su dva modela za generiranje opisa slika. Oba modela oslanjaju se na postojeću implementaciju mreže ResNet za obradu slika, a razlikuju se u implementaciji dekodera. Ostvareni dekoderi koriste jednu ili dvije LSTM ćelije, a u slučaju korištenja dvije ili više ćelija one su povezane u rezidualne blokove. Istaknuta je važnost odabira algoritma za odabir generirane riječi u svakom koraku te da nije uvijek najbolje birati najvjerojatniju riječ u svakom koraku. Priloženi su rezultati, mjerenja te primjeri slika iz skupa za treniranje s generiranim opisima.

Abstract (english)

Automatic image captioning is a problem that combines computer vision and natural language processing. It is applicable on social media platforms as well as in text-to-speech systems used to help people with impaired vision. Two image captioning systems are developed for this thesis. Both models rely on existing ResNet implementation to handle image processing, but the differ when it comes to their decoder implementations. Decoder use one or two LSTM cells and when two or more LSTM cells are used, they are stacked to form residual blocks. Importance of choosing the right search algorithm is emphasized as well as the fact that choosing the most likely word in each time step is not always the best option. Results, tests and examples of images from test set, along with their generated captions are included at the end.

• Keywords: generiranje opisa slika; računalni vid; obrada prirodnog jezika; duboko učenje; LSTM; RNN; CNN; ResNet; konvolucijske mreže; povratne mreže
• Keywords (english): image captioning; computer vision; natural language processing; deep learning; LSTM; RNN; CNN; ResNet; convolutional neural networks; recurrent neural networks
• Language: croatian
• URN:NBN: urn:nbn:hr:168:633150
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-14 08:56:03