Multimodalna analiza sentimenta korištenjem teksta i slike

master's thesis

• Title (english): Multimodal sentiment analysis using text and images
• Author: Tomislav Krog
• Mentor: Marko Đurasević (mentor)
• Committee member: Marko Đurasević (predsjednik povjerenstva)
• Committee member: Nikolina Frid (član povjerenstva)
• Committee member: Ante Đerek (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2024-07-16, Croatia

Abstract

Ovaj diplomski rad bavi se multimodalnom analizom sentimenta primjenom tekstualnih
i slikovnih značajki na slikama memeova. Memeovi su poseban izazov za analizu
sentimenta zbog svoje složenosti i dvosmislenosti. U radu su razvijeni i evaluirani
različiti modeli koji integriraju tekstualne i slikovne značajke radi poboljšanja točnosti
predikcije sentimenta. Primijenjene metode uključuju napredne tehnike fuzije značajki
kao što su BERT-ResNet-GMU, BERT-ResNet-AttnLSTM, te integracije s modelima GPT-
4-Vision i GPT-4o. Najbolji model prema macro F1 score-u bio je MoodModel v3, koji
koristi fuziju značajki putem dvosmjernog LSTM-a temeljenog na mehanizmu pažnje.
Rezultati istraživanja pokazuju da posebno prilagođeni modeli značajno poboljšavaju
performanse analize sentimenta u odnosu na osnovne modele. Rad također naglašava
izazove rada s nebalansiranim datasetima i predlaže smjernice za buduća istraživanja u
ovom području.

Abstract (english)

This thesis explores the multimodal sentiment analysis of memes by leveraging both
textual and visual features. Memes pose a unique challenge for sentiment analysis due
to their complexity and ambiguity. In this study, various models that integrate textual
and visual features were developed and evaluated to enhance the accuracy of sentiment
prediction. The applied methods include advanced feature fusion techniques such as
BERT-ResNet-GMU, BERT-ResNet-AttnLSTM, and integrations with GPT-4-Vision and
GPT-4o models. The best-performing model, according to the macro F1 score,was Mood-
Model v3, which employs feature fusion through a bidirectional LSTM based on an attention
mechanism. The research findings demonstrate that custom-implemented models
significantly improve sentiment analysis performance compared to baseline models.
The thesis also highlights the challenges of working with imbalanced datasets and offers
guidelines for future research in this field.

• Keywords: multimodalna analiza sentimenta; memeovi; emocije; BERT; ResNet; EfficientNet; GMU; LSTM; fuzija značajki; GPT-4; GPT-4o; analiza sentimenta
• Keywords (english): multimodal sentiment analysis; memes; emotions; BERT; ResNet; EfficientNet; GMU; LSTM; feature fusion; GPT-4; GPT-4o; sentiment analysis
• Language: croatian
• URN:NBN: urn:nbn:hr:168:803564
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2024-10-12 21:13:46