Arhitekture za raspoznavanje slika slojevima pažnje

undergraduate thesis

• Title (english): Architectures for Image classification through attention layers
• Author: Darijan Gudelj
• Mentor: Siniša Šegvić (mentor)
• Committee member: Siniša Šegvić (predsjednik povjerenstva)
• Committee member: Josip Šarić (član povjerenstva)
• Committee member: Ivan Grubišić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2023-07-06, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Vizualni transformeri postali su standardna arhitektura za obradu vizualnih podataka.
Obilježje transformerske arhitekture je rezidualni tok, koji olakšava optimizaciju
modela, no restriktira izradu novih značajki jer ima fiksnu dimanzionalnost kroz
cijeli model. Model treba odlučiti zadržati prethodno izračunate značajke ili dodati
nove. U ovom radu se proučavaju četiri različite arhitektonske modifikacije
vizualnog transformera s ciljem poboljšanja ponovne uporabe značajki i
parametarske učinkovitosti. Inspiracija se crpi iz konvolucijskih modela, posebno
DenseNet-a, koji je pokazao uspješno rješavanje sličnih problema. Provodimo
eksperimente na CIFAR-10 skupu podataka, pri tom pokazujući da je jedna od
predloženih varijanti, gusti vizualni transformer, pareto optimalnija od vizualnog
transformera.

Abstract (english)

Vision transformers have become a standard architecture for processing visual data. The characteristic of transformer architecture is the residual flow, which facilitates model optimization but restricts the creation of new features due to fixed dimensionality throughout the model. The model needs to decide whether to keep the previously calculated features or add new ones. This paper investigates four different architectural modifications of the visual transformer with the aim of improving feature reuse and parameter efficiency. Inspiration is drawn from convolutional models, particularly DenseNet, which has successfully addressed similar problems. We conduct experiments on the CIFAR-10 dataset, demonstrating that one of the proposed variations, Dense Visual Transformer, is Pareto-optimal compared to the Vision Transformer.

• Keywords: vizualni transformer; DenseNet; arhitektura; optimizacija
• Keywords (english): Vision Transformer; DenseNet; architecture; optimization
• Language: croatian
• URN:NBN: urn:nbn:hr:168:738470
• Study programme: Title: Computing; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-07-13 10:05:43