Implementacija i analiza arhitekture Mamba za neuronske jezične modele

master's thesis

• Title (english): Implementation and analysis of the Mamba architecture for neural language models
• Author: Lea Krsnik
• Mentor: Jan Šnajder (mentor)
• Committee member: Jan Šnajder (predsjednik povjerenstva)
• Committee member: Siniša Šegvić (član povjerenstva)
• Committee member: Marko Čupić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2024-07-04, Croatia

Abstract

Duboko učenje proteklih godina ostvaruje iznimne rezultate, pogotovo u području obrade prirodnog jezika kroz uporabu velikih jezičnih modela. Fokus istraživačke zajednice većinski je bio usmjeren na arhitekturu Transformatora kao temelj velikih jezičnih modela. Međutim, nedavna inovacija, arhitektura Mamba, pokazala je potencijal za postizanje usporedivih rezultata s boljom vremenskom i prostornom složenošću. U ovom radu, detaljno su razmotreni matematički temelji Mambe te optimizacijske tehnike za izvedbu na grafičkim procesorima. Za potrebe rada, arhitektura Mambe implementirana je u programskom okviru PyTorch te je nad tom implementacijom provedena evaluacija na različitim zadatcima obrade prirodnog jezika.

Abstract (english)

In recent years, deep learning has achieved remarkable results, especially in the field of natural language processing through the use of large language models. The research community has primarily focused on the Transformer architecture as the foundation of large language models. However, a recent innovation, the Mamba architecture, has shown the potential to achieve comparable results with better temporal and spatial complexity. This thesis provides a detailed examination of the mathematical foundations of Mamba and optimization techniques for implementation on graphics processors. For the purposes of this thesis, the Mamba architecture was implemented in the PyTorch framework, and evaluations were conducted on various natural language processing tasks.

• Keywords: Veliki jezični modeli; Transformatori; Mamba; Duboko učenje; Obrada prirodnog jezika
• Keywords (english): Large Language Models; Transformers; Mamba; Deep Learning; Natural Language Processing
• Language: croatian
• URN:NBN: urn:nbn:hr:168:591000
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2024-10-12 21:14:09