Analiza metoda za poboljšanje parametarske učinkovitosti neuronskih jezičnih modela

master's thesis

• Title (english): Analysis of methods for improving the parameter efficiency of neural language models
• Author: Mislav Perić
• Mentor: Jan Šnajder (mentor)
• Committee member: Jan Šnajder (predsjednik povjerenstva)
• Committee member: Marko Čupić (član povjerenstva)
• Committee member: Domagoj Jakobović (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2024-09-16, Croatia

Abstract

Ovaj rad se bavi analizom metoda za poboljšanje parametarske učinkovitosti neuronskih jezičnih modela u kontekstu transformatorske arhitekture koristeći modele RoBERTa-e i OPT-a. Dana je motivacija i pregled metoda parametarske učinkovitosti (PEFT) te je eksperimentima ispitana njihova uspješnost u režimu s malo podataka. U radu su ispitane metode: LoRA, (IA)3, prilagodba prefiksa i LoReFT. Rezultati pokazuju da metode PEFT-a uglavnom postižu rezultate usporedive s finim podešavanjem, ali podešavajući znatno manji broj parametara modela. Predloženi su smjerovi za buduća istraživanja.

Abstract (english)

This thesis is concerned with the analysis of methods for improving the parameter efficiency of neural language models in the context of the Transformer architecture using RoBERTa and OPT models. The motivation and overview of parameter-efficient fine-tuning (PEFT) methods are provided, and their success is experimentally tested in a low-data regime. The methods examined include LoRA, (IA)3 Prefix tuning, and LoReFT. The results show that PEFT methods generally achieve results comparable to fine-tuning, but with significantly fewer model parameters. Future research directions are proposed.

• Keywords: obrada prirodnog jezika; strojno učenje; jezični modeli; PEFT; oskudnost podataka
• Keywords (english): natural language processing; machine learning; language models; PEFT; data scarcity
• Language: croatian
• URN:NBN: urn:nbn:hr:168:008484
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2024-11-07 05:41:53