Primjena generativnoga jezičnog modela na zadatke odgovaranja na pitanja i jezičnog modeliranja

undergraduate thesis

• Title (english): Application of a generative language model to question answering and language modeling tasks
• Author: Jerko Šegvić
• Mentor: Jan Šnajder (mentor)
• Committee member: Jan Šnajder (predsjednik povjerenstva)
• Committee member: David Dukić (član povjerenstva)
• Committee member: Josip Jukić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2023-07-05, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Rad proučava mogućnosti generativnog jezičnog modela GPT-2. Detaljno je objašnjena ideja iza generativnih jezičnih modela i principi na kojima se temelje. Objašnjena je arhitektura modela i mehanizmi koje konkretan model koristi. Performanse modela mjerene su na tri različita skupa podataka bez ugađanja i s ugađanjem na podskupovima za treniranje. Skupovi podataka testiraju sposobnost modela za rješavanje jezičnih zadataka odgovaranja na pitanja i jezičnog modeliranja. Za svaki skup je objašnjena njegova struktura, naveden broj primjera na svakom podskupu te objašnjeno što je njime testirano. Da bi model uspješno savladao te zadatke, mora moći pratiti dulji kontekst i dobro razumjeti sadržaj teksta. Opisan je postupak treniranja i nevedeni su hiperparametri korišteni za treniranje na svakom skupu. Provedena je detaljna analiza rezultata prije i nakon ugađanja modela. Komentirani su nedostatci i predloženi koraci za daljnji razvoj.

Abstract (english)

This work studies the abilities of the generative language model GPT-2. The general idea behind generative language models and the principles on which they are based are explained in great detail. The architecture of GPT-2 and the mechanisms used by it are also explained. The performance of the model is measured on three different datasets in the zero-shot setup and with fine-tuning on the training subsets. The datasets test the model's ability to handle more complex language tasks. The structure of each dataset is explained in detail. For each dataset, the size of each subset is listed, and it is explained which tasks they are testing. To succeed on these tasks, the model must be able to follow the longer context and understand it well. The training procedure is described, and hyperparameters used to train each dataset are listed. A detailed analysis of the results before and after fine-tuning was carried out. The deficiencies of training procedures are commented. The steps for further development are also suggested.

• Keywords: umjetna inteligencija; strojno učenje; duboko učenje; obrada prirodnog jezika; generativni modeli; kauzalno jezično modeliranje; transformator; GPT-2
• Keywords (english): artificial inteligence; machine learning; deep learning; natural language processing; generative models; causal language modeling; transformer; GPT-2
• Language: croatian
• URN:NBN: urn:nbn:hr:168:981173
• Study programme: Title: Computing; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-10-11 10:38:13