Vox Calculi: rješavanje matematičkih problema korištenjem transformera i automatskog prepoznavanja govora

master's thesis

• Title (english): Vox Calculi: Transformer based approach for solving mathematical problems using automatic speech recognition
• Author: Ante Grgurević
• Mentor: Marina Bagić Babac (mentor)
• Committee member: Marina Bagić Babac (predsjednik povjerenstva)
• Committee member: Mirjana Domazet-Lošo (član povjerenstva)
• Committee member: Marko Horvat (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2023-07-10, Croatia

Abstract

U ovom radu predstavljamo Vox Calculi - sustav razvijen za rješavanje matematičkih problema iz transkripcija govora. Korištenjem najnovijih predtreniranih modela automatskog prepoznavanja govora (ASR), dobivamo točne transkripcije iz snimki korisnikovog glasa. Posebno razvijeni matematički parser za ovaj rad pretvara matematičke izraze iz prirodnog jezika u matematičke simbole i brojeve, ostavljajući ostatak transkripcije netaknutim. Zatim koristimo modele Transformer i TP-Transformer trenirane na DeepMindovom Mathematics skupu podataka kako bismo generirali odgovarajući matematički odgovor iz zadane ulazne sekvence. Detaljno prikazujemo evaluaciju ASR modela i modela Transformera koji postižu vrlo zadovoljavajuće rezultate, te sugeriramo neka buduća poboljšanja koja bi nadalje unaprijedila razvijeni sustav.

Abstract (english)

In this thesis, we introduce Vox Calculi, a system designed to solve mathematical problems through voice transcriptions. By leveraging cutting-edge pretrained automatic speech recognition (ASR) models, we accurately transcribe user's voice recordings. Additionally, we develop a dedicated Mathematical parser for this thesis, which converts natural language mathematical expressions into mathematical symbols and numbers while preserving the rest of the transcription. We employ Transformer and TP-Transformer models trained on DeepMind's Mathematics dataset to generate the appropriate mathematical answer from the provided sequence. We provide a comprehensive evaluation of both the ASR models and the Transformer models, which yield highly satisfactory results. Furthermore, we propose potential future improvements that could enhance the system further.

• Keywords: matematičko zaključivanje; obrada prirodnog jezika; automatsko prepoznavanje govora; duboko učenje; transformer modeli
• Keywords (english): mathematical reasoning; natural language processing; automatic speech recognition; deep learning; transformer models
• Language: english
• URN:NBN: urn:nbn:hr:168:467226
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-10-11 08:57:52