Vektorska reprezentacija teksta

undergraduate thesis

• Title (english): Vectorial text representation
• Author: Fran Mesić
• Mentor: Mario Bukal (mentor)
• Committee member: Mario Bukal (predsjednik povjerenstva)
• Committee member: Domagoj Vlah (član povjerenstva)
• Committee member: Josip Žubrinić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2023-02-06, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

U ovom radu prikazana je hijerarhija četiriju modela vektorizacije teksta. BoW (engl. Bag of words) model sastoji se od najjednostavnijeg prebrojavanja broja pojavljivanja pojedine riječi u tekstu. Njegova primjena je najčešće u klasifikaciji dokumenata.
„Bag of n-grams“ model proširuje BoW model tako što uzima više riječi odjednom i time
dobiva kontekst, ali pod cijenu veće dimenzionalnosti.
TF-IDF (engl. Term-Frequency Inverse-Document-Frequency) model unaprjeđuje BoW
model tako što skalira broj pojavljivanja te uzima u obzir i broj dokumenata u kojima se
pojam nalazi. Najčešća primjena je u području povrata informacija.
Konačno, BE25 model proširuje TF-IDF model tako što za relevantnost uzima u obzir i odnose duljina tekstova te optimizira neke od jednadžbi TF-IDF-a.

Abstract (english)

In this thesis, a hierarchy of four text vectorization models is presented. The BoW (Bag of words) model consists of the simplest counting of the number of occurrences of a single word in a text. Its application is most often in the classification of documents.
The "Bag of n-grams" model extends the BoW model by taking multiple words at once, thus getting the context, but at the expense of higher dimensionality.
TF-IDF (Term-Frequency Inverse-Document-Frequency) model improves BoW
model by scaling the number of occurrences and taking into account the number of documents in which the word appears. The most common application is in the area of information retrieval.
Finally, the BE25 model extends the TF-IDF model by taking into account text length relationships and optimizing some of the TF-IDF equations.

• Keywords: Vektori; Filtriranje riječi; Označavanje dijelova govora; Lematiziranje; Povrat informacija; Klasifikacija dokumenata; Vrećica riječi; Frekvencija termina Inverzna dokumentna frekvencija
• Keywords (english): Vectors; Word filtering; Marking parts of speech; Lematization; Information retreival; Classification of documents; Bag of words; Term Frequency Inverse document frequency
• Language: croatian
• URN:NBN: urn:nbn:hr:168:783062
• Study programme: Title: Computing; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-10-11 09:48:23