Primjena strojnog učenja pri čišćenju podataka

master's thesis

• Title (english): Applying maching learning to data cleansing
• Author: Vitomir Marjanović
• Mentor: Boris Vrdoljak (mentor)
• Committee member: Boris Vrdoljak (predsjednik povjerenstva)
• Committee member: Mario Brčić (član povjerenstva)
• Committee member: Mirta Baranović (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2021-07-06, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Electrical Engineering

Abstract

Cilj istraživanja provedenog ovim diplomskim radom je ispitati razne metode strojnog učenja pri postupku čišćenja podataka u svrhu unapređenja daljnjeg procesa analize podataka. Postupak čišćenja podijeljen je u tri koraka – uklanjanje dupliciranih vrijednosti, detekcija i uklanjanje anomalija te umetanje vrijednosti koje nedostaju. Za svaki od navedenih koraka ispitane su implementacije odabranih algoritama u programskim jezicima Python i R te je u konačnici napravljena njihova usporedba. Prilikom analize odabranih metoda, korišteni su skupovi podataka s karakteristikama koje odgovaraju scenarijima koje se ispituju kako bi rješenje bilo što uočljivije i kvalitetnije. Po završetku usporedbe odabranih metoda, prikazani su zaključci te potencijalni problemi prilikom implementacije navedenih metoda i samog postupka čišćenja u cijelosti.

Abstract (english)

The aim of the conducted research as a part of this master's thesis was to invesigate different machine learning methods in data cleansing in order to improve the following dana analysis process. Data cleansing process was divided in three steps – removing duplicated values, anomaly detection and anomaly removal and missing values imputation. For each of the named steps a selected set of algorithms implemented in Python and R programming languages were tested, and the testing results were provided and compaired to summarize it. In the process, the datasets for the testing were selected to have as many characteristics that describe the tested problem as possible so the output results are clearly visible and representative. By the end of the method comparison, some important conclusions were made, but also some of the underlying problems were pointed out when implementing the tested methods and conducting a data cleansing process as a whole.

• Keywords: čišćenje podataka; strojno učenje; uklanjanje duplikata; detekcija anomalija; imputacija; regresija; klasifikacija; python; scikit-learn; pandas; dedupe; r; mice
• Keywords (english): data cleansing; machine learning; duplicates removal; anomaly detection; imputation; regression; classification; python; scikit-learn; pandas; dedupe; r; mice
• Language: croatian
• URN:NBN: urn:nbn:hr:168:033600
• Study programme: Title: Information and Communication Technology; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka informacijske i komunikacijske tehnologije (magistar/magistra inženjer/inženjerka informacijske i komunikacijske tehnologije)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 15:31:14