Sustav za obradu polustrukturiranih tekstnih datoteka poslovnih transakcijskih podataka

master's thesis

• Title (english): System for processing of semi-structured text files of business transaction data
• Author: Ivan Derdić
• Mentor: Alan Jović (mentor)
• Committee member: Alan Jović (predsjednik povjerenstva)
• Committee member: Nikolina Frid (član povjerenstva)
• Committee member: Jan Šnajder (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2024-07-08, Croatia

Abstract

Ovaj diplomski rad bavi se razvojem sustava za obradu polustrukturiranih tekstual-
nih datoteka poslovnih transakcijskih podataka. U radu se analiziraju i implementiraju
tri modela obrade podataka: ETL (izvuci, obradi, spremi), ELT (izvuci, spremi, obradi)
i ETLL (izvuci, obradi i spremi, spremi). Implementacija koristi tehnologije Delta Lake
i Apache DataFusion, omogućujući učinkovitu obradu velikih količina podataka i osi-
guravajući svojstva ACID. Cilj rada je usporediti performance navedenih modela u kon-
tekstu brzine obrade. Eksperimentalni rezultati pokazuju prednosti i nedostatke svakog
modela, ovisno o specifičnim uvjetima i zahtjevima obrade podataka.
Model ELT2 ima najbolje performance u odnosu na modele ETL, ELT i ETLL. Model
ELT2 prvo sekvencijalno obradi datoteke u brončanu tablicu i zatim sve podatke iz bron-
čane tablice obradi u srebrne tablice. Model ELT2 je najbrži jer se podaci iz brončane
tablice učitavaju u memoriju samo jednom.

Abstract (english)

This thesis addresses the development of a system for processing semi-structured text
files of business transaction data. The study analyzes and implements three data process-
ing models: including ETL (Extract, Transform, Load), ELT (Extract, Load, Transform),
and ETLL (Extract, Transform, Load, Load) models. The implementation utilizes Delta
Lake and Apache DataFusion technologies, enabling efficient processing of large data
volumes while ensuring ACID properties. The aim is to compare the performance of
these models in terms of processing speed. Experimental results highlight the advan-
tages and disadvantages of each model, depending on specific conditions and data pro-
cessing requirements.
The ELT2 model has the best performance compared to the ETL, ELT, and ETLL
models. The ELT2 model first sequentially processes files into the bronze table and then
processes all the data from the bronze table into the silver tables. The ELT2 model is the
fastest because the data from the bronze table is loaded into memory only once.

• Keywords: Obrada podataka; model ETL; model ELT; model ETLL; Delta Lake; Apache DataFusion; Poslovni transakcijski podaci
• Keywords (english): Data processing; ETL model; ELT model; ETLL model; Delta Lake; Apache DataFusion; Business transaction data
• Language: croatian
• URN:NBN: urn:nbn:hr:168:789736
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2024-10-12 20:25:47