Upravljanje i obrada velike količine podataka koristeći Delta format

master's thesis

• Title (english): Managing and processing of large amounts of data using Delta format
• Author: Lovro Bučar
• Mentor: Mario Matijević (mentor)
• Committee member: Mario Matijević (predsjednik povjerenstva)
• Committee member: Siniša Šadek (član povjerenstva)
• Committee member: Krešimir Trontl (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2024-09-11, Croatia

Abstract

Ovaj rad primarno analizira Delta format, kako je nastao, koje ga karakteristike uzdižu nad konkurencijom i u kojim je slučajevima korištenja bi se trebao koristiti.
Glavna odlika te tehnologije je dnevnik zapisa (engl. transaction log) koji omogućuje sigurnije izvođenje transakcija i povratak u prijašnja stanja baze.
Kroz rad je također pokazano da taj format u sebi ima ugrađene metode optimizacije čitanja, preciznije, particioniranje zapisa i Z-Order.
Potonja optimizacija omogućuje efikasnije čitanje podataka za složenije upite.
Također je kroz rad opisana i medaljna arhitektura, njezina povijest, zašto je nastala i koji problemi su njome rješeni.
Praktični dio rada se sastojao od izgrađivanja sustava za pohranu podataka baziranog na medaljnoj arhitekturi i Delta formatu koji unosi podatke s dva izvora, obrađuje ih i sprema u tablice koje su optimizirane za čitanje.
Naposlijetku, radila su se mjerenja vremena izvođenja određenih operacija čitanja kojima se odredilo u kojim slučajevima treba koristiti te optimizacije.

Abstract (english)

This thesis primarily analyses Delta format, how it was made, which charactaristics lift it above its competition, and in which use cases it should be used.
The main feature of this technology is the transaction log that enables a safer performance of transactions, and returning to previous database states.
Through the thesis it is shown that this format has built-in reading optimisations, more specifically, data partitioning and Z-Order.
The latter optimisation allows more efficient reading for more complex queries.
Furthermore, thesis also describes the medallion architecture, its history, why it was created, and which problems were solved with it.
Practical part of the thesis was consisted of developing the data storage system based on the medallion architecture and Delta format which ingests data from two sources, processes and stores them into tables which are optimised for reading.
Lastly, the execution times of reading operations were measured, which showed when those optimisations should be used.

• Keywords: Delta format; Medaljna arhitektura; Data Lakehouse; Delta Lake; Obrada podataka
• Keywords (english): Delta format; Medallion architecture; Data Lakehouse; Delta Lake; Data processing
• Language: croatian
• URN:NBN: urn:nbn:hr:168:316045
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2024-10-12 20:13:06