Skriptni radni okvir za ekstrakciju, transformaciju i učitavanja podataka

master's thesis

• Title (english): Script-oriented framework for data extraction, transformation and loading
• Author: Luka Lazanja
• Mentor: Igor Mekterović (mentor)
• Committee member: Igor Mekterović (predsjednik povjerenstva)
• Committee member: Boris Milašinović (član povjerenstva)
• Committee member: Ljiljana Brkić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2020-07-10, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Proces ekstrakcije, transformacije i učitavanja podataka često se koristi kod izrade skladišta podataka. Većina postojećih alata bazira se na grafičkom sučelju, što nije pogodno za programere, zbog čega se predlaže korištenje skriptnih alata u izvršnom okruženju Node.js. U radu je napravljen pregled postojećih skriptnih alata te predstavljen skriptni radni okvir koji na jednostavan i efikasan način odrađuje integraciju. Okvir nudi razne mogućnosti, poput korištenja toka podataka, paralelnog izvođenja zadataka, vođenja dnevnika i generiranja algoritma toka izvođenja. Također, opisan je i algoritam za vremenski ograničenu detekciju razlika dvaju relacija velike kardinalnosti i ukomponiran u algoritam. Okvir je testiran na raznim primjerima koji su dostupni u izvornom kodu, od kojih je jedan stvaranje skladišta podataka iz poznate baze NorthWind.

Abstract (english)

Data extraction, transformation and loading process is often used in data warehouse creation. Most of the existing tools are based on graphical user interface, which is not very suitable for programmers. For that reason, this thesis suggests using script framework in runtime environment Node.js. Overview of most popular scripting tools has been made and another simplified and effective solution has been presented. Moreover, algorithm for time constrained detection of differences between two large relations has been studied and incorporated in the project. The framework has been tested on many examples, which are available in the source code, one of which includes creation of data warehouse from popular database NorthWind.

• Keywords: alat; tokovi podataka; upravljački programi; baze podataka; algoritam
• Keywords (english): tool; data streams; drivers; databases; algorithm
• Language: croatian
• URN:NBN: urn:nbn:hr:168:774056
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 10:17:58