Integracija podataka iz transakcijskih baza podataka u skladište podataka

master's thesis

• Title (english): Integration of data from transactional databases into data warehouse
• Author: Roko Zubčić
• Mentor: Boris Vrdoljak (mentor)
• Committee member: Boris Vrdoljak (predsjednik povjerenstva)
• Committee member: Ivica Botički (član povjerenstva)
• Committee member: Mirta Baranović (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2018-07-11, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Integracija podataka u procesu skladištenja podataka postiže se ETL procesom. Isti se sastoji od brojnih podsustava i faza izvršavanja unutar kojih se podaci s izvornog sustava dohvaćaju, transformiraju i pohranjuju u odredišne strukture. Porastom količine podataka koja se prikuplja rastu i zahtjevi za računalnim resursima okoline u kojoj se ETL proces izvršava. SQL Server Integration Services nudi mogućnost implementacije integracijskih rješenja koja se mogu postaviti i izvršavati na vlastitoj IT infrastrukturi nekog poduzeća. Azure Data Factory je platforma kao usluga koja omogućava migraciju takvih rješenja u oblak. U ovom radu opisana je implementacija ETL rješenja koje se snažno oslanja na metapodatke. Zadaci kojima se ostvaruju funkcionalnosti dohvata promjena s izvora, inkrementalnog unosa dimenzijskih i činjeničnih zapisa, upravljanja sporomijenjajućim dimenzijama i kasnodolazećim dimenzijskim zapisima te izvještavanja ostvareni su kao SSIS paketi. Opisana je i migracija takvog rješenja u Azure oblak te izvršavanje istog pomoću Azure Data Factory platforme.

Abstract (english)

ETL process is part of a data warehousing where data is integrated from source systems into data warehouse. There are multiple ETL subsystems implementing different ETL phases. Data is collected from its original data sources, it is transformed and loaded into a final set of structures. With continuous growth of data that is being collected and processed there is a greater need for more powerful IT environment that hosts ETL process. SQL Server Integration Services can be used for building data integration solutions. Azure Data Factory is platform as a service that enables migration of those solutions to the Azure cloud. This thesis describes implementation of metadata-driven ETL solution. Tasks for extracting the data from the source systems, incremental fact and dimension table loading, handling late arriving dimensions and notifying were implemented as SSIS packages. Finally it is described how to lift and shift those packages to the Azure cloud using Azure Data Factory.

• Keywords: Integracija podataka; ETL proces; Skladištenje podataka; SQL Server Integration Services; Azure Data Factory
• Keywords (english): Data integration; ETL process; Data Warehousing; SQL Server Integration Services; Azure Data Factory
• Language: croatian
• URN:NBN: urn:nbn:hr:168:381403
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2019-03-14 02:21:26