Upotreba tehnologija za raspodijeljenu obradu velikih količina podataka u knjigovodstvu

master's thesis

• Title (english): Using Big Data Computing Technologies for Bookkeeping
• Author: Filip Pažanin
• Mentor: Alan Jović (mentor)
• Committee member: Alan Jović (predsjednik povjerenstva)
• Committee member: Ante Đerek (član povjerenstva)
• Committee member: Željka Mihajlović (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2022-07-06, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

U tvrtki APIS IT zadatak je bilo smanjiti vrijeme potrebno za knjigovodstvenu obradu podataka. Podatci su bili veličine nekoliko gigabajta te se obrada trebala prevesti na paralelnu paradigmu i zatim pokrenuti na računalnom grozdu. Grozd je imao resurse nekolicine osrednjih stolnih računala. Proučavanjem tehnologija velikih količina podataka kao što su HDFS, YARN, Spark i Parquet zadatak je uspješno izvršen te se upotrebom niti polovice resursa dostupnog računalnog grozda smanjilo vrijeme obrade za 5 puta. U radu su opisane sve gore navedene tehnologije te su detaljno prikazani rezultati ispitivanja raznih konfiguracija obrade podataka na računalnom grozdu. Neke od konfiguracija odabrane su kao optimalne, a taj je izbor obrazložen u radu.

Abstract (english)

At APIS IT, the task was to reduce the time required for bookkeeping data processing. The data was a few gigabytes in size, so processing had to be translated to a parallel paradigm and then run on a computer cluster. The cluster had the resources of a few average desktop computers. By studying big data technologies such as HDFS, YARN, Spark and Parquet, the task was completed successfully. The solution included threading of half the resources of the available computer cluster, which reduced the processing time 5 times. The paper describes all the above technologies and presents in detail the results of various data processing configurations testing on the computer cluster. Some of the configurations were chosen as optimal and the choice was explained.

• Keywords: veliki podaci; knjigovodstvo; računalni grozd; Hadoop; YARN; Spark; Parquet; testiranje performansi; paralelizacija
• Keywords (english): big data; bookkeeping; computer cluster; Hadoop; YARN; Spark; Parquet; performance testing; parallelization
• Language: croatian
• URN:NBN: urn:nbn:hr:168:566872
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 15:50:17