Mjerenje performanci platforme Apache Beam u obradi prostorno vremenskih tokova podataka

master's thesis

• Title (english): Performance Measurement of the Apache Beam Platform in Processing of Spatio-Temporal Data Streams
• Author: Mateo Žaja
• Mentor: Igor Čavrak (mentor)
• Committee member: Igor Čavrak (predsjednik povjerenstva)
• Committee member: Ivana Bosnić (član povjerenstva)
• Committee member: Tomislav Lipić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2021-02-17, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Razvojem interneta stvari, počinje se detaljnije razmatrati problematika obrade velikih količina podataka. Sustavi za upravljanje tokom podataka posjeduju mehanizme koji omogućuju obradu podataka u stvarnome vremenu, stoga se koncept toka podataka počinje koristiti za obradu podataka iz domene interneta stvari. Ovisno o zahtjevima sustava, potrebno je odabrati odgovarajući alat za obradu toka podataka. Platforma Apache Beam nudi uniformno sučelje koje omogućava obradu prostorno-vremenskih tokova podataka pomoću većeg broja alata za obradu toka podataka bez poznavanja detalja njihove implementacije. Za mjerenje performanci platforme Apache Beam korišteni su podaci iz domene interneta stvari. Mjerenje se sastojalo od računanja propusnosti alata Apache Flink i Apache Spark koristeći varijabilan broj procesorskih jezgri HTC Cloud infrastrukture. Rezultati su pokazali da povećanjem broja procesorskih jezgri raste propusnost sustava, ali samo do gornje granice broja procesorskih jezgri. Za određivanje optimalnog broja procesorskih jezgri potrebno je poznavati zahtjeve sustava te empirijski testirati performance sustava koristeći varijabilan broj procesorskih jezgri.

Abstract (english)

With the development of the Internet of Things, the issue of processing large amounts of data is beginning to be considered in more detail. Data stream management systems possess mechanisms that allow real-time data processing, so the concept of data stream is beginning to be used in processing data from the domain of the Internet of Things. Depending on the system requirements, it is necessary to select the appropriate data stream management system. The Apache Beam platform offers a uniform interface that allows spatio-temporal data streams to be processed using a number of data stream processing tools without knowing the details of their implementation. Data from the Internet of Things domain was used to measure the performance of the Apache Beam platform. The measurement consisted of calculating the throughput of the Apache Flink and Apache Spark using a variable number of HTC Cloud processor cores. The results showed that increasing the number of processor cores increases system throughput, but only up to the upper limit of the number of processor cores. To determine the optimal number of processor cores, it is necessary to know the requirements of the system and empirically test the performance of the system using a variable number of processor cores.

• Keywords: Apache Beam; Apache Spark; Apache Flink; Apache Kafka; HTC Cloud; Internet stvari; Propusnost; Tok podataka
• Keywords (english): Apache Beam; Apache Spark; Apache Flink; Apache Kafka; HTC Cloud; IoT; Throughput; Data stream
• Language: croatian
• URN:NBN: urn:nbn:hr:168:178023
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 14:20:08