Sustav za procjenu skalabilnosti platforme Apache Beam u obradi velikih količina prostorno vremenskih podataka

master's thesis

• Title (english): A System for Scalability Assessment of Apache Beam Platform in Processing Large Volumes of Spatio-Temporal Data
• Author: Aleksandar Novak
• Mentor: Igor Čavrak (mentor)
• Committee member: Igor Čavrak (član povjerenstva)
• Committee member: Danko Basch (član povjerenstva)
• Committee member: Ivana Bosnić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2022-07-13, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

S povećanjem broja IoT uređaja i podacima koje dobivamo od njih, novi način obrade tih podataka je trebao biti stvoren. Ti alati koji procesiraju tokove podataka dali su ogromnu prednost nad tradicionalnim bazama podataka. Postoji velika lista alata koje nam pružaju rad s tokovima podataka, pa je pitanje koje od njih odabrati. Kako smo mi limitirani i ne može se očekivati da ćemo se naučiti sve te različite alate i načine kako pisati programe specifično za njih, solucija je došla u obliku platforme Apache Beam koja je dala unificirani model koji se može spojiti na veliki broj izvođača. S Apache Beamom jedina stvar o kojoj se moramo brinuti je kako ćemo napisati program, ne kako će se taj program izvoditi na nekom izvođaču u pozadini. Ovaj rad se fokusira na pokušaju pokretanja program na što više izvođača i testirati njihovu performansu. Sustav je bio testiran nad 3 izvođača Apache Flink, Apache Spark i Hazelcast Jet. Testiranje je pokazalo da postoji limitacija u pitanju koliko možemo povećati broj jezgri i čvorova kako bi dobili bolje performanse. Također je pokazalo limitaciju u Jetu, gdje ne može čitati više od limitiranog broja entiteta nad Kafka topicom s particijama.

Abstract (english)

With an increase of IoT devices and the data that they provide, an entirely new way of processing that data had to be created. These tools for processing data flows in real time, gave a huge advantage as opposed to traditional databases. There exists a large list of tools that can provide us with the processing capabilities, so it begs the question of which to choose. As we are limited with time and can't be expected to learn all the different tools and the ways in which we can create a program for them, Apache Beam came with a solution with a unified model that would connect a large set of runners. With Apache Beam, the only thing we need to care for is what we are implementing, not how it is going to run in the background on a runner. This theses focuses on trying to execute a program to test the throughput of the tools as well as the throughput of the system on as many runners as we can and test their performance. The system was tested with Apache Spark, Apache Flink and Hazelcast Jet. The tests showed that there was a limit to how much you can increase the number of nodes and their CPU core counts to increase their performance. It also showed a limitation of Jet whereby it cannot read more than a limited number of records from partitioned Kafka topics.

• Keywords: Apache Spark; Apache Flink; Apache Beam; Apache Kafka; Hazelcast Jet; IoT; Internet Stvari; AWS; Tok podataka
• Keywords (english): Apache Spark; Apache Flink; Apache Beam; Apache Kafka; Hazelcast Jet; IoT; Internet of Things; AWS; Data Stream
• Language: croatian
• URN:NBN: urn:nbn:hr:168:549249
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 15:43:25