Primjena algoritama strojnog učenja nad tokom podataka u stvarnom vremenu

master's thesis

• Title (english): Realtime Machine Learning over Streaming Data
• Author: Maja Puh
• Mentor: Ljiljana Brkić (mentor)
• Committee member: Ljiljana Brkić (predsjednik povjerenstva)
• Committee member: Vedran Mornar (član povjerenstva)
• Committee member: Boris Milašinović (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2019-07-03, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Konstantan porast količine podataka koji proizlaze iz različitih područja korisničkog djelovanja u tehnološkoj domeni izazvao je potrebu za fokusiranim razvojem prikladnih analitika. Da bi iz velikih količina nestrukturiranih podataka nastalo znanje, potrebno ih je dubinski analizirati, što omogućuje posebna disciplina rastućeg područja strojnog učenja. Nadalje, procesuiranje podataka koji se kontinuirano generiraju mora se paralelizirati da bi obrada u stvarnom vremenu bila moguća, a to je ostvareno kroz distribuirane sustave za obradu podataka. U ovom radu opisani su konceptualni modeli distribuiranih programskih okvira za obradu tokova podataka te su prikazani konkretni sustavi dostupni u Hadoop ekosistemu: Apache Kafka, Apache Spark i Apache Flink. Implementiran je sustav koji povezuje navedene okvire s ciljem evaluiranja i uspoređivanja njihovih performansi, prilikom primjene algoritama strojnog učenja. Apache Kafka koristi se za perzistenciju podataka u jednoj i drugoj eksperimentalnoj inačici, dok je varijabilni dio sustava korištenje Spark, odnosno Flink programskog okvira. Korišteni su algoritmi unaprijed dostupni u programskim modulima odabranih sustava: SVM, linearna regresija i algoritam k-sredina. Odabrana metrika evaluacije proizlazi iz općenitih zahtjeva postavljenih pred distribuirane sustave, a temelji se na usporedbi skalabilnosti jednog i drugog okvira prilikom obrade konkretnog skupa podataka koji simulira tok podataka. Kroz sve provedene eksperimente Apache Spark daje bolje rezultate, što ne umanjuje prosperitetne mogućnosti Apache Flinka.

Abstract (english)

Immense growth of avaliable data generated from various fields of user interaction within technological area of domain has made the need for structured development of suitable analytics crucial. To be able to provide knowledge from vast amount od unstructured data it is essential to use data mining, extended to the specialized fast-growing field of machine learning. Furthermore, processing of continuously generated data has to be parallelized to make real-time processing possible, which is achieved through distributed data processing engines. This thesis provides description of conceptual model of distribuited data streaming processing engines as well as representation of specific engines available through Hadoop ecosystem: Apache Kafka, Apache Spark and Apache Flink. System which interconnects all of the specified engines is implemented in order to carrry out evaluation and comparison of their performance, while applying machine learning algorithms. Apache Kafka is used for securing persistence of data in both versions of experimental setup, while the use of Spark and Flink excludes one another. Selection of applied machine learning algorithms depended on their availability within libraries of chosen engines, it consisted of SVM, linear regression and k-means. Evaluation metric is based upon usual demands concerning distributed systems; it compares scalability of selected engines while performing data processing on same dataset which simulates data stream. Apache Spark outperformed Apache Flink through all of the conducted experiments, which does not diminish future prosperity of Apache Flink.

• Keywords: stvarnovremenska obrada; tokovi podataka; sustav za obradu tokova podataka; strojno učenje
• Keywords (english): Real-time processing; data streaming; data streaming processing engine; Machine learning
• Language: croatian
• URN:NBN: urn:nbn:hr:168:860875
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2020-01-07 21:38:33