Analiza i vizualizacija velikih podataka u Apache Spark okruženju

master's thesis

• Title (english): Big data analysis and visualisation in Apache Spark framework
• Author: Josipa Kovačev
• Mentor: Mirta Baranović (mentor)
• Committee member: Mirta Baranović (predsjednik povjerenstva)
• Committee member: Igor Mekterović (član povjerenstva)
• Committee member: Ljiljana Brkić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2015-07-08, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Područje analize podataka suočava se s eksponencijalnim rastom količine podataka koje je teško obrađivati koristeći tradicionalne metode i alate. Apache Spark omogućava skalabilnu i brzu raspodijeljenu obradu velikih podatka u grozdovima računala.
Spark uz pomoć memorijskih apstrakcija drži podatke u radnoj memoriju tijekom cijelog rada što mu omogućava izbjegavanje ulazno-izlaznih uskih grla pri zapisavanju i čitanju s diska čime postiže veće brzine od sličnih platformi. Dozvoljene su samo krupno-zrnate transformacije čime je efikasno osigurana otpornost na greške kroz logiranje transformacija bez replikacije podataka.
Spark sadrži i posebne module koje je moguće kombinirati: Spark SQL za rad sa strukturiranim podacima, MLlib za strojno učenje, GraphX za operacije s grafovima i Spark Streaming za tokove podataka.

Abstract (english)

Modern data analysis is faced with the exponential growth of data volumes difficult to process using traditional methods and tools. Apache Spark provides scalable and fast distributed processing of Big Data in a cluster.
Spark uses memory abstraction called Resilient Distributed Datasets to perform all computations in memory which allows him to avoid disk I/O bottlenecks, thus achieving a greater speed than other similar platforms. RDDs provide an interface based on coarse-grained transformations enabling them to efficiently provide fault tolerance by logging the transformations rather than replicating the actual data across the nodes.
Spark contains special modules that can be combined: Spark SQL for working with structured data, MLlib for machine learning, GraphX for graph operations and Spark Streaming for data streams analysis.

• Keywords: Big Data; analiza podataka; veliki podaci; znanost o podacima; Apache Spark; RDD; Spark SQL; MLlib; GraphX; Spark Streaming; DataFrame; DStreams; grupiranje; algoritam k-srednjih vrijednosti; Google Charts; vizualizacija velikih podataka
• Keywords (english): Big Data; data analysis; Data Science; Apache Spark; RDD; Spark SQL; MLlib; GraphX; Spark Streaming; DataFrame; DStreams; clustering; k-means algorithm; Google Charts; Big Data visualisation
• Language: croatian
• URN:NBN: urn:nbn:hr:168:866555
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2018-02-06 09:49:15