Obrada podataka u programskom okviru Apache Spark

undergraduate thesis

• Title (english): Data Processing Using Apache Spark
• Author: Niko Krvavica
• Mentor: Vladimir Čeperić (mentor)
• Committee member: Vladimir Čeperić (predsjednik povjerenstva)
• Committee member: Alma Traživuk (član povjerenstva)
• Committee member: Vlatko Galić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2020-09-12, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

U prvom dijelu ovog rada prikazana je arhitektura programskog okvira Apache Spark-a, dok je u drugom dijelu prikazana obrada skupa podataka i korištenje klasifikacijskih i regresijskih algoritama koji se nalaze unutar Spark-ove knjižice MLlib. Cijeli postupak obrade, treniranja i evaluacije modela napisan je u programskom jeziku Python. Za prikaz korištenja klasifikacijskih algoritama korišteni su naivni Bayesov klasifikator i klasifikator stabla odluke dok su za prikaz korištenja regresijskih algoritama korišteni algoritam linearne regresije, algoritam GBT i algoritam slučajne šume.

Abstract (english)

In this paper, the first part explains the architecture of the Apache Spark framework, while the second part presents the processing of the data set and the use of classification and regression algorithms found within Spark's MLlib. The entire implementation of model processing, training and evaluation is written in the Python programming language. A naive Bayesian classifier and a decision tree classifier were used to illustrate the use of classification algorithms, while a linear regression algorithm, a GBT algorithm, and a random forest algorithm were used to illustrate the use of regression algorithms.

• Keywords: Apache Spark; regresijski algoritmi; klasifikacijski algoritmi; naivni Bayes; linearna regresija; algoritam stabla odluke; algoritam slučajne šume; GBT
• Keywords (english): Apache Spark; regression algorithms; classification algorithms; naive Bayes; linear regression; decision tree algorithm; random forest algorithm; GBT
• Language: croatian
• URN:NBN: urn:nbn:hr:168:121974
• Study programme: Title: Computing; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-14 20:54:05