Prikupljanje i analiza dnevničkih zapisa međusobno povezanih servisa u skoro realnom vremenu

master's thesis

• Title (english): Log aggregation and analysis of interconnected services in near real time
• Author: Iva Jutriša
• Mentor: Igor Mekterović (mentor)
• Committee member: Igor Mekterović (predsjednik povjerenstva)
• Committee member: Mario Brčić (član povjerenstva)
• Committee member: Ljiljana Brkić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2021-07-06, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Tema ovog diplomskog rada je izrada sustava za prikupljanje, obradu i prikaz dnevničkih zapisa međusobno povezanih servisa u skoro realnom vremenu. Na samom početku su obrađeni osnovni pojmovi vezani uz tokove podataka poput vremenskih domena, prozora i vodostaja. Zatim slijedi opis osnovnih funkcionalnosti tehnologija koje su korištene prilikom implementacije spomenutog prototipa sustava. Među njima se nalaze Apache Kafka, Spark Streaming i Spark Structured Streaming. Korisničko sučelje i web aplikacija ostvareni su korištenjem Vue.jsa i Dropwizarda. Sami sustav je testirana na dvama različitim tipovima dnevničkih zapisa, aplikacije za naplatu parkinga i nginx web poslužitelja sustava Edgar. Ostvaren je protočan cjeloviti cjevovod koji garantira „točno jednom” od odredišta do izvora, uz malo kašnjenje i ispravne rezultate.

Abstract (english)

The topic of this thesis is building a system for collecting, processing and displaying logs of interconnected services in almost real time. At the very beginning, the basic concepts related to data flows such as time domains, windows and watermarks are described. Then follows a descriptions of basic technology functionalities used in the implementation of said prototype system. These include Apache Kafka, Spark Streaming and Spark Structured Streaming. The user interface and web application were achieved using Vue.js and Dropwizard. The system itself was tested on two different log data types, of a parking billing app and the Edgar system’s nginx web server. A complete flow pipeline is achieved that guarantees "exactly once" from the destination to the source, with little delay and correct results.

• Keywords: tokovi podataka; Spark; Structured Streaming; Spark Streaming; Java; Kafka; proizvođač; potrošač; Dropwizard; Vue.js; obrada u realnom vremenu
• Keywords (english): data flows; Spark; Structured Streaming; Spark Streaming; Java; Kafka; Producer; Consumer; Dropwizard; Vue.js; real-time processing
• Language: croatian
• URN:NBN: urn:nbn:hr:168:924947
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 12:37:36