Analiza dnevničkih zapisa raspodijeljenih sustava u stvarnom vremenu

master's thesis

• Title (english): Real-Time Log Analysis for Distributed Systems Monitoring
• Author: Matej Čubek
• Mentor: Damir Pintar (mentor)
• Committee member: Damir Pintar (predsjednik povjerenstva)
• Committee member: Mihaela Vranić (član povjerenstva)
• Committee member: Luka Humski (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2023-07-13, Croatia

Abstract

U radu naslovljenom „Analiza dnevničkih zapisa raspodijeljenih sustava u stvarnom vremenu“ istražene su mogućnosti Apache Kafka platforme i komponenti ELK-a (ElasticSearch, Logstash i Kibana) za obradu i analizu dnevničkih zapisa raspodijeljenih mikro-servisnih aplikacija koje se izvode na Kubernetesu u stvarnom vremenu. Rad koristi OpenAI API za obogaćivanje dnevničkih zapisa s dodatnim kontekstom i informacijama. Implementirani su mehanizmi za stvarno-vremenske notifikacije i agregaciju dnevničkih zapisa koristeći Kafka Streams, što omogućuje pravovremeno otkrivanje i reagiranje na anomalije u sustavu. Korištenjem SaaS aplikacije CloudVane tvrtke Neos kao primjera, pokazano je kako se ove tehnologije mogu primijeniti u nadzoru dnevničkih zapisa u složenim distribuiranim sustavima. Ovaj rad naglašava važnost dnevničkih zapisa za otkrivanje anomalija i nadzor performansi te ulogu Apache Kafka i ELK platforme u tom kontekstu.

Abstract (english)

In the thesis titled „Real-time Log Analysis of Distributed Systems“, the potential of Apache Kafka platform and ELK stack components (ElasticSearch, Logstash, and Kibana) for real-time processing and analysis of distributed microservice applications running on Kubernetes was explored. The study utilizes the OpenAI API for enriching log files with additional context and information. The thesis introduces mechanisms for real-time notifications and log aggregations using Kafka Streams enabling timely detection and response to system anomalies. Using Neos’s SaaS application CloudVane as an example, the study demonstrates how these technologies can be implemented in monitoring log files in complex distributed systems. This work emphasizes the importance of log files for anomaly detection and performance monitoring, as well as the role of Apache Kafka and ELK platforms in this context.

• Keywords: Kubernetes; Apache Kafka; Tokovi Podataka; Dnevnički zapisi; Obrada u stvarnom vremenu; Elastic; ELK; Nadzor aplikacija; Grozd računala; Logstash; Filebeat; ElasticSearch; Kibana; Minikube; Oblak
• Keywords (english): Kubernetes; Apache Kafka; Streaming; Real-Time; Logs; Elastic; ELK; Monitoring; Cluster; Logstash; Filebeat; ElasticSearch; Kibana; Minikube; Cloud
• Language: croatian
• URN:NBN: urn:nbn:hr:168:430016
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-10-11 08:38:55