Filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark

undergraduate thesis

• Title (english): Filtering of Geospatial Data Streams using Apache Spark Platform
• Author: Darko Britvec
• Mentor: Krešimir Pripužić (mentor)
• Committee member: Krešimir Pripužić (predsjednik povjerenstva)
• Committee member: Lucija Petricioli (član povjerenstva)
• Committee member: Matija Šulc (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2019-07-12, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Cilj ovog rada je stvaranje učinkovitog raspodijeljenog sustava za filtriranje geoprostornog toka podataka korištenjem platforme Apache Spark. Arhitektura korištena u implementaciji programskog rješenja naziva se „objavi-pretplati“. Objave u sustavu predstavljaju poruke formata GeoJSON koje sadrže geoprostornu značajku (točku, liniju, poligon) te dodatne informacije. Pretplate u sustavu predstavljaju poligoni u prostoru uz koje vežemo dodatne značajke. Tok podataka implementiran je pomoću alata Apache Kafka. Za konzumaciju toka podataka koristi se Spark Streaming koji tok obrađuje u dijelovima koji se mogu primiti u određenom vremenskom intervalu (engl. micro-batch). Za filtriranje geoprostornih podataka korištena je programska knjižnica GeoSpark koja dodatno koristi knjižnice Java Topology Suite i Geotools. Učinkovito filtriranje prostornih podataka ostvareno je pomoću particioniranja skupa pretplata pomoću različitih strategija particioniranja. Izlaz sustava je poruka u formatu JSON koja sadrži identifikator objave i pripadne pretplate. U evaluaciji rješenja isprobane su strategije particioniranja pomoću Hilbertove krivulje, KDB-stabla, Q-stabla i R-stabla. Dodatno, za poboljšanje rezultata ispitano je i korištenje geoprostornih indeksa ostvarenih pomoću R-stabla i Q-stabla. U evaluaciji je također ispitan utjecaj korištenja pričuvne memorije za spremanje skupa pretplata.

Abstract (english)

The goal of this paper is to create an efficient distributed system for filtering of geospatial dana streams using the Apache Spark platform. Architecture used in the implementation is based on publish-subscribe pattern. Incoming messages in system carry a geospatial feature (eg. point, line or polygon on Earth) and some additional data formatted with GeoJSON specification. Geospatial subscriptions represent a set of polygons on Earth with some additional data. Geospatial data stream is implemented with Apache Kafka tool. Spark Streaming platform is used for consuming the incoming messages. It handles messages in portions (micro-batches) consumed within a time interval. GeoSpark library along with Java Topology Suite and GeoTools are used for processing the geospatial data. The effectiveness of geospatial filtering is improved with partitioning of the subscriptions dataset by using a different partitioning strategies. The system output consists of the identifiers of both messages corresponded subscriptions. Partitioning strategies used in the experiment are ones based on Hilbert curve, KDB-tree, Q-tree and R-tree. Additionally, evaluations examined the impact of using geospatial indexes based on Q-tree and R-tree and also the impact of using the cache functionality for persisting the subscription dataset in the memory of worker processes.

• Keywords: raspodijeljeni sustav; Apache Spark; objavi-pretplati; geoprostorni tok; GeoJSON; Apache Kafka; Spark Streaming; GeoSpark; Java Topology Suite; Geotools; particioniranje; Hilbertova krivulja; Voronoijev dijagram; R-stablo; Q-stablo; KDB-stablo; prostorni indeks
• Keywords (english): distributed system; Apache Spark; publish-subscribe; geospatial data stream; GeoJSON; Apache Kafka; Spark Streaming; GeoSpark; Java Topology Suite; Geotools; partitioning; Hilbert curve; Voronoi diagram; R-tree; Q-tree; KDB-tree; geospatial index
• Language: croatian
• URN:NBN: urn:nbn:hr:168:185017
• Study programme: Title: Computing; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2020-01-07 17:02:22