Fast distributed cross-matching of big astronomical data and parameter estimation of moving point source model

doctoral thesis

• Title (english): Brzo raspodijeljeno uparivanje velikih astronomskih podataka i procjena parametara modela gibajućega točkastog izvora
• Author: Petar Zečević
• Mentor: Sven Lončarić (mentor)
• Mentor: Mario Jurić https://orcid.org/0000-0003-1996-9252 (mentor)
• Committee member: Marko Subašić (predsjednik povjerenstva)
• Committee member: Zvonko Kostanjčar (član povjerenstva)
• Committee member: Vernesa Smolčić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; (Department of Electronic Systems and Information Processing); Zagreb
• Defense date and country: 2023-11-13, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing
• Universal decimal classification (UDC): 004 - Computer science and technology. Computing. Data processing

Abstract

The thesis develops two original methods: “Distributed Zones Algorithm” and “Online Multifit”. “Distributed Zones Algorithm” is a zone-based method for fast distributed crossmatching of big astronomical data. It is developed based on the Zones Algorithm and comprises a distributed data organization scheme and an efficient method of positionally joining data using a moving window. The algorithm is implemented within a system called AXS (Astronomical Extensions for Spark), based on Apache Spark and extended with astronomy-specific functionalities. The system’s cross-matching performance has been extensively tested in single-machine and “cloud” environments.
“OnlineMultifit” is a method for parameter estimation of a moving point-source model based on sequential Bayesian updating. It was developed by extending the Multifit method. OnlineMultifit utilizes Bayesian statistics to obtain the full posterior probability distribution of moving point source model parameters but it does so by processing one image at a time using “sequential updating” technique, updating the posteriors after each image. The method approximates posteriors as mixtures of Gaussians and the main challenge here is to reduce the errors introduced by these approximations. The final result is that the method uses less computational resources while obtaining full posterior estimates. Furthermore, the estimates can be stored in a compressed form (the means and covariances of Gaussian distributions).

Abstract (english)

Disertacijom se razvijaju dvije originalne metode: “algoritam baziran na distribuiranim zonama” i “slijedni Multifit”.
“Algoritam baziran na distribuiranim zonama” je metoda za brzo distribuirano uparivanje velikih astronomskih podataka, razvijena na temelju “algoritma baziranom na zonama”. Metoda se sastoji od sheme za distribuiranu organizaciju podataka i efikasne metode za poziciono uparivanje podataka pomoću pomičnog prozora. Algoritam je implementiran unutar sustava nazvanog AXS (eng. Astronomical Extensions for Spark), baziranog na Apache Spark platformi koja je proširena funkcionalnostima specifičnim za astronomiju. Performanse sustava u uparivanju detaljno su testirane na jednoj mašini te u okolini “u oblaku”.
“Slijedni Multifit” je metoda za estimaciju parametara modela gibajućeg točkastog izvora temeljena na slijednom Bayesovom osvježavanju. Razvijena je nadogradnjom Multifit metode. Slijedni Multifit koristi Bayesovu statistiku za dobivanje pune posteriorne distribucije vjerojatnosti parametara modela gibajućeg točkastog izvora, no to čini obradom jedne po jedne slike, metodom “slijednog osvježavanja”, osvježavajući posterior nakon svake slike. Metoda aproksimira posteriore kao skup Gaussovih distribucija te joj je glavni izazov u smanjenju grešaka koje takva aproksimacija unosi. Krajnji je rezultat taj da metoda koristi manje računalnih resursa, a procjenjuje potpune posteriorne distribucije vjerojatnosti. Štoviše, procjene se mogu spremiti u kompresiranoj formi (u obliku središnjih vrijednosti i kovarijanci Gaussovih distribucija vjerojatnosti).

• Keywords: astronomy; cross-matching; Apache Spark; Bayesian statistics; sequential updating
• Keywords (english): astronomija; uparivanje podataka; Apache Spark; Bayesova statistika; slijedno osvježavanje
• Language: english
• URN:NBN: urn:nbn:hr:168:820088
• Promotion: 2023
• Study programme: Title: Computer Science; Study programme type: university; Study level: postgraduate; Academic / professional title: Doktor znanosti (Doktor znanosti)
• Type of resource: Text
• Extent: 101 str. : graf. prikazi u boji ; 30 cm
• File origin: Born digital
• Access conditions: Open access
• Created on: 2024-05-10 13:52:00