Programski sustav za izdvajanje glavnog tekstualnog sadržaja iz web-stranica

master's thesis

• Title (english): Programming System for the Extraction of Main Textual Content from Web Pages
• Author: Dora Budić
• Mentor: Siniša Srbljić (mentor)
• Committee member: Siniša Srbljić (predsjednik povjerenstva)
• Committee member: Klemo Vladimir (član povjerenstva)
• Committee member: Goran Delač (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2017-07-03, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

U sklopu rada proučavaju se različite metode i dostupni alati otvorenog koda (eng. Open Source) za izdvajanje glavnog tekstualnog sadržaja iz web-stranica. Također, zbog nedostatka programskih sustava za usporedbu takvih alata, razvijen je takav programski sustav pomoću kojeg se, unosom željenog URL-a web-stranice i manualno izdvojenog glavnog sadržaja, pokreće automatizirano izdvajanje sadržaja te se prikazuju pojedinačne statističke metrike za svakog. Te statističke metrike čine: vrijeme izvođenja, preciznost, povrat te F1 rezultat. Alati se uspoređuju i na skupu stranica: sve unesene stranice, kategorije stranica (vijesti, blog, forum, stranice za prodaju), jezici stranica (hrvatski, engleski, njemački). Za implementaciju navedenog programskog sustava, zbog njegovih prednosti u izgradnji takvog sustava, koristio se Docker.

Abstract (english)

In this thesis, different methods and open source tools for extracting main textual content from web pages are researched. Also, due to the lack of programming systems for comparing such tools, a programming system has been developed which, by entering the desired URL of the web page and manually extracted main content, initiates automated content extraction and displays statistical metrics for each. These statistical metrics include: run time, precision, recall and F1 score. The tools are also compared to the sets of pages: all entered pages, categories of pages (news, blogs, forums, sales pages), languages of pages (Croatian, English, German). Docker was used to implement this programming system, due to its many advantages.

• Keywords: izdvajanje tekstualnog sadržaja; web-stranica; tehnike izdvajanja; alati za izdvajanje; Boilerpipe; Dragnet; Newspaper; Goose; Docker
• Keywords (english): textual content extraction; web site; extraction techniques; extraction tools; Boilerpipe; Dragnet; Newspaper; Goose; Docker
• Language: croatian
• URN:NBN: urn:nbn:hr:168:074194
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2019-03-13 19:24:21