Razvoj agenta korištenjem potpomognutog učenja u parcijalno primjetljivom okruženju

undergraduate thesis

• Title (english): An attempt of reinforcement learning in a partially observable environment
• Author: Laura Torić
• Mentor: Domagoj Jakobović (mentor)
• Committee member: Domagoj Jakobović (predsjednik povjerenstva)
• Committee member: Karlo Knežević (član povjerenstva)
• Committee member: Ivan Kovačević (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2020-06-29, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Svrha ovog projekta bila je riješiti problem učenja u složenom djelomično promatranom okruženju pomoću potpomognutog učenja. Igra za više igrača pod nazivom DotA 2 izabrana je kao okolina jer je i izazovna i vizualno ugodna. Agent je komunicirao s igrom putem dobivenog poslužitelja i dodatka za igru (engl. Steam Add-on) (zvanog Breezy), koji je na kraju i sam bio izazov. Krajnji rezultati pokazali su se razocaravajuće lošima. Prvotna ideja je bila spajanje Duboke Q-Mreže (engl. Deep Q-Network) s genetskim algoritmom, ali kasnijim je istraživanjem je odabrana metoda Hijerarhijske Duboke Q-Mreže (engl. Hierarchical Deep Q-Network) kao strategija koja obećava. Nažalost, zbog nedostatka resursa (dostupno samo jedno računalo), nedosljedne mrežne veze i velikog trajanja epizoda, izvelo se samo 250 epizoda za hDQN, što nije bilo dovoljno za postizanje željene konvergencije performansi. Unatoč nedostatku prihvatljivih rezultata, ovaj je projekt pokazao stvarnu važnost modeliranja ispravnih stanja i nagrade, kao i razlike između različitih strategija Q-učenja.

Abstract (english)

The purpose of this project was to tackle the problem of learning in a complex partially observable environment using reinforcement learning. A multiplayer game called DotA 2 was chosen as an environment because it was both challenging and visually pleasing. The agent was communicating with the game via provided server and steam add-on (called Breezy), which ended up being a challenge in itself. The end results turned out to be disappointingly substandard. The fist idea was to pair Deep Q-Network (DQN) with genetic algorithm, but upon later research Hierarchical Deep Q-Network (h-DQN) was chosen to be the the best promising strategy. Sadly, due to the lack of resources (only one PC available), inconsistent network connection and high episode duration, only 250 episodes ended up being played for h-DQN, which was not enough to achieve desired performance convergence. Despite the lack of acceptable results, this project showed a real importance of the right state and reward modelling, as well as differences between different Q-learning strategies.

• Keywords: Potpomognuto učenje; Duboke Q-Mreže; Hijerarhijske Duboke Q-Mreže; DotA 2
• Keywords (english): Reinforcement Learning; Deep Q-Network (DQN); Hierarchical Deep QNetwork (h-DQN); DotA 2
• Language: english
• URN:NBN: urn:nbn:hr:168:977951
• Study programme: Title: Computing; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva (sveučilišni/a prvostupnik/ prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-13 14:04:43