Raspoznavanje gesti ruke u videu

master's thesis

• Title (english): Hand Gesture Recognition in Video
• Author: Miroslav Bićanić
• Mentor: Zoran Kalafatić (mentor)
• Committee member: Zoran Kalafatić (predsjednik povjerenstva)
• Committee member: Vladimir Čeperić (član povjerenstva)
• Committee member: Tomislav Hrkać (član povjerenstva)
• Granter: University of Zagreb; Faculty of Electrical Engineering and Computing; Zagreb
• Defense date and country: 2022-07-15, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Computing

Abstract

Zajedno s ostalim alternativnim metodama interakcije čovjeka i računala, sustavi za raspoznavanje gesti ruke u današnje su doba sve popularniji u raznim područjima primjene. Mnogi od njih izgrađeni su pomoću modela i tehnika iz dubokog učenja koje zahtijevaju brojne primjere iz velikih skupova podataka, a primjeri najčešće dolaze u obliku RGB slika ili videozapisa. Takvi su primjeri nepotrebno veliki i redundantni za potrebe raspoznavanja gesti ruke, te je stoga u ovom radu prikupljen novi skup podataka s 14 dinamičkih i statičkih gesti u 812 primjera, gdje je svaki primjer predstavljen listom ključnih točaka ruke na svakom okviru videozapisa, dobivenom iz MediaPipe Hands biblioteke.

Na temelju ovog skupa podataka naučena su dva klasifikatora - klasifikator poze i klasifikator geste - nad kojima je izgrađen OSCON, sustav za upravljanje računalnim mišem u stvarnom vremenu pomoću gesti ruke. Neke od podržanih radnji u OSCON-u su pomicanje pokazivača miša, klizanje po aktivnom elementu grafičkog sučelja, pritiskanje lijeve ili desne tipke miša te zumiranje. Iako su računalni zahtjevi OSCON-a prilično veliki, na dovoljno jakom računalu OSCON radi bez imalo kašnjenja, uz vrlo rijetke pogrešne klasifikacije.

Abstract (english)

Along with other alternative methods of human-computer interaction, systems for hand gesture recognition are becoming more and more popular in various application areas. Many such systems are built with the help of deep learning models and techniques which require numerous examples from large datasets. These examples are most often given in the format of RGB pictures or videos, but such a format is needlessly large and redundant for hand gesture recognition. Thus, a new dataset was collected for the purposes of this thesis, containing 14 dynamic and static gestures in 812 examples, where every example is represented as a list of hand key-points for each frame of a video, as obtained from the MediaPipe Hands library.

Two classifiers were trained on the new dataset: a pose classifier and a gesture classifier. OSCON, a system for controlling the computer mouse in real time using hand gestures, was built on top of these two classifiers. Some of the supported mouse actions in OSCON are moving the pointer, scrolling the active element of the GUI, pressing the left and right mouse buttons and zooming. While the hardware requirements of OSCON are fairly high, on a strong enough computer OSCON works without delay, with very few misclassifications.

• Keywords: skupovi podataka; MediaPipe; detekcija ruke; ekstrakcija značajki; klasifikacija gesti; interakcija čovjeka i računala; umjetna inteligencija; duboko učenje; neuronske mreže; PyTorch; kontrola miša
• Keywords (english): datasets; MediaPipe; hand detection; feature extraction; gesture classification; human-computer interaction; artificial intelligence; deep learning; neural networks; PyTorch; mouse control
• Language: croatian
• URN:NBN: urn:nbn:hr:168:357612
• Study programme: Title: Computing; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka računarstva (magistar/magistra inženjer/inženjerka računarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Closed access
• Created on: 2023-01-14 08:28:34