General Reading » Hardware & System » Mouse Topics

Licence
First Posted 22 Nov 2001
Views 224,809
Bookmarked 146 times
Tags VC6, Win2K, Dev,
Intermediate

Mouse gestures recognition

By Konstantin Boukreev | 22 Nov 2001
Feedforward multilayer neural network and mouse gesture recognition

4.84 (90 votes)

1234 5
Top of Form
/w EPDw UKMTAy

• Download application - 158 Kb

• Download source - 101 Kb

Introduction
Recently I installed Opera 5 and was impressed on a Gesture UI. Moreover several weeks ago
I noticed a discussion on CodeProject's Lounge about it. To all appearances people want to
know about it too :). IMHO, the neural network most suitable for this purpose. As I a little
know neural network I tried to implement such feature themselves.

What is Neural Network ? Hm it's not easy to say. A rephrased definition Zurada, J.M.:
"Neural network software is a software which can acquire, store, and utilize
experiential knowledge."
I think I can point any person concerned to theory directly to several neural network sites.
Here is small list of web resources about Neural networks:
• FAQ of comp.ai.neural-nets
• Generation 5
• "Neural Computing: Theory and Practice" 1989 Philip D. Wasserman (in russian
only)
Implementation
Let's return to mouse gestures. After some research I have chosen a multilayer perceptron and
standard back-propagation algorithm for training. The main problem was in the
representation of an input data for neural network. The best result I found was in the
transformation of a mouse path into a vector of cosines and sines.
For example:
path {170:82 172:83 175:85 177:86 ...}
transformed into
vector {0.45 0.55 0.45 0.71 ... 0.89 0.83 0.89 0.71 ...}
Recognition algorithm.
1. record a mouse path
2. smooth a path to a base points
3. transform points to angles' vector
4. compute sines and cosines
5. pass values (cosines and sines) to network's inputs
6. apply softmax function on an output network vector
7. find and verify a winner
Neural network architecture.
• input layers : 32 sinapses
• hidded layer : 32 neurons
• output layer : 29 axons (one for each gesture)
• fully connected layers
 • transfer function : log-sigmoid
• incremental training algorithm, standard back-propagation method
• momentum, variable learning rate (slowly reduced)
• input noise
Application
Training

Before testing the recognition ability you must train the network (or you can load an file
image of trained net). You can customize the parameters of the training process, namely:
maximum number of cycles, a momentum value, a learning rate, a minimum value of mean
square error (in other words "target error"). The training process will stop after achieving
either of the conditions: maximum number of cycles or target error. During the training
process you can keep an eye on a error's graph, a current gesture (with noise) and 2D network
presentation.
Testing
As soon as you have a trained net, you can test it. Select the patterns (or test all of them), a
speed value and a noise level. Besides, you can familiarize oneself with ideal presentation of
gestures via setting minimal noise and minimal speed.
Recognition
For recognition of mouse gestures you must press right mouse button during moving a
mouse. For example for recognition "left" gesture, press right mouse button and move a
mouse to the left. If a neural network can recognize the gesture, then you will see the name,
probability and ideal presentation of winner. Because of freeware nature of GestureApp the
mouse path must have at least 16 points :(. Sorry I didn't implemented a "stretch a path"
feature so far.
Note: the direction is very important.
The network is trained to recognize the gestures but not 2D images. Hence, you can draw the
"circle" gesture a thousand different ways, but the only valid way is: press mouse button and
move a mouse to the right and down and so on. Once more: it's gesture, not 2D image.
Mouse gestures

Compatibility
Compatible with Win2k, WinXP, Win98, WinMe. Unfortunately doesn't work on WinNT
because of the need for the AlphaBlend API.
Acknowledgement
Special Thanks:
My wife Julia for her nice artwork ;)

And thanks to:

Pedro Pombeiro for Selection slider control
License
This article has no explicit license attached to it but may contain usage terms in the article
text or the download files themselves. If in doubt please contact the author via the discussion
board below.
A list of licenses authors might use can be found here
About the Author
Konstantin Boukreev I am freelance programmer. About 3 years of experience in C++

Web Developer and I would rather use ATL, STL, WTL but not MFC . Main
backgrounds are Win32 API, COM and Networking. Now I am
interested about AI (Neural Network, Fuzzy Logic and GA).
Russian Federation Currently based in Vladivostok, Russia.
Member

Bottom of Form
Comments and Discussions
43 messages have been posted for this article Visit
http://www.codeproject.com/KB/system/gestureapp.aspx to post and view comments on
this article, or click here to get a print view with messages.