Wednesday, 27 February 2013

Play Computer games with hand gesture recognition using Matlab

To do this experiment,just you need an webcam and matlab2012a or 2012b

Now we are gonna do live motion detection,the below live detection code is not written by me,i got it from this link

vidDevice = imaq.VideoDevice('winvideo', 1, 'YUY2_640x480', ...
'ROI', [1 1 640 480], ...
'ReturnedColorSpace', 'rgb');
optical = vision.OpticalFlow('OutputValue', 'Horizontal and vertical components in complex form');

maxWidth = imaqhwinfo(vidDevice,'MaxWidth');
maxHeight = imaqhwinfo(vidDevice,'MaxHeight');
shapes = vision.ShapeInserter;
shapes.Shape = 'Lines';
shapes.BorderColor = 'Custom';
shapes.CustomBorderColor = [255 0 0];
r = 1:5:maxHeight;
c = 1:5:maxWidth;
[Y, X] = meshgrid(c,r);

hVideoIn = vision.VideoPlayer;
hVideoIn.Name = 'Original Video';
hVideoIn.Position = [30 100 640 480];
hVideoOut = vision.VideoPlayer;
hVideoOut.Name = 'Motion Detected Video';
hVideoOut.Position = [700 100 640 480];

nFrames = 0;
while (nFrames < Inf)
rgbData = step(vidDevice);
optFlow = step(optical,rgb2gray(rgbData));
optFlow_DS = optFlow(r, c);
H = imag(optFlow_DS)*50;
V = real(optFlow_DS)*50;
lines = [Y(:)'; X(:)'; Y(:)'+V(:)'; X(:)'+H(:)'];
rgb_Out = step(shapes, rgbData, lines');
step(hVideoIn, rgbData);
step(hVideoOut, rgb_Out);
nFrames = nFrames + 1;

But after this Ive changed this a little bit and  made a different thing,hope you like it.
So the code i got from the above link,well im pasting it here,

Now again use the redbox around the motion


Now perform a mouse click operation whenever a redbox appears in the video,get the code here

So,this is how your computer player will fire or do some other stuff as per game settings

And move the mouse pointer as per box location,code is Here

By this your player will switch it's view,
Now after doing all this,

1.Run your matlab code
2.Minimize the video stream
3.Run the game you want to play

I've tried this with some popular games,like counter strike,IGI 

Well this is a very basic idea,you can implement it can also add some keyboard operation for performing other tasks,Keyboard Code here

try it it will definitely work... Thank u

Press like and leave comment


  1. hello have a good day i think this code shown all object in the video so need to modify this code if you can to work only for face or only for fingers this my email if we can send to me again by the ways my name is ahmad i hope we can be friends

  2. Wow ! What an eye opener this post has been for me. Very much appreciated, bookmarked, I can’t wait for more !

  3. I need to show input video for face ROI , like facial feature extraction using matlab .send me code to :

  4. Hi.
    I'm trying to implement this but unfortunately I'm still a newbie in Matlab.
    Can you please write me how to make the rectangles around detected moving objects and how to choose the proper one? So the mouse will track my hand and not head for example. My email is