Tuesday, July 10th, 2012, 317 days ago
Hand gesture detection and recognition using openCV 2
Before the example – why haar classifiers for hand gesture detection
As mentioned in one of early articles such like hand gesture detection and recognition using openCV, there are two main ways to detect hand gestures, including skin colour segmentation and haar classifier training, although recently some feature-detecting algorithms are used to detect hand gestures such like swift. Skin colour segmentation is a dead end – hand colours can be white, or black, or any other colours – depends on the varieties of environmental light conditions. So using a skin colour model (as in The colour range for HSV skin extraction) with a specific colour range can do hand gesture recognition quite well in strict scenarios but this still faces tough challenges for other use.
The source codes
// Name : opencv_handdetect.cpp
// Author : andol li, andol@andol.info
// Version : 0.1
// Copyright : 2012
// Description : using haartraining results to detect the hand gesture of FIST in video stream.
//
//============================================================================
#include <opencv/cv.h>
#include <opencv/cxcore.h>
#include <opencv/highgui.h>#include
#include
using namespace cv;
using namespace std;
const double scale = 1.1;
//1.0 api version
CvMemStorage* storage = 0;
CvHaarClassifierCascade* cascade = 0;
void detectAndDraw(IplImage *input_image);
const char* cascade_name = “fist.xml”;
//define the path to cascade file
string cascadeName = “fist.xml”; /*ROBUST-fist detection haartraining file*/
int main()
{
//1.0 api version
CvCapture *capture =0;
IplImage *frame, *frame_copy = 0;
cascade = (CvHaarClassifierCascade*)cvLoad( cascade_name, 0, 0, 0 );
if( !cascade ){
fprintf( stderr, “ERROR: Could not load classifier cascade\n” );
return -1;
}
storage = cvCreateMemStorage(0);
capture = cvCaptureFromCAM(0);
cvNamedWindow(“result”, 1);
if(capture){
for(;;){
if(!cvGrabFrame(capture)) break;
frame = cvRetrieveFrame( capture);
if(!frame) break;
if(!frame_copy) frame_copy = cvCreateImage(cvSize(frame->width, frame->height), IPL_DEPTH_8U, frame->nChannels);
if(frame->origin == IPL_ORIGIN_TL)
cvCopy(frame, frame_copy, 0);
else
cvFlip(frame, frame_copy, 0);
detectAndDraw(frame_copy);
if(cvWaitKey(10) >= 0) break;
}
cvReleaseImage( &frame_copy );
cvReleaseCapture( &capture );
}
return 0;
}
void detectAndDraw(IplImage *img)
{
double scale = 1.1;
IplImage* temp = cvCreateImage( cvSize(img->width/scale,img->height/scale), 8, 3 );
CvPoint pt1, pt2;
int i;
cvClearMemStorage( storage );
if(cascade){
CvSeq* faces = cvHaarDetectObjects(
img,
cascade,
storage,
scale, 2, CV_HAAR_DO_CANNY_PRUNING,
cvSize(24, 24) );
for( i = 0; i < (faces ? faces->total : 0); i++ )
{
CvRect* r = (CvRect*)cvGetSeqElem( faces, i );
pt1.x = r->x*scale;
pt2.x = (r->x+r->width)*scale;
pt1.y = r->y*scale;
pt2.y = (r->y+r->height)*scale;
cvRectangle( img, pt1, pt2, CV_RGB(200, 0, 0), 1, 8, 0 );
}
}
cvShowImage(“result”, img);
cvReleaseImage( &temp );
}
it can be downloaded from here SOURCE CODES OF HAND GESTURE DETECTION AND RECOGNITION USING OPENCV, relevant files/codes are for requests.
POSTS MAY BE OF INTEREST
Release of HAAR classifier trained for...
A method of detecting and recognising ...
First trial of classifier training – t...
The colour range for HSV skin extraction
A failed example of hand gesture recog...
Writing a simple hand gesture picture ...
Needed!!! High quality positive/negati...
Using hand gestures to control mouse p...
The first set of images captured from ...
Hand gesture recognition using Adaboos...
Detecting hand gestures using Haarcasc...
Transferring openCV from C++ to Java
Hi Andol!
Your blog is a godsend — I’m doing a open palm recognition project for a not-for-profit client. Could you share your haar cascade xml for palm/hand?
@sergi
thanks for your interests, the current haar xml file is still not robust enough for open palm detect in chaotic backgrounds – guess this is due to the lack of training samples – but it will be published as soon as it achieves satisfied results.
[...] http://www.andol.info/hci/2020.htm [...]
The archive doesn’t include fist.xml…
Hey Andol,
If you haven’t already read it I recommend this paper where they combine Hue thresholding with movement detection to get some improved results. http://www.inf.ufsc.br/~davigp/Artigos/artigoComparacaoMetodosDeteccaoDeFaces.pdf
Skin colour segmentation isn’t a dead end, it just needs a bit more tweeking
@ryan
Hi there, nice to have your arguments here – I have downloaded and read that paper – the adaptive skin colour detection still face some limitations – correct me if I am wrong:
1. there must be a colour image, a gray image may not have enough colour information to shape the boundaries.
2. the hand must be within a narrow range of normal light scenarios – a hand under blue light does not fit with the hisgram range of normal hand colours. if it is, then the hand colour range, particularly the Hue value, is far toooo broad.
3. the background with the hand must be static, otherwise the motion object’s colour extraction would be inaccurate.
ps- i like the board game project you did recently.
Yes your right with point 1 and 2, the hand has to be in the ‘normal’ colour ranges for the initial detection but I think after that as long as changes are gradual then the system would adapt to it, although I could be wrong, it’s been a while since I read the paper myself!
I think moving objects in the background would only be a problem is they are skin coloured, so people walking passed or maybe blowing branches, but so long as they aren’t skin coloured it shouldn’t be a problem. Also this would only be a problem if they are constantly moving and stay on the scene.
Hoping to have a implementation of this done at some point so will let you know how it goes.
Really need to get my C and OpenCV skills built up so thought it was a good thing to take on a project in my spare time. Just setting the scene just now it’ll get more interesting soon!
Hey Andol,
I am looking for a tool that can recognise body gesture from video(.avi file). Please share the link with me if u have one.
Thank you,
hey andol,
please include the fist.xml file too.
http://code.google.com/p/handtracking/downloads/detail?name=haarcascade3.xml
Can you link to the fist.xml??
@James
Please check my hand gesture recognition project in Github here:
https://github.com/yandol/GstHanddetect/tree/master/src/xml
cheers
libv4l2: error setting pixformat: Device or resource busy
HIGHGUI ERROR: libv4l unable to ioctl S_FMT
libv4l2: error setting pixformat: Device or resource busy
libv4l1: error setting pixformat: Device or resource busy
HIGHGUI ERROR: libv4l unable to ioctl VIDIOCSPICT
what are all these errors mean? how to solve it?
main.cpp:22: error: stray ‘\224′ in program
main.cpp:25: error: stray ‘\223′ in program
main.cpp:25: error: stray ‘\224′ in program
main.cpp:34: error: stray ‘\223′ in program
main.cpp:34: error: stray ‘\’ in program
main.cpp:34: error: stray ‘\224′ in program
main.cpp:39: error: stray ‘\223′ in program
main.cpp:39: error: stray ‘\224′ in program
main.cpp:85: error: stray ‘\223′ in program
main.cpp:85: error: stray ‘\224′ in program
main.cpp:20: error: expected primary-expression before ‘*’ token
main.cpp:22: error: ‘fist’ was not declared in this scope
main.cpp:25: error: ‘fist’ was not declared in this scope
main.cpp: In function ‘int main()’:
main.cpp:34: error: expected ‘)’ before ‘:’ token
main.cpp:34: warning: null argument where non-null required (argument 2)
main.cpp:39: error: ‘result’ was not declared in this scope
main.cpp: In function ‘void detectAndDraw(IplImage*)’:
main.cpp:85: error: ‘result’ was not declared in this scope
I’m getting this kind of errors while compiling, what can be the reason?
the ‘fist’ may not be properly defined or initialised or lacking the fist.xml file to run this code.
OpenCV Error: Parsing error (haar/fist.xml(4): Valid XML should start with ”) in icvXMLParse, file /home/frankie/Downloads/opencv-2.4.4/modules/core/src/persistence.cpp, line 2258
terminate called after throwing an instance of ‘cv::Exception’
what(): /home/frankie/Downloads/opencv-2.4.4/modules/core/src/persistence.cpp:2258: error: (-212) haar/fist.xml(4): Valid XML should start with ” in function icvXMLParse
——————————-
Hi sir,
after “run” opencv_handdetect.cpp with the “fist.xml”&”palm.xml”, both result the above error. Do u have any idea why? thx.
@pengin
be aware of openCV versions – as the fist.xml and palm.xml worked fine in 2.3.1 with many times of test.
openCV may have made some upgrades to the xml handling functions in new versions, such as 2.4.4, that may/will cause unexpected compatability problem.
to get over this issue, you can either try change the xml file to fit your xml file parser, such like the ” start of valid XML file as the error indicates, or try find and use another XML parser which can work well.
cheers
Hey Andol, I’m student, I’m working on a project using (OpenCV 2.4.1) that can detect hand gestures, but I’ve to recognize the hand by the shape, not by the color, your page It’s amazing. Do you have more Gestures?.
Thanks for share.
Hi,Andol.i am a student,i want to do a project that should be able to “convert sign language to text for deaf and dumb people”,can i have source code for that,it will be very helpful to me.
Thank You.
@Mohamed
what kind of sign languages?