A previous article has described a method to detect and recognise hand gestures using OpenCV, actually the c# wrapper of OpenCV called emgu CV. Details of this method can be found here a method of detecting and recognising hand gestures using Opencv. But Tongo’s project wrapped in emgu CV may not be easy to configure, so this article gives a new example transfering thet project to plain c++ OpenCV environment.
The example codes of the project, as well as the configurations, are described bellow.
To make a clear tutorial of how the previous project did the hand gesture detection and recognition, the functions used in the project are listed below, through which I hope it can make a clear picture of this project’s algorithm flow.
1) reading video files. The project did not read video images from live webcam, rather it provided a video file as the source of video images. Similar functions were used to query video image one by one by using the function – grabber.QueryFrame().
2) extracting contours and hulls of hand gestures. In this part, the method used HSV colour ranges to extract hand colours – the range can be adjusted dynamically according to background environments. A set of functions were applied to improve the accuracy of colour detection, such as cvErode and cvDilate. Here is a clip of source code in the project.
cb -= 109;
cr -= 152;
x1 = (819 * cr – 614 * cb) / 32 + 51;
y1 = (819 * cr + 614 * cb) / 32 + 77;
x1 = x1 * 41 / 1024;
y1 = y1 * 73 / 1024;
value = x1 * x1 + y1 * y1;
if (y < 100)
skinData[i, j, 0] = (value < 700) ? (byte)255 : (byte)0;
skinData[i, j, 0] = (value < 850) ? (byte)255 : (byte)0;
StructuringElementEx rect_6 = new StructuringElementEx(6, 6, 3, 3, Emgu.CV.CvEnum.CV_ELEMENT_SHAPE.CV_SHAPE_RECT);
CvInvoke.cvErode(skin, skin, rect_6, 1);
CvInvoke.cvDilate(skin, skin, rect_6, 2);
3) selecting the right contours. The skin detection generated many contours, including face shapes with similar colours as hands. So, the right contours need to be pick up from all these contours. In this project, the author used a simple method by comparing the size of contours – usually the hand in the image had the biggest contour size.
4) if the biggest contour existed, then counting the finger number from detected contours. During this progress some basic markings such as drawing a rectangle or eclipse are helpful to identify the right object. Counting the finger was realised via a function of defect detection. There were obvious defects between finger contours with up and down shapes.
LineSegment2D depthEndLine = new LineSegment2D(defectArray[i].DepthPoint, defectArray[i].EndPoint);
CircleF startCircle = new CircleF(startPoint, 5f);
CircleF depthCircle = new CircleF(depthPoint, 5f);
CircleF endCircle = new CircleF(endPoint, 5f);
//Custom heuristic based on some experiment, double check it before use
if ((startCircle.Center.Y < box.center.Y || depthCircle.Center.Y < box.center.Y) && (startCircle.Center.Y < depthCircle.Center.Y) && (Math.Sqrt(Math.Pow(startCircle.Center.X – depthCircle.Center.X, 2) + Math.Pow(startCircle.Center.Y – depthCircle.Center.Y, 2)) > box.size.Height / 6.5))
currentFrame.Draw(startDepthLine, new Bgr(Color.Green), 2);
//currentFrame.Draw(depthEndLine, new Bgr(Color.Magenta), 2);
currentFrame.Draw(startCircle, new Bgr(Color.Red), 2);
currentFrame.Draw(depthCircle, new Bgr(Color.Yellow), 5);
//currentFrame.Draw(endCircle, new Bgr(Color.DarkBlue), 4);
Based on the understanding of hand gesture detection and recognition algorithm flow described above, it is possible to map the flow in c++ projects. The c++ project used similar OpenCV functions to realise the detection and recognition, and it followed the same flow of detection. Below is a screenshot of the c++ project detecting and recognition simple finger gestures as numbers. The source code of this project is also included in the bottom of this post and the download page as well.
See the example codes in download page.