The initial outcomes of the OpenNI installation

A previous post has described an overall progress of installing OpenNI in Linux-arm -based pandaboard with Xtion pro. This post is a sequence of the instsallation, which demonstrates the success of OpenNI installation, and the working of Xtion pro with Linux-arm Ubuntu.

Subsequent developments

The first depth image in this post reminds me a potential breakthrough of hand detection – the fingers could be easily extracted from the backgrounds, no matter how complicated these are. Combined with the OpenCV 2.3.1 installed days ago, the hand detection could be developing some interesting applications.

In a early post openCV was used to segment natural hand gestures from complicated backgrounds in real environments, as the picture above showed (see the original post hand gesture detection and recognition using openCV).

The picture above is a screenshot from a recent paper, about using Ataboost with SIFT (scale invariant feature transform), to detect natural hand gestures. The SIFT is used in the paper to reduce the background noise in the training stage, and so experimental results demonstrated in the paper show the approach performs with high accuracy.

So, the the adaboost learning algorithm is used after the SIFT background noise reducing. Relevant functions corresponding to these two algorithms have not been checked in openCV, to see if there is any working functions to realise such algorithms. However, more details about the use of Adaboost and SIFT, the paper hand posture recognition using Adaboost with sift for human robot interaction is reachable in the download page.

Haarcascades training (haartraining) is seemed an quick tool to achieve accurate hand gesture detection and recognition. The face and body detection examples included in openCV’s installation example folders (\opencv\data\haarcascades\) demonstrate how fast the haarcascades files help to do the job. More information about how to train the haarcascades files can go to sonots.com.

Many face-image databases have been provided for haarcascades training, e.g. http://www.face-rec.org/databases/. But hand images are less contributed for haarcascades training. At least, these are more difficult than face images to find in the internet. I was also asked by many blog readers for the haarcascades training file to support their hand detection or recognition related projects.

So I post an example of using haarcascades training file 1256617233-1-haarcascade_hand.xml for hand gesture detection.

The source codes can be downloaded from (or go to the download page):

http://download.andol.info/1256617233-1-haarcascade_hand.xml
http://download.andol.info/1256617233-2-haarcascade-hand.xml
http://download.andol.info/haarcascades-based%20hand%20detection.cpp

Transferring the openCV from c++ to Java is what i am doing recently. Comparing with opencv c++ it feels more natural to programme in Java, because of the way of Object-oriented programming and some of Java’s advantages.

But how can the openCV be used in Java seamlessly?

The easiest way is to take advantage of existing Java openCV projects, such as JavaCV (http://code.google.com/p/javacv/) or openCV in Java (http://ubaa.net/shared/processing/opencv/). These wrappers do make the use of openCV relatively easier, as the face detection example below.

import hypermedia.video.*;
import java.awt.Rectangle;

OpenCV opencv;

void setup() {

    size( 320, 240 );

    opencv = new OpenCV(this);
    opencv.capture( width, height );
    opencv.cascade( OpenCV.CASCADE_FRONTALFACE_ALT );
    // load the FRONTALFACE description file
}

void draw() {

    opencv.read();
    image( opencv.image(), 0, 0 );

    // detect anything ressembling a FRONTALFACE
    Rectangle[] faces = opencv.detect();

    // draw detected face area(s)
    noFill();
    stroke(255,0,0);
    for( int i=0; i<faces.length; i++ ) {
        rect( faces[i].x, faces[i].y, faces[i].width, faces[i].height );
    }
}
[These lines of codes are from 

http://ubaa.net/shared/processing/opencv/opencv_detect.html]

Another advantage of using openCV in Java is that it can be integrated into mobile systems such as the Android. With Eclipse the wrapper classes can be used to programme intuitively. For example:

import static com.googlecode.javacv.cpp.opencv_core.*;
import static com.googlecode.javacv.cpp.opencv_imgproc.*;
import static com.googlecode.javacv.cpp.opencv_highgui.*;

public class Smoother {
    public static void smooth(String filename) {
        IplImage image = cvLoadImage(filename);
        if (image != null) {
            cvSmooth(image, image, CV_GAUSSIAN, 3);
            cvSaveImage(filename, image);
            cvReleaseImage(image);
        }
    }
}

For more details, here is a relevant tutorial video made by original author who explains who this works.

Here gives the source code of this video demonstration.

You can download the full code at:
1) Full Demo Solution (Code + video 43Mb)
http://db.tt/MukGcwB
2) Full Demo Code
http://db.tt/GkT6J9k

First we gather a data set of all the hand-shapes we wish to recognise. A naive approach to recognizing a new image D would be to simply compare it with all the images stored in the data set and find the target image T with the closest match. But because there are so many images in the data set this will take far too long. We can reduce the time by using a multi-scale approach. We divide up the data set into groups of images, which are similar to one another by blurring the images at different levels so that small differences between similar images will be eroded.

More information please refer here http://www.ercim.eu/publication/Ercim_News/enw46/shamaie.html

About openCV

A library of programming functions for real time computer vision,released under a BSD?license, and free for academic and commercial use (a). Current version is openCV 2.1, this version has been installed and compiled under my VS 2008 environment but with some errors, may not be very easy to configure this version. While the stabilities of openCV 1.0 have been firmly proved already and that is easier to install and configure. Install guides can be found as ?openCV install guide in openCV wiki,?installing openCV in windows, vc2008 install openCV 2.0/2.1(in Chinese)?and get started with openCV.

A easy start of ‘hello world‘ application using openCV can be very helpful. Under most occasions, openCV is used within windows console applications with a black command window and openCV’s own interfaces (highgui) to display images and draw patterns, as the ‘hello world’ example shows. As well there are some other interface components included in openCV library such as sliders, mouse movement events and key input events. Using these provided features of openCV, object detection could be done via capturing webcam images, analysing them and finally output the consequences.

One recommendation of learning openCV online is Yahoo openCV group. It is a group with huge number of active members posting hundreds of messages each day, may be a little massive, but a good place to raise questions and meet other people, highly recommended.

About detections & recognitions

Detections are core functions of openCV. Color knobs, objects, motions, gestures and faces are subjects of openCV detections and recognitions.

Color knob detections

detecting colors using openCV is basic, there could be several ways to achieve the color detections. For instance, as the color detection used in my hand recognition project, the target color is divided into three perspectives of H, B, and V. By combining these parameters, theoretically it is available to detect any color. But the fault of HSV method is that the background colours are usually too sensitive to be specified. So does another method which uses simple brightness threshold to get target colours, but it is limited as well. A better way of detecting a range of colours is to use ‘camshift’ which requires a source image which represents colour information for target colours. There is an example of using ‘camshift’ here.

Object detections

Object detection is kind of similar as colour extraction since to recognise the objects they should be detected first. Here is an example of using Actionscript 3.0 to tracking objects via webcam. In openCV it is simple as well. Tracking the objects, then using circle or line detection methods to extract rough object shapes. It could be very accurate of detecting object angles which are demonstrated here, and another object detection example illustration here. Detecting an array of object is nearly the same as detecting solo object, except some extra work is needed to loop all objects at one time and draw them out, this could be seem here as detecting combined objects.

Motion detections

There is an example of motion detections in openCV’s example folders, which is called motemlp.exe and a source file motempl.c. The picture on the left shows the motion directions and sub components’ directions are being marked clearly.

Gesture and face detections
Face detections are maturely applied in digital cameras presently. It is dependant on haar trainings and classifiers therefore the detection results seem robust. As well, a face detection example with source code can be found in openCV’s example folder. In this review face detection would not be emphasised too much as there are millions of tutorials telling how to detect the face in either static or dynamic circumstances.

Recognitions of hand gestures is the challenging part.  On current stage, hand gestures especially finger tips are still tough due to the natural shapes of hands are ambiguous for computer to identify which is which. Therefore some simple and intuitive gestures are commonly adopted to relatively improve the accuracy of recognitions. Recognition gestures requires two factors: a. correct contours and b. predefined gesture patterns. The first factor is reachable but the second requires more work to cheer up. Concerning the second factor, there have been some theoretical papers raising solutions of detecting hand gestures. Toughly, putting these theoretical methods into practical use and generate kind of applications providing open source codes still has long way to go. A progressive project can be referred here as hand detection and image descriptions.

Useful toolkits

HandVu

ARToolkit

There are several vital parameters in using the function cvsnakeimage, particularly parameters relating to curve energy which concerns judgements of finger tips. E.g.

A copy of full source code of demonstrating the use of the function could be found here cvSnakeImage for hand detection

secret powers from Mechanics of Destruction on Vimeo.

Relevant posts & comments please refer to  ‘Hand gesture detection and recognition using openCV’， and related comments here.

A full paper about this can be found here titled as ‘tracking a hand manipulating objects‘

As described by the authors, a method of individual local tracker is used to achieve extractions. To achieve the goal as the left picture shows, it is required to build a 3-dimensional frame to skip the covering of objects and overlapped parts of hands. Seems the authors do not narrowly focus on color image segmentation which has been commonly adopted by computer vision researchers, but they integrate an estimation of features as well as 2.5-dimensional maps. That means probably extra dimension may required. In previous posts, we have successfully located contours of hands yet the shape recognition is still under investigation. And this new method may inspire us something further in hands tracking and recognition.

As far as in my mind, there are three approaches that people have taken to detect hands using openCV, though not all of them have been tested by myself (a little shame). One is to use accurate HAAR-classifiers to locate and detect gestures which is considered as quite stable way but costing very much. Another is to use adaptive skin detection algorithms combined with motion analysis which seems easier to do than previous one. And the last is to use human skin colour segmentation to detect the contours of hands combined with hand convexity detection to recognise hands’ gestures [please refer to: hand gesture recognition using openCV].

HAAR-classifier database is rarely provided with open source projects. But we found some helpful papers which may give some thoughts of classifier training and test.

About hands database:

A colour hand gesture database for evaluating and improving algorithms on hand gesture and posture recognition

Application of classifiers:

Real-time hand tracking using a set of cooperative classifiers based on HAAR-like features

Alternative skin detection based approach gesture recognition through angle space:

Gesture recognition through angle space

As these are mostly asked questions about using openCV to detect and recognise all kinds of subjects such as colours or hands, a review which includes tutorials, progresses and examples (if possible) would be published shortly…

Quite much work has been doing to improve accuracy of pattern recognition, to enable computer applications being sufficiently intelligent facing this dynamic world. ARtoolkit is one of them. By training computers to recognize patterns as shown above, i have opportunities to shift my attention from recognition to applications, which means, representation forms of recognition affect interaction.

By extracting recognition codes from ARtoolkit, it seems easier to train a computer to recognize patterns like letters and other graphics, and its recognizing results seem pretty good. Based on that, creating appropriate applications become urgent.

In this post, as the progress goes, the hand detection technologies using openCV is introduced. As the pictures inllustrated below, this detection method is independent  with distance and background ( just not the background full of hands ), and the main segment method is color abstract which means getting the hand color pixels filtered. As the result tested within several environments, it workd well except slightly noise varying. But the noise is easy to take off.

The tested source code file is offered underneath the illustrations, feel free to use it.

SOURCE FILE HERE:  handDetection.CPP
Related post which may interest you:
Hand gesture detection and recognition