When I was building up a new silverlight project based on existing silverlight 3d template, I met a problem that said ‘error  : Unable to read the project file ‘OverlappingTabs.cspro … C:\Program Files\MSBuild\Microsoft\Silverlight\v4.0\Microsoft.Silverlight.CSharp.targets” was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.’

This problem happened because of the support of silverlight, as the visual studio 2010 did not have the required silverlight supporting files installed. To fix this problem, just setup ‘Microsoft® Silverlight® 5 Tools for Visual Studio® 2010 SP1‘.

This post demonstrates the breaking down of xtion pro into parts. For more pictures reading the whole post.

When I was writing a simple example of OpenCV below, an repeating error happened as ‘VIDIOC_QUERYMENU:Invalid argument’.

Is this because the Qt for drawing the output of openCV?

The code

The code that caused the above error is like this:

The error happened when initialising the webcam, but the webcam was working fine even though the errors were given in the console window.

The fix

From a post from stackoverflow, this problem was

with Qt handling its own event loop and thus the event loop of OpenCV is starved and never get run.

The way to get them to work together is quite simple: Display your OpenCV as a QPixmap (by convert your image to QImage then use QLabel to display it). Then add this QLabel to your QWidget. Your QWidget can either be embedded or become the main widget of your QApplication.

But it may not be so easy to get rid of, as other posts also mentioned that

You can’t mix openCV’s control of the event loop and qt’s app.exec

Either use the Qt flavour of the cvNamedWindow, or simple grab images from openCV and display them in a QLabel. Or better inherit from QWidget and write your own QImage painter

void OpencvWidget::paintEvent(QPaintEvent*)
{
//m_win is the window size
QPainter p(this);
p.drawImage(m_win,m_image,m_image.rect());

}

The best way to fix ‘VIDIOC_QUERYMENU:Invalid argument’

I have faced a similar problem recently (displaying webcam information using OpenCV and Qt as the final GUI). The best way that I find out to play the video (which is basically a set of images) is to use a GLWidget.

This is a brief summary of the work that has been done previously, including the ubuntu 11.04 install, Xtion depth camera setting, openCV 2.3.1 installing, and the final openNI install and configure.

Because this is an overview of the past work, the source codes are not included here.

Ubuntu 11.04 on pandaboard

The installation of Ubuntu 11.04 in the pandaboard did spend me some time. Since there was no hard drive in the pandaboard the SD card was specified to play that role. This means the whole Ubuntu 11.04 system should be written to the SD card. Especially, the becaue the pandaboard was based on ARM cup, it needed a special version of ubuntu – a pre-install ubuntu version for OMAP4. Downloading the system image, then the system install stared.

The hard part I met was not in the downloading, but the image system writing. Since the image was formated in RAW image, normal image burning software e.g. nero was not able to burn such image. Although the official instruction in ubuntu provided the instructions of burning the system image within three different systems windows, linux, and mac.

The instruction of windows system was not working. A disk writer software called ‘disimagewriter’ did not work functionally, at least in my and my coleagues’ computers. And currently, there was no other software that could do the same thing. I had a macbook but unfortunately the official instructions were not lucky to me.

Finally I found a shortcut. I downloaded a software to mount the RAW image as a system drive, and then used another software to copy the whole disk drive to another. In other words, I copied the RAW image -based drive to the SD card drive. It turned out successfully. The details can be reached here.

After an installation the ubuntu now was running on the pandaboard – feels pretty good, but the wifi signal sucked.

Webcams and the Xtion depth camera

Install openCV 2.3.1 in Ubuntu 11.04 with pandaboard

Ubuntu systems have good supports for webcams, but most of these are based on normal x86/x64 systems, not for ARM systems. I was worried about this problem, as the cameras to use with the system were quite special – not only the rgb webcams, but also the depth camera from ASUS Xtion pro camera.

A good way to use the cameras and to get proper drivers is to install some universal drivers, such as v4l, and ffmpeg.

During the install of openCV in the system, these libraries were particularly important.

Install openNI, NITE and depth camera driver for natural interaction app development

Install openNI for ARM-linux was quite easy, as there were three key components ready to download in the openni.org .

Downloading the unstable versions – openNI binary for ARM, sensor openNI for ARM, unzipping the packages and using the sudo ./install.sh’ command to complete the install.

* be aware of that, there is currently no openNI middleware package for ARM -based linux system, which means, some of the advanced features of openNI such like ‘user generators’ and ‘skeleton generators’ could not be used. The middleware for linux x86 was tested, but with no luck. The error’ one or more of the following nodes could not be enumerated‘ was actually caused by that factor.

Anyway, I ve got rgb and depth images to use. Hopefully the middleware for arm-linux comes out sooner.

The summary

It is hard to describe all details in a short post. As posted in the previous articles, the lessens learnt from the whole procedures are valuable, and more important, I wish these would be shared to give guidances to other people’s work.

ERROR – ‘One or more of the following nodes could not be enumerated:’

STATUS: not solved yet – still a big headache

Xn::UserGenerator and OpenNI

Install OpenNI in Linux with Arm is never a easy job – few people work on it – most of the work is based on windows and kinect rather than ubuntu and xtion and the arm board, and little experience is shared over the internet. So when I tried to run the usertracker example from the sample folder, I got this error with no luck or idea.

The thing is that, I installed OpenNI, NITE and driver, that referred to the install work of others – sorry not to mention a few here. The simpleviewer example worked fine, the image and depth maps were successfully captured.

But, the problem happened when – creating usergenerators, like:

Help needed

ImageGenerator and DepthGenerator have been tested with well working. Any other generators do not work, and keep giving the same error.

Any ideas would be very much appreciated!

When I moved the openCV development platform from the home to the office, a wired thing happened – the programme that I debugged smoothly in the home did not work at all in the office – even the simplest openCV camera view example, and the error said that – ‘HIGHGUI ERROR: V4L: index 0 is not correct‘ – but nothing of openCV configurations were changed at all!

So, where was the error from and, how to fix that wired error?

What caused this error?

Firstly I searched google for ‘HIGHGUI ERROR: V4L: index 0 is not correct’. This issue should be relevant with V4L.

Some pages the search results suggested to remove the v4l in Ubuntu, and then reinstall it again cleanly.

Some others suggested that re-configure and compile the openCV 2.3.1 might be solving the problem – as there was a file called ‘cvcap_libv4l.cpp’ in the source compiling folder, and a bug function was called in that file – find a line with ‘mman‘ function, and it supposed to be ‘libv4l_mman‘.

But, reconfiguring and re-compiling the openCV? That would take ages!

The unexpected solution

Before starting the re-compiling thing, be aware of this – the camera, yes, the camera!

Changing another camera may help? Why not bother to have a dead try?!

Oh-my— my camera was not plugged in!!!

Plugged in the camera, bingo~ everything worked fine.

Appendix – reading images from the webcam

This is a simple example of capturing live images from webcams using the new installed OpenCV 2.3.1 in the Ubuntu. The relevant information of OpenCV 2.3.1 install can see my previous posts -

Installing openCV in Ubuntu with pandaboard part 1
Installing openCV in Ubuntu with pandaboard part 2
Installing openCV in Ubuntu with pandaboard part 3 – the network proxy
Installing openCV in Ubuntu with pandaboard part 4 – compiling openCV 2.3.1
Installing openCV in Ubuntu with pandaboard part 5 – running openCV 2.3.1 examples, and
Tutorial: A Comprehensive Guide to Installing and Configuring OpenCV 2.3.1 on Ubuntu

This simple example includes lines of source codes using the new OpenCV API, for your convenience.

The source code

The source file compiling

Save the source file as ‘readwebcam’ for example, go to the directory containing the file.

Open the terminal, and type in the command like:

Then run it:

Here it is! Now let’s start rocking the hand gesture recognition!

This post describes an error and its solution in ffmpeg make for openCV install. It is based on Osman’s comprehensive troubleshooting of openCV 2.3.1 installing and configuring, but adds an additional error – ‘[libavutil/libavutil.so.51] Error 1‘ and the solution.

This error happens when running ‘make’ command after the ffmpeg configuration. It is a fatal error which directly stops the make process and fails the ffmpeg install.

why ‘[libavutil/libavutil.so.51] Error 1′ happens

Osman’s configurations for ffmpeg install are like this:

This was used in my first ffmpeg install, and was added with another line as:

But the result turned out this error – [libavutil/libavutil.so.51] Error 1

The solution

It was suggested over the internet that this problem was caused by the lack of ‘libvpx’ – this was essential to be used with ‘–enable-shared’.

So, clean the previous make using ‘make clean’ or ‘make distclean’, then re-configure the install as:

Then ‘make’ – bingo~ errors gone!

Thanks to the comprehensive troubleshooting guide to opencv installation posted by Osman, some install errors could be referred and fixed. But some errors are not included in that guide, as the chance of the error is various.

The point of this post is to add another minor error that encountered in my v4l making for opencv installing.

‘QMainWindow: No such file or directory’

‘No such file or directory’, this type of error used to be caused by the lack of some libraries not installed yet. The potential ways to fix these are to install the lacked libs, of course. But it may take a little time to check which libs are lacked, as the error message used to tell limited information.

In this example, ‘QMainWindow’ turned out to be relevant with libqt4-dev after a quick search in the internet. So, using the command below could easily solve that problem.

Carrying on the installation until it finishes. Then redo the v4l make, binggo~ the problem’s gone!

The initial outcomes of the OpenNI installation

A previous post has described an overall progress of installing OpenNI in Linux-arm -based pandaboard with Xtion pro. This post is a sequence of the instsallation, which demonstrates the success of OpenNI installation, and the working of Xtion pro with Linux-arm Ubuntu.

Subsequent developments

The first depth image in this post reminds me a potential breakthrough of hand detection – the fingers could be easily extracted from the backgrounds, no matter how complicated these are. Combined with the OpenCV 2.3.1 installed days ago, the hand detection could be developing some interesting applications.

About OpenNI

OpenNI has shown some lights to natural interactions by using depth map images together with RGB images from cameras. Natural human gestures, e.g. hand moving, limbs movements are all being developed fast. Microsoft Kinect is a typical example that demonstrated the huge potential of the OpenNI in terms of game applications. While, more work could be done with OpenNI, with other devices that could capture depth images, e.g. ASUS’s Xtion pro. In the OpenNI community supported by manufacturers e.g. Primesense, Microsoft and Asus more information could be found in openni.org.

About installing OpenNI

Many tutorials can be search out in the internet, there vary from different operation systems to different installation packages and to various installation errors and solutions as well. Mostly, these focus on three popular systems – windows, linux and mac os. In the windows systems, of course, the installation is relatively the easiest as a good population of developers are working on it. In the mac os, it is ok to install the OpenNI as it is intuitive and with good support from the community. In the Linux system the situation is more complicated as different sub systems were involved. Firstly, different versions of Linux and their configurations, secondly, the Linux running on a broad range of devices with diverse system structures.

And here in this post, it is about installing OpenNI in a Linux version – Ubuntu 11.04 Natty in a ARM -based device – the pandaboard.

The Installation steps

Jatin Sharma posted an article talking about how to install OpenNI on Beagleboard with Kinect – just seem this as a similar version of pandaboard and Xtion pro. That post is referable to the installations described in this article. So I would ‘borrow’ some of the procedures to better demonstrate my installations of OpenNI.

1. certain packages are required pior to the installations of OpenNI. Try the commands below to do fetch and install these packages.

* Jatin mentioned an error in ‘libglut3-dev’, as this lib has been replaced by ‘freeglut3-dev’, so you can directly try the later one. But actually I installed the ‘libglut3-dev’ successfully in Synaptic Package Manager, as well as ‘freeglut3-dev’. But never mind, as long as the required package is installed and working properly.

2. fetch the OpenNI package

Making a new directory to hold this package would be a good idea, that gives you much convenience. Let’s say to make it ‘~/Documents/OpenNI’

using the command below to get the package of OpenNI:

*depends on the versions, the contents of the OpenNi folder may be different.But the common thing is to get into the ‘Build’ folder. Mine is under ‘~/Documents/OpenNI/Platform/Linux/Build’.

* Please be aware of Jatin’s information that

Edit the make file in ~/kinect/OpenNI/Platform/Linux-x86/Build
In the latest version of OpenNI, the ARM platform has been included by the name Linux-ARM. However if we go into ~/kinect/OpenNI/Platform/Linux-ARM/Build/ and try to build it you’ll face errors thus better would be to use the old Linux-x86 platform files with some modifications.

Open ~/kinect/OpenNI/Platform/Linux-x86/Build/CommonMakefile and comment out these lines:

CFLAGS += -malign-double
and
ifeq ($(SSE_GENERATION), 2)
CFLAGS += -msse2
else
ifeq ($(SSE_GENERATION), 3)
CFLAGS += -msse3
else
($error “Only SSE2 and SSE3 are supported”)
endif
endif

We need to do this because ‘ -malign-double’ and ‘-msse3’ flags are valid only for x86 platform and will not work for gcc on ARM. Commenting out these statements makes the code usable for ARM.

Make and Make install

Future work

This installation has not yet been validated if the OpenNI samples are working or not. The sample validation and programming part will be published in another post.

The last tutorial about installing opencv in ubuntu with pandaboard 4 – compiling the opencv 2.3.1 did not complete the final compile steps, this post aims to complete that.

Continue the compiling of openCV 2.3.1

I assume you have finished the openCV compiling using cmake ., make, and make install commands in the terminal. Bear in my mind there is one thing I need to remind – be aware of the system date and time, as I left that incorrect and made some unnecessary compiling mistakes.

The system configurations

After the ‘make install’ commands was executed, the system configurations needed to be set up.

1. use the command below to set the opencv’s lib links in opencv.conf file.

adding the following link to the end of the file: /usr/local/lib

2. need to load the configuration file by using

3. open another file and add the configuration file path, adding the highlighted contents as in the below picture.

4. that’s done! Let’s launch an example to have a look if it works.

Running the example

1. make sure the build-essential is installed for opencv programme compiling, if not, using this:

sudo apt-get install build-essential

2. open the terminal, and go to the place holding the sample files, e.g. opencv-2.3.1/samples/c

3. change the target file mod, if you receive permission errors:

4. get the file compiled ready:

this command invokes the GNU C compiler to compile the file contours.c and make it an executable file called ‘contours’.

5. run the executable file contours:

dang-dang….. here it is…

Before the compiling

I was about to write a post about compiling the newest openCV 2.3.1 in Ubuntu with pandaboard, but because of the proxy network problem in Ubuntu software update and the right solution to the proxy network settings, this post was postponed.

So this post will be focusing on compiling openCV 2.3.1 in Ubuntu, with the right proxy networking settings and the synaptic package manager in working states.Sebastian Montabone has made a tutorial about installing openCV in Ubuntu, but that used openCV 2.0 and was not up-to-date. So, this tutorial aims to add an update of the newest version of openCV 2.3.1 installation.

Software updating

The compiling starts from software updating, which aims to prepare all necessary installation packages in the system. To make these packages needed for installations clear, I post these in forms of terminal commands, so you can type these commands one by one to see if they are installed already, or need updates.

Please be aware that, this post assumes the openCV 2.3.1 installation package has been downloaded in the local folder called ‘Download’, and has been unzipped into a folder called ‘OpenCV-2.3.1′.

Compiling steps

1. open the terminal, go to the directory where the openCV folder is located in, in this example, it is ‘/home/cw418/Downloads/OpenCV-2.3.1′.

2. type the cmake command in the terminal. please be aware there is a small ‘.’ after the cmake command.

3. after the cmake command is executed, type new make command.

This may take a while, so you can go and have a cup of tea first.
The tutorial continues on the next post – installing openCV 2.3.1 in ubuntu with pandaboard part 5.

The problem of networking proxy in ubuntu for openCV installation

In the last post instsalling openCV in Ubuntu with pandaboard a networking proxy problem was raised. The problem is relevant to software updates and installations in Ubuntu, before and after openCV installation. There were two general suggestions made in the previous post – finding a new network environment, or, properly setting up the proxy parameters.

The solution to ubuntu proxy settings

For the second suggestion, it seems more difficult than the first one, as it needs to be aware of different applications’ networking settings. This is my empirical experiences that took me nearly four days to get the right place.

1. I assume You have successfully got the proxy address, which used to be like this username:password@wwwproxy.domain.com:8080. This would be useful in the later setting.

2. There are two types of network proxy settings – application wide and whole system wide. I made mistakes in this stage so several days were wasted.

3. Click the power button on the top right corner, select the ‘System Settings’ from the popping up menu.

4. Go to ‘Network Proxy’ to start configuring the proxy parameters – if you have automatic scripts for proxy configurations then use that, if not, then fill in the proxy parameters in proper columns. Make sure you are aware of the bottom setting options – Applying system wide… click that button and confirm the passwords popping up for system authorisation.

5. Open firefox and check the network setting to see if it uses system configurations, if yes, see if firefox can open websites.

6. If firefox works, then go to ‘synaptic package manager’, go to ‘settings/preferences/network’, make sure the option is selected as ‘direct connection to the internet’, apply it.

7. Go and check the package manager updates, it should be working now.

Some errors caused by the incorrect proxy settings

Some typical errors caused by incorrect proxy settings include

- ‘failed to fetch’ error when trying to update the packages using apt-get update
- ’404 not found’ errors in reloading synaptic package manager

How to install openCV in Ubuntu without networks

In the last post of installing openCV in Ubuntu with pandaboard, the failure of software update of openCV was discussed. But if there was no alternative to switch to other working networks, then the only way to install openCV in Ubuntu is as the way showed in this post.

- first, downloading the OpenCV-2.3.1a.tar.bz2 from openCV wiki, selecting ‘other announcements’ then going to ‘for mac and linux’, the page will be directed to sourceforge in this page: http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/.

- copying the downloaded file to Ubuntu system, let’s make it in ‘Download’ folder. Then starting the manual installation – this part is rarely mentioned in tutorials over Internet.

How to install openCV in Ubuntu 11.04 manually

Once the openCV installation file has been ready in the folder, firstly it needs to unzip – using the commands in terminal:

- tar -xvjpf OpenCV-2.3.1a.tar.bz2

The untar progress will be running like this for a short while:

Untar openCV 2.3.1a in Downloads

After the openCV installation package is successfully untared in the Download folder, the source files can be seen. This is not the ready-to-use version, as the source files still need compiling.

–
To continue – part 3 – Compiling the openCV 2.3.1 within Ubuntu in pandaboard.
a networking-functional version of installing can be reached in Sebastian Montabone’s post ‘Installing OpenCV 2.2 in Ubuntu 11.04‘.

OpenCV at ubuntu with pandaboard

At the first beginning of post, I would like to remind a tiny, but sometimes may huge, problem in installing openCV at ubuntu with pandaboard – the network proxy setting. This tiny issue could be the fatal problem when it was not aware of. I had tried hundreds of times to ‘apt-get update’ but nothing worked out. After I searched the problem of ‘failed to fetch…’, I was not well informed about how severe that problem was. Until I tied to change the Ubuntu Software Center’s networking settings, I noticed there was an tap used for networking proxy purposes.

How serious is this issue

It is common that computers setting in universities are behind a boss proxy, it means all computers inside the university intranet have to get through the university server – data flows, firewalls and connection routines of course. The problem here is, the ubuntu system seems to recognise such super proxy not very well, to be honest, poorly. And what surprised me was that there was little posts in the internet that provided relevant information about how to set up the network connection before starting software update of installations.

As a result, the plan to install JAVA in ubuntu with pandaboard for openCV project development was non-doubtfully postponed – the reason was for the above.

How to overcome opencv at ubuntu installation

Well, the fastest and simplest way directing at the solution of this issue is to change to another network environments, or, use other network connection ways that does not need proxy settings. A good solution is already there – the wireless network. However, I have to mention that, the wifi signal received by the pandaboard was so poor – in most occasions there was only one signal bar – so tiny that I was always worried about if if would fade out. And the results proved that as well – the connection was unstable and slow.

Ok, no more chatty words, to address the network problem, either a wireless network is needed, or a open network connection that does not hold by a proxy is useful as well.

So, I am going to take the pandabaord home and get new software installed, until then I will be back to post ongoing openCV developments.

Some problematic screenshots

If you feel this post not providing the solution to the problem, please make yourself free to post any comments below.

Useful openCV mailing list

Someone in the comments asked where to get so much information about openCV learning and developing, well, one of my primary source to get openCV related information – as well as most people’s issues they are facing recently is the openCV mailing list – a good way to receive archived threads and discussions from lots groups such as yahoo openCV groups and google openCV groups, and even stackoverflow q&a forum, or maybe the canonical openCV wiki is also a good source.

Benefits to subscribe these sources’ mailing list

Well, it is obvious that if all of these groups and forums are subscribed by one reader, then the overwhelming information will keep him super hyper busy – all together these may generate up to thousands of openCV related items – including all kinds of problems, all levels of issues, and all possibilities of new openCV breakthroughs.

How to subscribe openCV mailing list

The simplest way to subscribe the openCV mailing list is to go to the groups or forum, make a registration, and probably there is a slot called ‘enter email address‘ waiting for your email address to fill in, after confirming the subscription – probably through emails the openCV mailing list will be like a loyal servant collecting and sending everyday or if you wish every week’s openCV discussions to your email, enjoy.

Opencv mailing list’s risks?

Well, if I have to say, the only risk is the too much information I receive everyday, I even have to set the mailing list subscription emails into a special category.

@丕子‘s comment reminded me to broaden the review area to cover multi-objects detection. Apologise for not making clear what is about to review in the second part, as a continue review of people counting using openCV, as in part 1 the review did not tell a clear content structure.
@zhijie thanks for your kind wishes.

Challenges for people counting using openCV

In part 1 the review mentioned some difficulties to apply openCV for people counting, including camera positions, specific detection feature (faces) and overlapping objects.

There are some other challenges as well – multiple people detection and tracking. This even concerns previous challenges, like the object overlapping. Someone suggested to use infrared photodetectors to avoid such issue, while that does not make sense here in using openCV and camera.

Methods of people counting using openCV

Andrey‘s suggestions gave three ideas from simple to advance. Simply enough, it was advised to calculate differences between successive frames of video stream and this detected line-crossing event by determining motion masks. A little more complicated, the idea was to use ‘running average‘ approach, and the most advanced one, was to integrate background subtraction technique to enhance the accuracy. The running average can be reached here openCV cvRunningAVG. However, there is still a problem – what if people gather together as a group then enter the room? In other words, how to keep tracking objects that may be combined and separated, unpredictably.

Counting grouped people using openCV

A short paper was recommended by someone in overstackflow.com, about human tracking by fast mean shift model seeking. Due to the limitation of time, the post is not demonstrating the details, but a picture is presented for demonstration. The paper can be reached via the link provided on the bottom of the post. This paper shows tracks behind people’s movements, grouping and separating have little influence on it.

Other ways of people counting using openCV

Other approaches were also introduced to deal with people group gathering problem. Someone (sorry I forgot the source) used track ID to mark different states of people. Every individual was given a unique track ID, which was binded with pixel-based contours. When two contours emerged together, the two track IDs would follow the common contour, until these people entered the room and were counted by the system.

A short summary

Neither people counting using openCV in part 1 and 2 is posted with some new cutting edge techniques, with which the people counting could be overwhelmingly improved. Rather, the state of people counting using openCV is very briefly introduced, with some concrete methods for people counting use. Wish these could be sparking some new thoughts.

Looking back the methods for people counting using openCV, some achievements have been made, especially in terms of counting a general number of people within specific settings. However, the future developments still face some problems, such as accurate grouped people counting, dwelling people counting and, probably, super density people counting – will it be possible to do that by counting heads?

Any comments are warmly welcome, including criticisms, as this review is widely open to anyone who is interested in people counting using openCV.

people counting openCV.pdf

Why using openCV for people counting?

There are alternatives to count people in a public place, for example using microcontrollers and lasers and arduino to design a cool and accurate system to count people in and out a room. But why does it bother to use webcams and, more importantly what the post focuses on, openCV?

1. it is easy to set-up. NOTHING is easier to mount a camera over the door and connected with power and data cables. Setting up a laser system? Think about the DIY work.

2. a lot of developers and programmers are working on that – making openCV more suitable and efficient to count people. A simple example is the blob extracting app.

The state of art of openCV for people counting

Blob extracting using openCV or flash (as3) have been researched by many developers. The accuracy has reached a high level, but limited to single objects – objects should not be overlapped – if people in the camera could be simply seem as objects.

Codeproject.com shows a new post discussing recognising people even though they are overlapped to some extent, as the picture below shows. The post compares different algorithms from simple threshold to image differences and erosion .

A pedestrian detector comes with recent versions of openCV (>=2.2), in modules/objdetect/src/hog.cpp, and samples/cpp/peopledetect.cpp. Unfortunately this new example has not be well documented officially.

If the camera for people counting is mounted statically, a substraction method is suggested by evident’s work. A relevant paper is linked http://portal.acm.org/citation.cfm?id=1561072&preflayout=flat.

Face detection for people counting using openCV is also concerned, as an example is described in opencv wiki – face detection using openCV. But the drawbacks are clear – this method needs faces for people counting. In other words, top-view cameras do not fit.

The rest part of review is published in the second part – a review of people counting using openCV part 2.

What webcams/motion sensing cameras are available for OMAP4 platforms?

As the Ubuntu 11.04 Natty has been successfully installed in the pandaboard, the next step of development work is to plug in webcams and motion sensing cameras (kinect and xtion). About installing necessary software such as openCV and openNI a new post will be dedicated to that. This post, mostly, desperately, is about seeking available supports to install motion sensing camera – especially the ASUS Xtion camera in OMAP4 pandaboard, which currently does not have many documents related to camera use in OMAP4 pandaboard.

In pandaboard’s website a compatible webcam Logitech webcam Pro 9000 is listed as one of accessories recommended for purchases. However this webcam is exclusively used in Android systems in pandaboard, in other words, there is no source found about Ubuntu based cameras.

Why cameras are not universally compatible?

My guess, to the incompatibility of cameras in pandaboard may be caused by the CPU – OMAP and normal Intel CPUs may have different structures and command sets. So they need different drivers to make cameras work. The problem is – where, and how.

Can you help?
There are some forum discusses seeking supports to camera drivers in OMAP4 platforms, pandaboard@googlegroups.com has quite many threads about this, but provides no canonical answers.

So this post is not about solving problems, instead, it calls for supports from anyone who may have useful information or experiences, to find a way to using universal cameras in OMAP4 pandaboard.

Intalling ubuntu in pandaboard for opencv development

Last post Doing openCV in Pandaboard described a odd mistake happened during the first boot of ubuntu system after the RAW image burning. The screen was frozen on the first booting, did not provide detailed information about any reasons.

I searched the internet and found two tools suitable to do the job – mounting the RAW image as a virtual disk in my windows xp pc, then burning the virtual disk image to the SD card. Two respective free-software was download – mount image pro for RAW image mounting as a virtual disk, and, minitool partition wizard for disk image copying/burning. These tools have been zipped, and can be reached via the link provided on the bottom of the post

At this time I installed these two tools, and successfully mounted the RAW image as a virtual disk 2, turned out it had three partitions indeed, the last one was not allocated/formatted yet, the first one was nearly 70 M and the second one was the main partition which had over 2 G contents. But the installation instruction in ubuntu official webpage said the disk space would be resized automatically. So I inserted the SD card and found it was recognised as disk 3. Here came the key operation – copying the disk 2 to disk 3, and leave the rest space as not resized.

Wait and wait and wait……

The copying was finally over, as the sd adapter did not flash any more. Then copying the MOL and u-boot.bin files to the first partition to guide the system boot.

Taking out the sd card from the host pc, and inserted it in the pandaboard. Exciting moment was finally coming…

Screenshots of ubuntu intallation on pandaboard

It was successful! Now the openCV development could start from here

The image mounting tool and disk copying tools can be reached via the link below, note the copyrights belonging to original publishers.

tools – mounting RAW image of ubuntu and burning disk copies

Pandaboard and ubuntu and openCV

In last post a brief show-off was given, with some snapshots of the new arriving pandaboard, but the post only had a quick mention of doing opencv jobs in pandaboard. Hence, I draft a new post to describe the environment of using opencv in pandaboard. Mostly it is about installing operation system (ubuntu 11.04) in the pandaboard in which to enable opencv development.

Issues of burning RAW image to SD card – ubuntu and openCV

Many tutorials have been given by bloggers, including video tutorials in youtube.com. Since the pandaboard used sd/mmc card memory as main storage, the ubuntu needs to be burned in the sd card to boot the board. Buring ubuntu in SD card had a simple glance at the official tutorial to install ubuntu in the pandaboard. However, that tutorial introduced an invalid software to burn RAW image in to the SD card, as it – win32 image writer kept popping ‘error 8′. All three versions had been tested, still gave same errors, and were finally given up.

Ubuntu RAW image’s contents – the head scratches

I looked into the RAW image extracted from the .gz file, found it consisted of three files, specifically, partitions, respectively named as ’0′, ’1.img’, and ’2′. So it seemed I need to project the three-partition image to the SD card. In windows xp, virtual image reading, disk partition copying – what tools could fix this headache?

Stucked ubuntu booting – any ideas?
I partitioned the SD card into three parts, as the RAW image did, same size, same files and same partition orders. Then copied related files to three partitions, burned the ’1.img’ to the second partition – only got cold black screen. Then I changed the first partition’s setting as active partition – windows used to boot from active partition. Try again, OOps, it worked this time…wait………stucked…lots errors…no booting ;{

(to be continued)

In a early post openCV was used to segment natural hand gestures from complicated backgrounds in real environments, as the picture above showed (see the original post hand gesture detection and recognition using openCV).

The picture above is a screenshot from a recent paper, about using Ataboost with SIFT (scale invariant feature transform), to detect natural hand gestures. The SIFT is used in the paper to reduce the background noise in the training stage, and so experimental results demonstrated in the paper show the approach performs with high accuracy.

So, the the adaboost learning algorithm is used after the SIFT background noise reducing. Relevant functions corresponding to these two algorithms have not been checked in openCV, to see if there is any working functions to realise such algorithms. However, more details about the use of Adaboost and SIFT, the paper hand posture recognition using Adaboost with sift for human robot interaction is reachable in the download page.

Haarcascades training (haartraining) is seemed an quick tool to achieve accurate hand gesture detection and recognition. The face and body detection examples included in openCV’s installation example folders (\opencv\data\haarcascades\) demonstrate how fast the haarcascades files help to do the job. More information about how to train the haarcascades files can go to sonots.com.

Many face-image databases have been provided for haarcascades training, e.g. http://www.face-rec.org/databases/. But hand images are less contributed for haarcascades training. At least, these are more difficult than face images to find in the internet. I was also asked by many blog readers for the haarcascades training file to support their hand detection or recognition related projects.

So I post an example of using haarcascades training file 1256617233-1-haarcascade_hand.xml for hand gesture detection.

The source codes can be downloaded from (or go to the download page):

http://download.andol.info/1256617233-1-haarcascade_hand.xml
http://download.andol.info/1256617233-2-haarcascade-hand.xml
http://download.andol.info/haarcascades-based%20hand%20detection.cpp

The preparation work
When I googled ‘pandaboard ubuntu’ I was firstly given the item of ‘ubuntu on pandaboar’. Following the link, finally, I was led to a ‘OMAP Ubuntu’ instruction page – it tells where I should start, to download the system image, and to burn it in a SD card.

Then downloading the right version of Ubuntu for OMAP4 platform. A number of copy versions are provided in the download page, in which the right one is ‘Texas Instruments OMAP4 preinstalled netbook image’, the download file after clicking the link is named like this ‘ubuntu-11.04-preinstalled-netbook-armel+omap4.img.gz’.

After the preinstalled system image downloading, need to unzip it, using winrar or winzip or other proper unzip tools.

(To continue – burning the downloaded image in a SD card)

Just received the pandaboard, to show off (;)) before I start to install Ubuntu on it, and to run openCV to see what will happen. New post soon, about installing Ubuntu on this board, then installing and running openCV on it. More posts will be published soon.

Transferring the openCV from c++ to Java is what i am doing recently. Comparing with opencv c++ it feels more natural to programme in Java, because of the way of Object-oriented programming and some of Java’s advantages.

But how can the openCV be used in Java seamlessly?

The easiest way is to take advantage of existing Java openCV projects, such as JavaCV (http://code.google.com/p/javacv/) or openCV in Java (http://ubaa.net/shared/processing/opencv/). These wrappers do make the use of openCV relatively easier, as the face detection example below.

import hypermedia.video.*;
import java.awt.Rectangle;

OpenCV opencv;

void setup() {

    size( 320, 240 );

    opencv = new OpenCV(this);
    opencv.capture( width, height );
    opencv.cascade( OpenCV.CASCADE_FRONTALFACE_ALT );
    // load the FRONTALFACE description file
}

void draw() {

    opencv.read();
    image( opencv.image(), 0, 0 );

    // detect anything ressembling a FRONTALFACE
    Rectangle[] faces = opencv.detect();

    // draw detected face area(s)
    noFill();
    stroke(255,0,0);
    for( int i=0; i<faces.length; i++ ) {
        rect( faces[i].x, faces[i].y, faces[i].width, faces[i].height );
    }
}
[These lines of codes are from 

http://ubaa.net/shared/processing/opencv/opencv_detect.html]

Another advantage of using openCV in Java is that it can be integrated into mobile systems such as the Android. With Eclipse the wrapper classes can be used to programme intuitively. For example:

import static com.googlecode.javacv.cpp.opencv_core.*;
import static com.googlecode.javacv.cpp.opencv_imgproc.*;
import static com.googlecode.javacv.cpp.opencv_highgui.*;

public class Smoother {
    public static void smooth(String filename) {
        IplImage image = cvLoadImage(filename);
        if (image != null) {
            cvSmooth(image, image, CV_GAUSSIAN, 3);
            cvSaveImage(filename, image);
            cvReleaseImage(image);
        }
    }
}

As asked many times about how to use openCV to detect body movements or other object movements, i borrow an example from openCV installation samples, which regards to motion detections particularly in motion directions. Some relevant posts have been posted before here, but to declare the use of openCV samples, this artical is posted. The picture demonstrating above has clearly showed the detection results which is, from left up to right down, just as the object moves, which is actually my head and part of shoulder.

Though this example is far too simple to put in practical use, this can be still useful for further applications as long as the codes are improved, at least the detection rules are remained. Here below is the list of source code which can be used directly as a .cpp file, check it out:

To download the source code, please refer to directory “OpenCV\samples\c\motempl.c“.

I found this by chance when i was searching something about body gesture recognitions. But this open library does attract me because it provides some features which can be very useful to detect and recognise objects, such like the picture above shows. Anyway i just provide a brief introduction to this library, and for more information please refer the official site here http://www.aforgenet.com/framework/samples/.

Shortly after further using and comparing with openCV in some object detections and recognitions, i will post another review post on this. Here is the brief introductions below.

AForge.NET framework is represented not only by a set of libraries and corresponding documentation, but also consists of collection of different sample applications, which are aimed to demonstrate usage of different framework’s API. The samples collection together with the provided framework’s documentation serves a good source of information and allows people to start working quickly with the framework and apply it for solving their own tasks in their different applications.

All the sources of AForge.NET framework’s sample applications are available in framework’s installation package, which contains all the components required to start working with the framework. Also binaries of all applications are available for downloading directly from the site, so each sample could be tested without installing the framework (look for downloads links on pages describing samples).

For more details, here is a relevant tutorial video made by original author who explains who this works.

Here gives the source code of this video demonstration.

You can download the full code at:
1) Full Demo Solution (Code + video 43Mb)
http://db.tt/MukGcwB
2) Full Demo Code
http://db.tt/GkT6J9k

First we gather a data set of all the hand-shapes we wish to recognise. A naive approach to recognizing a new image D would be to simply compare it with all the images stored in the data set and find the target image T with the closest match. But because there are so many images in the data set this will take far too long. We can reduce the time by using a multi-scale approach. We divide up the data set into groups of images, which are similar to one another by blurring the images at different levels so that small differences between similar images will be eroded.

More information please refer here http://www.ercim.eu/publication/Ercim_News/enw46/shamaie.html

A full paper about this can be found here titled as ‘tracking a hand manipulating objects‘

As described by the authors, a method of individual local tracker is used to achieve extractions. To achieve the goal as the left picture shows, it is required to build a 3-dimensional frame to skip the covering of objects and overlapped parts of hands. Seems the authors do not narrowly focus on color image segmentation which has been commonly adopted by computer vision researchers, but they integrate an estimation of features as well as 2.5-dimensional maps. That means probably extra dimension may required. In previous posts, we have successfully located contours of hands yet the shape recognition is still under investigation. And this new method may inspire us something further in hands tracking and recognition.

As far as in my mind, there are three approaches that people have taken to detect hands using openCV, though not all of them have been tested by myself (a little shame). One is to use accurate HAAR-classifiers to locate and detect gestures which is considered as quite stable way but costing very much. Another is to use adaptive skin detection algorithms combined with motion analysis which seems easier to do than previous one. And the last is to use human skin colour segmentation to detect the contours of hands combined with hand convexity detection to recognise hands’ gestures [please refer to: hand gesture recognition using openCV].

HAAR-classifier database is rarely provided with open source projects. But we found some helpful papers which may give some thoughts of classifier training and test.

About hands database:

A colour hand gesture database for evaluating and improving algorithms on hand gesture and posture recognition

Application of classifiers:

Real-time hand tracking using a set of cooperative classifiers based on HAAR-like features

Alternative skin detection based approach gesture recognition through angle space:

Gesture recognition through angle space

As these are mostly asked questions about using openCV to detect and recognise all kinds of subjects such as colours or hands, a review which includes tutorials, progresses and examples (if possible) would be published shortly…

Conclusively there are two ways of objects recognition, which are

• combining simple shapes’ recognition to detect complicated objects;
• using pre-trained data to match objects if they are as set.

The ways we used to detect circles and lines by openCV belong to the first category, and training needed recognition ways such as faces detection and hands detection belong to the second category. Recognition can be quite stable after good data training, but it may not recognize other features without pre-defined, so does the rules of faces detection with openCV.

But if recognition is used to design other applications such as objects controlling or multi-touch demonstrations, it could not be better choice adopting training way to set up recognition functionalities. As the picture shown above, i abstract the recognition part from ARtoolkit which is a popular open source library in internet. The training and recognition test results shows that  it can recognize nearly all the patterns pre-trained.

Quite much work has been doing to improve accuracy of pattern recognition, to enable computer applications being sufficiently intelligent facing this dynamic world. ARtoolkit is one of them. By training computers to recognize patterns as shown above, i have opportunities to shift my attention from recognition to applications, which means, representation forms of recognition affect interaction.

By extracting recognition codes from ARtoolkit, it seems easier to train a computer to recognize patterns like letters and other graphics, and its recognizing results seem pretty good. Based on that, creating appropriate applications become urgent.

The reason of that shortage is due to the lacking of appropriate algorithms. In AS3 objects can not be respectively recognized through colors or shapes before. Fortunately through combination of some kinds of image processing functions, we could still be able to find out the exact way of representing objects under webcam, just as processed images shown below.

This progress provides AS3 more potential to build up interactive prototypes through webcam and normal surfaces with kind of semantic objects laid on. Thus the AS3′s abilities of building up flexible interfaces may be well used in afterward works.

One very interesting aspect of human-computer interaction is methodologies and their effectiveness. There is a view bore in my mind that until interaction became invisibly natural, then interaction did its job. Comparing to thousand years’ history of human lives, computer interaction faces its challenge to integrate itself into natural lives.

We use object recognition to simulate original daily domestic interaction like reading mails and leaving notes. Once computers can recognize specific objects as different meanings relating to its huge stored information, we believe that would be a further step towards natural and efficient interaction.

This is a screenshot of demonstration hands detection using the code hsv.cpp. Comparing with the old version within black background and green contours lines, this version actually is based on that. Besides, two functions are added in this new version: using rectangle to locate the hands location, and using eslipe to illustrate the angle of hands motion.

hsv.cpp source code

Related post may interest you

Hand gesture detection and recognition using openCV

Some posts with prototype codes are occasionally requested to share the relevant codes. Due to this, i make a code collection in this post which could be convenient for all the readers. But please note that some codes still need additional image files or haar files that you should pay attention to.

haarFaceDetect
If you can not download the source code file, please refer to below
http://www.andol.info/download/haarFaceDetect.cpp

This is a prototype designed to test face detection using haar training algrithms. Needs image with faces and relevant haar trained xml file.

hsv
If you can not download the sourse code file, please refer to here below
http://www.andol.info/download/hsv.cpp
This is a prototype designed to detect hands ( not hands gesture ) in the dynamic motion images from web-cam. It uses color in a specific range to detect the hands, and then filters all the noises out to get the biggest contours of hands. Several backgrounds are tested by this codes.

HoughCircle
If you can not download the source code file, please refer to here below:
http://www.andol.info/download/HoughCircle.cpp
Static circle objects in the image are quite simple to grab. However, when it comes to combining recognition containing squares and circles, i have to admit that squares are easier to detect.

Recently, it is being applied to consumer electronic products because companies consider it an important key to their success.

However, there is a difference in the concept of usability between the two applications. Unlike the software user interfaces, the image/impression felt by the users are as important as the performance for a consumer electronic product to be successful. ”

As the post ‘10 videoes show some interesting AR applications‘ showed, integrating augmented reality technologies into interactive scenerial building is increasingly enhancing the experiences of interaction, both in the side of realastic world objects recognition and virtual applications.

These 2 pictures showed above are a demonstration of AR technology application with ARToolKit. Especially, i changed the default pattern for recognition and printed my own name on the card instead. After i trained the codes to recognize the new pattern, obviously, i got the satisfactory consequence. The codes recognized that new pattern rapidly and accurately, as well as using openGL to create practical coordinates and virtual 3D cube.

Although this isn’t the newest technology today, it indeed explored a new path to archive another advanced interaction method.

ARToolKit is a software library for building Augmented Reality (AR) applications. These are applications that involve the overlay of virtual imagery on the real world. For example, in the image to the right a three-dimensional virtual character appears standing on a real card. It can be seen by the user in the head set display they are wearing. When the user moves the card, the virtual character moves with it and appears attached to the real object.

— artoolkit programe

And the video show blow, is an extension of artoolkit, worth having a look.

During the past days, i was trying to get the computer recognizing blue and red colors to archive my prototype application. I tested two methods to distinguish the colors recognition effects, that were value-fixed approach and value-dynamic approach. 

What i was doing is to build a simple, flexible and adaptable intellegent colors recognition prototype which could work under varities of spatial environments. First, a fixed value of color was set. Any colors match this value would be recognized as specific one. It is simple and easy, although it is dependentable on the environment light. ( if the environment is controlable, this method would be good one. ) Second approach is that, i devided the picture into 3 channels which are H, S, and V. And then, from H ( Hue), i defined a range of color as one color and other ranges for another. This experiment looks not so difficult and more robust that previous one, however, there’s still one thing which should be cautious: light temperature. Light tempreature changes, color’s hue exceeds.

So, before there invents more advantaged devices, recognizing which colors from which colors is relatively limited.

Well, after watching this clip, i have to admit that augmented reality has the potential of combining the advantages of both well-spatial virtual reality and spatial augmented reality. And, i don’t konw if you have noticed that, actually, the demonstration in the video was made by Actionscript which included pattern recognition and 3D display rendering.

[ For more details, please go to slashgear ]

This is the 2nd part of usability evaluation for go.taobao.com. According to the 1st part evaluation, we found some of unfluent to browse the pictures which is probably due to the networking speed. So, in this review, networking was test 5 times destributed in different times in the day and night.

One random search word was selected on the page which looks quite hot. And, onece the first page was ready, regular click started at the rate of 1 click one second. Looks quite smooth at the first beginning~

Oops, looks something happened accidently, the picture displaying was sucked although the small thumbnails were still working.

Ok, waiting would be the only thing to do…

Getting into the picture detail entry, new window, detailed page and … oops, a small thumb’s missing.

Finally, the pictures caught up with my brouwsing. Wait… what is this, motel ? i found it when i was searching skirt ? Cool~

Again, similar lagging behind things happened, fortunately, i have understood it now.

So, the review conclusions:

1)  Precise picture matching, need be improved ( at least dont mistook motel to women’s dress, ;] ).

2) It is better to improve the rapidly picture browsing, don’t show sucking status in the front of users.

3) Small thumbnails are much faster to be loaded than big ones, why not fill the page with numrous of small pictures ( clear enough ) which would indicate bloody number of pictures.

4) The most important one, fliping from right to left seems not the perfectly effecient way to display information like those thousands of handreds pictures. If it flips fast, it looks blur and the users can not see the contents.

After several weeks’ hard working, finally, the memory leaking problem is fixed. 

In the last post which listed nearly all kinds of variaties declare and delete methods, here i add several tips for memory check while it is leaking:

1) int, double, float … : these kinds of variaties don’t matter so much;

2) IplImage* : this really matters, especially when cvCloneImage is conducted, if the destination variaty was not cleared, the memory would fly up.

3) CvMemStorage: be careful, to delete this data you have to use not ‘cvReleaseData’ but ‘cvReleaseMemStorage’, one declaration, one delete operation.

The main page is quite a little dark with simple search engine like style to lead user easily finding the entance. Some hot search words and pictures, on the left top corner, a even more simpler interface style switch button is there, but not so clear what it is used for until click it. 

Typing some words and click search, there comes results page. Looks the images are quite huge and slow to load, during the tests,  i tried several times to reload to stop page getting down, but at last i gave up to go back. Probably, the web speed would be the first problem.

Finally i saw the pictures, fantastic indeed, except bit of slow speed.

Browsing the page by click or wheel scrolling, either is good to use, nice work.

Oops, got down again. I wonder, why does Taobao sacrify so much waiting time to show such couple of pictures, patient challengable ? It would be nightmare under china’s web speed environments. 

So, i designed an illustration below to compare with the original one: getting smaller pictures which can be enlarged when mouse is over.

Here is just the first sight review, whether it is good to use or not, how easy to use and what exactly the interaction effectiveness it is, all those will come up soon. 

It can be a daunting task to hunt exactly where memory needs to be released. So I’ve compiled this short list of places where you should look out for memory leaks.

Create it, then Release it

If you create something, make sure you release it before “returning”. This is probably the very first thing you should check when fixing memory leak problems with OpenCV. For example, if you do a cvCreateImage, make sure you do a cvReleaseImage. There are many things you can create. Here are some functions that “create” and their corresponding “release” functions

One warning though: If you create something and want to return it, don’t release it. Lets say a function that creates a checkerboard image and returns it. If you release the image before returning it, you’re freeing all memory that stores the image data. And when you try accessing memory that isn’t yours, you get a crash.

Release returned structures

This is the second thing you should check for. Often, once you return a structure (say, an image).. you forget about it.

Multiple Memory Allocations

This is the third thing you should check for: Allocating memory, and then changing the pointer itself. Here’s some example code:

IplImage* image = cvCreateImage(whatever);
image = CreateCheckerBoard(whatever);
…
cvReleaseImage(&image);

This function creates a memory leak. First, you allocate some memory for image. Then, you call the function CreateCheckerBoard. This function itself creates new memory. And image now points to this new memory. The memory created in the first step is lost forever. No variable points to it. A memory leak. To fix this, you need to modify the code like this:

IplImage* image = NULL;
image = CreateCheckerBoard(whatever);
…
cvReleaseImage(&image);

If you return a sequence, release its storage

There are many instances where you use the CvSeq data structure. And often you might want to return this structure for further use. If you release its storage (a CvMemStorage structure) within the function itself, you’d free the memory where the sequence is stored. And then you’d try and access it in the calling function. Again, crash.

A temporary fix would be to just erasing the cvReleaseMemStorage statement… but that would mean lots of memory.
To fix this, you don’t release the memory in the function itself. You release it in the calling function like this:

cvReleaseMemStorage(&thesequence->storage);

storage is a member of the CvMemStorage structure that always points to the memory where its stored.

Again, this is just an example. There are more structures where a similar situation could arise.

Dependence on other structures

I quite recently discovered this memory leak. To explain this, I’ll use an example: Lets say you find out the contours of an image. OpenCV would return a “linked list” type structure calledCvSeq. You decide to access the third element of this linked list. OpenCV returns a pointer to the third element. All going great till this moment.

Now you decide to save all the points of this contour (the third element) in a data structure of your own. Since this is an array of points, you do something like:

mystructure->points = thirdcontour->points;

You set the pointer to equal to the thirdcontour. This is the bug. If you release the storage of the sequence (which you should), mystructure has a bad pointer. To fix this, allocate new memory to mystructure->points and then copy contents ofthirdcontour->points… something like this:

mystructure->points = (CvPoint*)malloc(sizeof(CvPoint) * thirdcontour->total);
memcpy(mystructure->points,thirdcontour->points,sizeof(CvPoint)*thirdcontour->total);

This creates new memory for your structure and then copies each element there. Once you’ve done this, you can release the storage of the sequence without fear.

1) Google is simple

Under the same technology & engineer conditions, Google makes simple products which are easy and fast to use; Yahoo, will make fantastic products too, but, probably, with some complicated functions which users dont need so much but which only confuse users. For example, GMAIL vs YMAIL; GTALK vs YAHOO! MESSAGER; and GOOGLE SEARCH vs YAHOO! SEARCH. That is the truth, if, one tool is able to archive one tast, the simpler, the better.

2) Yahoo! is aged

Just like human being, young people always have more strange ideas than aged ones, so is Yahoo!. Yahoo! was created on the first wave of internet which was populare with news, messages and information. While going to today, the third wave of internet has nearly pasted, Yahoo! still hasnt made imaginable products catering web2.0 users.

3) Google is creative

On the other side, Google is young and full of creation, especially its talent and energical engineers. They create all kinds of products, whether it is good or not, then, put the products into lab to test the response of usabilities. One by one, there increases the number of Google’s popular products and attracts users. 

Sometimes we dont know what users want, so we have to design all kinds of products according to the clues from web. Once we put the products on market, then, users will choose which ones are good and which ones aren’t — that is called rapid developing and fast evolution.

How to get the gestures needed?
Generally, in order to detect hand and extract gestures from dymanic background environments, some algrithms such as color segment and some kinds of ‘mean’ algrithms are neccessary, indeed. And, as far as i have learned until now, i found 2 methods to effectively extract target gestures from background images. Those methods are color segment, motion detection and more advanced sensing like infrared detecting integrating with motion or color. This step is quit easy to archive because of the limited algrithms and hand characters which include color, motion and what? Nothing! Probably temperature, but that is not so universal.

What chatacters could be used for recognizing gestures?
  So, if the gestures are extracted from the complicated background images, the next step would be recognize them. Recognition is not as simple as detection which you just do some filter work. It needs several unique characters from the detected images. Most of the time, these characters probably contain: 1) angle – detecting the rotation   2) size – detecting the area   3)  peak and valley which is specific for the open hand  4) moving speed and direction. 

Related Posts: Hand guesture recognition using openCV

Clicking the pictures and the detailed previews will display which show the hand contours extracting from video capture. These pictures illustrate the results of color based segmentation for hand gesture detection ( not recognition yet ). it is a  real time programme,  however, in this demonstration, the programme still depends on environments’ light , because of the color based segmentation method. If, combinned with motion detection and background substraction, it supposes to be robuster.

According to the algrithms of  HOUGH, that is more accurate to line detection rather than circle detection, although some times it looks not so much different in theory. Actually, if detecting circles in a black-white picture, it will be obviously not so accurate as detection for lines. Detected circles are always flashing in different possible positions.

To solve this problem and low the mistake detection, the target picture should do some filter: SMOOTH.

cvSmooth( gray, gray, CV_GAUSSIAN, 11, 11 );
cvCanny(gray, edge, (float)edge_thresh, (float)edge_thresh*3, 5); //get the circle edges
CvSeq* circles =  cvHoughCircles( gray, cstorage, CV_HOUGH_GRADIENT, 2, gray->height/50, 5, 35 );
cout << circles->total <<endl; //check how many circles detected
int i;
for( i = 0; circles->total>=2?i<2:i < circles->total; i++ ){
float* p = (float*)cvGetSeqElem( circles, i );
cvCircle( src, cvPoint(cvRound(p[0]),cvRound(p[1])), 3, CV_RGB(255,0,0), -1, 8, 0 );
cvCircle( src, cvPoint(cvRound(p[0]),cvRound(p[1])), cvRound(p[2]), CV_RGB(200,0,0), 1, 8, 0);
}

From the birth of mouse and keyboard, people have been using these kinds of input devices for decades years, and now, it is the time to change. More nature interaction methods are booming, like multiTouch, camera detection and gesture cognizing technologies. By these new technologies, people in the future could be able to interact with computers naturely as parts of own body. The nature hand gesture congnizing is a demonstration of that trendancy. 

We indeed need new interaction ways to communicate with the physical world with higher effeciency.

In this post, as the progress goes, the hand detection technologies using openCV is introduced. As the pictures inllustrated below, this detection method is independent  with distance and background ( just not the background full of hands ), and the main segment method is color abstract which means getting the hand color pixels filtered. As the result tested within several environments, it workd well except slightly noise varying. But the noise is easy to take off.

The tested source code file is offered underneath the illustrations, feel free to use it.

SOURCE FILE HERE:  handDetection.CPP
Related post which may interest you:
Hand gesture detection and recognition

After finished basic shapes recognition using openCV, we would step new progress — hand recognition. I saw some researchers said computer vision as key part of artificial intellegence due to its hardness. That probably is right, because when i started the hand detection ( before recognition ), i was blocked by the algrithms to accuratly detect the hand shape. Nothing simple algrithm will be applied directly here due to the variation of hand gestures.

When i googled how to detect hand gestures, i found  HandVu .

HandVu is a library of hand gesture recognition based on c++/c and openCV. According to the demoes published in web, this library looks powerful, although still is getting mature. When i tried to download the source files, i found the version hasnt been updated for years until 2006 and the downloaded binary didnt work right with alerts of *.dll files getting lost , that was probably caused by the openCV version.

Whatever, at least the HandVu library offered guidline of hand recognition.

In this post, i will present you the new progress of object recognition using openCV—angle recognition.

Actually, this is just a small step further than the last one which is rectangles’ recognition. From the four outlines of rectangle as well as the four corner points, we can get two relative points to calculat the angle between them. As inllustrated below, i chosed two points which are most left and right. Once the position of these two points are captured, the SINE value of the angle can be calculated like this

SIN (angle) = abs(p1.y-p2.y)/sqrt(
(p1.x-p2.x)*(p1.x-p2.x)+(p1.y-p2.y)*(p1.y-p2.y))

So, that’s it.

First-chance exception at 0x7c90eb74 in camera1.7.exe: 0xC0000008: An invalid handle was specified.

And, the debugging pointer was on the left side of the code:

capture = cvCaptureFromCAM(0);

Well, this line ran smoothly in Code:Blocks, why at here it went wrong.

However, i think, openCV’s default functions which are included in its library maybe probably have this abilities. As a result, today i found a openCV function called

CvConvexityDefect

-Structure describing a single contour convexity detect

and there is  a illustration picture on the side of the instruction like below:

So, until now, colors*shape*angle=>infinite, the future looks bright to code some application of interactive objects recognition. But these founded a problem that is memory leaking in C++. I am newbie of C++ and i used Actionscript to make codes which were automatically recyling all the memory trash. But now, the trash memory have to be delete manually, terrible nightmare to me.

However, when turns to rectangle recognition, it seems work right but not so much. Look carefully, there are multilines drawed on one rectangle detected, and the rectangle number displayed on the left side was totally wrong—too much rectangles, that was impossible because as i see that i just put 3 rectangles under the camera. As a result, i was sucked here and couldnt move forward.

If there’s any good idea, welcome to leave a comment or contact me.

In color recognition, it can detect any color, but there’s still a little bug that,  the detectation color should be preset before recognizing the objects in the destination image. I am not sure whether there’s any color filter in openCV like Actionscript3 which can just directly filter the necessary color objects.

Source code colorrecg.pdf

April 26th, 2007

Multitouch Audio Exploration

---

Jens Wunderling has been working on an interesting project for over a year now, he started with a single touch ELO display and now he has implemented it with multitouch.

loopArena is an AMAZING interactive audio tool, similar to a sequencer, however much more dynamic. Jen’s calls it an “agent based generative music interface”, which allows for multiple users to generate live music.  No typography, rasters and tools are needed because interaction is limited to dragging objects with the fingertips. I would imagine that this gives the artists a lot more freedom and allows one’s creativity to flourish.

He has been working with Ulrich v. Zadow of the MTC team. And as you can see from the video above the results are stunning. As for the technical details, the User Interface, Timing and MIDI Out are all made in Processing, Touch data is received from the libavg framework via OSC, and Processing sends Out MIDI data to Propellerheads Reason. You can read about how it all started as his Intermediate Diploma project.

There was a problem often happened in the prototype demonstration. When turned off the demonstrating computer and camera and then turned it on, there popped a interesting problem that was the camera was reset and the preset zoom area disappeared.

So, the best idea is to design a lable to tell prototype which part is target work place and which parts are not. After one day’s researching, finally, the solution is presented here (the picture above)— two circle dots are used to specify the invisible border of work place, out part of these two dots’ rectangle, there would not be work place any more, vice versa.

From black screen to graphic screen, this is big step over openCV. After the compiling problem was solved, the progress of objects recognition is much faster than ever, although there still are some unknow bugs. Mainly, the routine of interactive objects recognition prototype developing is from object dectetion to multi-objects recognition and at last it would be added artificial factors.

[The figure above illustrates prorotype recognizing multi-objects and displaying the number of objects detected.]

After several times’ failure, finally i fixed the problem of C++ compiling. Admittedly, that was under pressure. This new stage starting from openCV will mainly focuses on new interaction method and effectiveness evaluation.

This is the interface of prototype for central heating control.

This it just the Object-recognition effect i want. But i dont know how to do it. Any idea?

These are two pictures from object recognition in AS3. The first one is original picture captured from webcam and the second one is recognized picture which comes form the former picture.

Actually, the problem lays here: the objects which are recognized are unibody but not in parts, under this situation, when new objects being added, there would still be one whole part but not different parts which are fatal important to the respective object recognition.

I have tried to use different color to identify different objects, however, there still blocked by a problem: once the same color objects get more and more, the red frame (object zone) will also expand which directly declines the recognition accurate.

So, to solve the seperating objects problem, i need some experienced people’s help.

Note: please turn off your speaker or use earphone when test this prototype.

If you have MICPHONE connected with your computer while you browsing this page, it is a big chance to blow to your micphone to driven the windmill spin. Have fun.

Well, this is the second version of my windmill which name is Blowit which i gave it just moment ago. In this version, the power of driven has been improved and one interactive feature was added which is -micphone driven. So, it will be much mature prototype to experient this interaction way. However, i am still wondering to where i can use this prototype as it is designed to check the sound and use the audio level.

Related Reading:

Click-driven windmill

The first picture is original image  captured from webcam, it is little blur and grey which is not easy to identify. The second picture is the one processed by flash:as3, obviously, it can recognize the text and part of background which have high contrast to light color.

What i want to continue to do is to develop the color recognization and shape recognization, as well as adding more intellegent application in it. I have to admit that this is just the start of this interesting interaction way, once the further version was completed, i hope perhaps it would shock you.

‘Central heating control panel’ prototyping is going to end, especially after two evaluations of usability.The picture above is only interface of this interactive prototype which is designed to change users’ attitude and behavior towards energy consumption and green-house effect CO2 emission. We have tested the usability of this device to ask several participants learning it and teaching me back to check the degree of that. Comparing with the former version V1.2, it is in progress.

Windows(R) 7 preview version was released for evaluation. Some new features are added and some minused. However, the apperances looks still the same as Vista(R), transparency, little blur, colorful—looks nothing greatly changed. Although many creative features being added in this new future version, the interaction is still poor as the reasons below:

1) The interface of Windows(R) 7 is still colorfull and attention attractive, admittedly. It is good thing to some aesthetic extend, but not for great effectiveness—as you know, the interface distracts too much attentions from work to system expplore.

2) Too many tiny but smart functions are hidden and hard to find and use, before use it, not like the Windows XP(R) which is easy and simple to use and explore.

3) The icons in Windows(R) 7 is numberous and confusing, not uniform and same stylized.

Get more news of Windows(R) 7 from Windows(R) 7 official blog >>

This example is driven by mouse-click, and i am trying to design a new version which can be driven by voice. If that is done, we can blow the microphone to spin the windmill.

For weather report module.

This afternoon in informatic research group office, while i was reading Marker Weiser’s paper of ‘computers in the 21th century’, an idea suddenlly flashed into my head-the pitfall of VISTA’s interaction design. I believe that I found the reason why VISTA attracting so much attention of users from work task to interface.
I cite Marker Weiser’s explaination below:

The idea of integrating computers seamlessly into the world at large runs counter to a number of
present-day trends. “Ubiquitous computing” in this context does not just mean computers that can be
carried to the beach, jungle or airport. Even the most powerful notebook computer, with access to a
worldwide information network, still focuses attention on a single box. By analogy to writing, carrying a
super-laptop is like owning just one very important book. Customizing this book, even writing millions of
other books, does not begin to capture the real power of literacy.

Furthermore, although ubiquitous computers may employ sound and video in addition to text and
graphics, that does not make them “multimedia computers.” Today’s multimedia machine makes thecomputer screen into a demanding focus of attention rather than allowing it to fade into the background.

So, as it indicats, VISTA is making a big mistake.

What is different between VISTA and OX?

-Different system operation!

Of couse there are different OS, but, what else in interaction? VISTA is gogerous and wonderful, OX is elegant, beautiful and effective. What exactly the differences are?

Firstly, the interface difference is most obvious character between them. OX is grey, in the contract, VISTA is blue or gree or yellow? As i mentioned in the past post that the core of interaction activities is effectiveness which directly decides user experiences and system ability.  When people use long time VISTA, they will find that the fantastic background and colorful light effect are not so attractive any more. Strangely, the OX interface which is made up with grey and a little gradiant is always attractive.

Secondly, VISTA attracts people’s attention from work to system itself, it is too colorful to concentrate all the energy on work. Especially the beginner of VISTA, probably they would always look for some features but not work, too many settings, too many functions collected together and hard to find them. When goes to the term of OX, hardly people will lost their focus on work, why? The setting panel is easy and accessable, the features are useful and easy to manage, and the functions are so smart just like the system is only designed for this work. What else, it makes people enjoyable while doing the work, amazing!

The last difference is that, i think, VISTA’s effect is from OX. The animation, the window and the work tool, all of these have the shadow of OX. Do you think so?

HCI is interupted as Human-Computer Interaction. It is explicit to understand what that notion stands for-about interaction between people and computers. Is that right or does that completely summarize the right concept of HCI?

Not, I am afraid.

The reason i am not agree this notion all is that that interuption do not get the core of HCI. So, what exactly is the core of HCI? The core of HCI, in my opinion, is not UI (user interface), not UE (user experiences) and not just smoothing the task process.It is EFFECTIVENESS, all the work we are doing such like designing beautiful interface and simplifying process results in effectiveness improvement of HCI.

For example, the web site is rebuilt once again to improve user experiences and accelerate the data processing speed, as well as providing comfortable web environments for users to deepen the impression of the site service. Improving interaction means less commercial cost and more business effectiveness.

Another example is Windows (R) operation system. The visual ‘window’ concept came from early stage Apple (R) computer product. However, it is no doubt that the jump from command line input interaction to graphical user interface interaction indeedly improves the effectiveness of HCI which directly contributes the computer pervasive.

Here we can reach a conclusion, that is the core of HCI is effectiveness and once the HCI contributes great interaction effectiveness improvement, HCI would have an essential revolution.

-<Informatics applications and beyond>

-Eric Bergman

The ‘write once, run anywhere’ slogan was clearly stated by the software people who thought only of the algorithm, as opposed to the appearance of the screen, because it’s fundamentally impossible to do this. Although it is true i can make a satisfactory graphical user interface that could be written by JAVA and therefore look satisfactory on the Macintosh platform or a Microsoft platform, in fact, either would be probably inferiou to what Apple or Microsoft has provieded. And, second of all, it certainly would not cut accross dramatically different platform such as my cell phone or my Palm device and my high-defination display screen.

It’s increasing, the ‘write once, run anywhere’ argment. It was hard to see how that was ever said because for the last decade most of code has been user interface code. And, the concept failed at the user interface as soon as you work on a variety different platform with different hardware and different I/O devices.

This is a part of description of HTML structure for application in different devices. Admittedly, the problem of HTML data applying in different devices with different hardware system and display interpretation really blocks the internet being pervasive, at least pervisave in the mobile web devices such as mobile phone or PDA.

Not the end of internet, it is just beginning of it-we need to solve lots of compatity problems as mentioned above. Meanwhile, it is big challenge of Human-Computer Interaction to create nature interactive communication methods.

Radically condensed demonstration of oblong’s g-speak spatial operating environment

Vedio is attractive, but how about interactive video on street?

Double attractive interactive display.

The world we live in has become suffused with computer technologies. They have
created change and continue to create change. It is not only on our desktops
and in our hands that this is manifest; it is in virtually all aspects of our lives, in our
communities, and in the wider society of which we are a part.
What will our world be like in 2020? Digital technologies will continue to proliferate,
enabling ever more ways of changing how we live. But will such developments
improve the quality of life, empower us, and make us feel safer, happier and more
connected? Or will living with technology make it more tiresome, frustrating, angst-
ridden, and security-driven? What will it mean to be human when everything we do
is supported or augmented by technology? What role can researchers, designers and
computer scientists have in helping to shape the future?

Download: Being Human:HCI in 2020-Microsoft Research Report