Kinect on PC Windows - a window to the future (experience of use on a PC). Kinect: understanding the new device and learning to write applications for it Games for kinect windows

Sandbox

Natasha March 7, 2011 at 02:49

Installing Kinect on Windows 7

• Closet *

Introduction

This article was born in the process of creating a simple program in C# that receives an image from a regular USB cameras and saves the image once every 5 seconds hard drive. It so happened that I only have the built-in camera of my laptop, and to solve the problem, the camera must look through the window onto the street, I didn’t want to buy a new one, so my eyes turned to the Kinect on the TV. Accordingly, the first priority task is to connect the device to the computer, and we will deal with it now.

Looking ahead, I’ll say that I haven’t tried using Kinect instead of a camera yet, because... I found an ordinary usb webcam, but I won’t abandon this idea, because... There are already kinect wrappers for c#.

Prerequisites

I would like to immediately make a reservation that all operations were carried out in Windows 7 Ultimate installed on a MacBook via BootCamp. I am 99.9% sure that this does not matter, but it would not be amiss to mention.

So, what do we need:

• Kinect
• Computer
• OS Windows 7
• Visual Studio 2010 (with installed support C++)
• Internet (to download the necessary libraries)

Preparatory stage

Perhaps ready-made drivers for Kinect can be found somewhere on the Internet, but you must agree that what you create yourself is nicer, more understandable, and you have no one to blame for mistakes but yourself. So we'll start by downloading all necessary programs and libraries.

For convenience, I will create a folder C:\OpenKinect, in which all work will take place.

libusb-win32
This is a port of the libusb library, which allows access to various USB devices from client applications.
Download a file with a name like libusb-win32-bin-x.x.x.x.zip from the latest build, in my case it was build number 1.2.2.0 and unpack it to C:\OpenKinect\Libusb-win32 (the folder name is not important, I named it for convenience , this is true for all folders that we will create).

pthreads-win32
Pthreads is a set of libraries that implement simple work with POSIX threads. The project is very old, there have been no updates since 2006, so I am giving a direct link to the necessary file, which will need to be unzipped into the C:\OpenKinect\Pthreads-win32 folder.

You will also need to copy the file C:\OpenKinect\Pthreads-win32\Pre-built.2\lib\phtreadVC2.dll to the \Windows\System32\ folder

Glut
This is a library for working with OpenGL. Another old project, and another one is just a link to a file that needs to be unpacked into the folder C:\OpenKinect\Glut
glut-3.7.6-bin.zip

Copy the file C:\OpenKinect\Glut\glut32.dll to \Windows\System or to any folder that is mentioned in our environment variables (PATH)
In addition, you need to copy the file C:\OpenKinect\Glut\glut.h to the VC\include\GL folder of your VS2010, if there is no GL folder, then you need to create it. File C:\OpenKinect\Glut\glut32.lib in the VC\lib folder. In my case the paths look like this:

• C:\Program Files\Microsoft Visual Studio 10.0\VC\include\GL
• C:\Program Files\Microsoft Visual Studio 10.0\VC\lib

Building the project

Git
Naturally, we cannot do without the source files of the libfreenect project, so download Git, install it, and select options during installation "Run Git from the Windows Command Promt" And "Check-out Windows-style, commit Unix-style line endings".
Git-1.7.4-preview20110204.exe

The next step is to open command line and sequentially enter the following (the parser converts the address into a link, so I added an extra space between https:// and github.com, it needs to be removed):
cd c:\OpenKinect
git clone https://github.com/OpenKinect/libfreenect.git
cd libfreenect
git branch --track unstable origin/unstable
git checkout master
git pull
As a result, we will get a folder with sources C:\OpenKinect\libfreenect

CMake
A utility that will help us collect all our libraries into one project. Download the program from the website at the moment latest version 2.8.4, install, during installation select the option "Add CMake to the system PATH for all users"

After installation, run the cmake-gui.exe file. In the first line, select the path to the folder with the sources, for us it is C:/OpenKinect/libfreenect, the second line is the folder where the assembled project will be C:/OpenKinect/Drivers. Check the Grouped and Advanced boxes, click the Configure button, select Visual Studio 10 in the window that opens and click the Finish button. The program will run with an error, because the paths to our libraries are not indicated, we will indicate it as shown in the screenshot and double-click the Configure button.

Now we should have no errors and we can safely click the Generate button; as a result, in the C:\OpenKinect\Drivers folder we will have a project that we will need to build in VS2010.

Visual Studio 2010
We open our freshly baked project in VS2010 from the C:\OpenKinect\Drivers folder (libfreenect.sln file) and build it (for some reason the project was built only the second time). If everything went without errors, then we should get the folders C:\OpenKinect\Drivers\lib\Debug and C:\OpenKinect\Drivers\bin\Debug, the first of which needs to be added to PATH.

At this point the assembly stage is completed and we proceed directly to connecting the Kinect

Kinect connection

Everything is quite simple here. We connect Kinect, the system will detect it as Xbox NUI Motor and can’t find the drivers, let’s help her with this. Through the device manager we find our sufferer, select the “Update driver” option and specify the folder C:\OpenKinect\libfreenect\platform\windows\inf, the system will ask for confirmation for installation unsigned driver, we give our consent, after this the green diode on Kinect should blink and two more unidentified devices should appear in the list of devices "Xbox NUI Camera" And "Xbox NUI Audio", install drivers for them from the folder C:\OpenKinect\libfreenect\platform\windows\inf.

At this point all our work is completed, all that remains is to check how everything works.

Testing

To check the functionality of Kinect, we will use three utilities from the folder

• C:\OpenKinect\Drivers\bin\Debug:
• glpclview.exe - Shows the capabilities of the Kinect 3D camera
• glview.exe - Shows the operation of a regular camera
• tiltdemo.exe - Enables Kinect motors

Here is the result:

P.S. If you want to write your own program to work with Kinect, then in the C:\OpenKinect\libfreenect\wrappers folder there are wrappers for many languages.

When writing this article, information from the site was used

A lot of different reviews have already been written about Kinect. Therefore, my review will be somewhat chaotic. I don’t try to describe all the hidden sides of this device. I will focus on the practical component and a brief history.
And I’ll probably start the excursion into history with a review of game consoles.
There are a certain big trinity of manufacturers in the game console market.
Nintendo - console
Sony - PlayStation 3
Microsoft -

In 2006, the world was shocked by the news about the emergence of a new generation of game controllers; Nintendo, ahead of its time, released it to the market. Sales of the console, thanks to the concept of the new controller, have increased sharply. The competitors, not wanting to shepherd those behind, decided not to stand aside and began to prepare their solutions.
In 2010, Sony introduced its controller. The controller had higher recognition accuracy than its direct competitor. True, this controller is of no interest to you and me, because Sony was greedy and did not leave the opportunity to connect the Move directly to a PC.
And now, a month after Sony’s presentation, Microsoft triumphantly appears on stage with its revolutionary Kinect controller. The software giant's engineers discarded all stereotypes and created a fundamentally new type of controller.
Kinect has a whole series advantages compared to other competing products:

• does not require additional devices,
• leaves your hands free,
• recognizes movements of the whole body, not just the hands,
• recognizes the movements of up to 5 people simultaneously,
• recognizes voice
• does not require charging or replacing batteries,
• is able to identify movements and gestures of any complexity.

In fact, Kinect is a controller without a controller; it is a device capable of recognizing normal human movements and transmitting them to a computer for processing.
So, we will not go into details of the operation of the device; we will only focus on the fact that the controller is connected to the set-top box using a slightly modified USB port. This fact was noticed by the first users of the Kinect, so immediately after the release of the device, a real hunt began. The powers that be have announced thousand-dollar prizes for hackers who can make Kinect work on a PC.
A few days later they were written, and a couple of months later the official driver sets were presented by the chipset developers and Microsoft itself.
A year later, leaving out the unsuccessful attempts of developers, it is worth focusing on the two main sets of drivers and development kits for Kinect.
1) PrimeSense OpenNI/NITE Framework from the creators of the central chip. .
And
2) Microsoft's Kinect SDK.
PrimeSense was the first to create and open its SDK for PC programmers.
Therefore, all the first programs for Kinect work under the control of these drivers.
Watching programmers from all over the world create and modernize the software environment for working with the controller, Microsoft decided to reconsider its policy. They abandoned the idea of ​​Kinect being exclusive to the XBOX console. The controller was given green light to enter the PC arena.
On at this stage Microsoft prohibits the creation of commercial software for your controller. True, this situation will not last long. Already on 02/01/2012 we were promised the release of the final version of the development kit + start of licensing of commercial software. Moreover, Microsoft strongly promotes the development of software for its controller, assigning high monetary grants to development teams. Let's add to this the fact that Microsoft has obliged all game developers to release new games for XBOX exclusively with Kinect support. (Note: The vast majority of modern games are first developed for XBOX and then ported to PC.) All this suggests that in the coming months we will see the emergence of a whole range of programs that will be focused on controlling hand gestures and voice.
True, the most impatient can appreciate the benefits of Kinect now.
There are a number of smart programs that you can use today:
To do this, we first need to install the PrimeSense OpenNI/NITE software package and Microsoft's Kinect SDK.

After this, you will have to manually switch drivers in the task manager, depending on what software you choose:
1) – a set of gestures for mouse control and various programs. Uses . The transition to Microsoft's Kinect SDK has been announced in the near future.
P.S. The project has high usability. We are waiting for new versions from the developer.
2) kinvi– full-fledged virtual interface for working with a PC using a Kinect. Very powerful and progressive development, supports both types of SDK.
P.S. Unfortunately, I was unable to run it on Russian Windows. The program crashes with an error opening the config file.
3) Kinect Mouse Cursor- a simple but very useful program in the household, necessary for emulating a tailed controller. Powered by Microsoft's Kinect SDK drivers
4) FAAST– a shell for emulating a keyboard and mouse using a kinect. Great for creating configs for games that do not support a controller. Uses PrimeSense Framework. The transition to Microsoft's Kinect SDK has been announced in the near future.
P.S. Low usability. Easy to set up. Recently version 0.10 was released which has a number of improvements.
5) GlovePIE / FreePIE (new reincarnation)– a huge toolkit for capturing movements using a kinect and converting them into commands of any other controllers. Supports Microsoft's Kinect SDK and is currently the best project for using the controller with a Windows PC.
P.S. The project is free. The developer is not a rich man. Without external support It will be hard for him to carry such a colossus.
6) Win&I, kinclick and others. There are still a lot of old projects from different developers. As a rule, these projects have limited functionality and are currently frozen.
P.S. Win-and-I is a paid product. But those who are persistent can always find a cure for greed. 😉 Despite the famous developer, I didn’t like the product. There are too many bugs in Wibdows 7.

You can download the files in attachments or on the manufacturer’s website.

So, what awaits ordinary users in the near future?
Firstly, we can expect active growth in applications that support motion recognition, and secondly, beta will be released at the end of February windows version 8, which should have deep integration with kinect technologies (facial recognition, scrolling and movement using hand movements, voice control).
Based on the above analysis, you can be sure that Microsoft is preparing something huge. They are preparing a revolution in the world of controllers. Microsoft once managed to change the course of history by releasing OS Windows 3.1

So, how to teach Kinect to work with a PC?
1) First, buy a Kinect.
Its cost will range from 100 (for a used device in the states) to 300-400 for a fresh version of the device, which is adapted to work with a PC.
If you purchased the Kinect version for XBOX, then you will also need an adapter from the standard Kinect connector to USB.
You can easily buy an adapter in a store or make it yourself. In fact, the standard Kinect connector is the same 4 USB contact+ 1 contact 12 volts for running the camera motor. 12 volts can be taken from the power supply of your computer (any yellow wire).
2) need a TV. Of course, a monitor is also suitable, but for the full effect it is better to use a TV. In addition, the Kinect version for XBOX has a blind zone: 50 cm, which means it is advisable to be at least a meter away from the device.
3) position the kinect correctly. Preferably in the middle of the TV, 15 cm below or above the edge of the screen. From experience, even if you place it sideways and at an angle of 90 degrees, the device will work, but it’s better to follow the developer’s recommendations. For proper placement, you need to install the controller on a special stand - if you don’t have one, you can always.
4) decide on a set of drivers. I would recommend choosing the Microsoft kinect SDK. He's more promising. I would venture to guess that if Microsoft doesn’t screw things up ( has already started to break), then all competitive driver sets will fade into the shadows in the next 6 months and supporting their software will be pointless.
In addition, the best software already runs on Microsoft drivers. Although, one dog is buried here too. I would advise using the latest version. At the moment, only 11 days have passed since the release of the final version of the SDK, which means that everything existing programs are still adapted for version Beta 2 and are not compatible with final version. By the way, one of the advantages of the original SDK is that it comes with special software that allows you to use your controller as a high-quality 3D scanner. You can scan objects and then easily and easily send them to your location for printing.

This is all, Now you can fully use Kinect on your computer.
Let's try to use it in practice.
Methods for using kinect in everyday life are:
1) The easiest way. - This Use your Kinect as a high-quality camera and microphone.
How to do this?
– The microphone will work immediately after Microsoft installations SDK and additional manipulations are not required.
– With the camera everything is a little more complicated.
First you need to download the file
Then register it in the system with something like this line “ c:windowssystem32regsvr32.exe KinectCam.ax”
It may also turn out that you lack certain libraries on your computer. In this case, you need to download them and copy them to the folder with the file
Download all necessary files can be found in the attachment.
After all these manipulations, Skype and other programs will begin to see your Kinect as a regular camera.
Now, check the camera using Skype, and then go to C: Users AppData Local KinectCam config.ini
In this file you can set the camera angle, as well as other settings

2) To manage all kinds of software. Due to the fact that Microsoft prohibited the release of commercial software until the final SDK was released, worthwhile programs for Kinect now not so much.
I can recommend a couple of programs for controlling the cursor and windows in general: or Win&I. These programs are a bit old, but they have ready-made presets for several players, browsers and other programs. They didn’t want to work normally for me, the probable reason: new firewood or a new OS.
They can be used, but with a number of significant reservations.
You can also use the GlovePIE designer program or its new version FreePIE, the functionality is huge, but for her ready-made scripts no, you need to remake other people's scripts to suit your needs.

3) The third method is the most correct. Kinect was created precisely for games. Only in games can you reveal the full potential of this device.
I would roughly divide all the games into three groups:
3.1) Games specifically created for Kinect.

This is the most the right way enjoy the game. The developers have already thought through everything for you. They came up with gestures and adapted the gameplay to them. These are the kind of games that are released for XBOX and they are the ones we miss so much on PC.
I've only seen 4 similar games, and all of them are ordinary demos that only allow you to evaluate the capabilities of the controller:
-SkiRanger
– Shape Game Demo
– MotherNature
– ZombieHoldout
3.2) Simple games for PC.

Such as Tetris, Mario, etc.
These programs have simple controls, and most importantly, you can play them without using a mouse.
To control such games, a simple script is suitable. For example .

3.3) High quality 3D games, whose developers never thought about compatibility with Kinect

Such as: Skyrim, GTA 5, Mass Effect 3, Batman: Arkham City, and basically any other game you would like to play
The scripts for them are complex and multifaceted. First- and third-person games are simply born for this controller, only in them you can cast a spell with your hands or hide from the dragon’s flames with an air shield
I'll give you an example.

I would like to thank Mr. Mpoor for the development of this script.
The script is good, but requires calibration. You should stand in front of the kinect at such a distance that the debug line on the X and Z axis will display zeros. In order to go forward, step forward, and so on. Description more complex functions can be found in the script comments.
Here's a similar one.

The script is much simpler than the previous one and does not work as efficiently.

P.S. When playing Windows PC games on kinect, do not forget that the developers did not conceive the game for this type of control. Game dynamics, camera views, speed of movement and character reactions, all this is designed for the performance characteristics of a keyboard or mouse. With the help of a kinect, you can simulate any manipulator, but a kinect is not able to change the game itself, make your hands not tremble, your body not sway, and your head always stay straight. All this leads to a decrease in the dynamics of the gameplay, to erroneous clicks and inaccurate shots from fictional weapons =)
It is worth recognizing the fact that in non-adapted games the Kinect cannot help you play better or more efficiently, but it can fill gameplay completely new sensations, or at worst, it will help you burn a couple of extra calories. 😉

A new round of interest in the most unique controller on the planet has caused widespread dominance Virtual reality. Smartphones with 2K or more screens paired with Google Carfboard hit the jackpot, allowing millions of people to literally immerse themselves in an interesting and colorful world for pennies. For complete immersion, it is not enough to create a 3D image and high quality sound, it is also important to digitize human movements as accurately as possible, transferring them into virtual world. Kinect copes with this task with a bang, immersing its owner as deeply as possible into the world of computer 3D reality:

In 2010 the company PrimeSense involved in the project 3D visualization, released its own drivers and framework for Kinect called OpenNI. Soon after, she announced a partnership with ASUS in the production of new Kinect-like device Xtion.

In 2011 Microsoft released a non-commercial software development kit called Kinect SDK. In February 2012, a commercial version accompanying the sensor was released Kinect in the version for Windows.

In November 2013 Apple acquired an Israeli company PrimeSense.

Device Kinect

Kinect equipped RGB-camera, rangefinder, consisting of an infrared laser projector and an infrared CMOS sensor and a specialized microphone that provides localization of the sound source and suppression of ambient noise. Kinect also contains an LED, a three-axis accelerometer, and a small servo to control the tilt of the device.

Infrared CMOS-sensor ( complementary metal-oxide semiconductor, complementary metal-oxide-semiconductor structure) represents integrated circuit, containing a photodetector array that acts as an infrared image sensor. This device is also called IR camera, IR sensor, CMOS image depth, or CMOS-sensor.

RGB-the camera is VGA-camera (640 x 480 pixels) with 8-bit resolution. This doesn't sound impressive, but the point is that the magic happens in the distance sensor, which is completely independent of RGB-cameras.

Two proximity sensor elements, an IR projector and an IR camera, are located on the front panel Kinect. They work together using a chip from PrimeSense to recreate 3D-motion capture. To do this, a method called 3D scanning with structured light. The IR camera also has a resolution VGA(640 x 480 pixels) with 11-bit color depth, providing 2048 sensitivity levels.

From left to right: IR camera, RGB camera, LED and IR projector

Practical operating range Kinect ranges from 1.2 m to 3.5 m. If objects are too close to the sensor, they will not be scanned and will simply appear as black spots. If they are too far away, the scanning accuracy will be too low, making them appear as flat objects. For version Kinect under OS Windows,The camera range is from 40cm to 3m.

Differences between sensor versions Kinect

Kinect can be purchased in three different ways:

• Kinect For Windows - separate device supplied with drivers for the OS Windows(cost about 11 thousand rubles)
• Kinect For Xbox 360 as a separate peripheral device for use with Xbox 360(the price is much lower than the first option, about 5 thousand rubles)
• Kinect For Xbox 360 complete with Xbox 360 ( prices for a set start at about 9 thousand rubles ) . For gamers this is best option purchases.

IN present moment Set-top boxes are starting to appear on sale, complete with the second version of the sensor - Kinect 2. It has higher resolution, and, of course, it is a more advanced device. But it is not yet on sale as a separate device, and the price of new consoles with a second Kinect this is already at least 20 thousand rubles. In addition, I plan to use this sensor in conjunction with, and at the driver level it only supports the first version of the sensor. So for developers - the first Kinect For Xbox 360 is optimal choice to date.

There is one very important point that must be taken into account when purchasing a device.

If you buy Kinect in the version for Windows, or Kinect For Xbox 360 as a standalone peripheral device, then in this case you will have everything you need to connect to your computer.

But perhaps you already have Kinect which you use with your game console Xbox 360 or you plan to buy it along with a set-top box, then in this case, you will need a separate special adapter with a power supply, which is quite difficult to find.

IN latest versions Xbox 360, the console uses a special USB-port that provides sufficient current to power the sensor and is not required for such set-top boxes additional block nutrition. Plus, USB-sensor connector differs from standard USB. Problems additional food and compatibility with a standard USB connector can be solved by using a special adapter.

Kinect model

To find out which Kinect sensor model you have in your hands, you need to look at the base of the stand. I have model 1414.

Possibilities Kinect

So what can we get from Kinect and all these high-tech things hidden inside it? After Kinect properly installed and connected to the computer, we can access the raw data stream and other capabilities provided by the specific middleware. We can use drivers OpenNI And NITE as middleware, and also use this sensor in conjunction with ROS.

RGB image

We can use Kinect as a webcam with a resolution of 640x480 pixels. Get access to RGB-the image can be used using the library OpenNI.

IR image

Kinect has infrared CMOS, and again we can access the infrared CMOS-image with a resolution of 640×480 pixels, using OpenNI.

Depth map

The depth map is the result of operations performed by the chip PS1080 companies PrimeSense above the IR image obtained Kinect with IR CMOS. This VGA-The image has 11-bit resolution. These are 2048 different values, represented graphically by gray levels from white (2048) to black (0).

There is one detail to consider: the scene depth sensor measured is on a logarithmic scale, not linear.

Hand and skeleton tracking

Once the depth map is generated, we can use it directly in our applications or run it through middleware to extract more complex information from the raw depth map.

I recently bought an xbox 360 kinect to play around with the robot building maps of the area. I bought it, I bought it, but the connector disappointed me, it turns out there is a special version for PC with an adapter, and it’s more expensive, but you can’t fool an engineer if it somehow works in x-box, with usb interface, then everything should work on the computer. After reading the forums, everything became clear after a couple of minutes - just cut the cable and apply 12V to the necessary wires, throw the rest to regular USB. It was a pity to cut it, but the wait for an adapter with ALI is long, buying in Rostov is impractical, and I have a Playstation console, and there is still nowhere to connect this miracle. In general, the pinout of the special connector is as follows. It is clear that all these contacts do not contain separate wires, and when I cut the cable, I saw this picture. In total we have 5 wires - 4 standard for USB, one for supplying 12V.
We solder the FW connector of the “female” type, you will find many 12V blocks with such a connector of the “male” type, solder the remaining connectors according to standard USB colors.

We connect the ground (black wire) with both the USB cable that goes to the computer and the 12V connector. As a result, we get the following picture

First I wanted to check the functionality with Windows system, I spent a lot of effort installing the native SDK from the Microsoft website, I was convinced that it seemed to work, but I couldn’t think of anything to do with it all.

Then I rebooted into Linux and found out that there is such an open project - OpenKinect. Following all the instructions from github, I installed it on my computer. My goal was to check the functionality of the device and make sure that I cut the cable correctly. For this I used the RtabMAP utility

The page describes in detail how to install it for all types unix systems. After all this torment, I finally saw the treasured picture of the construction of a 3D model of my room

I checked it later in the office. Works! After construction, you can rotate the map - for the person who sees it for the first time, everything looks fantastic)