Coders working hard at developing new Kinect software at the DeveloperDeveloperDeveloper! coding event held at University of Technology, Sydney on July 2-3
When the Kinect for Xbox 360 was released in late 2010, it created a buzz in the video gaming industry that had not been seen in many years. A webcam-style add-on peripheral for Microsoft’s video game console, the Kinect eliminated the need for a traditional control pad and instead allows the user to interact with video games through body gestures and voice commands. While Kinect was a success, selling an excess of 10 million units as of March, it was still just a video game accessory that was locked to the Xbox 360 platform and its implementation did not go further than video games.
However, that did not stop numerous individuals from connecting the Kinect to their PC and start hacking the device to find ways to find new ways to utilise the motion sensing camera for their own personal coding projects. A watershed moment for the Kinect hacking scene was when Google’s April Fools joke for 2011, where Gmail was said to be controlled with body movements through Kinect, and was made a reality soon afterwards by a hack devised by University of Southern California’s ICT MxR Lab.
Recognising the high interest the Kinect has among developers, as well as the potential implementation the peripheral has beyond games, Microsoft Research recently released the Kinect for Windows Software Development Kit (SDK). A free beta release for non-commercial applications, the SDK eliminates the need to hack the Kinect and instead allows academic researchers and developers to directly create new software for it.
“This SDK is intended to provide deeper access to Kinect technology to a broader set of developers, academic researchers and enthusiasts to enable them to explore new ideas for developing creative applications,” Microsoft Australia academic developer evangelist, Lawrence Crumpton, said.
Microsoft celebrated the release of the SDK in the US with Code Camp, a coding marathon where developers had 24 hours to come up with innovative software for the Kinect using the new SDK. While there were plenty of entertainment-based implementations of the Kinect at Code Camp, what really caught the attention of the public and press were applications of the technology into the healthcare, science and education sectors.
“Because Kinect not only follows gestures but also can recognise speech, it can enable some amazing assistive technologies,” Crumpton said.
“There are people who have designed helmets using Kinect that help the blind navigate, while another developer has created a Kinect powered-shopping cart that is able to follow a disabled person around a shopping centre on its own.”
Possible applications
It is the potential implementations of the Kinect in the medical sector that had piqued the attention of researchers such as Dr Ross Clark of the Department of Physiotherapy at the University of Melbourne. Having already experimented with the Nintendo Wii balance board for diagnostics tests in the past, Clark is excited that advanced technology such as the Kinect is available and open to consumers.
“I think that anything that makes rehabilitation for people easy and enjoyable, which the Kinect does with its games by making it simple for the older population to use, is a great asset,” he said. “If correctly implemented, the Kinect could be fantastic for the rehabilitation of so many people in the future, especially with the aging population in Australia.”
Although the Kinect SDK has just been released, numerous universities and organisations such as the National Stroke Research Institute are already hard at work at developing applications that could benefit the general public. The Kinect’s ability to track body motion, particularly the upper arms, has presented researchers with tantalising new possibilities for study.
“With people who have had experienced strokes or shoulder surgery, you could use the Kinect to look at twin limb imbalances,” Clark explains. “This is a massive thing, as people who have a stroke get weak on one side, and therefore you can use the Kinect to train them to get better. That way they can actually see on the screen how well they are going and strengthen up that side.”
While the Kinect is currently limited to a free beta for research, Microsoft intends to release a commercial version of the SDK in the near future for businesses and individuals. Trevor Perry, Director of GunPoS, a supplier of point of sales (PoS) products, sees some interesting implementations of the Kinect in certain areas of business.
“As a digital signage product, I can see numerous ways to implement it, such as a customer coming up to a large screen and browsing menus by sliding and selection options on the screen using gestures,” he said. “This could have an eco-friendly element to it as well, where a digital sign would only turn on when the Kinect senses someone is nearby,” he said.
While Perry is enthusiastic about Kinect’s application into digital signage, he said development of the platform, both on the software and hardware side, still has a long way to go before it is ready for implementation into PoS.
“As a PoS product, the only potentially implementation that I see for it is the PoS system activating when someone is nearby, but it would need something advanced such as facial recognition to avoid the system turning on whenever someone walks past,” Perry said. “So for the Kinect to have any real application in the PoS industry, a good deal of development work still needs to be done.”
Development for the Kinect is still in its early days, and it’s currently in the hands of eager enthusiasts and early technology adopters that want to test the boundaries of Kinect’s depth sensing, human motion tracking, and voice and object recognition. Local coders had a chance to put the Kinect through the motions at an event held at the University of Technology in July by the DeveloperDeveloperDeveloper! (DDD!) Sydney development community.
“The Kinect is an extraordinary device and having access to an official developer kit opens doors to a whole new world of technology,” Datacom solution architect and DDD! Sydney organiser, Lewis Benge, said. “The developer kit for us opened up avenues for both the hardware and middleware that the hacked libraries previously couldn’t.”
DDD! Sydney is an event that has a history of promoting and support the developer community on the latest trends within the software engineering industry, which made the Kinect a good fit for this year’s event. The organisers were so impressed with the Kinect and the release of the SDK that an entire day of the event was focused entirely on development for the motion sensing peripheral.
“We feel Kinect is part of a larger revolution starting to appear in the form of ‘natural user interface,’” explained Benge. “Kinect is a huge advance in this space, and we are excited to be able to share this paradigm with the local developer community and hopefully drive some inspiration into them around how they can adapt their application design and architecture in the future to adopt this new way of thinking.”
While coders at Code Camp and DDD! are showing new and exciting ways to make use of the Kinect, Microsoft’s Crumpton knows that the implementation of Kinect is only limited by the imagination of the developers.
“When software can see and respond to an individual, ‘the person’ in the context of what you’re doing, you start to erase the boundaries between how people and machines can interact and you create new ways that software can really help people every day, no matter how much or how little they know about computers or technology,” he said.
Why Kinect is exciting
Microsoft’s software developers kit (SDK) is very exciting for one major reason: People have already done mind-blowing things with Kinect even without the SDK. Kinect has been hacked to make serious breakthroughs in technology for marketing, medicine, business, computer science, entertainment and robotics.
- Marketing: A Russian company called ARDoor has created an in-store “mirror” called the AR Door Kinect Fitting Room that superimposes clothes on shoppers. They can quickly cycle through different outfits to see how they’ll look.
- Medicine: An academic researcher at the Technical University of Munich created something he calls the “Magic Mirror”. Designed for teaching anatomy, the system creates the illusion of a mirror that shows the user’s insides through a “hole” in the body.
- Business: A graduate student at the University of North Carolina at Chapel Hill created a 3D video conferencing system using four Kinect devices. The system uses head tracking to simulate reality. For example, with a conventional videoconferencing system, the other user is just flat on the screen. If you’re looking at the user straight on, the view doesn’t change when you move to the side. But with the Kinect-based 3D system, if you move to one side, you see the side of his or her face, rather than just the front.
- Computer Science: MIT researchers built a working version of the user interface from the Steven Spielberg movie Minority Report, which involves computer control via very fine hand and finger gestures. Researchers at Germany’s Fraunhofer Institute created an elegant system for controlling a Windows PC with in-the-air hand gestures.
- Entertainment: One “Kinect hacker” invented a motion-controlled sound-and-laser system that enables an on-stage performer to control an audio track and laser lights just by moving around.
- Accessibility: A Portugese scientist created a shopping cart system called the wi-GO for people in wheelchairs that follows the user around automatically. A team of researchers at Sweden’s Lulea University of Technology invented an autonomous wheelchair for the blind. The wheelchair contains a map of its surroundings, which it uses to navigate. But it can also avoid obstacles and people. French researchers are working on a Kinect-based system for reading sign language. The user signs, and the system translates it into written or spoken language.
- Robotics: UC Berkeley scientists created an autonomous flying robot using Kinect. Researchers at Technical University Munich used Kinect to enable a robot to shop and cook. Engineers at the University of Pennsylvania even taught a robot to read signs and posters using Kinect.
