Hacked Kinect Understands Sign Language

French researchers have done what many thought impossible: created a truly useful hack for the Kinect. Using the body tracking capabilities of the Kinect, coupled with some clever coding, they have developed software to interpret sign language. Interestingly, this fulfills the original patent claim of the Kinect, and promises made at its introduction by Microsoft.

Key to the team’s work was using the Fast Artificial Neural Network, or FANN, to allow the computer to understand users’ signs. From the  YouTube comments made by the team:

For simples moves , like a punch or a kick, it easy to recognize : high vélocity of the hand or the foot. But for more complex moves (like the ones in this video), the problem is mathematicaly harder…

Neural network is a way to bypass the problem[.] It builds a mathematical model of the moves using move samples.

The system can only recognize two words at the moment — “hello” and “sorry” — but the team says that with all the software in place adding more words is a relatively easy task. While the XBox has supported text-based inputs for some time, sign language recognition could provide a fast and natural way for signers to communicate. It could also be used as a valuable teaching tool for those unfamiliar with the language.

Read on after the break to get a look at the Kinect hack in action.

(Image and story Übergizmo via Kotaku)