Real-Time Kinect Fingers Tracking and Recognition for Servo Gripper Controller

Abstract

The conventional method to control something like machines such as remote controllers or wearable sensors have its limitation and cannot cater the high demands in some scenarios. To overcome this situation, a real-time Kinect finger tracking and recognition is developed to control a servo gripper via a single board microcontroller, Arduino. This paper presents hand and finger tracking method that can determine the number of fingers and also the angle of the finger’s position. The hand gesture is captured by the Kinect and go through a series of image processing and finally, the information will send to the Arduino. The image processing method includes a detection of hand using depth sensor on the Kinect, then finds the finger and calculates the angle of the finger’s position. The convex hull algorithm is used to represent the region of the hand. The fingertips are recognized by calculating the angle of the fingertip and each angle is compared with threshold angle. The orientation and position of the fingers are obtained by finding the middle line of the finger and compares to the vertical line of the middle finger, then calculates the angle. The result shows that the number of fingers appeared in the display can be recognized and the orientation and position of each finger can be determined. A gripper also able to react simultaneously (open and close) based on the detected finger.