Research and Development of IMU Sensors-based Approach for Sign Language Gesture Recognition

Authors

  • A. Abdullah Faculty of Electrical Engineering, Universiti Teknologi Malaysia,81310 UTM Johor Bahru, Johor, Malaysia.
  • N. A. Abdul-Kadir Faculty of Biosciences and Medical Engineering, Universiti Teknologi Malaysia,81310 UTM Johor Bahru, Johor, Malaysia
  • F. K. Che Harun Faculty of Electrical Engineering, Universiti Teknologi Malaysia,81310 UTM Johor Bahru, Johor, Malaysia. Faculty of Biosciences and Medical Engineering, Universiti Teknologi Malaysia,81310 UTM Johor Bahru, Johor, Malaysia

Keywords:

IMU Sensor-based Approach, Sensor Fusion, Sign Language Recognition,

Abstract

This paper discusses a few Inertial Measurement Unit (IMU) sensor-based approaches for sign language gesture recognition. Generally, there are three main research areas for the IMU sensor-based approach which consist of the device structure, sensors fusion algorithm and calibration method, and finally gesture recognition and classification method. The device structure includes the number and placement of the sensors to cover the degrees of freedom. Sensors fusion algorithms, such as complementary filter, Kalman filter, and EKF are implemented to combine a variety of sensors used for data acquisition. Several gesture classification and recognition methods are also reviewed in this paper. Some of the limitations related to sensor-based technique such as device structure and classification technique are discussed as a research gap for future references.

References

M. Mohandes, M. Deriche and J. Liu, "Image-Based and SensorBased Approaches to Arabic Sign Language Recognition," in IEEE Trans. Human-Mach. Syst., vol. 44, no. 4, pp. 551-557, Aug. 2014.

J. Wu, L. Sun and R. Jafari, "A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors," in IEEE J. Biomed. Health Inform., vol. 20, no. 5, pp. 1281- 1290, Sept. 2016.

J. Gałka, M. Mąsior, M. Zaborski and K. Barczewska, "Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition," in IEEE Sensors J., vol. 16, no. 16, pp. 6310-6316, Aug.15, 2016.

Li Lei and Que Dashun, "Design of data-glove and Chinese sign language recognition system based on ARM9," IEEE Int. Conf. Electron. Meas. Instrum. (ICEMI), Qingdao, 2015, pp. 1130-1134.

T. H. S. Li, M. C. Kao and P. H. Kuo, "Recognition System for Home-Service-Related Sign Language Using Entropy-Based K - Means Algorithm and ABC-Based HMM," in IEEE Trans. Syst., Man, Cybern., Syst., vol. 46, no. 1, pp. 150-162, Jan. 2016.

A. Srisuphab and P. Silapachote, "Artificial neural networks for gesture classification with inertial motion sensing armbands," IEEE Region 10 Conf. (TENCON), Singapore, 2016, pp. 1-5.

M. Mohandes, S. Aliyu and M. Deriche, "Prototype Arabic Sign language recognition using multi-sensor data fusion of two leap motion controllers," IEEE Int. Multi-Conf. Syst. Signals Devices (SSD15), Mahdia, 2015, pp. 1-6.

Y. Park, J. Lee and J. Bae, "Development of a Wearable Sensing Glove for Measuring the Motion of Fingers Using Linear Potentiometers and Flexible Wires," in IEEE Trans. Ind. Informat., vol. 11, no. 1, pp. 198-206, Feb. 2015.

B. S. Lin, I. J. Lee, P. C. Hsiao, S. Y. Yang and W. Chou, "Data Glove Embedded with 6-DOF Inertial Sensors for Hand Rehabilitation," Int. Conf. Intell. Inf. Hiding Multimedia Signal Process., Kitakyushu, 2014, pp. 25-28.

B. Fang, F. Sun, H. Liu and D. Guo, "A novel data glove for fingers motion capture using inertial and magnetic measurement units," IEEE Int. Conf. Robot. Biomimetics (ROBIO), Qingdao, 2016, pp. 2099- 2104.

Z.F. Syed, P. Aggarwal, C. Goodall, X. Niu, N. El-Sheimy, “A new multi-position calibration method for MEMS inertial navigation systems,” in Meas. Sci. Technol. 18 (2007), 1897-1907.

J. Conroy, W. Nothwang and G. Gremillion, "Continuous time rate gyro calibration and motion capture system misalignment estimation using a nonlinear observer," IEEE Int. Conf. Multisensor Fusion Integr. for Intell. Syst. (MFI), Baden-Baden, 2016, pp. 487-492.

R. Mahony, T. Hamel and J. M. Pflimlin, "Nonlinear Complementary Filters on the Special Orthogonal Group," in IEEE Trans. Autom. Control, vol. 53, no. 5, pp. 1203-1218, June 2008.

S. Sabatelli, M. Galgani, L. Fanucci and A. Rocchi, "A double stage Kalman filter for sensor fusion and orientation tracking in 9D IMU," IEEE Sensors Appl. Symp. Proc., Brescia, 2012, pp. 1-5.

Rong Zhu and Zhaoying Zhou, "A real-time articulated human motion tracking using tri-axis inertial/magnetic sensors package," in IEEE Trans. Neural Syst. Rehabil. Eng., vol. 12, no. 2, pp. 295-302, June 2004.

Ghani, S. A., Rashid, M. I. M., Sulaiman, M. H., Noor, M. K. M., Subari, N., & Ramli, N. L., “ Self balancing unicycle controlled by using Arduino,” in ARPN J. Eng. Appl. Sci, 11(7), 4239–4244, 2016.

R. Faragher, "Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation [Lecture Notes]," in IEEE Signal Process. Mag., vol. 29, no. 5, pp. 128-132, Sept. 2012.

A. Pascoal, I. Kaminer and P. Oliveira, "Navigation system design using time-varying complementary filters," in IEEE Trans. Aerosp. Electron. Syst., vol. 36, no. 4, pp. 1099-1114, Oct 2000.

J. Bordoy, C. Schindelhauer, R. Zhang, F. Höflinger and L. M. Reindl, "Robust Extended Kalman filter for NLOS mitigation and sensor data fusion," IEEE Int. Symp. Inertial Sensors Syst. (INERTIAL), Kauai, HI, 2017, pp. 117-120.

A. M. Sabatini, "Quaternion-based extended Kalman filter for determining orientation by inertial and magnetic sensing," in IEEE Trans. Biomed. Eng., vol. 53, no. 7, pp. 1346-1356, July 2006.

W. Chou, B. Fang, L. Ding, X. Ma and X. Guo, "Two-step optimal filter design for the low-cost attitude and heading reference systems," in IET Sci., Meas. Technol., vol. 7, no. 4, pp. 240-248, July 2013.

Attal, F., Mohammed, S., Dedabrishvili, M., Chamroukhi, F., Oukhellou, L., & Amirat, Y., “Physical Human Activity Recognition Using Wearable Sensors,” in Sensors (Basel, Switzerland), 15(12), 31314–38, 2015.

I. Guyon and A. Elisseeff, “An introduction to variable and feature selection,” in J. Mach. Learn. Res., vol. 3, pp. 1157–1182, 2003.

X. H. Zhang, J. J. Wang, X. Wang and X. L. Ma, "Improvement of Dynamic Hand Gesture Recognition Based on HMM Algorithm," Int. Conf. Inf. Syst. Artificial Intell. (ISAI), Hong Kong, 2016, pp. 401-406.

M. Rossi, S. Benatti, E. Farella and L. Benini, "Hybrid EMG classifier based on HMM and SVM for hand gesture recognition in prosthetics," IEEE Int. Conf. Ind. Technol. (ICIT), Seville, 2015, pp. 1700-1705.

W. C. Stokoe, Jr., “Sign language structure: An outline of the visual communication systems of the American deaf,” in J. Deaf Stud. Deaf Edu., vol. 10, no. 1, pp. 3–37, 1960.

A. A. Ahmed and S. Aly, "Appearance-based Arabic Sign Language recognition using Hidden Markov Models," Int. Conf. Eng. Technol. (ICET), Cairo, 2014, pp. 1-6.

S. S. Shinde, R. M. Autee and V. K. Bhosale, "Real time two way communication approach for hearing impaired and dumb person based on image processing," IEEE Int. Conf. Comput. Intell. Comput. Res. (ICCIC), Chennai, 2016, pp. 1-5.

S. Bessa Carneiro, E. D. F. d. M. Santos, T. M. d. A. Barbosa, J. O. Ferreira, S. G. S. Alcalá and A. F. Da Rocha, "Static gestures recognition for Brazilian Sign Language with kinect sensor," IEEE SENSORS, Orlando, FL, 2016, pp. 1-3.

J. He, Z. Liu and J. Zhang, "Chinese sign language recognition based on trajectory and hand shape features," Vis. Commun. Image Process. (VCIP), Chengdu, 2016, pp. 1-4.

F. M. d. P. Neto, L. F. Cambuim, R. M. Macieira, T. B. Ludermir, C. Zanchettin and E. N. Barros, "Extreme Learning Machine for Real Time Recognition of Brazilian Sign Language," IEEE Int. Conf. Syst., Man, Cybern., Kowloon, 2015, pp. 1464-1469.

S. M. K. Hasan and M. Ahmad, "A new approach of sign language recognition system for bilingual users," Int. Conf. Elect. Electron. Eng. (ICEEE), Rajshahi, 2015, pp. 33-36.

F. Guesmi, T. Bouchrika, O. Jemai, M. Zaied and C. Ben Amar, "Arabic sign language recognition system based on wavelet networks," IEEE Int. Conf. Syst., Man, Cybern. (SMC), Budapest, 2016, pp. 3561-3566.

M. G. Vintimilla, D. Alulema, D. Morocho, M. Proano, F. Encalada and E. Granizo, "Development and implementation of an application that translates the alphabet and the numbers from 1 to 10 from sign language to text to help hearing impaired by Android mobile devices," IEEE Int. Conf. Autom. (ICA-ACCA), Curico, 2016, pp. 1-5.

T. N. T. Huong, T. V. Huu, T. L. Xuan and S. V. Van, "Static hand gesture recognition for vietnamese sign language (VSL) using principle components analysis," Int. Conf. Commun., Manag. and Telecommun. (ComManTel), DaNang, 2015, pp. 138-141.

N. H. Ariffin, N. Arsad and B. Bais, "Low cost MEMS gyroscope and accelerometer implementation without Kalman Filter for angle estimation," Int. Conf. Advances Elect., Electron. Syst. Eng. (ICAEES), Putrajaya, 2016, pp. 77-82.

M. Zhang, A. A. Sawchuk, “A feature selection-based framework for human activity recognition using wearable multimodal sensors,” Proc. Int. Conf. Body Area Netw. (ICST), Beijing, China, 7–8 November 2011; pp. 92–98.

B. Fish, A. Khan, N.H. Chehade, C. Chien, G. Pottie, “Feature selection based on mutual information for human activity recognition,” Proc. IEEE Int. Conf. Acoust., Speech and Signal Proc. (ICASSP), Kyoto, Japan, 25–30 March 2012; pp. 1729–1732.

Robotshop.com, ‘Leap motion 3D motion controller’, 2017. [Online]. Available: http://www.robotshop.com/en/leap-motion-3d-motion-controller.html. [Accessed: 25-July-2017].

Robotshop.com, ‘Myo gesture control armband’, 2017. [Online]. Available: http://www.robotshop.com/en/myo-gesture-control-armband-black.html. [Accessed: 25-July-2017].

Robotshop.com, ‘Sparkfun 9DOF sensor stick’, 2017. [Online]. Available: http://www.robotshop.com/en/sparkfun-9dof-sensor-stick.html. [Accessed: 25-July-2017].

Downloads

Published

2017-12-04

How to Cite

Abdullah, A., Abdul-Kadir, N. A., & Che Harun, F. K. (2017). Research and Development of IMU Sensors-based Approach for Sign Language Gesture Recognition. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 9(3-9), 33–39. Retrieved from https://jtec.utem.edu.my/jtec/article/view/3122

Issue

Vol. 9 No. 3-9: Biomedical Electronics and Informatics

Section

Articles

License

TRANSFER OF COPYRIGHT AGREEMENT

The manuscript is herewith submitted for publication in the Journal of Telecommunication, Electronic and Computer Engineering (JTEC). It has not been published before, and it is not under consideration for publication in any other journals. It contains no material that is scandalous, obscene, libelous or otherwise contrary to law. When the manuscript is accepted for publication, I, as the author, hereby agree to transfer to JTEC, all rights including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the author(s) specifically retain(s):

  1. All proprietary right other than copyright, such as patent rights
  2. The right to make further copies of all or part of the published article for my use in classroom teaching
  3. The right to reuse all or part of this manuscript in a compilation of my own works or in a textbook of which I am the author; and
  4. The right to make copies of the published work for internal distribution within the institution that employs me

I agree that copies made under these circumstances will continue to carry the copyright notice that appears in the original published work. I agree to inform my co-authors, if any, of the above terms. I certify that I have obtained written permission for the use of text, tables, and/or illustrations from any copyrighted source(s), and I agree to supply such written permission(s) to JTEC upon request.