Hand Gestures Replicating Robot Arm based on MediaPipe

Muneera Altayeb

Abstract


A robotic arm is any variety of programmable mechanical devices designed to operate items like a human arm and is one of the most beneficial innovations of the 20th century, quickly becoming a cornerstone of many industries. It can perform a variety of tasks and duties that may be time-consuming, difficult, or dangerous to humans. The gesture-based control interface offers many opportunities for more natural, configurable, and easy human-machine interaction. It can expand the capabilities of the GUI and command line interfaces that we use today with the mouse and keyboard. This work proposed changing the concept of remote controls for operating a hand-operated robotic arm to get rid of buttons and joysticks by replacing them with a more intuitive approach to controlling a robotic arm via the hand gestures of the user. The proposed system performs vision-based hand gesture recognition and a robot arm that can replicate the user's hand gestures using image processing. The system detects and recognizes hand gestures using Python and sends a command to the microcontroller which is the Arduino board connected to the robot arm to replicate the recognized gesture. Five servo motors are connected to the Arduino Nano to control the fingers of the robot arm; These servos are related to the robot arm prototype. It is worth noting that this system was able to repeat the user's hand gestures with an accuracy of up to 96%.

Keywords


Image processing; Arduino; Hand Gestures; Robot Arm; MediaPipe; Landmarks

References


Oudah, M., Al-Naji, A., & Chahl, J. (2020). Hand gesture recognition based on computer vision: a review of techniques. journal of Imaging, 6(8), 73.

Ansar, H., Jalal, A., Gochoo, M., & Kim, K. (2021). Hand gesture recognition based on auto-landmark localization and reweighted genetic algorithm for healthcare muscle activities. Sustainability, 13(5), 2961.

Khan, T., & Pathan, A. H. (2015). Hand gesture recognition based on digital image processing using MATLAB. International Journal of Scientific & Engineering Research, 6(9), 338-346.

Chaya, N. A., Bhavana, B. R., Anoogna, S. B., & Hiranmai, M. (2019). Real-time replication of arm movements using surface EMG signals. Procedia Computer Science, 154, 186-193.

Ss, Dheeban & Dhanasekaran Velayutha Rajan, Harish & A, Hari & Marimuthu, Prasanna & N, Senthil Kumar. (2016). Gesture Controlled Robotic Arm.

Afzal, W., Iqbal, S., Tahira, Z., & Qureshi, M. E. (2017). Gesture control robotic arm using flex sensor. Applied and Computational Mathematics, 6(4), 171-176.

Khajone, S. A., & Mohod, D. S. (2015). VM Harne Implementation of a Wireless Gesture Controlled Robotic Arm International Journal of Innovative Research in

Chaitanya, D., Suhasini, Y. A., & Nagarjun, G. (2017). Home Appliances Control Based on Hand Motion Gesture. International Journal of Emerging Technologies in Engineering Research, 5(11), 110-114.

Baranowski, L., Kaczmarek, W., Panasiuk, J., Prusaczyk, P., & Besseghieur, K. (2017). Integration of vision system and robotic arm under ROS. Engineering Mechanics, 114-117.

Rao, M. An integrated system design interface for operating 8-DoF robotic arm.2021 The 21st International Conference on Control, Automation and Systems (ICCAS 2021) Ramada Plaza Hotel, Jeju, Korea, Oct. 12∼15, 2021

Cisnal, Ana, et al. "RobHand: A Hand Exoskeleton With Real-Time EMG-Driven Embedded Control. Quantifying Hand Gesture Recognition Delays for Bilateral Rehabilitation." IEEE Access 9 (2021): 137809-137823.‏

Liu, Kai, et al. "Postural synergy based design of exoskeleton robot replicating human arm reaching movements." Robotics and Autonomous Systems 99 (2018): 84-96.‏

Altayeb, M., & Al-Ghraibah, A. (2023). Arduino Based Real-Time Face Recognition And Tracking System. International Journal, 12(4).

Harris, M., & Agoes, A. S. (2021, November). Applying Hand Gesture Recognition for User Guide Application Using MediaPipe. In 2nd International Seminar of Science and Applied Technology (ISSAT 2021) (pp. 101-108). Atlantis Press.

Joshi, P. (2015). OpenCV with Python by example. Packt Publishing Ltd.

Hughes, J. M. (2016). Arduino: a technical reference: a handbook for technicians, engineers, and makers. " O'Reilly Media, Inc.".

Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C. L., & Grundmann, M. (2020). Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.

Zhang, Y., Pu, X., Wang, X., Guo, H., Liu, K., Yang, Q., & Wang, L. (2022, November). Design concept of sign language recognition translation and gesture recognition control system based on deep learning and machine vision. In Second International Conference on Optics and Communication Technology (ICOCT 2022) (Vol. 12473, pp. 494-498). SPIE.

Liu, Z., Pan, C., & Wang, H. (2022). Continuous Gesture Sequences Recognition Based on Few-Shot Learning. International Journal of Aerospace Engineering, 2022.


Full Text: PDF

Refbacks

  • There are currently no refbacks.


 

Indonesian Journal of Electrical Engineering and Informatics (IJEEI)
ISSN 2089-3272

Creative Commons Licence

This work is licensed under a Creative Commons Attribution 4.0 International License.

web analytics
View IJEEI Stats