Conversational Assistant with Large Language Model Agent for Natural Interaction in Home Automation Environments

Authors

Ricardo Yauri
Universidad Nacional Tecnológica de Lima Sur
Rafael Espino
Universidad Tecnológica del Perú
DOI: https://doi.org/10.52549/ijeei.v13i4.7028
Download PDF
Published: 16-12-2025 • Issue: Vol. 13 No. 4: December 2025 • Section: Regular paper

Abstract

Digital transformation has driven the development of smart home automation systems for healthcare and remote assistance, which can integrate large language models, foundational models, and Retrieval-Augmented Generation techniques to design conversational assistants accessible to people with reduced mobility. In these cases, it is necessary to overcome usability barriers to control home devices because traditional home automation interfaces are not adapted to their specific needs. Previous research has identified systems such as voice control, gestures, and conversational agents based on generative artificial intelligence to facilitate assisted interaction in the home through home automation systems. This article describes a conversational agent, aimed at people with reduced mobility, to facilitate natural interaction with home automation environments, using technologies such as OpenAI, Langchain, and LlamaIndex. The results demonstrate the successful deployment of a web-based telecare platform based on conversational agents capable of retrieving technical documents through embedding, generating SQL queries, and MQTT topics for integration into real-world monitoring and assistance environments for people with reduced mobility.
Keywords: OpenAI; Foundational models; RAG; Distributed processing; LangChain

References

G. Rathee, S. Garg, G. Kaddoum, S. M. Alzanin, A. Gumaei, and M. M. Hassan, “Accessibility and ensured quality of life for disabled people using trusted edge computing,†Alexandria Eng. J., vol. 82, no. 1, pp. 35–42, Nov. 2023, doi: 10.1016/j.aej.2023.09.046.

A. Mahroo, L. Greci, M. Mondellini, and M. Sacco, “Assessment of a mixed reality smart home controller: HoloHome pilot study on healthy adults,†Virtual Real., vol. 27, no. 3, pp. 2673–2690, Sep. 2023, doi: 10.1007/s10055-023-00834-8.

S. Lindsay, K. Fuentes, S. Ragunathan, Y. Li, and T. Ross, “Accessible independent housing for people with disabilities: A scoping review of promising practices, policies and interventions,†PLoS One, vol. 19, no. 1, p. e0291228, Jan. 2024, doi: 10.1371/journal.pone.0291228.

M. Periša, P. Teskera, I. Cvitić, and I. Grgurević, “Empowering People with Disabilities in Smart Homes Using Predictive Informing,†Sensors, vol. 25, no. 1, p. 284, Jan. 2025, doi: 10.3390/s25010284.

E. Mori, Y. Qiu, H. Kataoka, and Y. Aoki, “A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans,†Sensors, vol. 25, no. 2, p. 477, Jan. 2025, doi: 10.3390/s25020477.

A. V. F. Porto and M. E. S. Furtado, “Framework to Specify Dialogues for Natural Interaction with Conversational Assistants Applied in Prompt Engineering,†in Lecture Notes in Networks and Systems, vol. 1068 LNNS, Springer Science and Business Media Deutschland GmbH, 2024, pp. 231–253. doi: 10.1007/978-3-031-66336-9_17.

H. Du et al., “Towards Proactive Interactions for In-Vehicle Conversational Assistants Utilizing Large Language Models,†in Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence, Aug. 2024, pp. 7850–7858. doi: 10.24963/ijcai.2024/869.

B. Saha and U. Saha, “Enhancing International Graduate Student Experience through AI-Driven Support Systems: A LLM and RAG-Based Approach,†in 2024 International Conference on Data Science and Its Applications (ICoDSA), Jul. 2024, pp. 300–304. doi: 10.1109/ICoDSA62899.2024.10651944.

T. Li, “AI-Driven Multi-Modal Interactive Learning Environment Using Deep Learning and Chain-of-Thought Reasoning,†Int. J. Pattern Recognit. Artif. Intell., vol. 39, no. 03n04, p. 10, Mar. 2025, doi: 10.1142/S0218001425520019.

S. K. Oswal and H. K. Oswal, “Examining the Accessibility of Generative AI Website Builder Tools for Blind and Low Vision Users: 21 Best Practices for Designers and Developers,†in 2024 IEEE International Professional Communication Conference (ProComm), Jul. 2024, pp. 121–128. doi: 10.1109/ProComm61427.2024.00030.

W. Wang, K. Wang, and H. Du, “Design and optimization of human-machine interaction interface for the intelligent Internet of Things based on deep learning and spatial computing,†Egypt. Informatics J., vol. 30, no. 1, p. 100685, Jun. 2025, doi: 10.1016/j.eij.2025.100685.

M. M. Linda, K. M, K. M, and U. C, “Gesture Control System Using Computer Vision Techniques,†in 2024 5th International Conference on Electronics and Sustainable Communication Systems (ICESC), Aug. 2024, pp. 1402–1406. doi: 10.1109/ICESC60852.2024.10689942.

A. Anand, A. Pandey, S. Mehndiratta, H. Goyal, P. Kaushik, and R. Rathore, “Smart AI basedVolume Control System: Gesture Recognition with OpenCV & Mediapipe Integration,†in 2023 International Conference on Smart Devices (ICSD), May 2024, pp. 1–6. doi: 10.1109/ICSD60021.2024.10751362.

A. Taqwa, Indra Griha Tofik Isa, and Indri Ariyanti, “Designing A WSNs-based Smart Home Monitoring System through Deep Reinforcement Learning,†J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 7, no. 5, pp. 1224–1232, Oct. 2023, doi: 10.29207/resti.v7i5.5143.

R. Fitriyan and S. Syafii, “Development design of an IoT-based smart home monitoring system with security features,†Indones. J. Electr. Eng. Comput. Sci., vol. 34, no. 2, p. 788, May 2024, doi: 10.11591/ijeecs.v34.i2.pp788-794.

S. Albert, M. Hamann, and E. Stokoe, “Conversational User Interfaces in Smart Homecare Interactions: A Conversation Analytic Case Study,†in Proceedings of the 5th International Conference on Conversational User Interfaces, Jul. 2023, pp. 1–12. doi: 10.1145/3571884.3597140.

D. Suhas, O. Rana, S. Lannon, and C. Perera, “Talking Buildings: Interactive Human-Building Smart-Bot for Smart Buildings,†in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15436 LNCS, Springer Science and Business Media Deutschland GmbH, 2025, pp. 399–415. doi: 10.1007/978-981-96-0579-8_28.

W. Guo, S. Yamagishi, and L. Jing, “Human Activity Recognition via Wi-Fi and Inertial Sensors With Machine Learning,†IEEE Access, vol. 12, no. 1, pp. 18821–18836, 2024, doi: 10.1109/ACCESS.2024.3360490.

C. Panagiotou, E. Faliagka, C. P. Antonopoulos, and N. Voros, “Multidisciplinary ML Techniques on Gesture Recognition for People with Disabilities in a Smart Home Environment,†AI, vol. 6, no. 1, p. 17, Jan. 2025, doi: 10.3390/ai6010017.

Y. Chen, H. Zhang, and S. Zhong, “Design and implementation of smart home system based on IoT,†Results Eng., vol. 24, no. 1, p. 103410, Dec. 2024, doi: 10.1016/j.rineng.2024.103410.

N. Nasaruddin, M. A. P. Pratama Tresma, M. Khamdan Muchamad, and Z. Fuadi, “Voice Frequency-Based Gender Classification Using Convolutional Neural Network for Smart Home,†IEEE Access, vol. 12, no. 1, pp. 104190–104203, 2024, doi: 10.1109/ACCESS.2024.3434547.

M. Javeed et al., “Intelligent ADL Recognition via IoT-Based Multimodal Deep Learning Framework,†Sensors, vol. 23, no. 18, p. 7927, Sep. 2023, doi: 10.3390/s23187927.

J. Kim and M. Sohn, “Graph Representation Learning-Based Early Depression Detection Framework in Smart Home Environments,†Sensors, vol. 22, no. 4, p. 1545, Feb. 2022, doi: 10.3390/s22041545.

Z. Severiche-Maury et al., “LSTM Networks for Home Energy Efficiency,†Designs, vol. 8, no. 4, p. 78, Aug. 2024, doi: 10.3390/designs8040078.

M. Nutakki and S. Mandava, “Resilient data-driven non-intrusive load monitoring for efficient energy management using machine learning techniques,†EURASIP J. Adv. Signal Process., vol. 2024, no. 1, p. 62, May 2024, doi: 10.1186/s13634-024-01157-9.

E. King, H. Yu, S. Lee, and C. Julien, “Sasha,†Proc. ACM Interactive, Mobile, Wearable Ubiquitous Technol., vol. 8, no. 1, pp. 1–38, Mar. 2024, doi: 10.1145/3643505.

T. Calò and L. De Russis, “Enhancing smart home interaction through multimodal command disambiguation,†Pers. Ubiquitous Comput., vol. 28, no. 6, pp. 985–1000, Dec. 2024, doi: 10.1007/S00779-024-01827-3/TABLES/2.

A. A. Ali, M. Mashhour, A. S. Salama, R. Shoitan, and H. Shaban, “Development of an Intelligent Personal Assistant System Based on IoT for People with Disabilities,†Sustainability, vol. 15, no. 6, p. 5166, Mar. 2023, doi: 10.3390/su15065166.

A. A. H. Al-Shateri, R. A. Rashid, A. Aburaya, and N. A. Muhammad, “Luna: A Benchmark Project in the Convergence of Artificial Intelligence and Internet of Things for Home Automation,†in 2024 IEEE International Conference on Advanced Telecommunication and Networking Technologies (ATNT), Sep. 2024, pp. 1–4. doi: 10.1109/ATNT61688.2024.10719226.

I. A. Pap and S. Oniga, “eHealth Assistant AI Chatbot Using a Large Language Model to Provide Personalized Answers through Secure Decentralized Communication,†Sensors 2024, Vol. 24, Page 6140, vol. 24, no. 18, p. 6140, Sep. 2024, doi: 10.3390/S24186140.

J. Lee et al., “Development of AI-generated medical responses using the ChatGPT for cancer patients,†Comput. Methods Programs Biomed., vol. 254, no. 1, p. 108302, Sep. 2024, doi: 10.1016/j.cmpb.2024.108302.

Author Biographies

Ricardo Yauri
Master of Science in Electronic Engineering with a Mention in Biomedical. He is associate professor at the Universidad Nacional Mayor de San Marcos (UNMSM) and Ph.D. student in Systems Engineering. He is professor at Universidad Tecnológica del Perú and Universidad Peruana del Norte. He has participated as a teacher in courses oriented to the Internet of things and applications in home automation and the Cisco academy for IoT. He was a researcher at INICTEL-UNI in the Embedded Systems and Internet of Things research group. He developed research projects on the implementation of low consumption IoT devices that involve inference techniques, machine learning algorithms. He can be contacted at email: ryauri@untels.edu.pe, ryaurir@unmsm.edu.pe
Rafael Espino
He is a graduate of the Faculty of Electronics of the National Universidad Nacional de Ingeniería, Perú. He is Specialist in the development of electronic systems with solid knowledge in photovoltaic systems and experience in implementing embedded firmware in electronic systems. He is currently a professor at the Universidad Tecnológica del Perú. He was a researcher at INICTEL-UNI in the Embedded Systems research group. With aptitudes for teaching, research, and project management. He can be contacted at C22165@utp.edu.pe

How to Cite

[1]
R. Yauri and R. Espino, “Conversational Assistant with Large Language Model Agent for Natural Interaction in Home Automation Environments”, IJEEI, vol. 13, no. 4, pp. 996–1007, Dec. 2025, doi: 10.52549/ijeei.v13i4.7028.