Enhancing Arabic Diacritization Using BERT with BiLSTM and BiGRU Heads

Authors

Ahmed Abdelhamed Elgayar
Faculty of Graduate Studies for Statistical Research (FGSSR), Cairo University, Giza, Egypt
Abdul-Hadi Nabi Ahmed
Faculty of Graduate Studies for Statistical Research (FGSSR), Cairo University, Giza, Egypt
Basma Ezzat El-Demerdash
Faculty of Graduate Studies for Statistical Research (FGSSR), Cairo University, Giza, Egypt
Ahmed E. Soliman
Faculty of Graduate Studies for Statistical Research (FGSSR), Cairo University, Giza, Egypt
DOI: https://doi.org/10.52549/ijeei.v14i2.7033
Download PDF
Published: 22-06-2026 • Issue: Vol. 14 No. 2: June 2026 • Section: Regular paper

Abstract

Arabic diacritization is the task of adding diacritical marks to Arabic text to aid correct articulation, resolve lexical ambiguity, and support downstream natural language processing tasks. This study proposes a hybrid deep learning architecture that augments a pre-trained transformer encoder (AraBERTv02) with a sequential prediction head built on either a Bidirectional Long Short-Term Memory (BiLSTM) or a Bidirectional Gated Recurrent Unit (BiGRU) network, replacing the conventional linear classification layer. This design introduces an explicit sequential inductive bias at the output level, enabling the model to capture local label dependencies that are not modeled by a standard linear head. The models were trained and evaluated on the Tashkeela Processed (TP) benchmark corpus of fully diacritized Modern Standard Arabic. The BiLSTM based model achieves a Diacritic Error Rate (DER) of 1.82% and a Word Error Rate (WER) of 2.58% while the BiGRU variant has a DER of 1.84% and a WER of 2.62% both outperforming the prior state-of-the-art on the same benchmark (WER: 3.34%). Error analysis reveals a systematic confusion of short vowels, especially in verb-initial contexts where fatha and damma alternate to mark active and passive voice, illustrating the inherent morphological complexity of Arabic diacritization. The proposed architecture is also shown to be suitable for real-world deployment with an inference time of 9.22 ms and 8.74 ms per sentence for BiLSTM and BiGRU models respectively. Thus, both variants are compatible with near real-time applications such as Arabic text-to-speech systems and educational tools.
Keywords: Arabic Diacritization; Transformer; Long Short-Term Memory; Gated Recurrent Unit; Natural Language Processing

References

Y. A. El-Imam, “Phonetization of Arabic: rules and algorithms,†Comput. Speech Lang., vol. 18, no. 4, pp. 339–373, 2004. https://doi.org/10.1016/S0885-2308(03)00035-4

K. Shaalan and Khaled, "Rule-based Approach in Arabic Natural Language Processing," International Journal on Information and Communication Technologies (IJICT), vol. 3, pp. 11–, Jun. 2010. https://doi.org/10.1142/9789813229396_0003

I. Zitouni and R. Sarikaya, "Arabic diacritic restoration approach based on maximum entropy models," Computer Speech & Language, vol. 23, no. 3, pp. 257–276, 2009. https://doi.org/10.1016/j.csl.2008.06.001

H. El Rifai, L. Al Qadi, and A. Elnagar, “Arabic text classification: the need for multi-labeling systems,†Neural Comput. Appl., vol. 34, no. 2, pp. 1135–1159, 2022. https://doi.org/10.1007/s00521-021-06390-z

A. Elnagar, S. M. Yagi, A. B. Nassif, I. Shahin, and S. A. Salloum, “Systematic literature review of dialectal Arabic: identification and detection,†IEEE Access, vol. 9, pp. 31010–31042, 2021. https://doi.org/10.1109/ACCESS.2021.3059504

A. B. Nassif, A. M. Darya, and A. Elnagar, “Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysis,†ACM Trans. Asian Low-Resour. Lang. Inf. Process., vol. 21, no. 1, pp. 1–25, Nov. 2021. https://doi.org/10.48550/arXiv.2112.00534

A. B. Nassif, A. Elnagar, O. Elgendy, and Y. Afadar, “Arabic fake news detection based on deep contextualized embedding models,†Neural Comput. Appl., vol. 34, no. 18, pp. 16019–16032, May 2022. https://doi.org/10.48550/arXiv.2205.03114

M. Lataifeh, A. Elnagar, I. Shahin, and A. B. Nassif, “Arabic audio clips: Identification and discrimination of authentic Cantillations from imitations,†Neurocomputing, vol. 418, pp. 162–177, 2020. https://doi.org/10.1016/j.neucom.2020.07.099

A. B. Nassif, I. Shahin, A. Elnagar, D. Velayudhan, A. Alhudhaif, and K. Polat, “Emotional speaker identification using a novel capsule nets model,†Expert Syst. Appl., vol. 193, p. 116469, May 2022. https://doi.org/10.48550/arXiv.2201.02994

I. Shahin, A. B. Nassif, N. Nemmour, A. Elnagar, A. Alhudhaif, and K. Polat, "Novel hybrid DNN approaches for speaker verification in emotional and stressful talking environments," Neural Computing and Applications, vol. 33, no. 23, pp. 16033–16055, Jun. 2021. https://doi.org/10.48550/arXiv.2112.13353

A. Al Sallab, M. Rashwan, H. M. Raafat, and A. Rafea, "Automatic Arabic diacritics restoration based on deep nets," in Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), Doha, Qatar, pp. 65–72, 2014. https://doi.org/10.3115/v1/W14-3608

M. Maamouri, A. Bies, T. Buckwalter, and W. Mekki, "The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus," 2004. https://api.semanticscholar.org/CorpusID:16205731

M. Diab, N. Habash, O. Rambow, and R. Roth, “LDC Arabic Treebanks and associated corpora: Data divisions manual,†arXiv preprint arXiv:1309.5652, 2013. https://doi.org/10.48550/arXiv.1309.5652

M. Rashwan, M. Elbadrashiny, M. Attia, and S. Abdou, "A hybrid system for automatic Arabic diacritization," 2009.

M. A. A. Rashwan, M. A. S. A. A. Al-Badrashiny, M. Attia, S. M. Abdou, and A. Rafea, “A stochastic Arabic diacritizer based on a hybrid of factorized and unfactorized textual features,†IEEE Trans. Audio Speech Lang. Process., vol. 19, no. 1, pp. 166–175, Jan. 2011. https://doi.org/10.1109/TASL.2010.2045240

G. A. Abandah, A. Graves, B. Al-Shagoor, A. Arabiyat, F. Jamour, and M. Al-Taee, “Automatic diacritization of Arabic text using recurrent neural networks,†Int. J. Doc. Anal. Recognit., vol. 18, no. 2, pp. 183–197, Jun. 2015. https://doi.org/10.1007/s10032-015-0242-2

R. Alnefaie and A. M. Azmi, “Automatic minimal diacritization of Arabic texts,†Procedia Computer Science, vol. 117, pp. 169–174, 2017. https://doi.org/10.1016/j.procs.2017.10.106

K. Darwish, H. Mubarak, and A. Abdelali, "Arabic Diacritization: Stats, Rules, and Hacks," in Proceedings of the Third Arabic Natural Language Processing Workshop, Valencia, Spain, Apr. 2017, pp. 9–17. https://doi.org/10.18653/v1/W17-1302

H. Mubarak, A. Abdelali, H. Sajjad, Y. Samih, and K. Darwish, "Highly Effective Arabic Diacritization using Sequence to Sequence Modeling," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 2390–2395, 2019. https://doi.org/10.18653/v1/N19-1248

A. Fadel, I. Tuffaha, B. Al-Jawarneh, and M. Al-Ayyoub, "Arabic Text Diacritization Using Deep Neural Networks," arXiv preprint arXiv:1905.01965, 2019. https://doi.org/10.48550/arXiv.1905.01965

T. Zerrouki and A. Balla, "Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems," Data in Brief, vol. 11, pp. 147–151, 2017. https://doi.org/10.1016/j.dib.2017.01.011

Z. Barqawi, "Shakkala, Arabic text vocalization," 2017. [Online]. Available: https://github.com/Barqawiz/Shakkala

A. Fadel, I. Tuffaha, B. Al-Jawarneh, and M. Al-Ayyoub, “Neural Arabic text diacritization: state of the art results and a novel approach for machine translation,†in Proc. 6th Workshop Asian Translation, Association for Computational Linguistics, 2019, pp. 215–225. https://doi.org/10.48550/arXiv.1911.03531

K. Darwish, A. Abdelali, H. Mubarak, and M. Eldesouki, “Arabic diacritic recovery using a feature-rich biLSTM model,†arXiv preprint arXiv:2002.01207, 2020. https://doi.org/10.48550/arXiv.2002.01207

B. AlKhamissi, M. N. ElNokrashy, and M. Gabr, "Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization," arXiv preprint arXiv:2011.00538, 2020. https://doi.org/10.48550/arXiv.2011.00538

H. Abbad and S. Xiong, "Multi-components System for Automatic Arabic Diacritization," in Advances in Information Retrieval, Cham: Springer International Publishing, pp. 341–355, 2020. https://doi.org/10.1007/978-3-030-45439-5_23

H. Abbad and S. Xiong, “Simple extensible deep learning model for automatic Arabic diacritization,†ACM Trans. Asian Low-Resour. Lang. Inf. Process., vol. 21, no. 2, pp. 1–16, Nov. 2021. https://doi.org/10.1145/3480938

Y. Hifny, "Recent Advances in Arabic Syntactic Diacritics Restoration," in 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7768–7772, 2021. https://doi.org/10.1109/ICASSP39728.2021.9414500

B. Thompson and A. Alshehri, "Improving Arabic Diacritization by Learning to Diacritize and Translate," arXiv preprint arXiv:2109.14150, 2021. https://doi.org/10.48550/arXiv.2109.14150

A. Masmoudi, C. Aloulou, A. G. S. Abdellahi, and L. H. Belguith, “Automatic diacritization of Tunisian dialect text using SMT model,†Int. J. Speech Technol., vol. 25, no. 1, pp. 89–104, 2022. https://doi.org/10.1007/s10772-021-09864-6

G. A. Abandah, A. E. Suyyagh, and M. R. Abdel-Majeed, “Transfer learning and multi-phase training for accurate diacritization of Arabic poetry,†J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 6, pt. B, pp. 3744–3757, 2022. https://doi.org/10.1016/j.jksuci.2022.04.005

R. Kharsa, A. Elnagar, and S. Yagi, “BERT-based Arabic diacritization: A state-of-the-art approach for improving text accuracy and pronunciation,†Expert Syst. Appl., vol. 248, p. 123416, 2024. https://doi.org/10.1016/j.eswa.2024.123416

N. Sengupta et al., “Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models,†arXiv preprint arXiv:2308.16149, 2023. https://doi.org/10.48550/arXiv.2308.16149

M. Bari et al., “ALLaM: Large Arabic language models,†arXiv preprint arXiv:2407.15390, 2024. https://doi.org/10.48550/arXiv.2407.15390

D. Huang et al., “AceGPT, localizing large language models in Arabic,†arXiv preprint arXiv:2309.12053, 2023. https://doi.org/10.48550/arXiv.2309.12053

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention Is All You Need," arXiv preprint arXiv:1706.03762, 2023. https://doi.org/10.48550/arXiv.1706.03762

K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," arXiv preprint arXiv:1512.03385, 2015. https://doi.org/10.48550/arXiv.1512.03385

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,†arXiv preprint arXiv:1810.04805, 2019. https://doi.org/10.48550/arXiv.1810.04805

W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,†arXiv preprint arXiv:2003.00104, 2021. https://doi.org/10.48550/arXiv.2003.00104

G. Inoue, B. Alhafni, N. Baimukan, H. Bouamor, and N. Habash, “The interplay of variant, size, and task type in Arabic pre-trained language models,†arXiv preprint arXiv:2103.06678, 2021. https://doi.org/10.48550/arXiv.2103.06678

S. Hochreiter and J. Schmidhuber, “Long short-term memory,†Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. https://doi.org/10.1162/neco.1997.9.8.1735

K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation," arXiv preprint arXiv:1406.1078, 2014. https://doi.org/10.48550/arXiv.1406.1078

AssemblyAI, “The 300ms rule: Why latency makes or breaks voice AI applications,†2025. [Online]. Available: https://www.assemblyai.com/blog/low-latency-voice-ai

Author Biographies

Ahmed Abdelhamed Elgayar

Ahmed Abdelhamed Elgayar is a Senior BI & Data Analytics Engineer with over five years of professional experience in business intelligence, data analytics, and data visualization, along with a broader background in civil engineering and project controls. He holds a Master’s degree in Data Science and a Postgraduate Diploma in Data Science with distinction from the Faculty of Graduate Studies for Statistical Research, Cairo University. His expertise includes Power BI, SQL, Python, ETL development, data warehousing, and the Microsoft BI stack (SSIS, SSAS, and SSRS). He has contributed to large-scale business intelligence projects across governmental and private sectors in the United Arab Emirates, Saudi Arabia, and Egypt. His research interests include Arabic natural language processing (NLP), deep learning, machine learning, and AI-based language processing systems.

Abdul-Hadi Nabi Ahmed
Abdul-Hadi Nabi Ahmed holds a Ph.D. and an M.Sc. in Statistics (Quantitative Methods) from Florida State University, USA, and an M.Sc. in Mathematical Statistics from Cairo University, Egypt. He also earned diplomas in Applied Statistics and Operations Research, and a B.Sc. in Communication Engineering. He has taught undergraduate and graduate courses in statistics, operations research, mathematics, and economics at institutions including Florida State University, Cairo University, King Saud University, King Abdulaziz University, and the American University in Cairo. He has supervised over 45 graduate theses, including 10 Ph.D. dissertations. His research interests include statistical modeling, stochastic processes, and quantitative methods in economics and engineering.
Basma Ezzat El-Demerdash
Basma Ezzat El-Demerdash is an associate professor at the Department of Operations Research and Management, Faculty of Graduate Studies for Statistical Research (FGSSR), Cairo University. She graduated with a B.Sc. in Operations Research and Decision Support from the Faculty of Computers and Information, Cairo University in 2007. Her outstanding academic performance earned her a position as a teaching assistant in the department. She then went on to receive her M.Sc. degree from the same department in 2011.Her thesis focused on evaluating higher education systems in Egypt using data envelopment analysis. Her academic career began as a teaching assistant in 2007. Since then, she has accumulated 17 years of teaching experience at the Faculty of Computers and Artificial Intelligence, Cairo University.
Ahmed E. Soliman
Ahmed E. Soliman is a Senior Team Lead Software Engineer and Data Scientist with extensive expertise in business process automation, data engineering, and deep learning. He holds a Master’s degree in Data Science from Cairo University. His research focus lies at the intersection of Natural Language Processing (NLP) and Large Language Models (LLMs), with specific contributions in Few-Shot Learning and Data Augmentation techniques. Throughout his career, Ahmed has led and contributed to numerous high-impact projects across government and private sectors in Egypt, Saudi Arabia, and the UAE. His technical proficiency spans Python-based machine learning, Deep Learning, neural networks, and advanced data visualization.

How to Cite

[1]
A. A. Elgayar, A.-H. N. Ahmed, B. E. El-Demerdash, and A. E. Soliman, “Enhancing Arabic Diacritization Using BERT with BiLSTM and BiGRU Heads”, IJEEI, vol. 14, no. 2, pp. 518–532, Jun. 2026, doi: 10.52549/ijeei.v14i2.7033.