Hybrid 3D U-Net Transformer for Precision Glioma Segmentation

Authors

Ahmed Bounegta
Laboratory of Information Processing and Telecommunication (LTIT), Department of Electrical Engineering, Faculty of Technology, University Tahri Mohammed Bechar, Bechar, Algeria
Mustapha Khelifi
Laboratory of Information Processing and Telecommunication (LTIT), Department of Electrical Engineering, Faculty of Technology, University Tahri Mohammed Bechar, Bechar, Algeria
Mohammed Beladgham
Laboratory of Information Processing and Telecommunication (LTIT), Department of Electrical Engineering, Faculty of Technology, University Tahri Mohammed Bechar, Bechar, Algeria
Abdellah Ouldammar
Laboratory of Information Processing and Telecommunication (LTIT), Department of Electrical Engineering, Faculty of Technology, University Tahri Mohammed Bechar, Bechar, Algeria
DOI: https://doi.org/10.52549/ijeei.v14i1.6189
Download PDF
Published: 31-03-2026 • Issue: Vol. 14 No. 1: March 2026 • Section: Regular paper

Abstract

Gliomas are one of the most prevalent malignant brain tumors, presenting considerable problems for patient prognosis and therapeutic approaches. Precise segmentation of these tumors is essential for diagnosis, surgical planning, and intraoperative radiologic monitoring (IRM). Defining glioma subregions, including the enhancing tumor (TE), tumor core (TC), and whole tumor (WT), enables targeted therapy and aids in monitoring tumor growth over time. This article presents the 3D U-Net Transformer, a deep learning architecture that integrates convolutional layers with transformer-based self-attention mechanisms. The model efficiently analyzes multimodal MRI scans, utilizing skip connections and attention modules to merge local spatial data with global context, thus improving segmentation performance. The 3D U-Net Transformer, validated on the BraTS 2020 dataset—a benchmark for brain tumor segmentation—surpassed traditional topologies like U-Net and UNet++. The model attained elevated Dice coefficients for TE, TC, and WT areas, accompanied by robust sensitivity and specificity metrics, hence, enhancing its clinical dependability. This sophisticated method enhances surgical procedures and clinical decision-making by providing accurate tumor delineation. The integration of transformer modules within U-Net architectures highlights the possibility of significant advancements in 3D medical imaging and real-time applications. The computer setup for this study comprised a high-performance PC featuring an Intel i7 CPU, 12 GB RAM, x64 architecture, Intel HD Graphics 3000, operating on Kaggle with a P100 GPU, and utilizing Python 3.8, Kaggle, and TensorFlow 2.4 on a 64-bit operating system.
Keywords: 3D U-Net Transformer; Glioma Segmentation; Multi-modal MRI; Deep Learning; BraTS 2020 Dataset; Self-Attention Mechanisms.

References

D. N. Louis et al., “The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary,†Acta Neuropathol. (Berl.), vol. 131, no. 6, pp. 803–820, Jun. 2016, doi: 10.1007/s00401-016-1545-1.

Q. T. Ostrom et al., “CBTRUS Statistical Report: Primary Brain and Central Nervous System Tumors Diagnosed in the United States in 2008-2012,†Neuro-Oncol., vol. 17, no. suppl 4, pp. iv1–iv62, Oct. 2015, doi: 10.1093/neuonc/nov189.

P. Kickingereder et al., “Automated quantitative tumour response assessment of MRI in neuro-oncology with artificial neural networks: a multicentre, retrospective study,†Lancet Oncol., vol. 20, no. 5, pp. 728–740, May 2019, doi: 10.1016/S1470-2045(19)30098-1.

N. J. Tustison et al., “Large-scale evaluation of ANTs and FreeSurfer cortical thickness measurements,†NeuroImage, vol. 99, pp. 166–179, Oct. 2014, doi: 10.1016/j.neuroimage.2014.05.044.

K. Chang et al., “Automatic assessment of glioma burden: a deep learning algorithm for fully automated volumetric and bidimensional measurement,†Neuro-Oncol., vol. 21, no. 11, pp. 1412–1422, Nov. 2019, doi: 10.1093/neuonc/noz106.

E. J. Bacon et al., “Neuroimage analysis using artificial intelligence approaches: a systematic review,†Med. Biol. Eng. Comput., vol. 62, no. 9, pp. 2599–2627, 2024.

V. Nagesh et al., “Radiation-induced changes in normal-appearing white matter in patients with cerebral tumors: a diffusion tensor imaging study,†Int. J. Radiat. Oncol. Biol. Phys., vol. 70, no. 4, pp. 1002–1010, 2008.

K. Witzmann, F. Raschke, and E. G. Troost, “MR image changes of normal-appearing brain tissue after radiotherapy,†Cancers, vol. 13, no. 7, p. 1573, 2021.

K. R. A. Van Dijk, T. Hedden, A. Venkataraman, K. C. Evans, S. W. Lazar, and R. L. Buckner, “Intrinsic Functional Connectivity As a Tool For Human Connectomics: Theory, Properties, and Optimization,†J. Neurophysiol., vol. 103, no. 1, pp. 297–321, Jan. 2010, doi: 10.1152/jn.00783.2009.

O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,†in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, vol. 9351, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds., in Lecture Notes in Computer Science, vol. 9351. , Cham: Springer International Publishing, 2015, pp. 234–241. doi: 10.1007/978-3-319-24574-4_28.

A. E. Orhan and X. Pitkow, “Skip Connections Eliminate Singularities,†2017, arXiv. doi: 10.48550/ARXIV.1701.09175.

Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++: A Nested U-Net Architecture for Medical Image Segmentation,†in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, vol. 11045, D. Stoyanov, Z. Taylor, G. Carneiro, T. Syeda-Mahmood, A. Martel, L. Maier-Hein, J. M. R. S. Tavares, A. Bradley, J. P. Papa, V. Belagiannis, J. C. Nascimento, Z. Lu, S. Conjeti, M. Moradi, H. Greenspan, and A. Madabhushi, Eds., in Lecture Notes in Computer Science, vol. 11045. , Cham: Springer International Publishing, 2018, pp. 3–11. doi: 10.1007/978-3-030-00889-5_1.

B. Kayalibay, G. Jensen, and P. Van Der Smagt, “CNN-based segmentation of medical imaging data,†ArXiv Prepr. ArXiv170103056, 2017.

B. H. Menze et al., “The multimodal brain tumor image segmentation benchmark (BRATS),†IEEE Trans. Med. Imaging, vol. 34, no. 10, pp. 1993–2024, 2014.

M. Prastawa, E. Bullitt, S. Ho, and G. Gerig, “A brain tumor segmentation framework based on outlier detection,†Med. Image Anal., vol. 8, no. 3, pp. 275–283, 2004, doi: 10.1016/j.media.2004.06.007.

M. Salvi et al., “A hybrid deep learning approach for gland segmentation in prostate histopathological images,†Artif. Intell. Med., vol. 115, p. 102076, May 2021, doi: 10.1016/j.artmed.2021.102076.

F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,†Nat. Methods, vol. 18, no. 2, pp. 203–211, 2021.

G. Litjens et al., “A Survey on Deep Learning in Medical Image Analysis,†Med. Image Anal., vol. 42, pp. 60–88, Dec. 2017, doi: 10.1016/j.media.2017.07.005.

A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,†Jun. 03, 2021, arXiv: arXiv:2010.11929. doi: 10.48550/arXiv.2010.11929.

A. Hatamizadeh et al., “UNETR: Transformers for 3D Medical Image Segmentation,†in 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA: IEEE, Jan. 2022, pp. 1748–1758. doi: 10.1109/WACV51458.2022.00181.

W. Wang, C. Chen, M. Ding, J. Li, H. Yu, and S. Zha, “TransBTS: Multimodal Brain Tumor Segmentation Using Transformer,†2021, arXiv. doi: 10.48550/ARXIV.2103.04430.

Y. Jiang, Y. Zhang, X. Lin, J. Dong, T. Cheng, and J. Liang, “SwinBTS: A Method for 3D Multimodal Brain Tumor Segmentation Using Swin Transformer,†Brain Sci., vol. 12, no. 6, p. 797, Jun. 2022, doi: 10.3390/brainsci12060797.

S. Perera, P. Navard, and A. Yilmaz, “SegFormer3D: an Efficient Transformer for 3D Medical Image Segmentation,†in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA: IEEE, Jun. 2024, pp. 4981–4988. doi: 10.1109/CVPRW63382.2024.00503.

V. Ashish, “Attention is all you need,†Adv. Neural Inf. Process. Syst., vol. 30, p. I, 2017.

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation,†in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, vol. 9901, S. Ourselin, L. Joskowicz, M. R. Sabuncu, G. Unal, and W. Wells, Eds., in Lecture Notes in Computer Science, vol. 9901. , Cham: Springer International Publishing, 2016, pp. 424–432. doi: 10.1007/978-3-319-46723-8_49.

X. Guan et al., “3D AGSE-VNet: an automatic brain tumor MRI data segmentation framework,†BMC Med. Imaging, vol. 22, no. 1, p. 6, Dec. 2022, doi: 10.1186/s12880-021-00728-8.

A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,†ArXiv Prepr. ArXiv200410934, 2020.

M. S. Alam, D. Wang, Q. Liao, and A. Sowmya, “A multi-scale context aware attention model for medical image segmentation,†IEEE J. Biomed. Health Inform., vol. 27, no. 8, pp. 3731–3739, 2022.

K. Kamnitsas et al., “Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation,†Med. Image Anal., vol. 36, pp. 61–78, Feb. 2017, doi: 10.1016/j.media.2016.10.004.

Y. Cai et al., “Swin Unet3D: a three-dimensional medical image segmentation network combining vision transformer and convolution,†BMC Med. Inform. Decis. Mak., vol. 23, no. 1, p. 33, Feb. 2023, doi: 10.1186/s12911-023-02129-z.

R. H. Helland et al., “Segmentation of glioblastomas in early post-operative multi-modal MRI with deep neural networks,†Sci. Rep., vol. 13, no. 1, p. 18897, 2023.

T. Magadza and S. Viriri, “Efficient nnU-net for brain tumor segmentation,†IEEE Access, vol. 11, pp. 126386–126397, 2023.

X. Fu et al., “3D bi-directional transformer U-Net for medical image segmentation,†Front. Big Data, vol. 5, p. 1080715, Jan. 2023, doi: 10.3389/fdata.2022.1080715.

A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,†in First conference on language modeling, 2024.

J. Ruan, J. Li, and S. Xiang, “Vm-unet: Vision mamba unet for medical image segmentation,†ACM Trans. Multimed. Comput. Commun. Appl., 2024.

Z. Nie, J. Yang, C. Li, Y. Wang, and J. Tang, “Diffbts: A lightweight diffusion model for 3d multimodal brain tumor segmentation,†Sensors, vol. 25, no. 10, p. 2985, 2025.

J. Wu et al., “Medsegdiff: Medical image segmentation with diffusion probabilistic model,†in Medical imaging with deep learning, PMLR, 2024, pp. 1623–1639.

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,†in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2881–2890.

A. Khan et al., “A recent survey of vision transformers for medical image segmentation,†IEEE Access, 2025.

J. Chen et al., “TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation,†Feb. 08, 2021, arXiv: arXiv:2102.04306. doi: 10.48550/arXiv.2102.04306.

X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local Neural Networks,†in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA: IEEE, Jun. 2018, pp. 7794–7803. doi: 10.1109/CVPR.2018.00813.

O. Oktay et al., “Attention U-Net: Learning Where to Look for the Pancreas,†2018, arXiv. doi: 10.48550/ARXIV.1804.03999.

S. Bakas et al., “Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge,†Apr. 23, 2019, arXiv: arXiv:1811.02629. doi: 10.48550/arXiv.1811.02629.

S. Bakas et al., “Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features,†Sci. Data, vol. 4, no. 1, p. 170117, Sep. 2017, doi: 10.1038/sdata.2017.117.

A. Kumar, “Study and analysis of different segmentation methods for brain tumor MRI application,†Multimed. Tools Appl., vol. 82, no. 5, pp. 7117–7139, 2023.

E. Ge, “Multimodal Deep Learning for Medical Imaging: A Survey and A New Approach to Brain Tumor Segmentation with Incomplete Dataâ€.

U. Baid et al., “The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification,†ArXiv Prepr. ArXiv210702314, 2021.

F. Isensee et al., “nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation,†Jul. 25, 2024, arXiv: arXiv:2404.09556. doi: 10.48550/arXiv.2404.09556.

R. Holden Helland et al., “Glioblastoma Segmentation from Early Post-operative MRI: Challenges and Clinical Impact,†in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2024, pp. 284–294.

L. R. Dice, “Measures of the amount of ecologic association between species,†Ecology, vol. 26, no. 3, pp. 297–302, 1945.

D. P. Huttenlocher, G. A. Klanderman, and W. J. Rucklidge, “Comparing images using the Hausdorff distance,†IEEE Trans. Pattern Anal. Mach. Intell., vol. 15, no. 9, pp. 850–863, 2002.

D. M. Powers, “Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation,†ArXiv Prepr. ArXiv201016061, 2020.

E. A. Kapp et al., “An evaluation, comparison, and accurate benchmarking of several publicly available MS/MS search algorithms: sensitivity and specificity analysis,†Proteomics, vol. 5, no. 13, pp. 3475–3490, 2005.

L. Hui and M. Belkin, “Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks,†ArXiv Prepr. ArXiv200607322, 2020.

C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso, “Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,†in International Workshop on Deep Learning in Medical Image Analysis, Springer, 2017, pp. 240–248.

O. Ogundairo, M. Arsalan, S. Pan, and F. Li, “A Comparative Analysis of Loss Functions for Handling Class Imbalance in Semantic Segmentationâ€.

Y. Bao, “Research on Transformer Point Cloud Segmentation Technology under Self Attention Mechanism,†Procedia Comput. Sci., vol. 262, pp. 1259–1266, 2025.

J. Wu, W. Ji, H. Fu, M. Xu, Y. Jin, and Y. Xu, “Medsegdiff-v2: Diffusion-based medical image segmentation with transformer,†in Proceedings of the AAAI conference on artificial intelligence, 2024, pp. 6030–6038.

How to Cite

[1]
A. Bounegta, M. Khelifi, M. Beladgham, and A. Ouldammar, “Hybrid 3D U-Net Transformer for Precision Glioma Segmentation”, IJEEI, vol. 14, no. 1, pp. 293–305, Mar. 2026, doi: 10.52549/ijeei.v14i1.6189.