Physics-guided LightM-UNet for BraTS Missing Modality synthesis

Abstract

Accurate brain tumour segmentation requires complete multi-modal magnetic resonance imaging (MRI) data comprising four modalities: T1-weighted (T1n), T1 contrast-enhanced (T1c), T2-weighted (T2w), and T2-FLAIR (T2f). However, in real-life scenarios, one or two modalities may be missing due to scan time constraints or patient condition. The aim of this thesis is to propose a novel Physics-guided LightM-UNet framework for missing modality synthesis based on the BraSyn 2025 dataset. The dataset comprises 1489 patients' training data for Glioma and Metastasis, and the validation set comprises data from 250 patients with Glioma and Metastasis. The test set includes data of 561 Glioma, Metastasis and Meningioma patients. To mitigate AI hallucinations and ensure clinically relevant modality synthesis, the vision state space models are integrated with a learnable Bloch layer to enforce MRI relaxation physics. The vision state space models ensure linear-time global context modelling. The core feature of the training pipeline is the Specialist agent strategy, where four independent generator models are trained, one for each modality. This ensures that there is no gradient conflict in multi-task formulation. This framework achieves an average Structural Similarity Index (SSIM) of 0.9305 across all agents and a Peak-Signal-to-Noise Ratio (PSNR) of 28.3320. Further, metrics evaluating perceptual fidelity, structural similarity, and statistical diversity were used, including High Frequency error Norm (HFEN), Learned perceptual image patch similarity (LPIPS), Fréchet inception distance (FID), and Normalised root mean square (NRMSE) to test the robustness of the proposed framework. The downstream segmentation task, which demonstrates the clinical relevance of the synthetic modality, is rigorously evaluated using a 4-stage evaluation matrix with nnUNetv2. A baseline dice score of approximately 0.87 is achieved with the nnUNetv2 model trained and tested on real data. Furthermore, a robust dice score of 0.73 is achieved when the segmentation model is trained and tested purely using synthetic data. The model trained on real sequences was further tested on a synthetic test set for segmentation, achieving a dice score of 0.6740vice versa, training achieved a dice score of 0.7014. These four stages of testing were conducted to demonstrate segmentation performance using state-of-the-art frameworks such as nnUNetv2. The Specialist agent models were deployed on NVIDIA Jetson AGX Orin by converting them using the TensorRT engine and FP16 precision to test real-time point-to-care functionality. A real-time inference time of approximately 61 to 64 ms when evaluated on the test dataset.