Cross-modal Deep Learning-based Clinical Recommendation System for Radiology Report Generation from Chest X-rays

Document Type : Original Article

Authors

1 Department of Information Technology, National Institute of Technology Karnataka, Mangalore, Karnataka, India

2 Department of Computer Science and Engineering, NMAM Institute of Technology, Nitte (Deemed to be University), Udupi, Karnataka, India

3 Deperatment of Radiology, Kasturba Medical College, Mangalore, Manipal Academy of Higher Education, Manipal, Karnataka, India

Abstract

Radiology report generation is a critical task for radiologists, and automating the process can significantly simplify their workload. However, creating accurate and reliable radiology reports requires radiologists to have sufficient experience and time to review medical images. Unfortunately, many radiology reports end with ambiguous conclusions, resulting in additional testing and diagnostic procedures for patients. To address this, we proposed an encoder-decoder-based deep learning framework that utilizes chest X-ray images to produce diagnostic radiology reports. In our study, we have introduced a novel text modelling and visual feature extraction strategy as part of our proposed encoder-decoder-based deep learning framework. Our approach aims to extract essential visual and textual information from chest X-ray images to generate more accurate and reliable radiology reports. Additionally, we have developed a dynamic web portal that accepts chest X-rays as input and generates a radiology report as output. We conducted an extensive analysis of our model and compared its performance with other state-of-the-art deep learning approaches. Our findings indicate significant improvement achieved by our proposed model compared to existing models, as evidenced by the higher BLEU scores (BLEU1 = 0.588, BLEU2 = 0.4325, BLEU3 = 0.4017, BLEU4 = 0.3860) attained on the Indiana University Dataset. These results underscore the potential of our deep learning framework to enhance the accuracy and reliability of radiology reports, leading to more efficient and effective medical treatment.

Keywords

Main Subjects

References

  1. Shetty, S. and Mahale, A., "Comprehensive review of multimodal medical data analysis: Open issues and future research directions", Acta Informatica Pragensia, Vol. 11, No. 3, (2022), 423-457. doi: 10.18267/j.aip.202.
  2. Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K.G. and Murphy, K., "Deep learning for chest x-ray analysis: A survey", Medical Image Analysis, Vol. 72, (2021), 102125. https://doi.org10.1016/j.media.2021.102125
  3. Ramirez-Alonso, G., Prieto-Ordaz, O., López-Santillan, R. and Montes-Y-Gómez, M., "Medical report generation through radiology images: An overview", IEEE Latin America Transactions, Vol. 20, No. 6, (2022), 986-999. doi: 10.1109/TLA.2022.9757742.
  4. Monshi, M.M.A., Poon, J. and Chung, V., "Deep learning in generating radiology reports: A survey", Artificial Intelligence in Medicine, Vol. 106, (2020), 101878. https://doi.org/10.1016/j.artmed.2020.101878
  5. Jing, B., Xie, P. and Xing, E., "On the automatic generation of medical imaging reports", arXiv preprint arXiv:1711.08195, (2017). doi: 10.18653/v1/P18-1240.
  6. Dey, A., "Cov-xdcnn: Deep learning model with external filter for detecting covid-19 on chest x-rays", in Computer, Communication, and Signal Processing: 6th IFIP TC 5 International Conference, ICCCSP 2022, Chennai, India, February 24–25, 2022, Revised Selected Papers, Springer. (2022), 174-189.
  7. Shetty, S. and Mahale, A., "Ms-chexnet: An explainable and lightweight multi-scale dilated network with depthwise separable convolution for prediction of pulmonary abnormalities in chest radiographs", Mathematics, Vol. 10, No. 19, (2022), 3646. https://doi.org/10.3390/math10193646
  8. Alahmari, S.S., Altazi, B., Hwang, J., Hawkins, S. and Salem, T., "A comprehensive review of deep learning-based methods for covid-19 detection using chest x-ray images", IEEE Access, (2022). doi: 10.1109/ACCESS.2022.3208138.
  9. Yang, S., Wu, X., Ge, S., Zhou, S.K. and Xiao, L., "Knowledge matters: Chest radiology report generation with general and specific knowledge", Medical Image Analysis, Vol. 80, (2022), 102510. https://doi.org/10.1016/j.media.2022.102510
  10. Carson, E.R., Cramp, D.G., Morgan, A. and Roudsari, A.V., "Clinical decision support, systems methodology, and telemedicine: Their role in the management of chronic disease", IEEE Transactions on Information Technology in Biomedicine, Vol. 2, No. 2, (1998), 80-88. doi: 10.1109/4233.720526.
  11. Abtahi, Z., Sahraeian, R. and Rahmani, D., "A stochastic model for prioritized outpatient scheduling in a radiology center", International Journal of Engineering Transactions A: Basics, Vol. 33, No. 4, (2020). doi: 10.5829/ije.2020.33.04a.11.
  12. Khatami, A., Babaie, M., Tizhoosh, H., Nazari, A., Khosravi, A. and Nahavandi, S., "A radon-based convolutional neural network for medical image retrieval", International Journal of Engineering, Transactions C: Aspects, Vol. 31, No. 6, (2018), 910-915. doi: 10.5829/ije.2018.31.06c.07.
  13. Gheitasi, A., Farsi, H. and Mohamadzadeh, S., "Estimation of hand skeletal postures by using deep convolutional neural networks", International Journal of Engineering, Transactions A: Basics, Vol. 33, No. 4, (2020), 552-559. doi: 10.5829/ije.2020.33.04a.06.
  14. Yuan, J., Liao, H., Luo, R. and Luo, J., "Automatic radiology report generation based on multi-view image fusion and medical concept enrichment", in Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22, Springer. (2019), 721-729.
  15. Nguyen, H.T., Nie, D., Badamdorj, T., Liu, Y., Zhu, Y., Truong, J. and Cheng, L., "Automated generation of accurate\& fluent medical x-ray reports", arXiv preprint arXiv:2108.12126, (2021). doi: 10.18653/v1/2021.emnlp-main.288.
  16. Tripathy, B., Sai, R.R. and Banu, K.S., Automated medical report generation on chest x-ray: Images using co-attention mechanism, in Hybrid computational intelligent systems. 2023, CRC Press.111-122.
  17. Zhou, X., Li, Y. and Liang, W., "Cnn-rnn based intelligent recommendation for online medical pre-diagnosis support", IEEE/ACM Transactions on Computational Biology and Bioinformatics, Vol. 18, No. 3, (2020), 912-921. doi: 10.1109/TCBB.2020.2994780.
  18. Liu, F., You, C., Wu, X., Ge, S. and Sun, X., "Auto-encoding knowledge graph for unsupervised medical report generation", Advances in Neural Information Processing Systems, Vol. 34, (2021), 16266-16279. https://arxiv.org/abs/2111.04318
  19. Sirshar, M., Paracha, M.F.K., Akram, M.U., Alghamdi, N.S., Zaidi, S.Z.Y. and Fatima, T., "Attention based automated radiology report generation using cnn and lstm", Plos one, Vol. 17, No. 1, (2022), e0262209. doi: 10.1371/journal.pone.0262209.
  20. Nicolson, A., Dowling, J. and Koopman, B., "Improving chest x-ray report generation by leveraging warm-starting", arXiv preprint arXiv:2201.09405, (2022). https://arxiv.org/abs/2201.09405
  21. Alfarghaly, O., Khaled, R., Elkorany, A., Helal, M. and Fahmy, A., "Automated radiology report generation using conditioned transformers", Informatics in Medicine Unlocked, Vol. 24, (2021), 100557. doi: 10.1016/j.imu.2021.100557.
  22. Babar, Z., van Laarhoven, T., Zanzotto, F.M. and Marchiori, E., "Evaluating diagnostic content of ai-generated radiology reports of chest x-rays", Artificial Intelligence in Medicine, Vol. 116, (2021), 102075. https://doi.org10.1016/j.artmed.2021.102075
  23. Shetty, S. and Mahale, A., "Multimodal medical tensor fusion network-based dl framework for abnormality prediction from the radiology cxrs and clinical text reports", Multimedia Tools and Applications, (2023), 1-48. https://doi.org/10.1007/s11042-023-14940-x.
  24. Pennington, J., Socher, R. and Manning, C.D., "Glove: Global vectors for word representation", in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). (2014), 1532-1543.
  25. Zhang, Y., Ding, D.Y., Qian, T., Manning, C.D. and Langlotz, C.P., "Learning to summarize radiology findings", arXiv preprint arXiv:1809.04698, (2018). http://arxiv.org/abs/1809.04698
  26. Shetty, S., Ananthanarayana, V. and Mahale, A., "Medical knowledge-based deep learning framework for disease prediction on unstructured radiology free-text reports under low data condition", in Proceedings of the 21st EANN (Engineering Applications of Neural Networks) 2020 Conference: Proceedings of the EANN 2020 21, Springer. (2020), 352-364.
  27. Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R. and McDonald, C.J., "Preparing a collection of radiology examinations for distribution and retrieval", Journal of the American Medical Informatics Association, Vol. 23, No. 2, (2016), 304-310. doi: 10.1093/jamia/ocv080.
  28. Gu, J., Cho, K. and Li, V.O., "Trainable greedy decoding for neural machine translation", arXiv preprint arXiv:1702.02429, (2017). doi: 10.18653/v1/D17-1210.
  29. Freitag, M. and Al-Onaizan, Y., "Beam search strategies for neural machine translation", arXiv preprint arXiv:1702.01806, (2017). https://doi.org/10.48550/arXiv.1702.01806
  30. Papineni, K., Roukos, S., Ward, T. and Zhu, W.-J., "Bleu: A method for automatic evaluation of machine translation", in Proceedings of the 40th annual meeting of the Association for Computational Linguistics. (2002), 311-318.
International Journal of Engineering
Volume 36, Issue 8
TRANSACTIONS B: Applications
August 2023
Pages 1569-1577

Files

Cited by

Share

How to cite

Statistics