Summary: Text-to-image generation of deep learning models such as OpenAI’s DALL-E 2 may be a promising new tool for image augmentation, generation, and manipulation in a healthcare environment.
Source: JMIR Publications
A new paper published in Journal of Medical Internet Research describes how generative models such as DALL-E 2, a new deep learning model for text-to-image generation, could represent a promising future tool for image generation, enhancement, and manipulation in healthcare.
Do generative models have sufficient knowledge of the medical domain to provide accurate and useful results? Dr. Lisa C Adams and colleagues explore this topic in their latest viewpoint entitled “What does DALL-E 2 know about radiology?”
First introduced by OpenAI in April 2022, OFF-E 2 is an artificial intelligence (AI) tool that has gained popularity to generate novel photorealistic images or artwork based on text input. DALL-E 2’s generative capabilities are powerful as it has been trained on billions of existing text-image pairs from the Internet.
To understand whether these capabilities can be transferred to the medical domain to create or augment data, researchers from Germany and the United States examined DALL-E 2’s radiological knowledge of creating and manipulating X-rays, computed tomography (CT), magnetic resonance imaging (MRI) and ultrasound images.
The study’s authors found that DALL-E 2 has learned relevant representations of X-ray images and shows promising potential for text-to-image generation. Specifically, the DALL-E 2 was able to create realistic X-ray images based on short text prompts, but it did not perform very well when given specific CT, MRI, or ultrasound image prompts. It was also able to reasonably reconstruct missing aspects in a radiological image.
It could do much more – for example, create a complete X-ray image of the entire body using only one image of the knee as a starting point. However, DALL-E 2 was limited in its ability to generate images with pathological abnormalities.
Synthetic data generated by DALL-E 2 could greatly accelerate the development of new deep learning tools for radiology, as well as address privacy issues related to data sharing between institutions. The study’s authors note that generated images should be subject to quality control by domain experts to reduce the risk of incorrect information entering a generated dataset.
They also emphasize the need for further research to fine-tune these models for medical data and incorporate medical terminology to create powerful models for data generation and augmentation in radiology research. Although DALL-E 2 is not available for the public to tweak, other generative models may be Stable diffusion is which could be adapted to generate a variety of medical images.
Overall, this view is published by JMIR Publications provides a promising outlook for the future of AI image generation in radiology. Further research and development in this area could lead to exciting new tools for radiologists and medical professionals.
Although there are limitations that need to be addressed, the potential benefits of using tools like DALL-E 2 and ChatGPT in research and medical education and training are essential. To this end, JMIR Medical Education is now inviting post for a new e-collection on the use of generative language models in medical education, as announced in a recent led by Dr. Gunther Eysenbach.
About this AI and DALL-E 2 research news
Author: Ryan James Jessup Jd/MPA
Source: JMIR Publications
Contact: Ryan James Jessup Jd/MPA – JMIR Publications
Image: Image credited to Microsoft Designer (based on DALL-E 2); Copyright: The authors × DALL·E 2; License: Creative Commons Attribution (CC-BY)
Original research: Closed access.
“What does DALL-E 2 know about radiology?” by Lisa C Adams et al. Journal of Medical Internet Research
Abstract
What does DALL-E 2 know about radiology?
Generative models, such as DALL-E 2 (OpenAI), could represent promising future image generation, enhancement and manipulation tools for AI research in radiology, provided these models have sufficient knowledge of the medical domain.
Herein, we show that DALL-E 2 has learned relevant representations of X-ray images with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements ; however, its capabilities for generating images of pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited.
Thus, the use of generative models to augment and generate radiological data seems possible, although further fine-tuning and adaptation of these models to their respective domains is required first.