Dual-Stream Contrastive Learning for Medical Visual Representations Using Synthetic Images Generated by Latent Diffusion Model

Deep learning-based medical image processing methods can enhance diagnostic accuracy while significantly accelerating clinical decision workflows. However, in order to learn better visual representations, such approaches usually need substantial amount of expert-annotated data, which are highly costly. To address this issue, we propose a novel approach called Dual-Stream Contrastive Learning with Cross-Scale Token Projection (DCL-CsTP), which aims to enhance visual representations and transferable initializations. Specifically, a latent diffusion model (LDM) is leveraged to generate high-quality synthetic medical images in order to expand the dataset. Then we utilize the proposed dual-stream architecture that consists of a global semantic relations stream and a local detail relations stream to learn discriminative medical image representations from the dataset. Furthermore, a cross-scale token projection is designed to enable the model to capture various scales of focus in medical images. Comprehensive experiments are performed on two downstream tasks: medical image classification and segmentation. For multi-classification of pneumonia, our DCL-CsTP method achieves 95.90% accuracy. For lesions segmentation, our DCL-CsTP method attains 89.73% dice coefficient on the International Skin Imaging Collaboration 2018 (ISIC 2018) dataset and 82.50% dice coefficient on the Kvasir-SEG dataset. The performance superiority of the model pre-trained by DCL-CsTP is conclusively demonstrated through the above experiments on various dataset, which shows that DCL-CsTP can enhance diagnostic precision and alleviate radiologists’ image screening burdens.