A Novel Shape Guided Transformer Network for Instance Segmentation in Remote Sensing Images

Instance segmentation performance in remote sensing images (RSIs) is significantly affected by two issues: how to extract accurate boundaries of objects from remote imaging through the dynamic atmosphere, and how to integrate the mutual information of related object instances scattered over a vast spatial region. In this study, we propose a novel shape guided transformer network (SGTN) to accurately extract objects at the instance level. Inspired by the global contextual modeling capacity of the self-attention mechanism, we propose an effective transformer encoder termed LSwin, which incorporates vertical and horizontal 1-D global self-attention mechanisms to obtain better global-perception capacity for RSIs than the popular local-shifted-window based swin transformer. To achieve accurate instance mask segmentation, we introduce a shape guidance module (SGM) to emphasize the object boundary and shape information. The combination of SGM, which emphasizes the local detail information, and LSwin, which focuses on the global context relationships, achieve excellent RSI instance segmentation. Their effectiveness was validated through comprehensive ablation experiments. Especially, LSwin is proven better than the popular ResNet and swin transformer encoders at the same level of efficiency. Compared to other instance segmentation methods, our SGTN achieves the highest average precision scores on two single-class public datasets (WHU dataset and BITCC dataset) and a multiclass public dataset (NWPU VHR-10 dataset).