Semantic Scholar Open Access 2022 755 sitasi

GIT: A Generative Image-to-text Transformer for Vision and Language

Jianfeng Wang Zhengyuan Yang Xiaowei Hu Linjie Li Kevin Lin +4 lainnya

Lihat Sumber DOI

Abstrak

In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at \url{https://github.com/microsoft/GenerativeImage2Text}.

Topik & Kata Kunci

Computer Science

Penulis (9)

Jianfeng Wang

Zhengyuan Yang

Xiaowei Hu

Linjie Li

Kevin Lin

Zhe Gan

Zicheng Liu

Ce Liu

Lijuan Wang

Format Sitasi

APA MLA BibTeX

Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z. et al. (2022). GIT: A Generative Image-to-text Transformer for Vision and Language. https://doi.org/10.48550/arXiv.2205.14100

Akses Cepat

Lihat di Sumber doi.org/10.48550/arXiv.2205.14100

Informasi Jurnal

Tahun Terbit: 2022
Bahasa: en
Total Sitasi: 755×
Sumber Database: Semantic Scholar
DOI: 10.48550/arXiv.2205.14100
Akses: Open Access ✓