arXiv Open Access 2020

ETC: Encoding Long and Structured Inputs in Transformers

Joshua Ainslie Santiago Ontanon Chris Alberti Vaclav Cvicek Zachary Fisher +5 lainnya

Lihat Sumber

Abstrak

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

Topik & Kata Kunci

cs.LG stat.ML

Penulis (10)

Joshua Ainslie

Santiago Ontanon

Chris Alberti

Vaclav Cvicek

Zachary Fisher

Philip Pham

Anirudh Ravula

Sumit Sanghai

Qifan Wang

Li Yang

Format Sitasi

APA MLA BibTeX

Ainslie, J., Ontanon, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P. et al. (2020). ETC: Encoding Long and Structured Inputs in Transformers. https://arxiv.org/abs/2004.08483

Akses Cepat

Lihat di Sumber

Informasi Jurnal

Tahun Terbit: 2020
Bahasa: en
Sumber Database: arXiv
Akses: Open Access ✓