arXiv
Open Access
2023
Dual PatchNorm
Manoj Kumar
Mostafa Dehghani
Neil Houlsby
Abstrak
We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.
Penulis (3)
M
Manoj Kumar
M
Mostafa Dehghani
N
Neil Houlsby
Akses Cepat
Informasi Jurnal
- Tahun Terbit
- 2023
- Bahasa
- en
- Sumber Database
- arXiv
- Akses
- Open Access ✓