arXiv Open Access 2025

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Jaap Jumelet Leonie Weissweiler Joakim Nivre Arianna Bisazza

Lihat Sumber

Abstrak

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.

Topik & Kata Kunci

cs.CL

Penulis (4)

Jaap Jumelet

Leonie Weissweiler

Joakim Nivre

Arianna Bisazza

Format Sitasi

APA MLA BibTeX

Jumelet, J., Weissweiler, L., Nivre, J., Bisazza, A. (2025). MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs. https://arxiv.org/abs/2504.02768

Akses Cepat

Lihat di Sumber

Informasi Jurnal

Tahun Terbit: 2025
Bahasa: en
Sumber Database: arXiv
Akses: Open Access ✓