Semantic Scholar Open Access 2023 16 sitasi

The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling

Joey Ohman S. Verlinden Ariel Ekgren Amaru Cuba Gyllensten T. Isbister +3 lainnya

Lihat Sumber DOI

Abstrak

Pre-training Large Language Models (LLMs) require massive amounts of text data, and the performance of the LLMs typically correlates with the scale and quality of the datasets. This means that it may be challenging to build LLMs for smaller languages such as Nordic ones, where the availability of text corpora is limited. In order to facilitate the development of the LLMS in the Nordic languages, we curate a high-quality dataset consisting of 1.2TB of text, in all of the major North Germanic languages (Danish, Icelandic, Norwegian, and Swedish), as well as some high-quality English data. This paper details our considerations and processes for collecting, cleaning, and filtering the dataset.

Topik & Kata Kunci

Computer Science

Penulis (8)

Joey Ohman

S. Verlinden

Ariel Ekgren

Amaru Cuba Gyllensten

T. Isbister

Evangelia Gogoulou

F. Carlsson

Magnus Sahlgren

Format Sitasi

APA MLA BibTeX

Ohman, J., Verlinden, S., Ekgren, A., Gyllensten, A.C., Isbister, T., Gogoulou, E. et al. (2023). The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling. https://doi.org/10.48550/arXiv.2303.17183

Akses Cepat

Lihat di Sumber doi.org/10.48550/arXiv.2303.17183

Informasi Jurnal

Tahun Terbit: 2023
Bahasa: en
Total Sitasi: 16×
Sumber Database: Semantic Scholar
DOI: 10.48550/arXiv.2303.17183
Akses: Open Access ✓