arXiv Open Access 2025

Which Pieces Does Unigram Tokenization Really Need?

Sander Land Yuval Pinter

Lihat Sumber

Abstrak

The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof. We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices. We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.

Topik & Kata Kunci

cs.CL

Penulis (2)

Sander Land

Yuval Pinter

Format Sitasi

APA MLA BibTeX

Land, S., Pinter, Y. (2025). Which Pieces Does Unigram Tokenization Really Need?. https://arxiv.org/abs/2512.12641

Akses Cepat

Lihat di Sumber

Informasi Jurnal

Tahun Terbit: 2025
Bahasa: en
Sumber Database: arXiv
Akses: Open Access ✓