arXiv Open Access 2024

Anthropocentric bias in language model evaluation

Raphaël Millière Charles Rathkopf

Lihat Sumber

Abstrak

Evaluating the cognitive capacities of large language models (LLMs) requires overcoming not only anthropomorphic but also anthropocentric biases. This article identifies two types of anthropocentric bias that have been neglected: overlooking how auxiliary factors can impede LLM performance despite competence ("auxiliary oversight"), and dismissing LLM mechanistic strategies that differ from those of humans as not genuinely competent ("mechanistic chauvinism"). Mitigating these biases necessitates an empirically-driven, iterative approach to mapping cognitive tasks to LLM-specific capacities and mechanisms, which can be done by supplementing carefully designed behavioral experiments with mechanistic studies.

Topik & Kata Kunci

cs.CL

Penulis (2)

Raphaël Millière

Charles Rathkopf

Format Sitasi

APA MLA BibTeX

Millière, R., Rathkopf, C. (2024). Anthropocentric bias in language model evaluation. https://arxiv.org/abs/2407.03859

Akses Cepat

Lihat di Sumber

Informasi Jurnal

Tahun Terbit: 2024
Bahasa: en
Sumber Database: arXiv
Akses: Open Access ✓