Is visual cortex really “language-aligned”? Perspectives from Model-to-Brain Comparisons in Human and Monkeys on the Natural Scenes Dataset
Abstrak
Recent progress in multimodal AI and “language-aligned” visual representation learning has re-ignited debates about the role of language in shaping the human visual system. In particular, the emergent ability of “language-aligned” vision models (e.g. CLIP) – and even pure language models (e.g. BERT) – to predict image-evoked brain activity has led some to suggest that human visual cortex itself may be “language-aligned” in comparable ways. But what would we make of this claim if the same procedures worked in the modeling of visual activity in a species that has no language? Here, we deploy controlled comparisons of pure-vision, pure-language, and multimodal vision-language models in prediction of human (N=4) and rhesus macaque (N=6, 5:IT, 1:V1) ventral visual activity evoked in response to the same set of 1000 captioned nat-ural images (the “NSD1000”). Preliminary results reveal markedly similar patterns in aggregate model predictivity of early and late ventral visual cortex across both species. Together, these results suggest that language predictivity of the human visual system is not necessarily due to “language-alignment” per se , but rather to the statistical structure of the visual world as reflected in language.
Penulis (8)
Colin Conwell
Emalie McMahon
Kasper Vinken
Jacob S. Prince
George A. Alvarez
Talia Konkle
Leyla Isik
Margaret S. Livingstone
Format Sitasi
Akses Cepat
- Tahun Terbit
- 2024
- Bahasa
- en
- Total Sitasi
- 1×
- Sumber Database
- Semantic Scholar
- DOI
- 10.1167/jov.24.10.1288
- Akses
- Open Access ✓