arXiv
Open Access
2025
JETHICS: Japanese Ethics Understanding Evaluation Dataset
Masashi Takeshita
Rafal Rzepka
Abstrak
In this work, we propose JETHICS, a Japanese dataset for evaluating ethics understanding of AI models. JETHICS contains 78K examples and is built by following the construction methods of the existing English ETHICS dataset. It includes four categories based normative theories and concepts from ethics and political philosophy; and one representing commonsense morality. Our evaluation experiments on non-proprietary large language models (LLMs) and on GPT-4o reveal that even GPT-4o achieves only an average score of about 0.7, while the best-performing Japanese LLM attains around 0.5, indicating a relatively large room for improvement in current LLMs.
Penulis (2)
M
Masashi Takeshita
R
Rafal Rzepka
Akses Cepat
Informasi Jurnal
- Tahun Terbit
- 2025
- Bahasa
- en
- Sumber Database
- arXiv
- Akses
- Open Access ✓