arXiv Open Access 2023

Are We Ready to Embrace Generative AI for Software Q&A?

Bowen Xu Thanh-Dat Nguyen Thanh Le-Cong Thong Hoang Jiakun Liu +6 lainnya

Lihat Sumber

Abstrak

Stack Overflow, the world's largest software Q&A (SQA) website, is facing a significant traffic drop due to the emergence of generative AI techniques. ChatGPT is banned by Stack Overflow after only 6 days from its release. The main reason provided by the official Stack Overflow is that the answers generated by ChatGPT are of low quality. To verify this, we conduct a comparative evaluation of human-written and ChatGPT-generated answers. Our methodology employs both automatic comparison and a manual study. Our results suggest that human-written and ChatGPT-generated answers are semantically similar, however, human-written answers outperform ChatGPT-generated ones consistently across multiple aspects, specifically by 10% on the overall score. We release the data, analysis scripts, and detailed results at https://anonymous.4open.science/r/GAI4SQA-FD5C.

Topik & Kata Kunci

cs.SE

Penulis (11)

Bowen Xu

Thanh-Dat Nguyen

Thanh Le-Cong

Thong Hoang

Jiakun Liu

Kisub Kim

Chen Gong

Changan Niu

Chenyu Wang

Bach Le

David Lo

Format Sitasi

APA MLA BibTeX

Xu, B., Nguyen, T., Le-Cong, T., Hoang, T., Liu, J., Kim, K. et al. (2023). Are We Ready to Embrace Generative AI for Software Q&A?. https://arxiv.org/abs/2307.09765

Akses Cepat

Lihat di Sumber

Informasi Jurnal

Tahun Terbit: 2023
Bahasa: en
Sumber Database: arXiv
Akses: Open Access ✓