Large language model-assisted research question development in public health: a case study in the Special Supplemental Nutrition Program for Women, Infants, and Children

Abstract Objective: To assess the feasibility of using large language models (LLM) to develop research questions about changes to the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) food packages. Design: We conducted a controlled experiment using ChatGPT-4 and its plugin, MixerBox Scholarly, to generate research questions based on a section of the U.S. Department of Agriculture (USDA) summary of the final public comments on the WIC revision. Five questions weekly for 3 weeks were generated using LLM under two conditions: fed with or without relevant literature. The experiment generated ninety questions, which were evaluated using the Feasibility, Innovation, Novelty, Ethics and Relevance criteria. t tests and multivariate regression examined the difference by feeding status, artificial intelligence model, evaluator and criterion. Setting: The United States. Participants: Six WIC expert evaluators from academia, government, industry and non-profit sectors. Results: Five themes were identified: administrative barriers, nutrition outcomes, participant preferences, economics and other topics. Feeding and non-feeding groups had no significant differences (Coeff. = 0·03, P = 0·52). MixerBox-generated questions received significantly lower scores than ChatGPT (Coeff. = –0·11, P = 0·02). Ethics scores were significantly higher than feasibility scores (Coeff. = 0·65, P < 0·001). Significant differences were found between the evaluators (P < 0·001). Conclusions: The LLM applications can assist in developing research questions with acceptable qualities related to the WIC food package revisions. Future research is needed to compare the development of research questions between LLM and human researchers.