Assessing the effectiveness of artificial intelligence education and training for healthcare workers: a systematic review

Abstract Background Artificial intelligence (AI) is increasingly integrated into healthcare, yet upskilling the health workforce remains a challenge. We addressed the research question: What evidence exists on the effectiveness of AI education and training programs in improving AI literacy among healthcare workers? Methods Following PRISMA guidelines and PROSPERO registration, five databases (PubMed, Scopus, CINAHL, Embase, ERIC) were searched on 20 August 2024, focusing on studies with an intervention of AI training or education for the healthcare workforce, in any study design that reported an evaluation. Results 27 studies were included. Programs improved AI literacy outcomes mapped to levels 1–3 of the Kirkpatrick-Barr training evaluation hierarchy including improved learner reactions, shifts in attitudes and perceptions, enhanced knowledge and skills, and behavior changes. Programs did not map to level 4, where healthcare workers learn to metacognition levels, including organizational change and patient benefit. Programs were short in length (44%), delivered in academic settings (56%), to doctors (44%) or medical students (44%), at entry-to-practice level (56%). Most taught an introduction to AI (67%), with technical AI skills less frequent. Conclusions These programs are a promising start but often lack sufficient depth to build advanced competencies. Improving AI literacy in healthcare will require appropriate course design, an evolving understanding of this rapidly changing area, and evaluating learning effectiveness. As the adoption of AI accelerates across healthcare, health systems may seek to standardise and assess the efficacy of these courses.