Cracking the code: AI’s role in mapping evidence syntheses to the United Nations sustainable development goals

Abstract Background Understanding the alignment and contributions of research to the United Nations Sustainable Development Goals (UN SDGs) is essential for guiding global progress toward these critical targets. Several SDG mapping approaches have been proposed and tested by organisations and researchers but have not produced consistent results. With its capacity to analyse vast datasets and identify patterns, AI-powered search functionality has been presented as an innovative mechanism for tracking, analysing and reporting SDG research to assess progress towards targets and facilitate evidence-based decisions. This study aimed to assess the reliability of automated mapping approaches utilised by online research databases in mapping published evidence syntheses to the UN SDG-3, Good Health & Wellbeing. Methods This study mapped systematic and scoping reviews published in JBI Evidence Synthesis to SDG- 3, Good Health and Wellbeing. Four unique raters independently assessed 204 evidence syntheses based on relevance to SDG-3. These four raters included AI in three established databases and a manual ‘human’ assessment. Inter-rater reliability was assessed using Light’s Kappa. Results Concurrence occurred for 52% of publications. Inter-rater reliability indicated ‘minimal agreement’ among the four raters in mapping the 204 evidence syntheses to SDG-3. Discrepancies in the publications mapped to SDG-3 across the four raters may be explained by the different taxonomies used by the databases, different machine learning algorithms, and constantly evolving search strategies. Our results indicate significant room for improvement in achieving greater consensus among approaches, including AI algorithms designed to identify such publications. Conclusion The findings of this study point to the need for SDG mapping tools that are practical and effective, as the 2030 agenda deadline nears and progress on numerous targets lags. Identifying evidence across the global ecosystem is critical, but AI’s reliability to contribute to this has yet to be established with confidence. It signals to the scientific community, policymakers, funders and others the importance of critically reflecting on how research is captured and aligned to the SDGs.