Automatic feature‐based markerless calibration and navigation method for augmented reality assisted dental treatment

Abstract Augmented reality (AR) is gaining traction in the field of computer‐assisted treatment (CAT). Head‐mounted display (HMD)‐based AR in CAT provides dentists with enhanced visualisation by directly overlaying a three‐dimensional (3D) model on a real patient during dental treatment. However, conventional AR‐based treatments rely on optical markers and trackers, which makes them tedious, expensive, and uncomfortable for dentists. Therefore, a markerless image‐to‐patient tracking system is necessary to overcome these challenges and enhance system efficiency. This paper proposes a novel feature‐based markerless calibration and navigation method for an HMD‐based AR visualisation system. The authors address three sub‐challenges: firstly, synthetic RGB‐D data for anatomical landmark detection is generated to train a deep convolutional neural network (DCNN); secondly, the HMD is automatically calibrated using detected anatomical landmarks, eliminating the need for user input or optical trackers; and thirdly, a multi‐iterative closest point (ICP) algorithm is developed for effective 3D‐3D real‐time navigation. The authors conduct several experiments on a commercially available HMD (HoloLens 2). Finally, the authors compare and evaluate the approach against state‐of‐the‐art methods that employ HoloLens. The proposed method achieves a calibration virtual‐to‐real re‐projection distance of (1.09 ± 0.23) mm and navigation projection errors and accuracies of approximately (0.53 ± 0.19) mm and 93.87%, respectively.