A Practical Approach to Transitioning 5G-AKA Toward Fully and Hybrid Post-Quantum Security for 5G and Beyond Communication

Advancements in 5G are largely driven by its adoption across a variety of applications, including smart homes, smart healthcare, autonomous vehicles, transportation, and Industry 5.0. However, this widespread usage raises significant security and privacy concerns, particularly regarding authentication. To secure 5G communications, 3GPP has standardised the 5G-Authentication and Key Agreement (5G-AKA) protocol. Despite this, state-of-the-art research has revealed that 5G-AKA suffers from several vulnerabilities. In response, numerous alternative authentication schemes have been proposed. Nevertheless, many of these protocols continue to be susceptible to classical security threats, such as malicious serving network (SN) attacks, ephemeral secret leakage, traceability, anonymity violations, and, most critically, a lack of Perfect Forward Secrecy (PFS). Additionally, recent advancements in quantum computing pose a substantial risk to the security of 5G-AKA and its variants, rendering them insecure in the post-quantum era. Although a few post-quantum cryptography-based protocols have been introduced, studies show that they are either computationally expensive or still vulnerable to quantum attacks. In order to tackle these challenges, we propose two versions of a novel authentication protocol that not only provide comprehensive security guarantees but are also computationally efficient. Version 1 utilises a hybrid cryptographic approach (AES + KEM) and achieves the lowest computational cost—even lower than Version 2, which relies solely on KEM. Both versions offer the same level of security, but Version 1 is significantly more efficient in terms of resource usage. We have validated our proposed protocols using the Real-Or-Random (ROR) model and the Scyther tool. Furthermore, we evaluated their performance against existing protocols under both known and unknown attack scenarios. The results show that our protocols incur lower computational, communication, and energy overhead, as well as strong resilience against unknown attacks, outperforming several existing authentication schemes. Furthermore, we implement a testbed to demonstrate the real-time applicability of both proposed versions.