A Retrieval-Augmented Generation Method for Question Answering on Airworthiness Regulations

Civil aviation airworthiness regulations are the fundamental basis for the design and operational safety of aircraft. Their provisions exhibit a high degree of specialization, cross-disciplinary complexity, and hierarchical structure. Moreover, the regulations are frequently updated, posing unique challenges for automated question-answering systems. While large language models (LLMs) have demonstrated remarkable capabilities in dialog and reasoning; however, they still face challenges such as difficulties in knowledge updating and a scarcity of high-quality domain-specific datasets when tackling knowledge-intensive tasks in the field of civil aviation regulations. This study introduces a retrieval-augmented generation (RAG) approach that integrates retrieval modules with generative models to enable more efficient knowledge acquisition and updating, encompassing data processing and retrieval-based reasoning. The data processing stage comprises document conversion, information extraction, and document parsing modules. Additionally, a high-quality airworthiness regulation QA dataset was specifically constructed, covering multiple-choice, true/false, and fill-in-the-blank questions, with a total of 4688 entries. The retrieval-based reasoning stage employs vector search and re-ranking strategies, combined with prompt optimization, to enhance the model’s reasoning capabilities in specific airworthiness certification regulation comprehension tasks. A series of experiments demonstrate the effectiveness of the retrieval-augmented generation approach in this domain, significantly improving answer accuracy and retrieval hit rates.