A Systems Engineering Methodology for System of Autonomous Systems: Test and Evaluation

ABSTRACT Recent advances in artificial intelligence and machine learning (AI/ML) have resulted in autonomous systems that reduce operators' workload and involvement in hazardous missions. Integrating these systems into an existing system of systems (SoS) can evolve it into a system of autonomous system (SoAS). SoAS brings new Systems Engineering (SE) challenges for architecture development, integration, testing, and evaluation that originate from the level of autonomy (LoA). LoA refers to the level of autonomous capabilities of a system depending on its AI/ML technology. This paper examines SoAS test and evaluation challenges, such as uncertainty and emergent behaviors. We propose a test and evaluation method that provides stakeholders with a decision analysis tool to make data‐driven decisions by exploring the SoAS design space, comparing different alternative solutions under uncertainty, and selecting the most suitable one that addresses their mission needs. The proposed method is a step towards AI for Systems Engineering (AI4SE). It applies the Bayesian network (BN) to SoAS by integrating ML algorithms and Model‐Based Systems Engineering (MBSE) executable models. The decision analysis tool provides two types of examinations as follows: predictive analysis to select the most suitable SoAS alternative solution that yields the desired performance; and prescriptive analysis to identify root causes of an SoAS undesired emergent behavior and define preventive strategies for future operations. We present a conceptual example of a search‐and‐rescue mission to demonstrate the implementation and effectiveness of the proposed method in evaluating SoAS with varying LoAs.