Test and Evaluation Challenges in Artificial Intelligence-Enabled Systems for the Department of the Air Force

The Department of the Air Force (DAF) is in the early stages of incorporating modern artificial intelligence (AI) technologies into its systems and operations. The integration of AI-enabled capabilities across the DAF will accelerate over the next few years.

At the request of DAF Air and Space Forces, this report examines the Air Force Test Center technical capabilities and capacity to conduct rigorous and objective tests, evaluations, and assessments of AI-enabled systems under operational conditions and against realistic threats. This report explores both the opportunities and challenges inherent in integrating AI at speed and at scale across the DAF.

Table of Contents

Front Matter Summary 1 Introduction 2 Definitions and Perspectives 3 Test and Evaluation of DAF AI-Enabled Systems 4 Evolution of Test and Evaluation in Future AI-Based DAF Systems 5 AI Technical Risks Under Operational Conditions 6 Emerging AI Technologies and Future T&E Implications 7 Concluding Thoughts Appendixes Appendix A: Statement of Task Appendix B: Public Meeting Agendas Appendix C: Committee Member Biographical Information Appendix D: Acronyms and Abbreviations Appendix E: Testing, Evaluating, and Assessing Artificial IntelligenceEnabled Systems Under Operational Conditions for the Department of the Air Force: Proceedings of a Workshop-in Brief