The views and opinions expressed or implied in WBY are those of the authors and should not be construed as carrying the official sanction of the Department of Defense, Air Force, Air Education and Training Command, Air University, or other agencies or departments of the US government or their international equivalents.
By Capt Hannah M. Ferrarini & Capt Christian P. Ferrarini
/ Published September 03, 2021
Talent Management (TM) is on the brink of massive disruption. Technology has advanced to the point where the Air Force will be able to provide personalized attention to every single Airman, providing them the development, opportunities, and pathways to success to a degree unmatched by current methodologies. The use of artificial intelligence (AI), combined with strategic insights, will transform every aspect of how we do business through informed data-driven decisions. This prospectus gives a brief overview of the current state of AI and, through a practical analysis of where we are today, why the Air Force has every reason to be optimistic about the future.
The goal of AI is to empower computer systems to perform higher levels of intellectual functions that were previously thought only possible by humans. While there are many advances that fall under the umbrella of AI, the primary tool the Air Force will need to harness for TM is Machine Learning (ML). ML is an approach that is best suited when it is not practical, possible, or preferential to create an algorithm to tell the computer how to perform a specific task. Instead, ML algorithms empower the system to find patterns, truths, and weights for significance from the large sets of data poured into them. Common ML algorithms someone might interface with are song recommendations and navigation routes. More complex than this example would be creating an algorithm for every career field and Airmen considering all factors to best match them to their assignment. Preferences are one thing, knowing what is best in a holistic sense of individual development and the needs of the future force are quite another.
The advantages of using AI/ML converge on three opportunities: knowledge, value, and force development. AI/ML provides knowledge that would otherwise be hidden, obtained through analyzing the entirety of information on the collective force. The value that AI/ML provides is the sense of worth gained by the individual Airmen. Through a deeper understanding of our force as individuals, we can put them in the right place, with the right skill set, at the right time in their career. Put another way, AI will allow the Air Force to create a plan for every Airman. Finally, AI/ML allows for advanced force development. Much like how AI can win a chess match against the grandmasters of the game through its advanced understanding of strategy and probabilities, AI/ML can help the Air Force create a strategy for developing the force to meet the needs of the future. Through prediction and thoughtful planning, AI can assist the senior leaders of today with the long-range career development insight necessary to provide the Air Force, the generals, and the senior noncommissioned officers to lead and win future wars.
The Air Force has already shown an appetite for developing AI in the realm of TM to take advantage of emerging technologies. While at Air University, Lt Col William Clayton explored how ML can be applied to Professional Military Education boards. His 2020 study found that there is potential for AI to increase the efficiency of selection board processes through a ML algorithm scoring concurrently with board members. In 2021, the RAND Corporation released its findings on how ML can assist in TM by developing a performance index. Over the course of two years, they developed an AI scoring system to attempt to mimic how senior officers use Officer Performance Reports (OPRs) to determine how well officers perform in comparison to one another. Ultimately, these efforts faced challenges due to data and vision. While not proving ready for full TM implementation, the efforts of RAND and Lt Col Clayton demonstrate that the Air Force is actively exploring the possibilities. Fortunately, the Department of Defense (DOD) is actively taking steps to reconcile these shortcomings.
Starting at the top, Congress understands the urgent need to progress quickly with AI development. The 2021 National Defense Authorization Act (NDAA) includes further specifications on the already established charter for the DOD to further develop AI technology. The NDAA charges the Joint AI Center to research, develop, and transition activities to mature AI systems. The Director of Information Innovation Office at the Defense Advanced Research Projects Agency, Mr. William Scherlis, recently spoke to the current state of AI development within the DOD at the Defense One Genius Machines 2021 summit. There, he acknowledged that the military is only in the early stages of AI development, but it needs to start moving fast. To this end, current and future AI development teams need access to vast amounts of high-quality data.
Before going further into the specific types of data required, it is important to know why exactly data matters for AI/ML development. AI/ML mimics the act of learning to reach novel conclusions. Just like humans, AI/ML needs information from which to learn—i.e. data. Data refers to a description of something (words, numbers, etc.) that allows it to be recorded, analyzed, and recognized. Programmers empower systems to search for patterns and truths about data provided, which can ultimately be trained to a desired outcome. To have AI/ML useful for TM, data is needed and in large amounts. Opportunely for the DOD and the Air Force, data on the force is not in short supply, just a little messy.
Messy can mean a lot of different things when talking about data. To oversimplify the issue, with the advent of computers the DOD did not anticipate and effectively plan for how they should regulate all the information and data they produced. As a result, the state of data in the DOD needed to be reformed. To address the issues, the DOD released a comprehensive data-management plan entitled DoD Data Strategy in 2020. This document provides “overarching vision, focus areas, guiding principles, essential capabilities, and goals necessary to transform the Department into a data-centric enterprise.” The strategy further defined future acquisition and utilization of data AI training as one of its Eight Guiding Principles. Moving forward, data must be Visible, Accessible, Understandable, Linked, Trustworthy, Interoperable, and Secure (VAULTIS) to be useful for analysis and algorithm development. As previously stated, today’s legacy systems fail to meet those criteria. Therefore, the DOD committed to only procuring systems that are data-interoperable, software upgradable, and cloud-ready moving forward. Beyond the goal of developing comprehensive data-management and acquisition policies for future, the DOD must find a way to utilize already acquired data held in legacy systems.
To make data more VAULTIS, Air Force Digital Transformation Activity is pursuing a cloud-based data repository called Talent Management Data Environment (TMDE). The goal of TMDE is to allow the Air Force to consolidate information from hundreds of personnel legacy systems and enable access to structured and unstructured data in a centralized location. Currently, there are 118 systems that contain personnel data relevant to TM. TMDE will take all the data from these databases and centralize them, allowing for advanced analytics and correlations to be derived. Right now, to get data, analysts are hindered by the arduous process of extracting data they think they might need for their algorithms. The roll-out of TMDE will be invaluable to future AI development efforts. Until that time, the Air Force will be hindered in any effort to develop TM AI.
The research efforts of RAND and Lt Col Clayton highlighted that a lack of usable data is a detrimental roadblock to TM AI development. The problems with the current state of data can be simplified into three key areas: ownership, unlocking, and attribution. The problems of ownership refer to limited access to information currently stove-piped in the 118 personnel legacy systems. Many of the Air Force organizations that contracted these systems and the private companies that built them believe that they own the data and often restrict access to their information. This creates an obvious problem for data analysis because each system only provides a piece of the picture for the talent management needs of each Airmen.
The problems of unlocking are associated with taking the information provided by legacy systems and turning it into usable datapoints. Data associated with Airmen is stored in formats that do not lend themselves to large-scale data analysis. One striking example of this is the previous method of storing OPRs. Before recent changes, OPRs were uploaded into the Automated Records Management System as low-quality pictures as opposed to formatted PDF files with meta-data. Especially in terms of TM efforts, OPRs are the definitive way that the Air Force tracks and measures performance. Without that data unlocked, TM insights are shallow at best. Luckily for the Air Force, Natural Language Processing (NLP) programs can be used to datify the content of the reports. In other words, the text from the OPR can be transformed into a format that can be analyzed. However, if the quality of an uploaded picture is too low, computer programs are unable to extract the data points that analysts require. This challenge combined with the computer processing effort to run an NLP makes it a daunting task to gather enough datapoints over an adequate timeframe to create predictive algorithms. Ultimately, the data in OPRs as well as other text documents will need to become unlocked before AI will be able to be applied to TM.
Attribution refers to ensuring that collected datapoints are connected to the right Airmen. This problem exposes the lack of standard organization and interconnectivity of legacy systems. Each system collects valuable data on each Airmen, but it does not organize the information in the same way. This is because they were not required to be able to interact with each other or export data in a standard manner. For instance, some systems organize users by name, others go by DODID number, and others go by SSN. To exacerbate the problem, how systems store information like SSN can differ (XXX-XX-XXXX, XXXXXXXXX, -----XXXX). This can make it extremely difficult for data-extraction software and even humans to distinguish what information belongs to what Airmen. Similar to the issue of unlocking, until data-extraction software algorithms become advanced enough, it will take human intervention to organize and attribute data stored in legacy systems. Future systems such as TMDE will mitigate many of these problems, but the Air Force must work towards a short-term solution in the meantime.
While the Air Force continues to develop their own solutions, the private sector races ahead. Companies such as IBM successfully integrated AI into the entire TM life cycle. For example, IBM utilized AI to map, with 85-95 percent accuracy, the skills of each employee by scraping the digital interface of resumes, sales info, and training.1 This information helps IBM understand what skills are needed in the organization in order to quickly close skill gaps and find the ‘hidden gems’ in the organization. This is just one of the many ways AI/ML is successfully being employed. Other companies are trying to mimic IBM’s success with mixed results.
It is essential to remember that models are only as good as the data to which they are trained. There is a possibility that data can be riddled with biases and AI will perpetuate them. Amazon’s resume vetting AI epitomized this challenge when their algorithm failed to value resumes from female applicants equally. They found that the data, collected over a ten-year period, used to train their AI consisted of resumes of mostly men. The program was eventually disbanded due to the inability to train the AI to be unbiased. As the Air Force starts seeing the results of their AI/ML, the leaders need to ensure that the outcomes used reflect what is truly valued.
As the data challenges continue to be resolved, Air Force leaders need to begin to grapple with their vision for the force of the future. As mentioned previously, AI/ML systems must be trained to outcomes. The input data is not enough on its own to provide meaningful insights, it must be focused to a goal. For example, what makes a ‘successful’ assignment? While a binary metric for this question is not required for AI/ML to thrive, consideration must be made for what the goals are of the assignment and a way to datify if those outcomes are achieved. Or perhaps, how will the Air Force measure if an officer has the leadership aptitude required for command? These questions can be applied over the whole range of the TM spectrum but are not easy to answer.
AI is here. The opportunities it presents are apparent and worth being excited about. This is not to say that every venture in AI in TM will provide exponential gains or that every aspect should be pursued. There is evidence, however, that the journey will be worth it. Through incremental implementation and mending data, the Air Force can create an environment where AI can thrive. An environment where leaders fine-tune how the Air Force rewards talent, producing outcomes aligned with the organization’s values. Leaders that are equipped with insights, garnered through AI, will be able to develop Airmen like never before. Airmen that will be ready for whatever fight the future throws at them.
Captain Hannah Ferrarini, USAF
Captain Hannah Ferrarini (MS, Colorado State University; BS, USAF Academy) is a Force Support Officer assigned to the 87th Air Base Wing at Joint Base McGuire-Dix-Lakehurst with various experiences across the A1 disciplines; contributing to her insights and perspectives in this piece.
Captain Christian Ferrarini, USAF
Captain Christian Ferrarini (MPP, Harvard; BS, USAF Academy) is KC-10 Pilot assigned to the 2d Air Refueling Squadron at Joint Base McGuire-Dix-Lakehurst. He has over 950 flight hours logged in T-6, T-1, and the KC-10 aircraft, and has flown in a wide array of national and international refueling missions including those in Operation Inherent Resolve. His extensive background in policy served a critical role in this piece.
This paper was written as part of the SOS Air University Advanced Research (AUAR) elective, Artificial Intelligence section
 Michael Kanaan, T- Minus AI: Humanity’s Countdown to Artificial Intelligence and the New Pursuit of Global Power (Dallas, Texas: BenBella Books, Inc., 2020), 191.
 Kanaan, T- Minus AI, 120–21.
 Kanaan, T- Minus AI, 121.
 Kanaan, T- Minus AI, 118.
 Kanaan, T- Minus AI.
 William Clayton and Joseph King, “Improving the Development Education Selection Board Process with Machine Learning,” Institute for Defense Analyses, May 2020.
 David Schulker et al., “Can Artificial Intelligence Help Improve Air Force Talent Management? An Exploratory Application,” RAND Cooperation, 2021, https://doi.org/10.7249/RRA812-1.
 Adam Smith, “H.R.6395 - 116th Congress (2019-2020): National Defense Authorization Act for Fiscal Year 2021” (116th Congress, 1 January 2021), 2019/2020, https://www.congress.gov/.
 David Vergun, “Artificial Intelligence Is a Work in Progress, Official Says,” US Department of Defense, 22 January 21, https://www.defense.gov/.
 Kanaan, T- Minus AI, 118.
 Viktor Meyer-Schonberger and Kenneth Cukier, Big Data (New York: Houghton Mifflin Harcourt Publishing Company, 2013), 78.
 Kanaan, T- Minus AI, 121.
 David L Norquist, “DOD Data Strategy,” n.d., 16.
 Norquist, “DOD Data Strategy.”
 Sergio Rios, Talent Management in the AF, interview by author, March 2021.
 Carlos Davila, The Status of TMDE, interview by author, March 2021.
 Meyer-Schonberger and Cukier, Big Data, 36.
 Meyer-Schonberger and Cukier, Big Data, 78.
 Justin Joffrion, Data Challenges, interview by author, March 2021.
 Jeffrey Dastin, “Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women,” Reuters, 10 October 2018, https://www.reuters.com/.
 Dastin, “Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women.”
Wild Blue Yonder Home