When AIs Fail to Act Human: A Legal Framework to Determine the Reasonableness of Artificial Intelligence Actions Published Aug. 13, 2021 By Capt Christy Heath Wild Blue Yonder -- Introduction Imagine your boss has just requested that you take a package to a customer and instructs you to utilize the company’s self-driving car to carry out this feat. Your boss hands you the package before giving you the customer’s address to input it into the car’s navigational system, which you do. The car carefully drives out of the parking lot and onto a nearby road, but as it approaches an intersection, you realize an overgrown shrub is obstructing the stop sign. The car fails to stop and proceeds through the intersection. Shortly after, the self-driving car enters an on-ramp for a busy interstate. You observe a temporary construction area in the process of being dismantled, but clearly speed limits are not impacted because cars are proceeding at normal speeds. Before workers can remove a temporarily posted sign indicating a much lower speed limit, your car reduces its speed.Transiting vehicles quickly swerve around you in order to avoid an accident and honk their horns disapprovingly. As you exit the interstate, you approach a traffic intersection and see the traffic light is red. Your car fails to stop and proceeds right into oncoming traffic, almost colliding with a school bus. You later find out someone, in an attempt to create public distrust in all self-driving vehicles, placed a translucent piece of tape over the traffic light. The tape did not impact the color of the light for drivers but did cause self-driving cars to interpret the color of the traffic light as green. When you finally approach the customer’s address, the customer is waiting and walks out to greet you. The self-driving car, however, down not slow-down in spite of the presence of the customer and, instead, runs into them, causing significant injuries. You call 911 and the injured customer is subsequently transported to the hospital by ambulance. Legal Framework: A Four-Prong Analyses This scenario contains several instances in which the self-driving car’s Artificial Intelligence system (AI) failed to act in accordance with how, generally, a human would have acted or would have known how to act. Each point of failure invokes a different level of liability or culpability, and potential application of law. In order to evaluate the reasonableness of an AI’s actions, surveyors should turn to a four-prong analysis: (1) was the information received by the AI prima facie true; (2) did the AI know how to interpret the information; (3) was the information received by the AI honest; and (4) were the actions of the AI reasonable? These four prongs should help determine not only what went wrong, but where to allocate fault, and what area of law should apply in response to facts surrounding that failure. Was the information received by the AI prima facie true? This question intends to answer whether, at face value, the information received by the AI was what it appeared to be. It does not intend to answer whether the information was honest. To illustrate this point, think about the following: you paint a tree’s leaves purple. Later, a passerby observing your unique creation states, “Look at that! I’ve never seen a tree with purple leaves.” If you were asked whether the individual saw a tree with purple leaves (was the information received prima facie true?), the answer would be yes because the tree does appear to have purple leaves, even though the color of the leaves is not honest. If the individual looked at the tree and said, “Look at that tree with blue leaves!” then the information received by the individual would be prima facie false. In the self-driving car scenario, examine the first instance of AI failure or when the self-driving car does not stop for the stop sign due to the overgrown shrub. On its face, the visual information presented to the self-driving car indicated that there was no stop sign; so, the information received by the self-driving car was prima-facie true—it appeared there was no stop sign. Thus, we can answer this question in the affirmative. Now assess the scenario’s instance of AI failure involving the traffic light. The light appeared to be red, but the car interpreted the light as green. In this instance, the answer to the question of whether the information received by the self-driving car was prima facie true is no. The light appeared to be red, but the car did not receive that information properly. Did the AI know how to interpret the information? After determining if the AI received prima facie true information, it must be decided whether it knew how to interpret that information. For instance, if the self-driving car had been imported from China and contained programming language only housing information about China’s traffic rules, then the self-driving car may not have known how to interpret American road signs or traffic lights. In our scenario, despite the stop sign obscured by the overgrown shrub, perhaps the self-driving car had been programmed with certain reasoning algorithms based on additional visual information it received (it was at an intersection, there were signs posted in other directions indicating it was a 2-way or 4-way stop, etc.) that should have resulted in the car stopping because it knew how to interpret that additional information. If such was the case, further investigation would be necessary to explain why the AI failed to act in agreement with its programming. Even when an AI knows how to interpret information, a question may still arise about whether the AI should have interpreted the information in the way that it did, in light of the present circumstances. When the self-driving car failed to proceed at normal interstate speeds as it approached temporary road construction, the AI obviously interpreted the posted speed limit sign correctly, but there may be some argument about whether it should have interpreted the speed limit sign in that way. Should the self-driving car’s AI system have been programmed to mimic the action of the vehicles around it? Should it have been programmed to anticipate when a human was going to remove a speed limit sign? These are questions that can be explored when determining whether the AI knew how to interpret the information at all. Was the information received by the AI honest? If the information received by the AI was what it appeared to be, and the AI knew how to interpret that information, then why did the AI fail to act in accordance with that information? The purpose of the third prong is to determine whether an intervening force manipulated the actions of the AI by presenting fraudulent information. In the self-driving car scenario, the car approached a red traffic light, which it had presumably been programmed to understand how to interpret; yet the car failed to interpret the red traffic light as it appeared. The scenario explains that the light had been tampered with in such a way to cause the self-driving car to interpret the signal in a different way from the way it appeared. Yet, the self-driving car is not completely absolved of responsibility based on that information alone. If the answer to this prong is no, a sub-analysis should be explored: (1) did the AI know how to distinguish false information and, if not, (2) should the AI have known how to distinguish false information? If the AI knew how to distinguish the false information, but failed to conform its behavior accordingly, it is likely that the culprit is a defect or malfunction. Whether or not the AI should have known how to distinguish the false information is a foreseeability question that points to a failure in the AI’s design. Were the actions of the AI reasonable? In law, the customary way to measure whether an act perpetrated by a human was reasonable is called the “reasonable person standard.” This standard is often used in criminal, tort, and contract law as a way to determine negligence and intent. It is a legal doctrine positing that, in any given situation, each person has a duty to behave as a reasonable person in the same or similar circumstances. Whether an action is reasonable is often considered through two lenses: subjective and objective. The subjective lens examines the action from the point of view of the actor. The objective lens examines the action from the point of view of the audience. The purpose of the first three prongs of the analysis is to help determine the answer to the fourth prong. For instance, in the self-driving scenario, the car received prima facie true information when the customer came out to receive their package—there was actually a person in the path of the car. The AI should have been programmed to interpret that information (though consultation might be necessary to confirm), and the information was honest—there was actually a person in the path of the car. The question of whether the actions of the AI were reasonable must then be assessed. In this instance, the reasonableness is clear: both from the point of view of the self-driving car, and from the point of view of the audience, the actions of the self-driving car (its AI system) are unreasonable. Once a determination has been made that the self-driving car acted unreasonably, understanding why the car acted unreasonably can then be unpacked. For instance, in the injured customer scenario, one might inquire whether the car recognized that there was a person in its path. If not, was it because the car was not programmed correctly? If so, then this can likely be chalked up to a design issue; liability would fall on the engineers who should have known that people often walk into roadways for various reasons. If the car was programmed to recognize there was a person in its path, did it not stop because of a malfunction? If so, then the manufacturer might be at fault. What if the customer darted out in front of the car while it was still moving? In that instance, then the customer might be liable. What if someone hacked the self-driving car’s programming and disabled a feature in the car to recognize persons in the road? Then the hacker would be criminally liable. The engineer of the car could also be civilly liable, if the engineer knew about a weakness in the AI’s design that caused it to be susceptible to a potential risk of hacking. The Four-Prongs in Relation to DOD AI Ethical Principles In relation to the Department of Defense (DOD), understanding and exploring these four prongs could also prove or disprove adherence to ethical principles recently recommended by the DOD. These recommendations insist that the DoD’s use of AI follow five main principles: Responsible: DOD personnel will exercise appropriate levels of judgment and care while remaining responsible for the development, deployment, and use of AI capabilities. Equitable: The DOD will take deliberate steps to minimize unintended bias in AI capabilities. Traceable: The DOD’s AI capabilities will be developed and deployed such that relevant personnel possess an appropriate understanding of the technology, development processes, and operational methods applicable to AI capabilities, including with transparent and auditable methodologies, data sources, and design procedure and documentation. Reliable: The DOD’s AI capabilities will have explicit, well-defined uses and the safety, security, and effectiveness of such capabilities will be subject to testing and assurance within those defined uses across their entire life cycles. Governable: The DOD will design and engineer AI capabilities to fulfill their intended functions while possessing the ability to detect and avoid unintended consequences, and the ability to disengage or deactivate deployed systems that demonstrate unintended behavior.1 As you work through the self-driving car scenario above and begin to answer questions about whether the car acted reasonably in each instance of failure, you will find that many of the answers will prove adherence to, or divergence from, these ethical principles. For instance, in the scenario, the self-driving car demonstrated that it was not reliable when it did not stop at the stop sign or traffic light. But what about the principle of governable? Traceable? We don’t have all the information from the scenario to determine those principles, but the four-prong analyses is an appropriate place to start investigating. Conclusion The sooner we know how to determine the reasonableness of an AI’s actions, the sooner we can develop AI in accordance with that understanding. The proposed four-prong analysis is a simplistic legal framework to assist AI engineers in understanding the requirements necessary for an AI to function legally and seamlessly in a human setting. Ensuring AIs behave reasonably in everyday situations promotes trust in AI operations, and trust in AI is an area in need of attention and cultivation. AIs do not need to act perfect, and humans do not need to like or love AI to trust it; they only need to know that AIs will act reasonably. The more consistently an AI behaves reasonably—i.e., more human—the more its incorporation into every day human life will become readily accepted. Captain Christina Heath, USAF Captain Christina Heath (Juris Doctor, Florida Coastal School of Law) is the Chief of Military Justice at the 78th Air Base Wing, Robins Air Force Base, Georgia. She manages a nine-person justice team and oversees the administration of all punitive and administrative disciplinary actions involving military members assigned to the installation. This paper was written as part of the SOS Air University Advanced Research (AUAR) elective, Artificial Intelligence section. Notes 1 Defense Innovation Board, “AI Principles: Recommendations of the Ethical Use of Artificial Intelligence by the Department of Defense,” visited 15 December 2020, https://media.defense.gov/.