By Patrick Hall* and Andrew Burt**
* Patrick Hall is a principal scientist at bnh.ai, a boutique law firm focused on AI and analytics, and an adjunct professor in the Department of Decision Sciences at GWU.
** Andrew Burt is a managing partner at bnh.ai and chief legal officer at Immuta.
Artificial intelligence can fail. It can also be attacked. When a failure or attack spins out of control, this is a major AI incident. There have been over 1,000 public reports of AI incidents in recent years. Yet many organizations are operating nascent AI efforts without incident response plans – using AI, in other words, without any clear understanding of what to do when it fails.
Why? Organizations simply aren’t thinking about AI failures, they’re focused on AI successes instead. Indeed, there’s a great deal of hype on the positive side of this technology, and deservedly so. It can make and save money, and it can even be transformational. However, AI is probably more likely to fail than traditional enterprise software, at least as of today. It’s for this reason that some have called the technology the “high-interest credit card of technical debt.” Governments around the world are increasingly interested in enforcing best practices for AI. And numerous damaging attacks against AI systems have already been published in machine learning and security research journals.
Our bet is you’ll be hearing more about AI incidents in the coming years. Below, we’ll go over why AI is (and is not) different from more traditional software systems, some of the primary lessons we’ve learned writing AI incident response plans, and we’ll introduce the free and open bnh.ai Sample AI Incident Response Plan to help you make your organization better prepared for AI incidents.
How AI Is (and Is Not) Different
What’s so different about AI? Basically, it’s much more complex than traditional software, it has a nasty tendency to drift toward failure, and it’s often based on statistical modeling. What does that really mean?
More complexity: For starters, AI systems can have millions or billions of rules or parameters that consider combinations of thousands of inputs to make a decision. That’s a lot to debug and it’s hard to tell if an AI system has been manipulated by an adversary.
Drift toward failure: Most AI systems are trained on static snapshots of the world encapsulated in training datasets. And just in case you haven’t noticed, the world is not a particularly static place. As the world changes, the AI system’s understanding of reality becomes less and less valid, leading to degrading quality of decisions or predictions over time. This is known as “model decay” or “concept drift,” and it applies to nearly all current AI systems.
Probabilistic outcomes: Most AI systems today are inherently probabilistic, which means that their decisions and predictions are guaranteed to be wrong at least some of the time. In standard software, wrong outcomes are bugs. In AI, they are features. This makes testing and establishing tolerances for failure more difficult.
The combination of these three characteristics present a number of testing difficulties, potential attack surfaces and failure modes for AI-based systems that often are not present in more traditional software applications.
If that’s what’s different, then what’s the same?
In the end, AI is still just software. It’s not magically exempt from the bugs and attacks that plague other software, and it should be documented, tested, managed, and monitored just like any other valuable enterprise software asset. This means that AI incident response plans, and AI security plans more generally, needn’t reinvent the wheel. Frequently they can piggyback on existing plans and processes.
What We Learned About AI Incident Response
Drafting AI incident response plans have been eye-opening, even for us. Inputting to paper for our customers all the various ways AI can fail and its many attack surfaces, we’ve learned several big lessons.
Neither MRM Nor Conventional lR is Enough
The basics of our AI incident response plans come from combining model risk management (MRM) practices, which have become fairly mature within the financial industry, with pre-existing computer incident response guidance and other information security best practices. MRM helps protect against AI failures. Conventional incident response provides a framework to prepare for AI attacks. These are both great starts, but as we detail below, a simple combination of both is still not quite right for AI incident response. This is why our Sample AI Incident Response plan includes guidance on both MRM and traditional computer incident response, plus plans to handle novel AI risks in the context of the burgeoning AI regulation landscape in the US.
MRM practices, illustrated in, among other places, the Federal Reserve’s Supervisory Guidance on Model Risk Management, known as SR 11-7, are an excellent start for decreasing risk in AI. (In fact, if your organization is using AI and is not familiar with the SR 11-7 guidance, stop reading this article and start reading the guidance.) Broadly, MRM calls for testing of AI systems, management of AI systems with inventories and documentation, and careful monitoring of AI systems once they are deployed. MRM also relies on the concept of “effective challenge” – which consists of models and processes being questioned and reviewed by humans in multiple lines of technology, compliance, and audit functions. However, MRM practices do not specifically address AI security or incident response, and they often require resources not available to smaller organizations.
We’ll address incident response for smaller organizations in the next section, but from an information security perspective, traditional incident response guidance is helpful – though not a perfect fit. For instance, AI attacks can occur without traditional routes of infiltration and exfiltration. They can manifest as high usage of prediction APIs, insider manipulation of AI training data or models, or as specialized trojans buried in complex third-party AI software or artifacts. Standard incident response guidance, say from SANS or NIST, will get you started in preparing for AI incidents, but they also weren’t specifically designed for newer attacks against AI and could leave your organization with AI security blindspots.
When Going Fast and Breaking Things Goes Wrong
MRM practices require serious resources: lots of people, time, and technology. Standard MRM may not be feasible for early-stage or small organizations under commercial pressure to “go fast and break things.” Common sense indicates that when going fast and breaking things, and without conventional MRM, AI incidents are even more likely. With AI incident response, smaller organizations without the capability for heavy-handed supervision on the build side of AI can spend limited resources in a manner that allows them to stay agile while also confronting the reality of AI incidents.
Attitude Adjustments
There is a lot of hype surrounding AI and the profession of data science. This hype, coupled with lax regulatory oversight has led to a wild west of AI implementations that can favor the kitchen sink over the scientific method.
A hype-driven sense of entitlement can sometimes lead to friction and resistance from front line AI practitioners. We’ve found that some practitioners are unwilling or unable to understand that, despite their best intentions, their AI systems can fail, discriminate, get hacked, or even worse. There’s not much to say about this except that it’s time for the commercial practice of AI to mature and accept that with increasing privilege comes increased responsibility. AI can and is already starting to, causes serious harm. As of today, compliance, legal, security, and risk functions in large organizations may have to make manual attitude adjustments and insist that AI groups are subject to the same level of oversight as other IT groups, including incident response planning for AI attacks and failures.
Don’t Deploy AI Without an Incident Response Plan
The final takeaway? AI is not magic — meaning organizations can and should govern it. If AI is the transformative technology it is hyped to be (and we do believe it is), then deploying AI with no incident response plans is a recipe for disaster. After all, we don’t fly commercial jetliners without detailed plans for systems failures; we don’t run nuclear reactors without emergency plans; if the activity is important to us, we think and plan in advance about its risks.
And that means we also need to be prepared for AI to fail. Having an AI incident response plan in place can be the difference between an easily manageable deviation in AI system behavior and serious AI-driven harm and liabilities.
About the Authors
Andrew Burt is a managing partner at bnh.ai, a boutique law firm focused on AI and analytics, and chief legal officer at Immuta. He is also a visiting fellow at Yale Law School’s Information Society Project.
Previously, Andrew was a Special Advisor for Policy to the head of the FBI Cyber Division, where he served as lead author on the FBI’s after-action report on the 2014 Sony data breach, in addition to serving as chief compliance and chief privacy officer for the division.
A frequent speaker and writer, Andrew has published articles on law and technology for the New York Times, the Financial Times , and Harvard Business Review, where he is a regular contributor. He holds a JD from Yale Law School.
Patrick Hall is a principal scientist at bnh.ai, a boutique law firm focused on AI and analytics. Patrick also serves as an advisor to H2O.ai and as an adjunct professor in the Department of Decision Sciences at The George Washington University.
Before co-founding bnh.ai, Patrick led H2O.ai’s efforts in responsible AI, resulting in one of the world’s first widely deployed commercial solutions for explainable and fair machine learning. He also held global customer-facing roles and R&D research roles at SAS Institute. Patrick studied computational chemistry at the University of Illinois before graduating from the Institute for Advanced Analytics at North Carolina State University.
