The ‘Black Box’ Problem of AI
by Davide NG Fania, MD, XTN Cognitive Security
Powerful machine-learning techniques have taken the tech world by storm in recent years. AI-based technologies continue to improve and enhance processes in many fields of application, such as voice and image recognition, machine translation, medical diagnostics, customer interaction and even the way we buy and sell, particularly online. Machine learning (ML) is a subfield of artificial intelligence (AI) interested in making systems autonomously learn how to perform tasks (based on examples, generally speaking- training datasets), and this has resulted in many people believing AI and ML are the same things. However, there is much more to AI than ML alone, just as there are many more aspects of biological intelligence than independent learning skills.
The financial market is one of the key industries expected to be thoroughly transformed by these technologies. Financial institutions are introducing AI into many areas, such as
fraud analysis, loan related risk evaluation, and customer support relations. Having AI support existing processes can produce cost efficiencies combined with better and far more secure service experience for most users; however, new technological opportunities may be hindered by a significant problem. It’s often impossible to explain how ML algorithms (for example deep learning) reached a confident decision. From a service provider perspective, being able to pinpoint the reason for an AI-driven decision is crucial. Furthermore, some regulatory agencies are turning their attention to algorithmic accountability; this is absolutely the case for GDPR (or PSD2 even if limited to the EU) where financial institutions are required to explain why a specific decision has been taken by one of their algorithms.
Comprehensively deploying AI into financial services environments requires keeping in mind intelligibility of results as part of the equation and approaching ML as always evaluating costs and benefits in solving specific problems. In the case of fraud protection, there are several areas of interest in applying AI technology to support human processes:
- Behavioral analysis: many ML algorithms are very useful to detect patterns or anomalies through vast amounts of data (events, transactions) that allow rapid (or near real-time) recognition of typical behaviors and the identification of suspicious
Digital identity validation: ML can be used to implement passive biometrics analysis. These techniques are used to identify users based on biometrics information coming from sensors (e.g., keyboard, mouse or smartphone embedded sensors). Thanks to these checks the end- user could benefit from having less identity-related challenges while accessing the service (no pin codes or 2FA, if not required).
- Replicating human decision flow: AIcanbeusedtosubstitute a human analyst evaluating a suspicious Training these
There are open debates about the use of AI. For example, the AI Now Institute at NYU pressed the core social domains, such as
specific problems extremely well, but at the same time other issues can be better addressed with alternative approaches ML algorithms can be done those responsible for criminal (for example rules-based or by feeding the system with actual decisions made by the analyst and then engaging the algorithm to continue the task autonomously. As previously mentioned, this is surely an area where the decision rationale needs to be crystal clear and tracked as part of the evaluation justice, healthcare, welfare, and education, to halt black-box AIs because their decisions cannot be clearly explained. This situation is further exacerbated by business innovation because the majority of products and services of artificial intelligence companies use complex neural statistical models).
Being able to understand the decision of an AI system, avoiding the “all-black-box” approach, is one of our main goals. Using an extreme example, if a neural network is being used for medical diagnosis and comes up with a process.networks. Decisions are taken “you have cancer” result, you To put this in a context in the real world, think about an end user that is unable to perform an ATM withdrawal, unable to access the online services or to use the mobile app due to some fraud protections restrictions, while the bank (usually help desk) is unable to explain why. Alternatively, think about a potential customer applying for a loan and being rejected without any understandable reason.
These scenarios could have a serious impact on the financial institution’s reputation (mobile app adoption and trust in the service) and produce damaging business impacts. In regulated markets, companies need to be able to explain both internally as well as to external parties (auditors and end-customers) why they’re making decisions and above all for the right reasons. by a “black-box” solution, whose companies declare their algorithms a trade secret and typically do not disclose their operation, where even their designers cannot explain how they arrive at the answers.
Consider for a moment the rationale of why a hybrid approach should be used to comply especially in relation to the financial services market requirements. A modern working AI system often contains parts based on some symbolic, logic-based framework and elements based on ML. IBM’s Watson is a typical example, as much as Google search engine whose search results are enriched by a knowledge base called Knowledge Graph. ML is undoubtedly a good practice, but we do not consider it a silver bullet problem solver. In XTN we believe ML solves would absolutely want to know how it came up with that! XTN provides products that can evaluate unusual events from the very beginning, starting from day one. Up and running ability is achieved thanks to the default training dataset that comes from our decades of expertise and experience in fraud prevention and protection. Over time our technology will always improve its ability and refine accuracy, but we’ll still be watching the customer’s back because our designers can surely explain how they get the answers, and that’s what the market wants and more importantly, demands!
Especially in the financial industry, we believe the customer should always manage the countermeasure, the reaction could impact user experience and should be as limited as possible on the client side. We have designed our solutions to provide risk and behavioral evaluations to react dynamically modulating the service or activating awareness campaigns without impacting the user experience. The service provider always has to be able to analyze the case, to understand the issue and eventually engage the end-user.
For this reason, we at XTN believe that using several ML algorithms (both supervised and unsupervised) to solve specific problems while maintaining visibility on partial results (i.e. detecting “anomalous behavior” due to a new attack scenario, typically a zero-day attack), and a rule-based last line of defense, is the best approach from the business standpoint. A holistic framework to consider fraud protection with a multi-channel and multi-layer approach we call Advanced Behavioral Fraud Prevention. This means the ability to benefit from AI power with a new generation of digital identification dynamic indicators (used to ensure digital users are who they say they are), providing superior protection using non- invasive frictionless solutions (specifically designed for every single endpoint). By taking this approach you are able to trust the digital customer and protect the business, while granting an unparalleled result speed for real-time services, reducing false positives, time- consuming activities for fraud teams, and guaranteeing significant containment of management costs.
About the Author
Davide NG Fania is the Managing Director of XTN Cognitive Security. He’s a manager with proven decades experience in the areas of Information Technology, Biomedical eMedical Devices Automation, and that bases its success stories on strong creativity to innovation and holistic view of the business, a supporter of the global market and strategic partnerships. Davide can be reached online at linkedin.com/in/dngfania and at our company website https://www.xtn-lab.com
