Model Validation Solutions, 2019

Model validation: an overview

In analyzing model validation solutions and technology, the main themes are diversity, specificity and maturity. Model validation is a complex area, covering a wide variety of highly specific tools and techniques of varying maturity. Because of this diversity and complexity, model validation will be a key research area for Chartis in 2019 and beyond.

This introductory report provides a broad overview of model validation techniques and processes, and – linking with our other key areas of research – examines how Artificial Intelligence (AI) may play a role in their evolution. In future reports we will look in more detail at specific areas of model validation – such as derivatives, Anti-Money Laundering (AML) and credit – examining the quantitative methods underlying each, as well as the fundamental and important nuances that differentiate model validation, model governance and model risk.

A new kind of complacency

Since the financial crisis of 2008, the way that Financial Institutions (FIs) view and approach model validation has changed significantly.

FIs’ models were often complex and hard to understand – but for FIs themselves these traits were examples of the models’ effectiveness. As a result, they didn’t see much need to validate them, leading to a level of complacency.

All that changed after the financial crisis. Following pressure from regulators, model validation and management were placed firmly in the spotlight, and the accuracy and validation of models became a crucial part of FIs’ risk governance processes.

Despite the heightened focus on model validation, however, FIs are still complacent. Now they rely too much on models and the outputs of their often opaque internal processes. But several high-profile incidents (such as the case of the ‘London Whale’) have highlighted how FIs need a deeper understanding of these processes. And increasingly regulators – frustrated by the opacity of many models, and their results (particularly in Financial Crime Risk Management [FCRM]) – want a clearer picture of exactly how FIs’ models work.

In a world where models are becoming bigger and more complex, and FIs depend increasingly on quantitative decision-making processes, achieving effective model validation is still a challenge.

Although validation processes are continuously developing, there is still some doubt over whether current model validation techniques and processes are resolving model risk at a fundamental level or merely mitigating it.

Current versus future

In analyzing the overall model validation landscape, it’s helpful to view its evolution in two phases: current and future. Current model validation, using relatively well established technologies and techniques, is not without its challenges: notably around workflow, the ease with which models can be interpreted, and how to tackle the opportunities and challenges presented by data. To help FIs address these issues, Chartis has developed some best practices for model validation within the context of the current technological and regulatory environment.

But we are also seeing a fundamental shift in the model validation landscape, driven by more sophisticated technology – an evolution that is challenging the structures on which established model validation has been built. As Artificial Intelligence (AI), Machine Learning (ML) and Natural Language Processing (NLP) tools and techniques become more prevalent, they will play a vital role in the future of model development and validation.

Much of this evolution is being influenced by the need for explainability and interpretability now demanded by regulators – and CROs, who increasingly have to understand why a system produced specific results, and be able to defend those results. It is seeing us move – albeit gradually – into a new era of AI development, away from the ‘black box’ models of the past toward more ‘glass box’ models (that employ so- called Explainable AI, or XAI). As the overarching imperative for transparency and accountability in FIs develops, XAI could become a critical element in meeting any future regulations and ultimately preventing systemic technological risk.

FIs are taking their first steps toward more AI- driven validation, and some vendors are exploring this new avenue (particularly in the area of FCRM, where ML tools are being used extensively) – but these technologies are developing fast. While they will likely transform the model validation sector, they also risk increasing the problems of complexity, opacity and interpretability that bedevil the use of models and threaten their credibility.

There are also more practical problems. Because of the heavy investment many FIs have already made in existing AI technologies, when XAI becomes more mainstream, integrating it won’t be a simple exercise. To fully embrace the latest wave1 of AI and ML, FIs will probably need to have a costly rethink; they may even have to roll back their current AI implementations.

A more holistic view

The reality is that a truly interactive relationship between model development, validation and users is still a long way off. At the moment, most validation is carried out using non-AI tools, and will continue to be for some time. Nevertheless, the development of XAI highlights the challenges that affect current non-AI model validation.

For vendors, this presents a problem. Chartis believes that to compete successfully they should start to take a more ‘holistic’ view and cover both options: ensure they are at the forefront of developing and implementing new validation techniques, while still providing services for traditional models. In our view, effective model risk governance should involve the validation of all aspects of model development and implementation. This is reflected in our definition of ‘model validation’, which covers data validation, internal model validation, the appropriateness of particular models and their transparency.

Successful vendors of the future, we believe, will offer comprehensive solutions that fit this wider definition of validation. Vendors also have an opportunity to move beyond the labels and restrictions of ‘developer’ and ‘validator’ to provide more ‘holistic’ expertise. Conversely, by treating the development and validation of models as strictly separate processes, vendors risk overlooking the close relationship that validation should have with development at every stage of the process.

Ultimately, however, a fundamental challenge for both FIs and vendors remains. For model validation to be possible and effective in FIs’ new, AI-led future, vendors and users will have to dramatically change their attitude and approach to modeling, if they are to make the end-to-end process of model development and validation transparent and effective.

This report uses Chartis’s RiskTech Quadrant^® to explain the structure of the market. The RiskTech Quadrant^® uses a comprehensive methodology of in-depth independent research and a clear scoring system to explain which technology solutions meet an organization’s needs. The RiskTech Quadrant^® does not simply describe one technology solution as the best risk management solution; it has a sophisticated ranking methodology to explain which solutions would be best for buyers, depending on their implementation strategies.

This report covers the leading providers of model validation technology: Bloomberg, Finastra, FINCAD, Moody’s Analytics, MSCI, Murex, Numerix, RiskSpan, SAS and Xenomorph.

Note that because this introductory report provides a broad overview of model validation types and processes, we scored the vendors in the quadrant according to general criteria – only those vendors that offered validation according to our criteria were briefed. Note also that some vendors did not respond to our invitation to brief for this report, and some we contacted declined to brief us.