Model validation is a strict regulatory requirement for many financial services businesses, essential for risk management and front office profitability. It’s also critical for many other industries as well. In the medical device, aviation and mining industries, model failure could not only lead to financial loss, but cost lives. In so many industries, models must be thoroughly validated both before deployment, and at regular intervals after deployment. They must also be stress tested on unusual scenarios to ensure they will remain robust.
In order to speed up and broaden the scope of model validation, many organisations are considering building automated model validation tools. In this article, we’ll look at how automated model validation differs from manual validation, and how both are an essential part of a validation ecosystem.
What is model validation?
Model validation is a process to regularly test and monitor mathematical models. This spans the conceptual soundness of the model, the correctness of the coding implementation, and the interactions between systems such as data interfaces. The context could be option pricing or risk models in finance, machine learning models to detect mineral deposits in mining, automated trading systems, or medical monitoring devices. Increasingly, AI and machine learning models need to be validated. And of course, the wide-spread use of impressive but error-prone AI tools like ChatGPT necessitates a whole new world of model validation. The process typically involves:
- Reviewing the correctness of the mathematical methodology and documenting it
- Assessing the any model limitations or boundaries of validity
- Building an independent model, either of the same methodology or a different one, and ensuring the output of both models are within some acceptable tolerance
- Monitoring the ongoing appropriateness of the model considering changes in downstream and upstream systems, and changes in the environment the model operates in
- Stress testing the model to ensure robustness under unusual scenarios (eg financial downturns)
- Documenting all findings
What is automated model validation?
Automated model validation is the use of software tools and frameworks to autonomously and systematically test and monitor models. Typically these tools:
- Automatically pull in and format data from source systems (where a human validator might need to examine a GUI one number at a time)
- Verify huge numbers of model outputs by comparing against independent model implementations
Benefits of automated model validation
A downside of automated model validation systems is they may require a significant initial investment to build. So what are the advantages?
- Once set up, automated model validation is far less time intensive than model validation. An automated system could examine thousands of trades in a trading book, where a manual validator has time to check a dozen.
- Automated systems make it easy to quickly retest the models after system updates, or set up regular periodic validations or continuous monitoring.
- Scalability: automated systems can have a scale beyond the scope of human validators
- Automated systems may free up human validators to focus more on conceptual model review
- To the extent that less manual work is required, they may reduce costs.
Benefits of manual model validation
- Manual validation involves reviewing the assumptions and methodology of the model, often in a changing environment. This is something which doesn’t happen at all with automated model validation.
- How can you be sure the independent implementation used by the automated validation system is correct? To an extent, automated validation actually results in two models that need to be validated. And where both models have similar assumptions or methodologies, it’s entirely possible for the models to agree and both be wrong.
- Manual model validators also provide another very important function – the preparation of detailed validation reports clarifying the methodology, assumptions and limitations of the model where documentation is usually sorely lacking.
- Human validators can improve model design, not just test model outputs.
- Credibility with the regulator: in the case of regulated industries, human expert analysis may be considered essential.
Why a hybrid approach is best practice
The best approach to model validation is a hybrid approach, combining manual model validation with automated model validation. This is where manual model validators carefully check model assumptions, work to improve and maintain the accuracy of the automated model, and produce high quality documentation about the models. At the same time, the automated system allows for much more frequent and large scale testing and monitoring of trades and models. Having found serious errors in models that had already been validated several times, this author can tell you that the need for expert validators will never go away.
We offer model validation consulting services, including both manual model validation and the design of automated model validation systems. Contact us to learn more.