\(\)You might assume that one of the largest financial consulting firms in the world with a net revenue in the tens of billions would be able to do some pretty good capital and risk modelling work. As a mathematician working in finance, I’ve come to realise that even the largest and most prestigious financial organisations routinely make critical errors in the design and validation of financial models. This is true not just for complex models, but even very simple ones.
In this article we’ll take a look at some risk modelling work performed by one of the “big four” financial consulting firms. The model is supposed to estimate how much capital a bank should hold to protect itself against large losses. As we’ll see, they could have hired Chim-Chim the monkey to design and validate a risk model for them with similar results (but at a significant cost advantage!)
So here’s what they did. A business manager, let’s call him “Bob”, used his knowledge of the business and memory of past losses to estimate the loss size that could occur in each part of the business, and on average how many years would pass before such a loss occurred. They divided each loss size by the corresponding numbers of years to yield a set of data points, to which they fitted a lognormal distribution by matching mean and variance, as given by the following formulae:
\[mu = \log(m^2 / \sqrt{v + m^2}),\]
\[\sigma = \sqrt{log(v/m^2 + 1}).\]
The bank determined that it would hold as capital the 99.9% quantile of the resulting distribution.
Let’s say bob comes up with loss estimates of $50 million every 5 years, $30 million every 3 years, and $25 million every 2 years. If you determine the parameters of the lognormal distribution from the above formulae and find the 99.9% quantile, you’ll get $16.2 million. The model provided by the financial consulting firm predicts that the bank should hold $16.2 million in capital, which will allegedly protect it against all but the 1 in a thousand year loss.
So what’s wrong with this approach?
First, think about the numbers – if we suffer a loss of $50 million every 5 years, how can $16.2 million cover the once in a thousand year loss! It doesn’t even cover the once in 5 year loss from only one of the three parts of the business! Clearly there’s something very wrong with this modelling work!
Next, let’s see if the model output changes in a reasonable way when we make a change to the input. Suppose Bob realises that he has forgotten about one important risk. You see, bob occasionally loses his pen. He estimates that he loses his pen about once a year, and that it costs the bank about $1 to replace it. Our consulting firm adds this data point to the model inputs and redoes the calculation. So what is the result now? Has the amount of capital gone up by a few bucks? Nope, it’s now $45.3 million! In order to insure bob against the risk of losing his pen, the bank has to hold an additional $29.2 million in capital! In fact, the cost of insuring that pen is almost twice as much as the $16.2 million required to ensure the entire business!
Ok, so this model is crazy. But what’s wrong from a mathematical perspective?
These analysts know about fitting a lognormal distribution to data, but there’s no depth to their understanding, causing them to design models that produce entirely meaningless numbers. When you fit a lognormal distribution to some data, you are making the assumption that the data is lognormally distributed. The data you are using therefore has to be representative of amounts chosen at random from the loss distribution. Because the standard deviation of the distribution is fixed to the standard deviation of the estimates, the closer together the estimates are, the smaller the amount of capital the model estimates!
But the data they were using was a fairly arbitrary collection of numbers gathered by considering different loss scenarios in different parts of the business. How spread out these numbers were was not something they were even conscious of – yet it was the critical determinant of risk. If Bob comes up with a few losses close together, the model won’t require the bank to hold much capital – even if those losses are very large. But if Bob comes up with losses far apart – even much smaller ones – then the model will require the bank to hold a huge amount of capital!
You would think banks would be outraged that they were paying so much money for models that were nothing but random number generators, and concerned that they weren’t holding the right amount of capital to prevent their business from collapsing. And it’s certainly not very good for the security of their customer’s money. How much money do you this financial consulting firm, one of the top four, charged for Chim-Chim the monkey numbers? If only the CEO knew how much money he paid external consultants to pretend to do risk modelling.
Can you believe these large financial consulting firms get paid a lot of money to validate risk models?
So, if you’re seeking professional financial modelling consulting, you could hire me to produce quality, meaningful capital numbers that will protect your business against unexpected losses. Or you could hire one of the big four to take a random stab in the dark while tripping over the washing basket – for ten times the cost!
