Thursday, March 7, 2013

Accurate and Reliable or just Politically Acceptable?

Models approved by the FCHLPM are said to be accurate and reliable.  Which is not saying much.  Accuracy traditionally means that a prediction shows a close correspondence with observations.  The dictionary definition would be that a prediction is "correct."   But given the time frames and probabilistic nature of catastrophe model predictions it is not possible to show that the models are accurate (no less skillful).  Thus, the FCHLPM redefines accurate to mean “the models meet the standards that have been developed” because "‘accurate’ cannot necessarily mean that a model conforms exactly to known facts since that contradicts the nature of the modeling process" (p. 39).

That a model is reliable means that it “will consistently produce statistically similar results upon repeated use without inherent or known bias.”  My understanding of this is that every time the user clicks "run" the model comes up with a similar result.  But even if interpreted differently it no less means that a model that is consistently wrong is no less reliable.  Consider a model that consistently predicts no hurricane losses or one that consistently predicts an annual loss of $500 B. In each case, such a prediction is as reliable as it is counterintuitive.

Below are graphs of accurate and reliable model loss predictions (for hypothetical data sets) approved by the FCHLPM.  Predicted loss is adjusted to the long term modeled mean for AAL and 250PML, respectively.  There are some interesting things to note...    
First, that red line that seems to be doing its own thing is produced by Applied Research Associates (ARA).  This company holds a great many Federal contracts for different technological things and it boasts that it's HurLoss model
Since 1998 it has been accepted by the American Society of Civil Engineers (ASCE) for setting national hurricane wind speed standards.
As such, this model is referred to often when building codes are being considered. So for instance, when the Florida Building Codes were under review, the ARA model was used for the review process.  Note that this model's risk estimate peaked in 2002.  Anyone remember what happened in Florida in 2002?  A massive overhaul of the building codes making them more stringent for wind.  Interesting, I think.

What is more?  All the models appear to converge around an average value in 2007.  Anyone remember what happened that year?  Citizens "capped" the conception of hurricane risk with changing eligibility requirements for coverage.  Coincidence?  Perhaps.  But more likely, legislative action created some politically acceptable risk bound.  

Another interesting point of note, is that the Florida Public Model began making submissions to the Commission in 2007.  It's average annual loss (aka Pure Premium) prediction was the highest of all the models.  This is notable because the model is legislatively mandated to serve as the minimum benchmark for Citizens rates.  Further consider that the long term risk, 250 PML, for the public model is at the low end.  According to this model then, Citizens risk of deficit from a low frequency, high severity hurricane event is small as compared to the other models and that risk is balanced by a higher estimate of annual risk.  

So, are any of these models any more accurate or reliable then the next?  Do any of them show a true or real depiction of Florida's hurricane risk?  No they do not.

What they suggest is that modeling companies have different clientele with different needs.  Each model, shows a different idea about the Florida hurricane risk that can be used to support chosen political positions and policies.