MODULE 05 · APPLIED
Model Validation & Uncertainty
No Model Is Perfect
All catastrophe models are simplifications of reality. They make assumptions, rely on incomplete data, and cannot fully capture the complexity of natural hazard systems. Understanding where models are reliable, where they are limited, and how to communicate uncertainty is one of the most important — and most demanding — skills in professional catastrophe modelling.
Advantages of Cat Models
Despite their limitations, cat models offer significant advantages over relying on historical loss data alone:
- Comprehensive event simulation: Models simulate significantly more realistic plausible events than are contained in any historical record. The stochastic catalog is calibrated to reflect scientific understanding of event frequency and severity, smoothing gaps in historical records.
- Current exposure analysis: Models allow users to import and analyse current exposure and policy terms — avoiding the pitfalls of adjusting historical loss experience to reflect changes in the number, type, and value of structures.
- Regular updates: Commercial cat models are updated regularly, incorporating advances in meteorology, seismology, hydrology, and structural engineering. They can reflect the most current information on land use, soil type, flood defences, and building codes.
- "What if" scenario analysis: Models allow users to develop forward-looking views and assess the impact of specific risk management strategies — reinsurance structures, portfolio changes, or mitigation investments.
- Sensitivity testing: The model framework encourages systematic testing of assumptions, providing additional viewpoints and stress test scenarios.
- Multiple viewpoints: Several commercial models are available, each reflecting different scientific assumptions. Using multiple models provides additional perspective on risk.
Key Advantage Summary
Cat models overcome the fundamental limitation of historical data — that the past is too short and too variable to reliably estimate the full probability distribution of future catastrophe losses — by combining physical science with engineering and financial analysis in a probabilistic simulation framework.
Limitations of Cat Models
- Wide output ranges: Significant uncertainties exist around model estimates, and large ranges of output are common across different models. A wide range does not mean any single model is wrong — it reflects genuine scientific uncertainty. However, it can be difficult to explain to consumers, regulators, and executives.
- Data collection costs: Collecting detailed building characteristics (construction type, age, occupancy) is expensive and time-consuming. Incomplete exposure data degrades model output quality significantly.
- Excluded causes of loss: Some damage that occurs concurrent with a catastrophe event may not be captured by the model — for example, demand surge (the increase in repair costs when many properties need rebuilding simultaneously), or certain types of contingent business interruption.
- Software update volatility: As science evolves, model vendors update their products. Large changes between model versions can cause significant swings in estimated losses from year to year, creating instability in pricing and capital management.
- Expertise requirements: Given the complexity of cat models, meaningful use requires either significant internal investment in training and infrastructure, or reliance on a reinsurance broker or other third party.
- Proprietary assumptions: Some core model assumptions are considered proprietary and are not disclosed to users. This limits users' ability to independently verify model behaviour in all circumstances.
The Right Expectation
Catastrophe models are tools to help insurers understand and manage risk. It is impossible and unrealistic to expect a cat model to produce perfect answers. However, the appropriate alternative is not to abandon modelling and rely on historical data — it is to understand the model's limitations, use multiple models where possible, and apply expert judgment alongside quantitative outputs.
Types of Uncertainty
Model uncertainty exists at multiple levels:
- Hazard uncertainty: Where, when, and at what intensity will future events occur?
- Exposure uncertainty: Are the insured properties accurately described in the exposure database?
- Vulnerability uncertainty: How accurately do the damage functions represent actual building behaviour?
- Secondary uncertainty: For a given mean damage ratio, how much variability around that mean exists at the individual building level?
- Model uncertainty: Do the model's structural assumptions — its scientific framework — accurately reflect reality?
Validation Methods
Model validation compares model outputs against empirical evidence. Key validation approaches include comparing modelled losses against actual paid losses from historical events, comparing modelled hazard footprints against instrumental measurements, and comparing outputs across multiple vendor models. The goal is not to prove the model is correct, but to understand where it performs well and where it is likely to diverge from reality.
Knowledge Check — Lesson 5.1
Answer all questions. You need 75% to pass.