← Foundation Course /Module 04 - What the Model Doesn't Know /Lesson 4.1
MODULE 04 · WHAT THE MODEL DOESN'T KNOW

What the Model Doesn't Know

📖 ~25 min · Lesson 12 of 15 · 3 micro-checks + quiz · Foundation Level · Free
Solent Re - London · The Loss That Wasn't in the Model

Solent Re is a London-based specialty reinsurer writing global property catastrophe alongside marine, energy, and a growing cyber book. After a major flood event in Central Europe, the claims team reports a final net loss of USD 118M against a modelled loss estimate of USD 71M, a 66% overshoot.

The head of cat modelling is asked to explain the discrepancy. Her analysis identifies three contributing factors. First, a large industrial park on the western edge of the affected area had been geocoded to postcode level, the actual buildings were in a lower-lying area with 40% higher flood inundation depth than the postcode centroid. Second, the model's flood vulnerability functions for Eastern European industrial construction were calibrated on Western European data, a structural mismatch. Third, and largest: two of the affected locations carried business interruption cover that the cat model had not been configured to capture. The BI losses alone accounted for USD 28M of the gap.

"The model was right about what it modelled," she tells the board. "The problem is what it didn't model at all."

Every cat model has a scope, a defined boundary of what it covers and what it does not. What are the main categories of things that fall outside that boundary, and why does a number the model produces never tell you what it left out?

The Model's Boundary Is Not Its Cover Page

Every cat model comes with documentation listing what it covers, which perils, which geographies, which building types. But the boundary of a model is rarely as clear as its documentation suggests. A model described as covering "European flood" may cover riverine flood in France and Germany while treating flash flood, coastal surge, and groundwater flood very differently, or not at all. A model described as covering "US tornado" may use wind speed as its primary intensity metric without fully capturing the damage from large hail that frequently accompanies severe convective storms.

The most dangerous aspect of a model's boundary is that crossing it produces a number, not an error. When a location falls outside the model's calibrated scope, a construction class it has no data for, a geography where its hazard grid is sparse, a secondary peril its vulnerability functions were not built to capture, the model does not stop. It applies a default and continues. The output looks identical to output produced within the model's well-calibrated scope. There is no asterisk, no warning, no indication that the number was produced outside the model's area of reliable knowledge.

The Central Problem

A cat model output of USD 71M is the same number whether it was produced with complete, well-calibrated data within the model's core scope, or with sparse data at the edge of the model's domain using default assumptions for everything the data did not provide. The number does not carry its own quality label. The analyst who produced it is the only source of that information, and if the caveats do not travel with the output, the downstream decision-maker cannot know what the number actually represents.

Three Types of Model Limitation

It helps to separate model limitations into three distinct categories. They are different in nature and different in what can be done about them, but all three contribute to the gap between what a model says and what actually happens when a loss occurs.

Aleatory
Uncertainty

The irreducible randomness of the physical world. Even a perfect model cannot predict which specific events will occur in a given year, or exactly how strong the ground shaking will be at a specific point from a known earthquake. This uncertainty is captured in the stochastic catalogue, it is the entire reason we run tens of thousands of synthetic events rather than a single deterministic prediction. No amount of better data or better science eliminates it. It is the randomness the EP curve and AAL are designed to quantify.

Epistemic
Uncertainty

Uncertainty from incomplete knowledge, things the model does not know but could in principle know better. The choice of Ground Motion Prediction Equation for earthquake, the calibration of flood vulnerability functions to a specific regional building stock, the rate assumptions in the stochastic catalogue for a peril with limited historical data, all of these reflect the limits of current scientific knowledge, not the limits of knowledge in principle. Better data, better calibration, and more research can reduce epistemic uncertainty over time. It is the primary source of vendor model divergence: two vendors making different but defensible scientific choices about the same uncertain physical processes.

Non-Modelled
Items

Losses from perils, geographies, or coverage types that the model does not attempt to represent at all. This is categorically different from uncertainty, it is absence. If a model does not include a business interruption vulnerability function, the BI loss is not uncertain within the model, it is simply not there. The model produces a number as if BI did not exist. Non-modelled items do not show up as uncertainty ranges or confidence intervals. They show up as a gap between the modelled loss and the actual loss after the event, as happened with Solent Re's USD 28M BI shortfall.

Micro-Check · Before You Continue

Solent Re's cat modelling head identifies three contributors to the USD 47M gap: geocoding error, vulnerability function mismatch, and unconfigured BI cover. Which of these is aleatory uncertainty, which is epistemic uncertainty, and which is a non-modelled item?

None of the three is aleatory uncertainty. Aleatory uncertainty is the randomness of the physical world, which events occur and at what intensity. Once the flood event actually happened, the question of what loss it produced is no longer aleatory.

The vulnerability function mismatch is epistemic uncertainty, the model applied damage functions calibrated to Western European industrial construction to Eastern European buildings. This reflects a gap in knowledge that better regional calibration data could in principle reduce. The model was doing its best with the science available to it, but the science was not well-suited to this geography.

The geocoding error is a data quality problem, strictly speaking, neither aleatory nor epistemic model uncertainty. It is an input error that caused the model to calculate the wrong hazard intensity. The model was working correctly; it was given the wrong location.

The unconfigured BI cover is a non-modelled item, the model was not told to include business interruption, so it produced no BI loss estimate at all. The USD 28M BI loss is not uncertain within the model; it simply does not exist in the model's output.

What Vendors Do Not Model - The Common Gaps

Non-modelled items fall into four broad categories. Each represents a class of real-world loss that standard vendor cat models either do not attempt to capture or capture only partially. Understanding these gaps does not require knowing how to handle them professionally, it requires knowing they exist and that the model's output is silent about them.

Coverage Gaps

Business Interruption and Contingent BI

Standard cat models focus on physical property damage. Business interruption, the revenue lost while a building is unusable after a cat event, is modelled separately and often not configured in routine runs. Contingent BI, which arises when a supplier or customer's premises is damaged rather than the insured's own, is rarely modelled at all. For commercial and industrial risks, BI can exceed the property damage loss.

Peril Gaps

Secondary and Correlated Perils

A major earthquake can trigger landslides, fires, and tsunami. A tropical cyclone produces storm surge as well as wind. A severe hailstorm accompanies a tornado outbreak. Standard models typically capture the primary peril intensity but may underrepresent the secondary perils that frequently accompany it. Fire-following-earthquake and tsunami are the most commonly cited secondary peril gaps for earthquake models.

Geography Gaps

Emerging Markets and Thin Data Regions

Vendor models are calibrated to available data. In markets with limited historical loss records, sparse building stock information, and incomplete hazard data, much of sub-Saharan Africa, parts of Southeast Asia, Central Asia, the models' calibration is thinner and their uncertainty bands wider. Output for these regions carries higher epistemic uncertainty than the same vendor's output for US hurricane or Japan earthquake.

Demand Surge

Post-Event Cost Inflation

After a major catastrophe, the sudden demand for construction materials and skilled labour drives up repair costs, sometimes by 20–40% above pre-event levels. Standard model outputs are calibrated to normal market conditions. The additional cost of rebuilding in a disaster-affected economy, demand surge, is a systematic source of underestimation for large, geographically concentrated events.

Emerging Risks

New Perils Without Established Models

Some risks do not yet have mature cat model frameworks. Cyber catastrophe, a systemic attack affecting many policyholders simultaneously, has no established stochastic catalogue comparable to those for hurricane or earthquake. Pandemic business interruption, climate-change-driven non-linear hazard shifts, and novel liability scenarios share the same challenge: the physical science or loss history needed to build a calibrated probabilistic model does not yet exist or is too short to be reliable.

Legal / Social

Loss Amplification

In some jurisdictions, post-event legal and social dynamics inflate insured losses well above the physical damage the model estimates. Assignment of benefits disputes, class action litigation, expanded policy interpretations by courts, and regulatory intervention can all cause actual paid claims to significantly exceed what the cat model calculated from pure physical damage assessment. US litigation-driven loss amplification has been a material source of model underperformance in certain property markets.

Micro-Check · Before You Continue

Solent Re writes a policy on a large pharmaceutical manufacturing facility in coastal Bangladesh. The facility is exposed to tropical cyclone wind and storm surge. The cat model produces an AAL of USD 1.4M and a 1-in-100 year OEP of USD 18M. What non-modelled items should the cat modelling head flag when presenting these figures, and why might the actual loss significantly exceed these estimates in a major event?

At minimum, three non-modelled items warrant explicit flagging.

Business interruption: A pharmaceutical manufacturing facility has high revenue per square metre and a complex production process. Structural damage that might be repairable in weeks could halt production for months if specialised equipment is damaged, regulatory recertification is required, or supply chain disruption compounds the physical loss. BI losses for this type of occupancy can easily exceed the property damage estimate, and if BI cover is not configured in the model, none of that loss appears.

Storm surge and secondary peril: Bangladesh's coastal geography means that cyclone events frequently produce severe storm surge flooding well inland. If the model's primary intensity metric is wind speed and its storm surge representation is limited or absent, the flood damage component of a coastal cyclone event may be underestimated.

Emerging market calibration: Vendor models for Bangladesh cyclone carry higher epistemic uncertainty than the same vendors' US hurricane models. The hazard catalogue, vulnerability functions, and financial module defaults are all less well-calibrated for this geography, meaning the model's output should be treated with wider confidence bounds than it carries for mature market perils.

Why Vendors Give Different Answers

When two vendors produce materially different loss estimates for the same portfolio, it is tempting to conclude one of them is wrong. Usually, neither is wrong, they are making different but defensible choices within areas of genuine scientific uncertainty. Understanding why divergence occurs is the first step to using multi-vendor analysis productively.

Illustrative Vendor Divergence - Solent Re European Flood Portfolio

AAL
Vendor A
USD 8.4M
Vendor B
USD 9.9M
1-in-100 OEP
Vendor A
USD 48M
Vendor B
USD 71M
1-in-250 OEP
Vendor A
USD 82M
Vendor B
USD 140M

Divergence typically grows with return period. At AAL, dominated by moderate, frequent events where both vendors have reasonable calibration data, the gap is 18%. At 1-in-100 the gap is 48%. At 1-in-250 it reaches 71%. This pattern is normal and expected: both vendors are well-calibrated for the body of the distribution where historical data is abundant, and diverge at the extreme tail where data is sparse and model assumptions dominate.

The Four Primary Sources of Vendor Divergence

What Divergence Tells You

The gap between vendors is not noise to be averaged away, it is information. Where vendors agree closely, the science is relatively settled and the output is more reliable. Where they diverge significantly, the divergence marks the boundary of current scientific knowledge on that peril and geography. A practitioner who understands why the vendors diverge for European flood at the 1-in-250 year level understands something important about the limits of what current cat models can reliably say about that risk.

The Protection Gap, When the Model Is Right and the Problem Is Bigger

Beyond the technical limits of what models can measure lies a structural reality: most economic catastrophe loss is not insured. Cat models estimate insured losses, the losses falling within the coverage terms of the policies in the portfolio. The total economic loss from a major event is typically far larger. The gap between total economic loss and insured loss is the protection gap, and it is one of the defining challenges in global catastrophe risk management.

Economic Loss vs Insured Loss - The Protection Gap (Illustrative)

Total Economic Loss - 100%
Insured Loss - 38%
Insured loss - captured by cat model
Uninsured economic loss - invisible to cat model

The insured fraction varies enormously by country and peril. In the US, it may exceed 60% for hurricane wind. In many developing markets, it may be under 5% for earthquake. The cat model only sees the insured portion, the same physical event produces very different model output depending on where it strikes.

The protection gap matters to practitioners for three reasons. First, it explains why a model can be technically correct, accurately estimating insured losses, while massively understating the total economic impact of an event. The model is not wrong; it is answering a narrower question than the physical scale of the disaster suggests. Second, it explains why insurance penetration matters as a context for model output, a portfolio covering 5% of the economic exposure in a region is very differently positioned from one covering 40%, even if both produce similar model outputs for the covered losses. Third, it identifies where future market growth will occur: the protection gap is the unfilled demand for insurance coverage, and closing it is the business case for many emerging market initiatives.

Micro-Check · Before You Continue

A major earthquake strikes a densely populated city in a country with 8% insurance penetration for property. The cat model estimates Solent Re's insured loss from the event at USD 22M, which is consistent with the modelled AAL and EP curve. A colleague says: "The model worked, it got the loss about right." A senior underwriter says: "The model only tells us part of the story." Who is more nearly correct, and what is the underwriter pointing at?

Both are saying something true, but the underwriter is pointing at something the colleague is missing. The colleague is correct that if the model estimated Solent Re's insured loss at USD 22M and the actual insured loss came in near that figure, the model performed well for the purpose it was designed for. The underwriter is pointing at something larger: with 8% insurance penetration, the total economic loss from the same earthquake may be USD 275M or more, only USD 22M of which is insured. The cat model captures the insured portion with reasonable accuracy; it says nothing about the USD 253M of uninsured economic damage, the displacement of tens of thousands of uninsured homeowners, or the fiscal pressure on a government without an insurance backstop. The model worked in a narrow technical sense. But "the model worked" can never mean "the risk was adequately covered" in a low-penetration market, those are completely different statements.

Common Misconceptions

01
"If the model produces a number, that peril is covered"

The model produces a number for every event in its catalogue regardless of whether the coverage in the policy actually responds to that peril. A wind-only policy on a flood-prone property will produce a cat model output, for the wind component, that looks complete, while the flood exposure the model does not cover goes entirely unrepresented. The output covers what the policy covers and what the model models. Both limits apply simultaneously.

02
"One vendor is right, the other is wrong"

Vendor divergence reflects genuine scientific uncertainty, not error on one side. Both vendors are typically making defensible choices within the limits of available data and science. The divergence is most informative when treated as a measure of that uncertainty rather than as evidence that one model needs correcting.

03
"Non-modelled items are a small rounding error"

In Solent Re's Central European flood loss, the non-modelled BI component alone accounted for USD 28M of a USD 47M gap, more than half the total discrepancy. For commercial and industrial risks with high revenue intensity, BI can exceed property damage. For events in litigation-prone markets, loss amplification can add tens of percent to the modelled loss. Non-modelled items are systematic, directional, and sometimes larger than the modelled loss itself.

04
"The protection gap is someone else's problem"

Practitioners working in emerging markets, structuring parametric products, pricing climate-linked covers, or assessing sovereign risk programmes directly encounter the protection gap as a core professional challenge. It is also the context within which every insured loss estimate must be understood, a USD 22M insured loss in a market with 8% penetration represents a very different physical event from a USD 22M insured loss in a 60% penetration market.

Key Terms

Aleatory Uncertainty

The irreducible randomness of the physical world, which events occur and at what intensity. Quantified by the stochastic catalogue and the EP curve. Cannot be reduced by better data or science. The fundamental reason cat modelling uses probability distributions rather than single predictions.

Epistemic Uncertainty

Uncertainty from incomplete scientific knowledge, things the model does not know but could in principle know better. The primary source of vendor model divergence. Can be reduced over time with better data, better calibration, and more research.

Non-Modelled Items

Losses from perils, geographies, or coverage types the model does not attempt to represent. Categorically different from uncertainty, not uncertain within the model, simply absent. Common examples: business interruption, secondary perils, demand surge, loss amplification.

Protection Gap

The difference between total economic loss and insured loss from a catastrophe event. The portion of physical damage that is not covered by any insurance policy. Cat models capture only the insured portion. In many developing markets, the protection gap exceeds 90% of total economic loss.

Demand Surge

Post-event inflation of construction and repair costs driven by sudden excess demand for labour and materials in the affected area. Typically 20–40% above pre-event market rates for major concentrated events. A systematic non-modelled item that causes actual losses to exceed standard model output.

Loss Amplification

The inflation of insured losses beyond physical damage through legal and social dynamics, litigation, assignment of benefits disputes, expanded judicial interpretations of policy language. A significant non-modelled source of underperformance in certain property markets.

Model Vintage

The version or release date of a vendor cat model. Models are updated periodically to incorporate new science, post-event recalibration, and new data. Two models of different vintages will diverge even if they use similar methodologies, because their underlying calibration data differs.

Knowledge Check - Lesson 4.1

Five questions · 4 of 5 correct (80%) to pass

1. Solent Re's cat model produces a 1-in-100 year OEP of USD 48M for Vendor A and USD 71M for Vendor B on the same European flood portfolio. The head of cat modelling presents both figures to the underwriting committee. A board member asks: "Which one should we use for capital planning?" What is the most accurate framing of the answer?

AUse the higher figure, regulatory capital requirements mandate using the most conservative available estimate, so Vendor B's USD 71M is the correct basis for capital planning regardless of which model is considered more accurate.
BNeither figure is definitively correct, the USD 23M gap represents the range of scientific uncertainty in European flood modelling at the 1-in-100 year return period. For capital planning, this range should be presented honestly: the true 1-in-100 year loss is somewhere within this band, and capital held at USD 48M would be adequate if the lower estimate is closer to reality, but insufficient if the higher estimate is. Many capital frameworks use an average of available vendor estimates, apply a loading above the higher estimate, or use the higher figure as the base with explicit acknowledgement that the gap represents epistemic uncertainty rather than model error. The board should understand they are holding capital against a range, not a precise number.
CUse Vendor A, a lower loss estimate is more appropriate for capital planning because it represents the baseline expectation and capital should only be held against expected losses, not conservative outlier estimates.
DCommission a third vendor model, with three estimates the middle value can be used as a tiebreaker, producing a defensible consensus figure for capital planning purposes.

2. A major windstorm affects Solent Re's Northern European portfolio. The cat model estimated a gross loss of USD 55M. The actual gross loss comes in at USD 84M. The cat modelling team's post-event analysis identifies: USD 12M from demand surge, USD 9M from BI losses not configured in the model, USD 5M from a portfolio of rooftop solar panels the model had no damage function for, and USD 3M from higher-than-modelled damage to older timber-frame buildings. Which of these is epistemic uncertainty and which is a non-modelled item?

AAll four are non-modelled items, the model failed to capture each of them, meaning by definition they fall outside its scope and are therefore non-modelled.
BThe USD 3M from older timber-frame buildings is epistemic uncertainty, the model had a damage function for this building type but it was less well-calibrated than for more common construction classes, producing an underestimate within the model's scope. The other three are non-modelled items in different categories: demand surge (a systematic post-event cost factor the model was not configured to include), BI losses (a coverage type not configured in the run), and solar panel damage (a building sub-type with no damage function at all). The distinction matters: epistemic uncertainty is reducible through better calibration and data; non-modelled items require the model to be extended to cover them, a different kind of problem requiring a different kind of solution.
CDemand surge is aleatory uncertainty, it depends on the unpredictable post-event behaviour of labour and materials markets. The other three are non-modelled items.
DAll four are epistemic uncertainty, they represent areas where the model's knowledge was incomplete, which is the definition of epistemic uncertainty. Better data and better models would have captured all four.

3. Solent Re is underwriting a large portfolio of commercial properties in Vietnam, a market with limited historical loss data for typhoon, sparse building stock records, and a growing but still thin cat model vendor presence. The model produces an AAL of USD 6.2M. What should the cat modelling head communicate to the underwriter about the reliability of this figure relative to, say, the same vendor's output for a US Gulf of Mexico hurricane portfolio?

AThe Vietnam figure is equally reliable, modern cat models apply the same rigorous methodology globally. Regional data availability affects the precision of individual location estimates but not the statistical reliability of portfolio-level AAL figures.
BThe Vietnam figure carries materially higher epistemic uncertainty than the US Gulf output, for compounding reasons. The hazard catalogue for Vietnam typhoon is calibrated on a shorter and sparser historical record than the Gulf hurricane track database. Vulnerability functions for Vietnamese commercial construction are less well-calibrated than those for US commercial stock. The financial module defaults for Vietnamese policies may not reflect actual market practice. And the absence of multiple competing vendor models means there is no divergence comparison available to bracket the uncertainty range. The USD 6.2M AAL should be presented with explicit acknowledgement that the confidence interval around it is wider than for a mature market peril, and the underwriter should understand that the number is the model's best estimate, not a well-calibrated probability in the same sense as the US hurricane output.
CThe Vietnam figure is less reliable at the location level but equally reliable at portfolio level, with 200 locations in Vietnam, the law of large numbers ensures the portfolio AAL converges to the true expected value even if individual location estimates are uncertain.
DThe figures are incomparable, the Vietnam output should simply be excluded from the portfolio analysis and replaced with an engineering-based assessment, because vendor models are not suitable for thin-data markets.

4. Why does vendor model divergence grow with return period, and what does this tell a practitioner about where to place more weight on judgment versus model output?

ADivergence grows with return period because rare events are physically larger, and larger events produce more correlated losses across the portfolio, increasing the variance in model outputs. It tells practitioners to place less weight on model output generally for large events.
BDivergence grows because higher return periods correspond to rarer events where historical calibration data is sparse or absent. Both vendors have reasonable loss data for moderate events that occur every few years; they have little or no historical data for 1-in-250 year events that may never have occurred in the instrumental record. At the tail, model outputs are driven by extrapolation from physical science models rather than empirical calibration, and those extrapolations reflect different scientific choices that compound as the return period increases. This tells practitioners that model output is most reliable and least in need of supplementary judgment in the body of the distribution (AAL, 1-in-10 to 1-in-50 year losses), and least reliable at the extreme tail, exactly where capital adequacy decisions are most consequential. The wider the vendor gap at a given return period, the more important it is to understand the sources of that divergence rather than mechanically averaging or choosing between them.
CDivergence grows because vendors use different financial module configurations at higher return periods, they apply different demand surge loadings and legal inflation adjustments to extreme events. The divergence reflects policy and financial assumptions rather than hazard science differences.
DDivergence grows for computational reasons, extreme tail events require more simulation iterations to estimate reliably, and different vendors use different iteration counts. More iterations reduce the statistical noise in the estimate, so vendors with larger simulation budgets produce more stable tail estimates.

5. A cat model accurately estimates Solent Re's insured loss from a major earthquake at USD 38M. Total economic damage from the same event is estimated at USD 620M. In what sense is the model both "right" and "insufficient" as a description of the event's risk?

AThe model is right and sufficient, its job is to estimate insured losses for Solent Re's portfolio, and it did that accurately. The uninsured economic loss is not within the model's remit and should not be considered a limitation of the model.
BThe model is right in the narrow sense that it accurately estimated the loss for which Solent Re is financially liable, USD 38M. It is insufficient as a description of the event's risk in at least two senses. First, the USD 620M total economic loss reflects the actual scale of physical and human impact, the destruction of infrastructure, displacement of people, and economic disruption that no insurance product covered. A practitioner who describes the event as a "USD 38M loss" is describing a financial liability, not a catastrophe. Second, the USD 38M insured loss in a USD 620M economic event implies roughly 6% insurance penetration, a fact that is relevant for understanding the market's role, the demand for future coverage, and the societal context in which the claim sits. The model answers the question it was designed for. The protection gap is the reminder that this question is smaller than the event.
CThe model is insufficient because it should have estimated the full USD 620M economic loss, a well-calibrated model should be able to separate insured from uninsured components and report both. The USD 38M figure represents a modelling failure rather than a design boundary.
DThe model is right for reinsurance purposes but insufficient for primary insurance purposes. Primary insurers need total economic loss estimates to price reconstruction coverage; reinsurers only need the insured layer, making the model adequate for Solent Re's specific use case.