ECL model validation and back-testing

How does back-testing work?

A lender's model predicted one-year PDs for three grades a year ago. Comparing them with the share of each grade that actually defaulted:

Grade B's observed rate is 30% above prediction. One year can be noise, so the validator checks whether the gap is statistically significant given the number of borrowers, whether it persisted in earlier years, and whether something has changed in the grade, such as a new customer segment. If the gap is real, the model is recalibrated or an overlay applied until it is.

What are discrimination and calibration in ECL model validation?

Discrimination is the model's ability to rank borrowers: do those it rates riskier actually default more often? It is usually measured with the Gini coefficient or the area under the ROC curve. Calibration is whether the level of predicted PDs matches observed default rates, as in the back-test above. A model can rank well but be poorly calibrated, or the reverse; ECL needs both.

What else does validation cover?

• Conceptual soundness: are the method and assumptions appropriate for the portfolio and for IFRS 9?
• Data quality: are the inputs complete, accurate and representative?
• Stability: has the portfolio changed so much that the model no longer fits?
• Forward-looking components: do the links between economic variables and losses still hold?
• Benchmarking: does a simpler or alternative model give similar results?
• Implementation: does the production system calculate what the model documentation says?

How often should ECL models be validated?

A full validation before a model is first used and periodically afterwards, often every two or three years for significant models, with lighter annual monitoring of performance in between. A material change to the model, its data or the portfolio triggers a new validation.

Are scenarios and overlays validated too?

They should be reviewed, even though they rely more on judgement than models do. Validators check that scenario variables move consistently, that weights are supported, and that overlays are not double counting risks the models already capture. A large and growing share of the allowance coming from overlays is itself a validation finding.

What is independent validation?

Validation is one strand of a bank's wider accounting framework; see bank accounting.

Banks have a model risk management framework in which models are validated by a team independent of the people who built them, before first use and periodically afterwards, with findings tracked to resolution. Supervisors expect this for significant models. Validation findings often lead to management overlays while the model is fixed; see post-model adjustment governance.

What does a good discrimination result look like?

There is no universal pass mark: retail scorecards with rich data typically rank borrowers better than models for low-default portfolios such as large corporates. What matters most is whether performance is stable over time and adequate for the portfolio, as judged against the model's own history and peers.

How can a company validate its provision matrix?

By back-testing: take last year's allowance on each ageing bucket and compare it with the write-offs that actually arose from those receivables over the following year. If last year's allowance was 25,000 and write-offs from those balances were 40,000, the loss rates were too low and need updating. Doing this every year, and recording the result, is simple, effective and exactly what auditors look for.

Where to go next

See how auditors test ECL, probability of default and historical loss rates.

This guide is general information. It is not tax or legal advice for your situation.