Methodology

The principle

An ill-fitting comparison is worse than none, because it looks like an answer. So every figure we consider showing passes one question first: is it comparable at all to what the user told us.

The five fields without which there is no figure

A figure that does not carry all five does not enter the system — not merely that it is not displayed:

• Unit — what it measures.
• Definition — precisely what it includes and excludes. “Income” is not one term.
• Reference period — which month or year it describes.
• Population — who it is about.
• Geography — national, district or local. A national figure is never shown as though it were local.

Match confidence

When we find a peer group, we also say how well it fits. High, medium or low confidence is displayed alongside the figure. A low-confidence comparison may be shown — but never without saying that it is one.

What we refuse to do

• Fill a missing value with a plausible-looking estimate. A missing value stays missing.
• Derive a success rate from a curated collection. A hand-gathered set is not a sample and has no denominator.
• Present one organisation's figures as a picture of the market.
• Uprate an old figure by a multiplier and present the result as official. An estimate is labelled an estimate.
• Reach information by any route not documented and permitted by whoever publishes it.

When there is not enough to compare

There are questions where we cannot give an answer we can stand behind. In that case we say so, and explain why. We do not substitute an approximate figure, and we do not leave a screen blank without explanation.

Versioning

A methodological change that alters a figure already shown is a material change and will be documented. This page will grow as there are more comparisons to explain.