Damned lies and statistics: Helping numbers make sense

Click to enlarge.

Comparability, reliability, accuracy: all are hallmarks of the OECD’s renowned statistical databases. This is just as well, since good policy and business decisions depend on good numbers. But should you believe what you read?

What would your reaction have been if you had read the following sentence in a 1995 newspaper article: “Every year since 1950, the number of American children gunned down has doubled”? You probably would have been shocked, but would you have spent that much time thinking about the reliability of such an impressive “statistic”? For if you had believed it, as one often does when reading such headlines, you would have been making a big mistake. Suppose that in 1950 only one child in America was shot dead, then doubling this number 45 times, you would reach a number of 35 trillion children gunned down in 1995.

Fortunately, this is an implausible figure. This widely quoted example is recalled by Joel Best in his most interesting book, Damned Lies and Statistics, a title that itself was inspired by a remark that some accord to Disraeli, others to Mark Twain. The example is even more pertinent insofar as it is based on a syntactic mistake. In fact, the correct wording in the original source was “The number of American children killed each year by guns has doubled since 1950” and it was the misquote that disseminated the “false statistic”.

The example points to several other lessons, too. For a start, data (i.e., a numeric value) without appropriate metadata (i.e. information about the meaning of data) do not give any meaningful information. Also, where explanatory metadata is provided, the attention paid by media and other users to it is frequently much lower than the attention they pay to the data itself. This means a “trained” brain may indeed be necessary to be able to assess the reliability of a statistical value.

Unfortunately, this creates a general attitude of leaving to media and other experts the role of selecting the “correct statistical information”, with the userpublic left to judge the credibility of the source, assuming it has the tools to do so. In other words, the layman’s capacity to evaluate the correctness of a statistic is quite limited. Nor can we pretend that each citizen should become a professional statistician by being able to interpret correctly the growing amount of data quoted by the media or available on the Internet.

Therefore, we normally have to rely on somebody else to gather and “read” statistical information for us. And for that, we tend to trust public sources. Yet, there are cases in which we think we are able to measure economic or social phenomena much better than statisticians.

Let’s take the case of inflation. Day after day we play our role of consumers. In doing so, we deal with a large range of prices and we normally compare these prices both between different stores and over time. Does this daily activity give us the capacity to evaluate inflation trends correctly? Most of us would answer, “Absolutely!”. And when official figures (normally based on thousands of observed prices for a large basket of goods and services) are released by the national statistical office, we consumers can then judge whether they are correct, excessive or undershooting, right? No doubt the popular media will exploit public sentiment and at least one headline will trumpet the quip about lies.

Statisticians will say that consumers were mistaken, of course. But are they right? It is often a matter of perception. Consider what happened in 2002 when 12 countries adopted the euro. Several consumers’ associations queried the “true” inflation rate, arguing that the impact of the new currency was to push prices upward. The euro was introduced in January, when the official increase in consumer prices since the same month of 2001 was already around 2.5%. During the spring, newspapers and other media began highlighting that some items (mainly food) were showing large price increases, due to the “rounding up” effect, or to the speculative behaviour of retailers. In some countries “private” surveys – many of which were more keen on scoring political points than explaining the underlying truth – were hastily launched (sometimes only covering food items) and they showed that the “true” inflation rate was much higher than the official one (calculated by Eurostat), at 1.8% in June and 2.1% in September.

Curiously, very seldom were the methods of these private evaluations scrutinised by the media, and certainly not to the extent that official figures have been. The main reason for these divergences is that consumers buy certain products daily or weekly, like food and clothes, and other products very rarely, if at all, like televisions, computers and cars. Therefore, if there are very different trends in prices for various groups of products, it can be difficult to evaluate the overall trend. And if households with different income levels buy different baskets of good and services, and if the distribution of price movements is heterogeneous (with large increases on “basic” items and price-falls on hi-fis and computers, etc.), then the inflation rate as perceived by various groups of households will be very different.

In addition, consumers often have trouble evaluating the effect on prices of the quality of goods and services, in particular products with a high technological content, like cellular phones, for which prices have dropped sharply since the start of the 1990s. Unlike consumers, national statistical offices spend resources adjusting price movements for these quality changes. Perhaps it is technical adjustments like these that lay statisticians open to accusations of “massaging” the inflation figures for political ends. Frankly, such trickery is unlikely, given the large number of people involved in compiling consumer price statistics in each country.

And anyway, there is a very long tradition of strong political independence in almost all European statistical offices. Competition from other (often private) sources of information poses another problem for official statisticians. First, it makes the job of identifying reliable sources that much harder. This is particularly true of social statistics, and in overall measures of global well-being. Policymakers and the general public often like to use comparative indicators to measure if their situation is improving or worsening, or better than in other countries. Thus, newspapers and journals are full of reports about indicators on well-being, competitiveness, development, etc., where several time series are combined to derive a single measure and performances are ranked in ascending or descending order. But social indicators are really too complex for this kind of reductionist treatment. At the very least, caveats should be made clear, which is too often not the case.

Such ranking of indicators may be appealing, but they are subjective, reflecting only the compiler’s view of the relative importance of the various aspects taken into account. On the other hand, statisticians have to respond to user needs, by finding correct and transparent ways to measure factors that contribute to the well-being of our societies, using a variety of tools, including reliable and explainable indicators.

So, while there are cases where statistics and common sense can diverge, unraveling the truth and explaining it accurately is exactly the role of official statistics. Making an effort to communicate results is very much part of the package, too. Only by bridging the gap between the complexity of statistical methods, and the needs of users in having clear and understandable results, can official statistics be seen by public opinion as the most trustworthy source of information.

But if statisticians ignore these pressures, then people will continue to base their judgements on unreliable figures or indicators. In doing so, they will become even more convinced that all statistics, whatever their source, are beneath even the damnedest of lies.

References 

Best, J. (2001), Damned Lies and Statistics. Untangling Numbers from the Media, Politicians and Activist, University of California Press.

©OECD Observer No 237, May 2003




Economic data

E-Newsletter

Stay up-to-date with the latest news from the OECD by signing up for our e-newsletter :

Twitter feed

Suscribe now

<b>Subscribe now!</b>

To receive your exclusive paper editions delivered to you directly


Online edition
Previous editions

Don't miss

  • Africa's cities at the forefront of progress: Africa is urbanising at a historically rapid pace coupled with an unprecedented demographic boom. By 2050, about 56% of Africans are expected to live in cities. This poses major policy challenges, but make no mistake: Africa’s cities and towns are engines of progress that, if harnessed correctly, can fuel the entire continent’s sustainable development.
  • “Nizip” refugee camp visit
    July 2016: OECD Secretary-General Angel Gurría visits the “Nizip” refugee camp, situated between Gaziantep and the Turkish-Syrian border, accompanied by Turkey’s Deputy Prime Minister Mehmet Şimşek. The camp accommodates a small number of the 2.75 million Syrians currently registered in Turkey, mostly outside the camps. In his tour of the camp, Mr Gurría visits a school, speaks with refugees and gives a short interview.
  • OECD Observer i-Sheet Series: OECD Observer i-Sheets are smart contents pages on major issues and events. Use them to find current or recent articles, video, books and working papers. To browse on paper and read on line, or simply download.
  • Queen Maxima of the Netherlands gives a speech next to Mexico's President Enrique Pena Nieto (not pictured) during the International Forum of Financial Inclusion at the National Palace in Mexico City, Mexico June 21, 2016.
  • How sustainable is the ocean as a source of economic development? The Ocean Economy in 2030 examines the risks and uncertainties surrounding the future development of ocean industries, the innovations required in science and technology to support their progress, their potential contribution to green growth and some of the implications for ocean management.
  • OECD Environment Director Simon Upton presented a talk at Imperial College London on 21 April 2016. With the world awash in surplus oil and prices languishing around US$40 per barrel, how can governments step up efforts to transform the world’s energy systems in line with the Paris Agreement?
  • Happy 10th birthday to Twitter. This 2008 OECD Observer interview with Henry Copeland said you’d do well.
  • The OECD Gender Initiative examines existing barriers to gender equality in education, employment, and entrepreneurship. The gender portal monitors the progress made by governments to promote gender equality in both OECD and non-OECD countries and provides good practices based on analytical tools and reliable data.
  • Once migrants reach Europe, countries face integration challenge: OECD's Thomas Liebig speaks to NPR's Audie Cornish.

  • Message from the International Space Station to COP21

  • The carbon clock is ticking: OECD’s Gurría on CNBC

  • If we want to reach zero net emissions by the end of the century, we must align our policies for a low-carbon economy, put a price on carbon everywhere, spend less subsidising fossil fuels and invest more in clean energy. OECD at #COP21 – OECD statement for #COP21
  • They are green and local --It’s a new generation of entrepreneurs in Kenya with big dreams of sustainable energy and the drive to see their innovative technologies throughout Africa. blogs.worldbank.org
  • Pole to Paris Project
  • In order to face global warming, Asia needs at least $40 billion per year, derived from both the public and private sector. Read how to bridge the climate financing gap on the Asian Bank of Development's website.
  • How can cities fight climate change?
    Discover projects in Denmark, Canada, Australia, Japan and Mexico.
  • Climate: What's changed, what hasn't, what we can do about it.
    Lecture by OECD Secretary-General Angel Gurría, hosted by the London School of Economics and Aviva Investors in association with ClimateWise, London, UK, 3 July 2015.
  • Is technological progress slowing down? Is it speeding up? At the OECD, we believe the research from our Future of ‪Productivity‬ project helps to resolve this paradox.
  • Is inequality bad for growth? That redistribution boosts economies is not established by the evidence says FT economics editor Chris Giles. Read more on www.ft.com.
  • Interested in a career in Paris at the OECD? The OECD is a major international organisation, with a mission to build better policies for better lives. With our hub based in one of the world's global cities and offices across continents, find out more at www.oecd.org/careers .

Most Popular Articles

Poll

What issue are you most concerned about in 2016?

Unemployment
Euro crisis
International conflict
Global warming
Other

OECD Insights Blog

NOTE: All signed articles in the OECD Observer express the opinions of the authors
and do not necessarily represent the official views of OECD member countries.

All rights reserved. OECD 2016