How an arcane statistical law could have prevented the Greek disaster

The EU can use a mathematical law as an early warning system for manipulated macroeconomic data, writes Hans Christian Müller in a guest post for “Economics Intelligence”.

If we had known what we know today, Greece would not have been able to enter the Euro area. The macroeconomic data the country reported to Eurostat in Luxembourg were heavily tweaked. Unfortunately, however, this only became clear years after it was too late.

Fact or fiction? Greek GDP numbers (Source: Philly boy92 via Wikipedia)

An early warning system could have avoided such a rude awakening – and, astonishingly, it could be set up rather easily. A simple mathematical law delivers circumstantial evidence about data manipulation, as four German economists show in a paper entitled Fact and Fiction in EU-Governmental Economic Data” that is forthcoming in the “German Economic Review”.

Gernot Brähler (University of Ilmenau), Stefan Engel (University of Eichstätt-Ingolstadt), Max Goettsche and Bernhard Rauch (both: University of Regensburg) apply a rule that is known as “Benford’s Law” among statisticians.

[/caption]

To the statistical layman, this rule appears weird at first glance, but it is mathematically proven. If you want to spot manipulated data, you just have to count how often numbers in a dataset start with a 1 or a 2. Intuitively, one would suspect that all digits from 0 to 9 should be evenly distributed.

However, in reality this is not the case. In sufficiently large datasets the first digit of a number on average is much more often a 1 or a 2 than a 8 or 9. While about 30 percent start with a 1, the frequency decreases the higher the digit. The 9, for instance, occurs in the first place only in less than five percent of the cases.

This law applies to a variety of data, as the US-physicist Frank Benford showed back in 1938: From Baseball results to the lengths of rivers to street numbers from the phonebook – and to macroeconomic data, as recent evidence reveals. According to statisticians, it is almost impossible to manipulate data in a way that a certain outcome is guaranteed and Benford’s Law is met at the same time. Hence, tax authorities in several countries are using Benford’s Law as a default testing device.

In 2009,  Karl-Heinz Tödter, a researcher with Bundesbank, suggested that scientific referees could check the coefficient tables within empirical economic studies with this method. This prompted a vivid debate among German economists.

The authors of the forthcoming paper analysed official statistics of the EU member states from the last eleven years by counting the first digits.They looked at 130 different values per country and year. Among other things, they looked at the total level of debt, the cash reserves of the government and the pensions of retired civil servants.

The result is straightforward. Judged by Benford’s Law, Greece produces the dodgiest data. The distribution of the digits differ the most from the Benford distribution. This is a strong indication of creative accounting by the Greek government.

The authors interpret this result carefully and stress that nobody should jump to conclusions:

“In summary, deviation from Benford’s law is only an indicator of manipulation.”

As with every statistical result, the outcome could be pure chance. Nevertheless, the method is well suited as a first routine test for detecting poor data. Asserts Bernhard Rauch:

“The results provide a clear order in which the country data you should then be investigated further.”

There are several signs indicating that the results are not just coincidental. For example, statistics from the Czech Republic, Sweden and the UK meet the Benford distribution particularly close and hence appear unsuspicious. All those countries have no interest in joining the Euro and therefore do not have to meet the convergence criteria. On the other hand, statistics from Latvia, Belgium and Romania (the latter notorious for its corruption) appear very suspicious, as well.

After the Euro zone debt crisis broke out, Brussels expanded Eurostat’s responsibilities. Europe’s statistical office does not have to trust the numbers they receive from the individual member states. Nowadays, Eurostat’s staff is also allowed to visit the national agencies and to inspect all interesting files. However, this is a very tedious and resource consuming effort and Eurostat has only limited manpower. While its tasks were upscaled, its workforce was not.

Benford’s Law could raise a flag with regards to potential manipulation and could guide the supervisory effort. For example, the Benford test shows that in the year of 2000 the Greek statistics were particularly shady. One year later Greece finally joined the Euro area.

Note: This is a guest post written by Hans Christian Müller

 

Update: The paper later was picked up by Tim Harford in the Financial Times and Ben Goldacre in the “Guardian”.

20 Comments

Filed under Financial Crisis, General Economics

20 Responses to How an arcane statistical law could have prevented the Greek disaster

  1. Great post!!! Very entertaining. At least from the perspective of a mathematician ;-) That said in general I do not like this Greek bashing from German mercantilists. I hope your post does not add fuel to the fire of some hopelessly wrong German economists and politicians.

  2. Nick P.

    Re

    1) “If we would have known what we know today, Greece would not have been able to enter the Euro area.”
    Would any country have entered? Just asking?

    2) “The macroeconomic data the country reported to Eurostat in Luxembourg were heavily tweaked.”
    Heavily is a relative and imprecise term. Did Germany not “tweak” the data, for example?

    3) “Unfortunately, however, this only became clear years after it was too late”.
    Was sort of known all the time and was officially reported and denounced by new government (New Democracy) when it came into power in 2004. .

    @Stephan:
    “That said in general I do not like this Greek bashing from German mercantilists. I hope your post does not add fuel to the fire of some hopelessly wrong German economists and politicians.” << Very well said.

    • Nick, Stephan: This post was not meant as Greek bashing at all, and I do not consider myself a mercantilist as well (I don’t think Hans Christian does). I’m a staunch critic of the German stance regarding Greece, as my Target2 dispute with HWS should have shown.

      I’m not aware of any outright tweaking of statistics by Germany. Without any doubt, the German government has violated and watered down the Stability and Growth pact in 2003.

  3. @Olaf
    Again: this post is very interesting and entertaining! Many thanks to Hans Christian and you for publishing and investigating these sort of stuff. Unfortunately in these times I feel always obliged to add this caveat because there are numerous self-nominated German geniuses around looking for further ammunition to torpedo an already sinking ship. I’m aware of the fact that you are not a mercantilist. Which is a notably rare exception for a German economist.

  4. Regarding Target2 and HWS: The only thing which can’t be said of HWS is that he is not stubborn. This guy is obsessing about his Target2 rants and just published his next one on VoxEU. Unbelievable.

  5. Nikos

    A very interesting article indeed.
    I would like to point out a factual error in your last paragraph: “the Benford test shows that in the year of 2000 the Greek statistics were particularly shady. One year later Greece finally joined the Euro area.” Greece’s entrance into the Eurozone was decided the summer of 2000 and that judgment was based, according to the treaties, on the economic data of the years 1998 and 1999. In other world the “shady” statistics of the year 2000 played no part in Greece’s adoption of the euro. Not to be misunderstood, i am not saying that the statistics of 1998, 1999 were factually correct, just that those of 2000 did not mater for the entrance.

    As a general comment I’d like to say something that almost always goes unmentioned in the discussion about Euro and Greece. For Greece joining the Euro wasn’t about the economic gains of the common currency but about the country’s strategic decision to become an integral part of Europe’s core, for geopolitic and security reasons, as well as means of giving the fable internal modernization forces a powerful external ally in their effort to change the country. A battle ultimately lost and a chance wasted, but thats another story….

    Finally, i believe that even if Greece did not join the Euro the disaster for the country’s economy could not be avoided. What could have been avoided is the Europeanization of said disaster.

  6. nigecus

    i think a friend told me the finance ministery is using such algorithms to check firms’ tax statements. people who manipulate data tend to favor certain numbers.

  7. informavorette

    This is an entertaining read, but totally unsuited for as an indicator.

    You see, once the watchdogs start using this as an indicator, they will have to admit using it. It will come out when they accuse somebody and are asked for evidence. So it will work the first time.

    After that, the next country/corporation/whatever with a dodgy report will continue using false numbers in their reports. Only, instead of generating randomly distributed numbers, they will generate numbers with a distribution which complies with Benford’s law. It is trivial to do so. After that, there will be no more indications of doctoring in the data.

    The old law still holds: in a social system, never use an indicator which can be directly influenced.

  8. @Nikos: not just the Europeanization of the disaster, but the Hellenization of Europe’s problems, too. The fashion right now is to respond to every European country’s problems (and to a lesser extent, the US’s) as though they were the same as Greece’s, when they couldn’t be more different.

    @informavorette: but that’s the point: it’s *not* trivial to fake this, at least if you need to meet any other constraints, too (like a desired picture). Various national (and other) revenue offices have been using this for a long time, non-secretly, and it’s still a useful tool for them.

  9. Anonymous Coward

    “To the statistical layman, this rule appears weird at first glance, but it is mathematically proven.”

    This statement is clearly false. Benford’s law is not a theorem. And therefore cannot be “mathematically proven”. It’s just an observation about patterns in real-world data. The author would have found this out if he’d bothered to read the wiki article he links to.

  10. Just a note. There is no real proof for Benford’s law. I can give you a random distribution of numbers that do not follow the law. However, Hill (1997) showed that : “if distributions are selected at random, and random samples are taken from each of these distributions, the significant digits of the combined sample will converge to the logarithmic distribution”

    So, this is like the central theorem for random samples from random distributions, which is much less strong than saying Benford’s law is proven. Because for some distributions it just doesn’t apply.

  11. Benford’s Law shows up Big Fraud pretty well. It works because numbers in spreadsheets can’t be just *anything*, they have natural ranges and maximums. “Paperclip: $2Bn” won’t fool anyone, for example.

    When you accept the values have this limited range, then Benford’s Law becomes easier to understand.

    Eg. Imagine something has a value from 1 to 9, with equal probability. Benford’s Law can’t help you here, you’ll see each digit equally often over a large dataset (or you should do…!).

    But now imagine the maximum number for this entry is 10. Now the digit 1 will appear twice as often as the others, because it appears in both the 1 and 10.

    If the maximum value is 20, now fully half the results include the number 1 (1, 10-19). But also, another result includes the number 2…

    You can see the pattern. As you increase the maximum number, you gradually get more results with 1s in them, then 2s, then 3s, etc. until you finally reach the number 99.

    Just like when the range was only 1-9, now it’s equal probability again – but it’s a rare happening. Once the max value goes to 100, then yet again the number 1 appears more often than its colleagues, and the pattern repeats.

    More often than not, the 1s will indeed be the most popular digit, followed by the 2s, 3s, etc. Benford’s Law.

    PS. And why they didn’t use this with the Greece stats is just unbelievable. Should be Economics 1.0.

    PPS. Who cares about the cases in which Benford isn’t applicable? It most certainly applies regarding national economic stats.

  12. Benford’s (or Newcomb’s if you like) Law is not proven. It is an observation on data. Furthermore it is not arcane. It may not have received the coverage of other empirical observations, but it is not a secret. And anyone who has even bothered to read Nigrini’s book about the Law knows that it can be used to spot possible cases of fraud or other kind of number manipulation.

    But even given all that, the Greek disaster might have not been avoided for the simple reason that management often makes decisions against scientifically backed advice.

  13. Pingback: pinboard September 21, 2011 — arghh.net

  14. Pingback: An Application Of Benford’s Law | Irreal

  15. Andrew

    “On the other hand, statistics from Latvia, Belgium and Romania (the latter notorious for its corruption) appear very suspicious, as well.” Great to see that Europeans found the only source of “notorious corruption”. I just wonder how these mean wolves (Greeks and now Romanians) manage to deceive so bad the good working Germany and other civilized and pristine European systems. They are either amazingly stupid or equally corrupt. Which one?

  16. The EU knows very well what happens with the statistics with every country… So “oh we didn’t know, those greek bastards …” is a good excuse to reduce the whole financial crisis of a system to the sins of a single country …

    I think if politics would be fueled more by science, there could be no guarantee that things would be better.. people who are into science know for a fact that scientists produce crap as well.. and they also like to deceive reviewers to get their names on paper :)

  17. As a financial fraud investigator we used Benford’s Law on a regular basis to see if “selection pressure” has been applied to the data. It is a powerful and simple tool to guide one to the problem areas / transaction / manipulators etc… This adds creditability to the Greek Ministry of Fiance whose members say they did manipulate the numbers for the benefit of the Greeks. An interesting humanitarian motive – if hollow in its ring – much like Madoff manipulated the numbers for the benefit of his investors.

    In the end – Greece needs to be healed – financially and politically – part of the healing process to to take the politicians and the ministers who fiddled with the data and introduce them to Madoff. I am, however, confident both Greece and the EU lack the will to make such a s decisive move.

    Unless you otherwise object I would like to post this – pots to our humble blog site.

    Great stuff

  18. People who fabricate data are also aware of Benford’s Law, and can employ it to “create” a good Benford fit. Sales receipts will also follow Benford’s Law, and it can be used to monitor fraud in expense accounts as well — simply require receipts for all purchases over $100 and see how many $9X purchases you get. There are also some measurements that can be recognized by the pattern in the second digits.