One of the most senior economists of the German speaking world faces serious questions about his scientific modus operandi. Bruno Frey and his research team are accused of self-plagiarism. Additionally, they at least showed an amazing degree of sloppiness with regard to literature research. Five older publications from different authors on exactly the same research question are missing from the references.
This blog (among others, especially Andrew Gelman’s as well as“Economic Logic” and the EJMR forum) has played a role in making the whole thing public. On Wednesday, 6 July the University of Zurich has started a formal investigation against Frey, based on the “suspicion of unethical scientific conduct”.
Since the information about the affair and the views of the people involved is scattered among a number of posts, I thought it might be a good idea to publish a summary.
What it is all about?
Bruno Frey (University of Zurich), Benno Torgler (Queensland University of Technology) and Torgler’s Ph.D. student David Savage simultaneously published a series of papers dealing with the sinking of the Titanic, but neither cross-reference their own work nor cite a number of older papers by other researcher addressing exactly the same topic.
The articles by Frey, Torgler and Savage appeared in the “Journal of Economic Behaviour and Organization”, “PNAS”, “Rationality and Society” and the “Journal of Economic Perspectives” in 2010 and 2011. They used individual-level passenger data showing the age, gender, ticket class and nationality of 2207 people sailing on the Titanic and employed an econometrical analysis on the determinants of survival.
For several months the authors have been criticised because they simultaneously published nearly identical papers in four different journals without mentioning their other work on the same topic to the editors. This is a clear violation of the submission guidelines of the journals and economists consider this being unethical. The “Journal of Economic Perspectives” rebuked the authors publicly and will publish the conversation with Frey in its forthcoming issue. The editor of the JEBO apparently black-listed the authors with regard to further publications. (see comment by Barkley Rosser).
However, it has become increasingly clear that Frey, Torgler and Savage by no means have been the first economists who econometrically address the survival probabilities of the people sailing in the Titanic. At least five older publications with a very similar research outline and similar results exist. They have all been published before the first working papers of Frey, Torgler and Savage came out and most of them are rather easy to find using the query “Titanic” or “Titanic survivor” on databases like Repec, EconBiz and Google Scholar. However, none of these publications is cited in any of Frey’s et al papers.
The oldest paper on the topic I could get hold of has been published in 1986 by Wayne Hall (“Social class and survival on the S.S. Titanic”) in “Social Science & Medicine”. Hall also related passengers’ chances of surviving the sinking of the Titatic to their sex and their social status: He came to the conclusion that “females were more likely to survive than males, and the chances of survival declined with social class as measured by the class in which the passenger travelled”. If you type in the word “Titanic” into Repec, this paper is the fifth result.
Oddly enough, the author of the 1986 paper is a Professor at the University of Queensland, Brisbane while Frey’s co-authors Benno Torgler and David Savage are both affiliated with the Queensland University of Technology in Brisbane. That’s quite a coincidence, isn’t it?
David Autor (MIT) and currently the editor of the “Journal of Economic Perspectives” (JEP), was furious. The JEP, where one version of the papers by Frey, Torgler and Savage was published, has been dealing with the issue of “self-plagiarism” by Frey et al. for several months. However, the JEP-editors were not aware of the paper by Hall. That “comes as unhappy news to my coeditors and me”, Autor wrote me:
“My reading is that there is considerable overlap between Frey et al. and this 1986 paper.”
However, Autor stresses that he does not accuse them of plagiarism:
“It would be extremely difficult to establish that Frey et al. knew of the Hall paper and had borrowed from it. Absent such proof, I would not make such an allegation. However, Frey et al. should have known about this article. It is their responsibility as scholars to fully research the literature in their field of study, and to acknowledge prior contributions accurately and fairly.”
Bruno Frey and Benno Torgler assure that they neither knew the journal nor the paper nor the author and argue that the connection to Brisbane is just happenstance. In an email, Torgler wrote me:
“Thanks for referring us to other studies. We were not aware of them. We haven’t seen the Hall paper published in Social Science & Medicine beforehand, otherwise we would have cited it.”
Torgler pointed out that they had done a literature research on various issues related to the topic and refers to the numerous articles cited in their papers. He asserted:
“However, it seems that we have missed the ones you have mentioned which is a substantial shortcoming from our part.“
A few hours later Torgler wrote me that they were “puzzled by the fact that we were not able to find the Hall paper” and therefore took a closer look at Repec. He pointed out that the journal “ Social Science & Medicine” apparently was only added to Repec in April 2010 “and therefore after we had done our literature search as the access statistics start then”. He might in fact have a point with regard to his, as the access statistics for the journal and the Hall paper published by RepEc suggest.
Wayne Hall, the author of the 1986 paper, was taken by surprise by the sudden spark of interested in his work. He wrote me:
“It is clear that they [Frey et al.] did not look very hard for previous publications on this topic. My paper pops up on the top of lists on Google Scholar and PubMed when you use “Titanic” and “survival” as search terms. My paper has also been cited in one of Michael Marmot’s books on social class differences in mortality and in a widely used as an introductory sociology text. Their analysis of the reasons for the social class differentials on the Titanic shows little evidence that they have examined previous scholarship on this issue. That would fit with their failure to do any searches that located my paper. As my article shows, the mortality difference between the three passenger classes was discussed at the Mersey Inquiry, in the newspapers of the period, and in the popular literature on the sinking such as Lord’s ,A Night to Remember’.”
In fact, Hall’s paper is by far not the only similar paper Frey, Torgler and Savage missed. They managed to overlook an entire class of statistics publications, too.
The statistical analysis of the passenger data of the Titanic and the determinants of survival have been a favoured case study for statistics courses. In the last 15 years at least four different publications discussed how the passenger data can be used for teaching statistics. It seems to have started with two oddly entitled papers published in the Journal of Statistics Education in 1995 by Robert Dawson (“The “Unusual Episode” Data Revisited”) and in 1997 by Jeffrey S. Simonoff (“The “Unusual Episode” and a Second Statistics Course”).
With regard to these papers, the credit for discovery goes to an anonymous writer who wrote a comment on the blog “Economic Logic and/or to people discussing the subject on the forum “Economic Job Market Rumours”.
Robert Dawson, who I asked to compare his 1995 paper with the work by Frey et al wrote me at he contributed neither data nor methodology and considers it a “minor omission” that his paper isn’t mentioned in the bibliography. However, Dawson stresses:
“I am more concerned by the absence of Jeffry Simonoff’s name from the bibliographies. His paper “The ‘Unusual Episode’ and a Second Statistics Course” uses logistic regression on “my” data in much the same way that Frey, Torgler, Savage use probit regression (though he does not explicitly interpret the results from an economist’s standpoint); these techniques are close enough that the omission of his article is at the very least careless and unfortunate.”
A quick search on Google Scholar for “Titanic survivor” reveals something even more striking. The first result refers to the “Handbook of Statistical Analyses using SPSS”, published in 2004 by Sabine Landau and Brian S. Everitt. They wrote an entire chapter on the topic “Logistic Regression: Who Survived the Sinking of the Titanic?”. On 23 pages, the authors use passenger data from the Titanic as a case study for the question how to do regressions with SPSS.
Compared to Frey et. al., Landau and Everitt as well as Hall used different datasets: For 1309 passengers they have information about survival, age, gender, ticket class and the number of family members accompanying. Frey, Torgler and Savage also look at crew members and in total have information for 2207 people. The basic research question, however, is exactly the same: “Which of the explanatory variables are predictive of the response, survived or died?”
The Titanic example seems to be so popular in statistics, that a dataset of the passengers and the survivors has been available as a package that’s part of the default distribution of the popular statistics package R, as “Tobias” pointed out in a comment on my blog and somebody also noted on Twitter.
In 2006, another publication using the passenger data as a case study for teaching statistics came out. This time, it appeared in the Journal of Economic Education, was written by Robert Dixon and William Griffiths (both, interestingly, are affiliated with universities in Australia, as well, as Hall, Torgler and Savage are) and was entitled “Survival on the Titanic: Illustrating Wald and Lagrange Multiplier Tests for Proportions and Logits”.
The abstract (here’s a free working paper version) says:
“Students are interested in lecture examples and class exercises involving data connected to the maiden voyage and the sinking of the liner Titanic. Information on the passengers and their fate can be used to explore relationships between various tests for differences in survival rates between different groups of passengers. Among the concepts the authors examined are…”
While it is true that “Social Science & Medicine“ is not part of EconLit (an influential database set up by the American Economic Assoication) and was probably added to RepEc and EconBiz (a database run by the German National Library of Economics) in 2010, the Journal of Economic Education is listed in EconLit. Currently, a EconLit search for “Titanic” lists the paper by Dixon and Griffiths as the 15th search result – seven of the previous 14 results refer to different versions of the paper by Frey, Torgler and Savage.
I’m really stunned. How is it possible that a parcel-learned economic journo like me finds more academic literature about a topic than a research team of three economists, among them two full professors? And how could this stuff pass the hurdle of four different peer-reviewed academic journals? From my point of view, the journals should really rethink their procedures.
David Autor, the editor of the Journal of Economic Perspectives, sees it a differently: “I disagree that this shows a weakness in the peer review process”, he wrote me.
“Referees are not in the business of policing for plagiarism (of self or others). The scientific publishing process relies on the fact that researchers face severe sanctions for fraud or plagiarism. Generally, one such incident can end a noteworthy career, even one decades in the making.”
So let’s wrap the main points up. Which contribution did Frey, Torgler and Savage make, what are their shortcomings and what are the open questions remaining?
- The basic idea to do a statistical analysis of survival probabilities of passengers on the Titanic is not new at all. It is possible that Frey, Torgler and Savage had the idea independently from the older literature. One has to apply the presumption of innocence here.
- However, it is striking that Frey, Torgler and Savage did a literature research which completely overlooked the body of existing literature dealing with the same topic. At least, this shows an egregious degree of sloppiness, especially given that Frey is such a senior and experienced figure.
- Given the eagerness the authors had with regard to publishing their results in four different journals simultaneously, this sloppiness appears even more outrageous.
- No matter why the older literature on the topic is missing, the lack of cross-references between their own papers and the submission of very similar papers at the same time to different journals is a clear violation of the submission guidelines of the journals that are involved and constitute some kind of “self-plagiarism” that is considered unethical.
- This unethical behaviour with regard to their own work tarnishes the credibility with regard to the missing papers from other authors.
- This kind of modus operandi is outrageous given the fact that the third author (David Savage) is a Ph.D student who ought to have been protected better by his advisor Benno Torgler.
- Frey’s conduct is even more amazing given his frequent statements criticising the publishing process in economics and his own work on fraud in academia (Ironically, Frey was working on a big research project dealing with fraud in academia, which currently is put on hold, as Frey’s coauthor on that study told me yesterday). To me, with the benefit of hindsight, his criticism sounds quite hypocritical.
- The fact that the author of the initial paper from 1986 works in the same city (albeit at a different university) as Benno Torgler and David Savage is at least a weird coincidence. Even more stunning is the fact that the authors of the 2006 paper are affiliated with Australian Universities as well. Of course, this can also be pure happenstance, but it is really odd.
- One unique contribution of Frey, Torgler and Savage is the comparison between the sinking of the Titanic and the Lousitania. (Interestingly, this also is the only point where they explicitly claim to be the first ones …)
- A second unique contribution is to tell an economic story around the results (the stuff about norms, selfishness and so on)
- Apart from these two contributions, their work basically is a replication of the older papers using slightly different methods and better data. Replication is very important, and the results of Hall et al are confirmed, which is a good thing too. However, I have serious doubts the four journals would have accepted the papers if the editors had been aware of the true character of the papers.
Remark on my own behalf: Some commets were questioning my own intentions and accused me of “exploiting” this issue. I don’t really have dog in this fight. I’ve liked Bruno Frey’s work, respect him personally and I frequently wrote about him and his papers (see here, for example) favourably.
I still think he has done a lot for Economics, especially with regard to ending the narrow-mindedness of the discipline.
However, his conduct described here really irritates and disappoints me. I think it’s my duty as a journalist writing about Economics Science to make it public. I try hard to present the facts as accurately as possible and give everyone involved the opportunity to make his view public.
If there are any mistakes, distortions or omissions in my posts, please let me know. I’m happy to correct them.
If you happen to like this blog, why don’t you like me on Facebook as well?