When Economists Make Mistakes

In 2010, Harvard University economists Carmen Reinhart (then at the University of Maryland) and Kenneth Rogoff published a paper concluding that economic growth stagnated when a country had very high public debt. "Growth in a Time of Debt" has been cited more than 250 times and was widely referenced by U.S. and European policymakers advocating austerity measures following the Great Recession.

So it made headlines in 2013 when Thomas Herndon, a graduate student at the University of Massachusetts Amherst, discovered a spreadsheet error in Reinhart and Rogoff’s work — an error that Herndon, in a paper with his professors Michael Ash and Robert Pollin, said disproved the negative relationship between large debts and growth. (Reinhart and Rogoff have acknowledged the error but don’t believe it alters the substance of their conclusions.)

"Growth in a Time of Debt" was published as part of the proceedings of the American Economic Association (AEA) 2010 annual meeting. While conference papers are reviewed by editors, they aren't subject to a formal peer review, the traditional imprimatur of academic publishing. But peer review isn't necessarily intended to catch simple data errors, and sometimes economics articles — including peer-reviewed ones — are later found to contain mistakes of one kind or another. Such incidents have raised the question: Who’s checking?

To conduct a peer review, the editor of a journal asks other experts, known as referees, to read a paper to ensure that it's an important contribution to the field and that the conclusions are credible. Referees remain anonymous to the authors, so they feel free to offer their honest opinions.

Traditionally, social science journals also have maintained the authors' anonymity during the review process to prevent a referee from being swayed by an author’s reputation (or lack thereof). But the wide dissemination of working papers online has made it easy to learn an author's name by entering the paper's title into a search engine. That led the AEA to drop such "double-blind" reviewing for all of its journals. An added benefit is that knowing who the authors are could help referees identify potential conflicts of interest.

Sometimes the system can be gamed. In July, SAGE Publications announced it was retracting 60 papers from the Journal of Vibration and Control, a well-regarded acoustics journal, after the discovery of a "peer-review ring." A scientist in Taiwan had created more than 100 fake identities in an online reviewing system, which authors then used to write favorable reviews of each other's — and sometimes their own — papers.

Nothing so nefarious is known to have happened in economics, but sometimes a discipline becomes clubby, says Penny Goldberg, an economist at Yale University and the editor of the American Economic Review. "Then you can end up in a bad equilibrium where people support each other and recommend acceptance even if the papers aren't very strong."

Authors aren't perfect, either. In a survey conducted by Sarah Necker of the Walter Eucken Institute in Germany, 2 percent of economists admitted to plagiarism, 3 percent admitted fabricating some data, and 7 percent admitted using tricks to increase the statistical validity of their work. Between one-fifth and one-third acknowledged practices such as selectively presenting findings in order to confirm their hypothesis or not citing works that refuted their argument. Referees and editors aren't necessarily on the lookout for such practices. "As an editor, my role is not to be the police," says Liran Einav of Stanford University. Einav is a co-editor of the journal Econometrica and an associate editor of several other journals. "If someone is doing something bad, hopefully the market is efficient enough that eventually they will get caught."

Editors also are under pressure to meet publication deadlines, as William Dewald, emeritus professor at Ohio State University and a former editor of the Journal of Money, Credit, and Banking, wrote in a chapter for the 2014 book Secrets of Economic Editors. As a result, "Some weaker papers slip through." And authors are under their own pressure to publish as many papers as possible, which may lead them to make mistakes.

When mistakes are found, it's often because someone tries to replicate the original study. Some top journals, including those published by the AEA, require authors to share their data and programs (with some exceptions for proprietary data), and many economists post data on their personal websites as well.

Some economists have argued that replications should be much more common in order to keep the profession honest. But there's the potential to overdo it. "Replication is very important and should be strongly encouraged, but we should realize if we spend too much time replicating other people's studies, it could generate a lot of noise," says Einav. "You could easily see how it gets into a spiral and researchers just spend all their time responding to the replicators. That’s not very productive." Also, Einav adds, the mere possibility of replication might be enough to make economists extra careful.

Ultimately, the market is the final test. "If a question is interesting and policy relevant, then people will try to replicate the results," says Goldberg. "This is constructive — this is how science progresses."