Portside Date: May 29, 2015

Author: Tabitha M. Powledge

Date of source: May 29, 2015

Plos Blogs

There’s an interesting meta-question growing out of the flap over that Science paper that’s just been retracted.  I speak, of course, of the one by UCLA grad student Michael LaCour and Columbia political scientist Donald Green, published last December, the one purporting to show that people’s political attitudes (in this case opposition to gay marriage) can be overturned in the course of persuasive conversation with an advocate.

The meta-question grows out of the size and decibel level of the flap–if not the biggest ever, then certainly among the top ten. Could this be the beginning of a real reversal in the problem of fraud and misconduct in science?

And perhaps, she ventured hopefully, could it mean reduced gullibility of the media? Will it change the reflexive practice of swallowing whole any old paper if it’s on a hot topic like gay marriage, especially one in a major journal like Science?

In this case credulity swept over even savvy journalists and statistically expert political scientists–even while they noted with astonishment that the reported results were contrary to what previous researchers had found. In a post last week at Vox, Dylan Matthews acknowledged that originally he had called the results “kind of miraculous.” His new piece appeared under the forthright hed “This was the biggest political science study of last year. It was a complete fraud.”

At The Monkey Cage, political scientist Andrew Geller confessed in his mea culpa last week that he had marveled repeatedly in his original piece at the size of the reported effects, and even theorized about what could explain them. What did not cross his mind was the most parsimonious explanation: phony data. “The message, I suppose, is to be aware of the possibility that someone’s faking their data, next time I see an effect that’s stunningly large.”

How the truth came out

The fraud story came out, as is so often the case these days, via that essential science blog Retraction Watch. Ivan Oransky followed up a tweeted tip from Science News deputy managing editor Lila Guterman. Guterman, trolling Twitter in the wee hours, had seen a tweet about a paper posted on a Stanford web site that had uncovered fraud in the LaCour-Green paper. Here’s the analysis paper.

The irony about the fraud’s discovery is that paper authors David Broockman and Joshua Kalla, both grad students, delved into the Science paper’s data not because they smelled a rat, but because they were excited about the study and wanted to extend it. They weren’t even trying to replicate it.

The original study had been based on a project run by Dave Fleischer, who heads the Leadership Lab at the Los Angeles L.G.B.T. Center. The project sent gay canvassers into the Los Angeles neighborhoods where voters had supported Proposition 8, which banned same-sex marriage in California. “The canvassers followed standardized scripts meant to convince those voters to change their minds through non-confrontational, one-on-one contact,” says Maria Konnikova in her recap at the New Yorker.

This canvassing part of the study appears to have been carried out as planned. The LaCour-Green idea was to follow up the canvassing some months later to see if the persuasion methods had worked and attitudes toward gay marriage had changed. The December 2014 paper claimed that they had, dramatically.

More juicy details in  Naomi Shavin’s piece at the New Republic.  At the Chronicle of Higher Education, Steve Kolowich did a Q&A with data sleuth Broockman.  Data journalist Carl Bialik’s recap at FiveThirtyEight goes into some detail about the statistical clues that alerted Broockman and Kalla that the data were bogus.  At Americablog, Jon Green describes work by Tim Groseclose suggesting that LaCour faked data in other work too.

Speculations about fallout and the future of fraud

In an interview with Jesse Singal at Science of Us, senior author of the retracted paper Donald Green says “I’m quite confident that people are going to do this experiment. I want to do this experiment.” The irony is, Green says, is that there really was an experiment–the canvassing undertaken by the L.G.B.T group. What’s needed is to evaluate the results, as the December paper pretended to do. “Dave Fleischer and his canvassing team, they really did bust their hump to do these interventions, to give treatment messages, placebo messages, with gay canvassers, with straight canvassers — that’s all true, and it happened not once but in two separate studies,” Green says. “But the outcomes were never measured, so now we just need to do it with real survey data.”

To the idea that the study will be done again and actually carried out this time, Daniel Flynn retorted, “Codifying gay marriage has never been about canvassers, gay or straight, persuading Americans. Voters, after all, rejected same-sex marriage in California, Wisconsin, Oregon, and other blue states only to watch judges order them to embrace it. America’s evolution on gay marriage came as a conversion by the sword.” Flynn’s post appeared at The American Spectator, the prominent opinion journal on the right.  He complimented the Broockman-Kalla exposé, despite the fact that their “Twitter feeds betray a decidedly left-wing bent.”

Much of what has been written has been kind to Green, presenting him as a straight-arrow scientist hoodwinked by perfectly natural trust in a convincing grad student, LaCour, who turned out to be a con man. Charles Seife–best known, probably, for helping to bring down the science writer Jonah Lehrer–is not so forgiving of Green’s (and the journal’s) failure to inquire more deeply into LaCour’s nonexistent data. “Science magazine didn’t shoulder any blame, either. In a statement, Editor in Chief Marcia McNutt said the magazine was essentially helpless against the depredations of a clever hoaxer.”

That these things are true indicates that something is very amiss in science, Seife says. “Despite the artful passing of the buck by LaCour’s senior colleague and the editors of Science magazine, affairs like this are seldom truly the product of a single dishonest grad student. Scientific publishers and veteran scientists — even when they don’t take an active part in deception — must recognize that they are ultimately responsible for the culture producing the steady drip-drip-drip of falsification, exaggeration and outright fabrication eroding the discipline they serve.”

At Poynter, James Warren tries to extract lessons for journalists from this tale. Nothing you haven’t heard before, and also not really helpful here. Stop trusting small sample sizes, for example. Good advice, but completely irrelevant for flagging this study as bogus. As I noted above, the fakery fooled real experts. Including the co-author.

Ivan Oransky offered me in a small ray of hope in an email. He pointed out that the sort of statistical analysis Broockman and Kalla did to expose the fraudulent data in this study has become more common. He and Retraction Watch partner Adam Marcus described how this can work in their piece in the new Nautilus. It tells (as the hed says) “How the Biggest Fabricator in Science Got Caught.” That would be anesthesiologist Yoshitaka Fujii, who has now retracted 183 Potemkin papers.

In an email to me, Marcus emphasized that journals are more likely to catch fishy results if they pay close attention to the data in a submitted paper. But he also notes that it has not been the traditional role of peer reviewers or editors “to identify sketchy results.”

I didn’t know that.  Marcus is a journal editor as well as a founder of Retraction Watch, so I’ll take his word for it. But I’m a mere science journalist and have believed for years that this was exactly the sort of thing journal editors and peer reviewers were supposed to be doing, identifying sketchy results. Isn’t that the point of gatekeeping? Isn’t that why publishing in a peer-reviewed journal is desirable? Isn’t it a kind of seal of approval? If not, then what is journal editing and peer-reviewing about?

Marcus does have advice about changing that, however. He told me, “[A]s the tools of statistics are increasingly brought to bear in science publishing — as they were in the Fujii case, retroactively — the harder it will be to cheat. So editors and publishers should welcome these tools and find ways to use them more often.”

At Covering Health, the blog of the Association of Health Care Journalists, Tara Haelle has similar advice for journalists, based on something Oransky said in an interview: Keep a biostatistician in your back pocket.

Excellent advice. Also completely impractical for most journalists in most media in most circumstances. Some biostatisticans may be able (and willing) to spot fishiness in a cursory reading of  some manuscripts, especially if the imploring journalist is a friend or relation. Very few would be eager to plunge into a time-consuming extended analysis–especially as volunteers.

The New York Times and a handful of other top media would probably be willing to pay for such a service in a very small number of very occasional, very high profile cases. But a tame biostatistician on call is never going to be a real possibility for most science and medical journalists.  Science writers will have to pray for more scientist-heros like Broockman and Kalla–and more devoted journalistic (and apparently insomniac) tweet-scanners like Lila Guterman.

Our update is now ended, back to the original post

Oransky and Marcus argued in a New York Times op-ed last Saturday that the central problem in reducing fraud is doing something about incentives. By incentives they mean the drive to publish on a hot topic in a hot journal, which is the key to getting and keeping a job in science.

“But as you well know, there’s a scandal like this at least once a year, in fields from social psychology to stem cells, and yet the fraud continues,” Oransky told me.

So, back to my original question. Could this exceptionally noisy example begin a real reform process and do something about fraud and misconduct in science? Also, is there hope that those who write about science and medicine will give up simple regurgitation and get their skepticism on?

Nah.

Tabitha Powledge is an award-winning long-time science journalist, book author, and media critic. On Science Blogs is her weekly look at the craft and content of science blogging.