Science and Data Journalism at Nate Silver’s New 538
Science was supposed to be part of the mix when Nate Silver, the highest profile statistician in history, first announced his move last year. That move being, you’ll recall, that he was taking his 538 political blog away from the New York Times to ESPN and expanding it into the data journalism site. In the intervening months, though, announcements about his plans emphasized his traditional strongest interests, sports and politics. I thought maybe science had been dumped, or at least postponed.
Not so. The newly launched 538 does include science, although the choice of staff writers for the science section is . . . peculiar. There are no science journalists or science writers–so far, at least. The choice of topics has been strange, too.
Knight Science Journalism Tracker Paul Raeburn walks you through some of his objections to the science coverage, which he characterizes as tepid and sometimes silly and also tardy. Such as a piece on calories burned during sex (which, he points out, is a topic already covered by Woman’s Day.)
Raeburn saves his biggest guns for 538′s lead science item of that day, a very odd piece on how to evaluate health news that advises paying attention to your gut feelings about the validity of the story. That was particularly odd advice because it was written by Jeff Leek, an associate professor of biostatistics and oncology at Johns Hopkins and a blogger at Simply Statistics.
Raeburn takes this idiocy–by a statistician!–down easily, and the rest of the site’s opening science pieces as well. But there seems to be no answer to the question that baffled me too: Why would a news site that’s supposedly built on data journalism advise people to base judgements about newsworthiness on “gut feelings,” the antithesis of data?
The climate at 538
One of 538′s listed science contributors is Roger Pielke, Jr., a professor of environmental studies at the University of Colorado Boulder. He is thought of as a climate change skeptic in the sense I outlined here last year–one who acknowledges that global warming is happening but does not favor drastic measures to slow it down. This choice of Pielke, as you may imagine, had a lot of the climate folks foaming at the mouth.
At the Columbia Journalism Review’s Observatory, Alexis Sobel Fitts presents a critique of the 538 science section, including Leek’s piece. But it’s also a takedown of Pielke’s post, which argues that recent increase in the cost of weather damage is a function of increased infrastructure in vulnerable places and has nothing to do with climate change. Judith Curry, a climate scientist and sometime contrarian who blogs at Climate Etc. also considers 538 in general and Pielke in particular, taking exception to Fitts’s comments. Curry also includes links to and quotes from other critiques, pro and con.
538′s data journalism lacks data
Raeburn’s view that 538 so far lacks the data and analyses it claims to be about is shared by others. Mark Coddington, at the Nieman Journalism Lab, links to several critiques of the new 538; many voice that same objection. Most of these posts are not about 538′s science coverage, and some simply seem offended by the very idea of data journalism. Psychologist Adam Waytz, at the SciAm blog The Moral Universe, thinks what he calls “quantiphobes” resist expressing subjective subjects in objective terms. He says, “numbers make things more fact-like, and facts can evoke discomfort.” Scary.
The same objection was posted most pungently by Noah Smith, who says at Noahpinion,
I’m a big Nate Silver fan, but let me join the chorus of people looking at his new “data-driven” blog site and saying “WTF?”. As far as I can tell, it’s barely data-driven at all! . . .
The problem with the new FiveThirtyEight is not one of data vs. theory. It is one of “data” the buzzword vs. data the actual thing. Nate Silver is a hero of mine, but this site is not living up to its billing at all.
Smith’s is a detailed and persuasive blast at Pielke, noting that Pielke’s post contains no statistical analysis and doesn’t count costs that should be included, such as reinforcing buildings and relocating crops. Smith objects to other posts at 538 on economic subjects for the same reason: lack of data and data analysis.
The pregnant pause at 538
In adddition to Pielke, Silver has also hired University of Chicago health economist Emily Oster to write about science. Oster’s claim to fame is that she resented the arbitrary-sounding edicts of her obstetrician, such as limits on weight gain. The current advised limit of 35 pounds that so chafed her sounds delightful and doable to me. In my pregnant day, back in the last century, it was just 20 pounds. I didn’t make it.
Anyway, instead of dumping her doc and finding one less autocratic, Oster wrote a book, Expecting Better, that investigated whether there were scientific bases for the pregnancy shibboleths. Her most controversial claim is that there is no evidence that light drinking damages a fetus. (“Light drinking” she defines as 1 drink a day after the first trimester. Isn’t 1 drink daily the recommended limit for all women, pregnant or not?)
Her opening shot at 538 is about pregnancy too, taking on recent studies suggesting that acetaminophen during pregnancy can have long-term consequences for the children. One study suggests behavior problems and slow motor development at age 3, and the other attention-deficit hyperactivity disorder at age 7.
Oster argues that it’s difficult to get a good sense of the risks outlined in these studies because of the traditional way results are reported. She’s right. It’s a problem with many medical studies and a very useful point to keep hammering at. “For example, a 30 percent increase in risk on a baseline of 10 percent means an increase from 10 percent to 13 percent. A 30 percent increase in risk on a baseline of 1 percent is an increase from 1 percent to 1.3 percent. In terms of the number of people affected, these are quite different.” The reported risks of damage in the developmental study turned out to be much larger than the small ones in the ADHD study.
Nor were these randomized controlled trials, raising the question of whether acetaminophen-users are different from nonusers. At a guess, they had more pain, or at the very least lower pain thresholds. Oster wonders whether it was not the drug itself but whatever prompted using the drug that encouraged later ill effects. In the ADHD study the differences were significant; the users had more psychiatric diagnoses than the nonusers.
Also, she notes, the results in both studies could just be wrong. Her example comes from physics, the recent erroneous faster-than-light results. It struck me as an odd to choose a mistaken study from physics for comparison, since there are plenty of examples of errors in biomedical studies. Oster’s bottom line: “We have a tremendous amount of evidence that Tylenol is safe in pregnancy, and now a little bit of evidence that maybe there are some risks. We should, perhaps, be more cautious than before, but only a little.”
This piece is not difficult or technical writing–in fact there’s not much data in it, which seems to be the trend at 538. But it’s more than 1300 words long. An experienced science writer could have explored the same cautions about these studies more crisply, engagingly, and briefly. Half the length or less. Indeed, science writers do this sort of thing every day. It might have involved interviewing Oster or one of her colleagues plus some docs, but science writers do that every day too. And there are a not-insignificant number of science writers who could have done this analysis on their own.
So far, the pieces I’ve seen at 538 are not data-intensive analyses of the sort that Silver specialized in at the Times. I’m not sure who Silver thinks 538′s audience is, but it’s a rare reader–even among pregnant women–who will stick around to wade through 1300+ words. My summary above hit most of the important points and even included a couple of quotes from Oster’s piece. 310 words.
Oster seems to be on the staff, so presumably she will be writing about science often. I assume she will be extending her subject matter beyond pregnancy even though she wrote a book about it. A single topic seems a pretty narrow focus for a science section. Her hire is one way of dealing with the XX Issue, and it’s hard to think of a more XX Topic than pregnancy. But there are tons of other XX subjects loaded with potential for data analysis and commentary. They range from science about women–much to deride and pick apart there–to women in science, where there is much to mourn. OTOH, Oster is a health economist, so she probably won’t be writing only about women.
The future of 538
Having been through a number of startup pub launches myself, two of which have survived for decades, I caution against issuing final judgements about 538 in the first wretched days, or weeks, or even months. I can tell you from experience that if it looks as if the folks at 538 are making it up as they go along, that’s because they are. It’s what you do at a startup.
I expect they’ll get better. Maybe they’ll even figure out how to do data journalism, which would be a boon for everybody. One commentary I read suggested that 538 badly needs a news editor. More than 1, I should think. There’s a lot of work to do.