Seth’s blog

A few days ago I blogged about how Tim Lundeen, via careful  and repeated measurement — let’s call it self-experimentation — uncovered a serious and previously-unreported side effect of a drug he was taking. Tim’s example illustrates an important use of self-experimentation: discovering unreported side effects, which I believe are common.

By coincidence today I came across a talk about the very subject of unmentioned side effects: Alison Bass speaking about her new book, Side Effects: A Prosecutor, a Whistleblower, and a Bestselling Antidepressant on Trial. Near the end, Bass said,

It’s not the just the antidepressants, it’s not just the antipsychotics. This is happening with a lot of other drugs. With Vioxx, with Vytorin, an anti-cholesterol drug, with Propries [?] and Marimet [?], anti-anemia drugs. Where again and again the drug companies know that there are more severe side effects and they’re not letting the public know about that. It just keeps happening, unfortunately.

Just as it would be foolish to think the problem is limited to mental-health drugs, it would be foolish to think the problem is limited to side effects, that drug company researchers do everything right except fail to report side effects. Tim’s example shows how hard it is to learn about unreported side effects — so it is only realistic to think that there are other big problems with drug company research we don’t know about. Bass mentioned one I didn’t know about. A company did a clinical trial of Paxil. The goal was to see if the drug helped with Measures of Depression A and B. Turns out it didn’t: no effect. So the company changed the measures! They shifted to reporting different measures that the drug did seem to improve. Creating the hypothesis to be tested after the data supposedly supporting that hypothesis had already been collected. Without making this clear. (Which I presciently mentioned here, in response to an interesting comment by Andrew Gelman.) And if you think that drug companies do research like this — in ways that seriously damage people’s lives — but everyone else, such as academia, is really good, that is as realistic as thinking the problem with drug company research is restricted to side effects. Self-experimentation has all sorts of limitations, yes, but (a) you know what they are and (b) it is cheap enough so that you can gather more data to deal with the problems. Drug company research and lots of other research is too expensive to fail — or even be honest about shortcomings.

This is an aspect of scientific method that scientists rarely discuss: the effect of cost on honesty. Is there an economic term (a Veblen good, perhaps?) for things whose quality goes down as their cost goes up?

If we can believe a movie based on a true story, the doctors consulted by the family with an epileptic son in …First Do No Harm knew about the ketogenic diet but (a) didn’t tell the parents about it, (b) didn’t take it seriously, and (c) thought that irreversible brain surgery should be done before trying the diet, which was of course much safer. Moreover, these doctors had an authoritative book to back up these remarkably harmful and unfortunate attitudes. The doctors in …First, as far as I can tell, reflected (and still reflect) mainstream medical practice.

Certainly the doctors were evidence snobs — treating evidence not from a double-blind study as worthless. Why were they evidence snobs? I suppose the universal tendency toward snobbery (we love feeling superior) is one reason but that may be only part of the explanation. In the 1990s, Phillip Price, a researcher at Lawrence Berkeley Labs, and one of his colleagues were awarded a grant from the Environmental Protection Agency (EPA) to study home radon levels nationwide. They planned to look at the distribution of radon levels and make recommendations for better guidelines. After their proposal was approved, some higher-ups at EPA took a look at it and realized that the proposed research would almost surely imply that the current EPA radon guidelines could be improved. To prevent such criticism, the grant was canceled. Price was told by an EPA administrator that this was the reason for the cancellation.

This has nothing to do with evidence snobbery. But I’m afraid it may have a lot to do with how the doctors in …First Do No Harm viewed the ketogenic diet. If the ketogenic diet worked, it called into question their past, present, and future practices — namely, (a) prescribing powerful drugs with terrible side effects and (b) performing damaging and irreversible brain surgery of uncertain benefit. If something as benign as the ketogenic diet worked some of the time, you’d want to try it before doing anything else. This hadn’t happened: The diet hadn’t been tried first, it had been ignored. Rather than allow evidence of the diet’s value to be gathered, which would open them up to considerable criticism, the doctors did their best to keep the parents from trying it. Much like canceling the radon grant.

The ketogenic diet.

In a science classroom at a middle school I saw a poster about “the scientific method.” There were seven steps; one was “analyze your data.” According to the poster, you use the data you’ve collected to say if your hypothesis was right or wrong. Nothing was said about using data to generate new hypotheses. Yet coming up with ideas worth testing is just as important as testing them.

It’s like teaching the alphabet and omitting half of the letters. Or teaching French and omitting half the common words. While no one actually teaches only half the alphabet or only half of common French words, this is how science is actually taught. Not just in middle school, everywhere. The poster correctly reflects the usual understanding. I have seen dozens of books about scientific method. They usually say almost nothing about how to come up with a new idea worth testing. An example is Statistics For Experimenters, a well-respected book by Box, Hunter, and Hunter. One of the authors (George Box) is a famous statistician.

The curious part of this omission is how unnecessary it is. Every scientific idea we now take for granted started somewhere. It would be no great effort to find where a bunch of them came from.

Did the professionalization of science — people could make a living doing science — cause harm because although more science was done scientists — the professional ones — were no longer free to pursue the truth in any direction? Because their jobs and status were at stake? It’s plausible. Recall that Mendel and Darwin were amateurs. A more recent example is Alister Hardy, the Oxford professor who conceived the aquatic ape theory of evolution. He didn’t pursue it because he feared loss of reputation. The more sophisticated conclusion, I suppose, isn’t that professionalization was bad but that loss of diversity was bad. We need both amateur and professional scientists because each can do stuff the other can’t. Right now we only have professional ones. No one encourages amateur science; there is no way they can publish their work. (Unless, like Elaine Morgan, who wrote several books about the aquatic ape theory, you’re a professional writer.)

These thoughts were prompted by this remarkable blog post, which has nothing to do with science. What an amazing piece of writing, I thought. I don’t even agree with it, and here I am staring at it. A work of genius? No, lots of blog posts are really good. This one was merely better than most. Would something this brazen and effective appear in any major magazine, newspaper, TV show, radio ad, etc.? No, not even. Do we realize that, all these years, stuff like this has been missing from our media consumption? No, we don’t. Before there was news, there was gossip, I realized; news (such as newspapers) was a kind of professionalization of gossip. The blog post I admired was a bit of riveting creative gossip. Blogs are just new-fangled gossip. Bloggers are endlessly scandalized, indignant, judgmental, just as gossips are. Just as gossip is usually “passed on,” most blog posts have links and many posts consist almost entirely of “passing on” something. Just as gossip can be anything, bloggers can say what they really think, as Tyler Cowen pointed out. That’s why they’re so successful, so easy to write and read. Gossip is good for our mental ecology, just as science is. Mark Liberman’s Language Log blog is a blend of (good) gossip and science; as you can see from my interview with him, it filled a gap. I hope blogs will provide a kind of support structure on which amateur science can grow.

I am fascinated by how human nature interferes with science. This article in the Wall Street Journal helped me understand one way this happens.

A civility campaign in Howard County, Maryland, centered on a book called Choosing Civility: The Twenty-five Rules of Considerate Conduct (2002) by P. M. Forni, a John Hopkins professor of romance languages. Rule 7, for example, is “don’t speak ill.” The book bothered Heather Kirk-Davidoff, a pastor. She visited Professor Forni. “Jesus didn’t say, ‘I am the rule,’ right?” she told him. Professor Forni agreed. “Yes, Jesus said, ‘I am the way.’ If I had met you before, probably I would have used way. The 25 Ways of Being Considerate and Kind,” he said.

Hmm. The way versus the rule: similar. The way versus a way: big difference. Neither the professor nor the pastor noted that a better title would omit the: 25 Ways of Being…

The writer of a book about civility — in that very book — fails to grasp a big point about civility. The pastor who points out the problem makes a similar omission. Our tendency to turn tools into rules must be strong.
If you invent a useful tool, you have made the world a better place. If you denigrate non-users, the improvement is less obvious. Randomization, for example, is a tool. Many scientists treat it like a rule. Were I to write a book on scientific method, it would contain a paragraph beginning: “A few years ago, the head librarian of the Howard County, Maryland, county library bought 2300 copies of a book called . . .”
Twisted skepticism.