Seth’s blog

If we can believe a movie based on a true story, the doctors consulted by the family with an epileptic son in …First Do No Harm knew about the ketogenic diet but (a) didn’t tell the parents about it, (b) didn’t take it seriously, and (c) thought that irreversible brain surgery should be done before trying the diet, which was of course much safer. Moreover, these doctors had an authoritative book to back up these remarkably harmful and unfortunate attitudes. The doctors in …First, as far as I can tell, reflected (and still reflect) mainstream medical practice.

Certainly the doctors were evidence snobs — treating evidence not from a double-blind study as worthless. Why were they evidence snobs? I suppose the universal tendency toward snobbery (we love feeling superior) is one reason but that may be only part of the explanation. In the 1990s, Phillip Price, a researcher at Lawrence Berkeley Labs, and one of his colleagues were awarded a grant from the Environmental Protection Agency (EPA) to study home radon levels nationwide. They planned to look at the distribution of radon levels and make recommendations for better guidelines. After their proposal was approved, some higher-ups at EPA took a look at it and realized that the proposed research would almost surely imply that the current EPA radon guidelines could be improved. To prevent such criticism, the grant was canceled. Price was told by an EPA administrator that this was the reason for the cancellation.

This has nothing to do with evidence snobbery. But I’m afraid it may have a lot to do with how the doctors in …First Do No Harm viewed the ketogenic diet. If the ketogenic diet worked, it called into question their past, present, and future practices — namely, (a) prescribing powerful drugs with terrible side effects and (b) performing damaging and irreversible brain surgery of uncertain benefit. If something as benign as the ketogenic diet worked some of the time, you’d want to try it before doing anything else. This hadn’t happened: The diet hadn’t been tried first, it had been ignored. Rather than allow evidence of the diet’s value to be gathered, which would open them up to considerable criticism, the doctors did their best to keep the parents from trying it. Much like canceling the radon grant.

The ketogenic diet.

In a science classroom at a middle school I saw a poster about “the scientific method.” There were seven steps; one was “analyze your data.” According to the poster, you use the data you’ve collected to say if your hypothesis was right or wrong. Nothing was said about using data to generate new hypotheses. Yet coming up with ideas worth testing is just as important as testing them.

It’s like teaching the alphabet and omitting half of the letters. Or teaching French and omitting half the common words. While no one actually teaches only half the alphabet or only half of common French words, this is how science is actually taught. Not just in middle school, everywhere. The poster correctly reflects the usual understanding. I have seen dozens of books about scientific method. They usually say almost nothing about how to come up with a new idea worth testing. An example is Statistics For Experimenters, a well-respected book by Box, Hunter, and Hunter. One of the authors (George Box) is a famous statistician.

The curious part of this omission is how unnecessary it is. Every scientific idea we now take for granted started somewhere. It would be no great effort to find where a bunch of them came from.

Did the professionalization of science — people could make a living doing science — cause harm because although more science was done scientists — the professional ones — were no longer free to pursue the truth in any direction? Because their jobs and status were at stake? It’s plausible. Recall that Mendel and Darwin were amateurs. A more recent example is Alister Hardy, the Oxford professor who conceived the aquatic ape theory of evolution. He didn’t pursue it because he feared loss of reputation. The more sophisticated conclusion, I suppose, isn’t that professionalization was bad but that loss of diversity was bad. We need both amateur and professional scientists because each can do stuff the other can’t. Right now we only have professional ones. No one encourages amateur science; there is no way they can publish their work. (Unless, like Elaine Morgan, who wrote several books about the aquatic ape theory, you’re a professional writer.)

These thoughts were prompted by this remarkable blog post, which has nothing to do with science. What an amazing piece of writing, I thought. I don’t even agree with it, and here I am staring at it. A work of genius? No, lots of blog posts are really good. This one was merely better than most. Would something this brazen and effective appear in any major magazine, newspaper, TV show, radio ad, etc.? No, not even. Do we realize that, all these years, stuff like this has been missing from our media consumption? No, we don’t. Before there was news, there was gossip, I realized; news (such as newspapers) was a kind of professionalization of gossip. The blog post I admired was a bit of riveting creative gossip. Blogs are just new-fangled gossip. Bloggers are endlessly scandalized, indignant, judgmental, just as gossips are. Just as gossip is usually “passed on,” most blog posts have links and many posts consist almost entirely of “passing on” something. Just as gossip can be anything, bloggers can say what they really think, as Tyler Cowen pointed out. That’s why they’re so successful, so easy to write and read. Gossip is good for our mental ecology, just as science is. Mark Liberman’s Language Log blog is a blend of (good) gossip and science; as you can see from my interview with him, it filled a gap. I hope blogs will provide a kind of support structure on which amateur science can grow.

I am fascinated by how human nature interferes with science. This article in the Wall Street Journal helped me understand one way this happens.

A civility campaign in Howard County, Maryland, centered on a book called Choosing Civility: The Twenty-five Rules of Considerate Conduct (2002) by P. M. Forni, a John Hopkins professor of romance languages. Rule 7, for example, is “don’t speak ill.” The book bothered Heather Kirk-Davidoff, a pastor. She visited Professor Forni. “Jesus didn’t say, ‘I am the rule,’ right?” she told him. Professor Forni agreed. “Yes, Jesus said, ‘I am the way.’ If I had met you before, probably I would have used way. The 25 Ways of Being Considerate and Kind,” he said.

Hmm. The way versus the rule: similar. The way versus a way: big difference. Neither the professor nor the pastor noted that a better title would omit the: 25 Ways of Being…

The writer of a book about civility — in that very book — fails to grasp a big point about civility. The pastor who points out the problem makes a similar omission. Our tendency to turn tools into rules must be strong.
If you invent a useful tool, you have made the world a better place. If you denigrate non-users, the improvement is less obvious. Randomization, for example, is a tool. Many scientists treat it like a rule. Were I to write a book on scientific method, it would contain a paragraph beginning: “A few years ago, the head librarian of the Howard County, Maryland, county library bought 2300 copies of a book called . . .”
Twisted skepticism.

Before the invention of statistical tests, such as the t test, science moved forward. People gathered data, computed averages, drew reasonable conclusions. As far as I can tell, modern ways of analyzing data improved the linkage between data and conclusion because they reduced a big source of noise: How the data were analyzed. Procedures became standardized. Hypothesis testing improved. Hypothesis formation, however, did not improve. Knowing how to do a t test and the philosophy behind it will not help you come up with new ideas. Yet data can be used to generate new ideas, not just test the ones you already have.

Our understanding of outliers is in a kind of pre-t-test era. People use them in an unstructured way. As Howard Wainer’s analysis of his blood sugar data indicates, better use of them will improve hypothesis formation. A kind of standardized treatment should help generate ideas, just as the t test and related ideas helped test ideas. Here are some questions I think can be answered:

1. Cause. What causes outliers? It’s a step forward to realize that outliers are often caused by other outliers. Howard has found that unusually high blood sugar readings are caused by eating unusual (for him) foods.

2. Inference. I’m fond of saying lightning doesn’t strike twice in one place for different reasons. The longer version is if two outliers could have the same explanation, they probably do. I think this principle can be improved.

3. Methodology. To test ideas, you want variation to be low. To generate ideas, you want outlier rate to be high. Howard could make progress in understanding what controls his blood sugar by deliberately testing foods that might produce outliers. In genetics, x-rays and chemical mutagens have been used to increase mutation rates; mutations are outliers. (Discovery of a white-eyed mutant fruit fly led to a wealth of new genetic ideas.) In physics, particle accelerators increase the outlier rate in order to discover new subatomic particles. There are no comparable procedures for psychology. Self-experimentation increased my rate of new ideas because it increased my outlier detection rate. It increased that rate for three reasons: 1. I kept numerical records. 2. I analyzed my data using the same methods as Howard. 3. I did experiments. Travel is like experimentation; there too it helps to keep numerical records and analyze them. The question: What are the basic principles for increasing outlier rate?
Part 1.