Seth’s blog

Self-experimentation is an example of the more general idea that non-experts can do valuable research. Another example is that two New York teenagers have shown that fish sold in New York City is often mislabeled. They gathered samples from 4 sushi restaurants and 10 grocery stores and sent them to a lab to be identified using a methodology and database called Barcode of Life. They found that “one-fourth of the fish samples with identifiable DNA were mislabeled  . . . [and concluded] that 2 of the 4 restaurants and 6 of the 10 grocery stores had sold mislabeled fish.”

The article, by John Schwartz, appeared in the Science section, which makes the following sentence highly unfortunate:

The sample size is too small to serve as an indictment of all New York fishmongers and restaurateurs, but the results are unlikely to be a mere statistical fluke.

This is a Samantha-Powers-sized blunder. It could hardly be more wrong. How much you can generalize from a sample to a population depends on how the samples were chosen. Sample size has very little to do with it. (John Tukey had the same complaint about the Kinsey Report: Stop boasting about your sample size, he said to Kinsey. Your sampling methods were terrible.) To know to what population we can reasonably generalize these results we’d need to know how the two teenagers decided what grocery stores and restaurants to sample from. (Which the article does not say.) If the 14 fish sellers were randomly sampled from the entire New York City population of grocery stores and restaurants, it would be perfectly reasonable to draw broad conclusions.

I have no idea what it could mean that the results are “a mere statistical fluke”.

The effect of these errors is that Mr. Schwartz places too low a value on this research. It’s impressive not only for its basic conclusion that there’s lots of mislabeling but also for showing what non-experts can do.

The end of the article did see the big picture:

In a way, Dr. Ausubel said, their experiment is a return to an earlier era of scientific inquiry. “Three hundred years ago, science was less professionalized,” he said, and contributions were made by interested amateurs. “Perhaps the wheel is turning again where more people can participate.”

A few days ago I blogged about how Tim Lundeen, via careful  and repeated measurement — let’s call it self-experimentation — uncovered a serious and previously-unreported side effect of a drug he was taking. Tim’s example illustrates an important use of self-experimentation: discovering unreported side effects, which I believe are common.

By coincidence today I came across a talk about the very subject of unmentioned side effects: Alison Bass speaking about her new book, Side Effects: A Prosecutor, a Whistleblower, and a Bestselling Antidepressant on Trial. Near the end, Bass said,

It’s not the just the antidepressants, it’s not just the antipsychotics. This is happening with a lot of other drugs. With Vioxx, with Vytorin, an anti-cholesterol drug, with Propries [?] and Marimet [?], anti-anemia drugs. Where again and again the drug companies know that there are more severe side effects and they’re not letting the public know about that. It just keeps happening, unfortunately.

Just as it would be foolish to think the problem is limited to mental-health drugs, it would be foolish to think the problem is limited to side effects, that drug company researchers do everything right except fail to report side effects. Tim’s example shows how hard it is to learn about unreported side effects — so it is only realistic to think that there are other big problems with drug company research we don’t know about. Bass mentioned one I didn’t know about. A company did a clinical trial of Paxil. The goal was to see if the drug helped with Measures of Depression A and B. Turns out it didn’t: no effect. So the company changed the measures! They shifted to reporting different measures that the drug did seem to improve. Creating the hypothesis to be tested after the data supposedly supporting that hypothesis had already been collected. Without making this clear. (Which I presciently mentioned here, in response to an interesting comment by Andrew Gelman.) And if you think that drug companies do research like this — in ways that seriously damage people’s lives — but everyone else, such as academia, is really good, that is as realistic as thinking the problem with drug company research is restricted to side effects. Self-experimentation has all sorts of limitations, yes, but (a) you know what they are and (b) it is cheap enough so that you can gather more data to deal with the problems. Drug company research and lots of other research is too expensive to fail — or even be honest about shortcomings.

This is an aspect of scientific method that scientists rarely discuss: the effect of cost on honesty. Is there an economic term (a Veblen good, perhaps?) for things whose quality goes down as their cost goes up?

If we can believe a movie based on a true story, the doctors consulted by the family with an epileptic son in …First Do No Harm knew about the ketogenic diet but (a) didn’t tell the parents about it, (b) didn’t take it seriously, and (c) thought that irreversible brain surgery should be done before trying the diet, which was of course much safer. Moreover, these doctors had an authoritative book to back up these remarkably harmful and unfortunate attitudes. The doctors in …First, as far as I can tell, reflected (and still reflect) mainstream medical practice.

Certainly the doctors were evidence snobs — treating evidence not from a double-blind study as worthless. Why were they evidence snobs? I suppose the universal tendency toward snobbery (we love feeling superior) is one reason but that may be only part of the explanation. In the 1990s, Phillip Price, a researcher at Lawrence Berkeley Labs, and one of his colleagues were awarded a grant from the Environmental Protection Agency (EPA) to study home radon levels nationwide. They planned to look at the distribution of radon levels and make recommendations for better guidelines. After their proposal was approved, some higher-ups at EPA took a look at it and realized that the proposed research would almost surely imply that the current EPA radon guidelines could be improved. To prevent such criticism, the grant was canceled. Price was told by an EPA administrator that this was the reason for the cancellation.

This has nothing to do with evidence snobbery. But I’m afraid it may have a lot to do with how the doctors in …First Do No Harm viewed the ketogenic diet. If the ketogenic diet worked, it called into question their past, present, and future practices — namely, (a) prescribing powerful drugs with terrible side effects and (b) performing damaging and irreversible brain surgery of uncertain benefit. If something as benign as the ketogenic diet worked some of the time, you’d want to try it before doing anything else. This hadn’t happened: The diet hadn’t been tried first, it had been ignored. Rather than allow evidence of the diet’s value to be gathered, which would open them up to considerable criticism, the doctors did their best to keep the parents from trying it. Much like canceling the radon grant.

The ketogenic diet.

In a science classroom at a middle school I saw a poster about “the scientific method.” There were seven steps; one was “analyze your data.” According to the poster, you use the data you’ve collected to say if your hypothesis was right or wrong. Nothing was said about using data to generate new hypotheses. Yet coming up with ideas worth testing is just as important as testing them.

It’s like teaching the alphabet and omitting half of the letters. Or teaching French and omitting half the common words. While no one actually teaches only half the alphabet or only half of common French words, this is how science is actually taught. Not just in middle school, everywhere. The poster correctly reflects the usual understanding. I have seen dozens of books about scientific method. They usually say almost nothing about how to come up with a new idea worth testing. An example is Statistics For Experimenters, a well-respected book by Box, Hunter, and Hunter. One of the authors (George Box) is a famous statistician.

The curious part of this omission is how unnecessary it is. Every scientific idea we now take for granted started somewhere. It would be no great effort to find where a bunch of them came from.

Did the professionalization of science — people could make a living doing science — cause harm because although more science was done scientists — the professional ones — were no longer free to pursue the truth in any direction? Because their jobs and status were at stake? It’s plausible. Recall that Mendel and Darwin were amateurs. A more recent example is Alister Hardy, the Oxford professor who conceived the aquatic ape theory of evolution. He didn’t pursue it because he feared loss of reputation. The more sophisticated conclusion, I suppose, isn’t that professionalization was bad but that loss of diversity was bad. We need both amateur and professional scientists because each can do stuff the other can’t. Right now we only have professional ones. No one encourages amateur science; there is no way they can publish their work. (Unless, like Elaine Morgan, who wrote several books about the aquatic ape theory, you’re a professional writer.)

These thoughts were prompted by this remarkable blog post, which has nothing to do with science. What an amazing piece of writing, I thought. I don’t even agree with it, and here I am staring at it. A work of genius? No, lots of blog posts are really good. This one was merely better than most. Would something this brazen and effective appear in any major magazine, newspaper, TV show, radio ad, etc.? No, not even. Do we realize that, all these years, stuff like this has been missing from our media consumption? No, we don’t. Before there was news, there was gossip, I realized; news (such as newspapers) was a kind of professionalization of gossip. The blog post I admired was a bit of riveting creative gossip. Blogs are just new-fangled gossip. Bloggers are endlessly scandalized, indignant, judgmental, just as gossips are. Just as gossip is usually “passed on,” most blog posts have links and many posts consist almost entirely of “passing on” something. Just as gossip can be anything, bloggers can say what they really think, as Tyler Cowen pointed out. That’s why they’re so successful, so easy to write and read. Gossip is good for our mental ecology, just as science is. Mark Liberman’s Language Log blog is a blend of (good) gossip and science; as you can see from my interview with him, it filled a gap. I hope blogs will provide a kind of support structure on which amateur science can grow.