Seth’s blog

Alex Tabarrok discusses a proposal to make referee reports and associated material publicly available. I think it would be a good thing because it would make writing a self-serving review (e.g., a retaliatory review) more dangerous. If Reviewer X writes an unreasonable review, the author is likely to complain to the editor. If the paper gets published, the unreasonableness will be highlighted — and nominal anonymity may not be enough to hide who wrote it. On the other side, as a reader, it would be extremely educational. You could learn a lot from studying these reports and the replies they generated, especially if you’re a grad student. I would like to know why some papers got accepted. For example, my Tsinghua students pointed out serious flaws in published papers. Were the problems noted by reviewers and ignored, or what?

My experience is that about 80% of reviews are reasonable. Many of those are ignorant, but that’s no crime. (A lot of reviewers know more than me.) The remaining 20% seem to go off the rails somehow. For example, Hal Pashler and I wrote a paper criticizing the use of good fits to support quantitative models. The first two reviewers seemed to have been people who did just that. Their reviews were ridiculous. Apparently they thought the paper shouldn’t be published because it might call their work into question. A few reviews have appeared to be retaliation. In the 1990s, I complained to the Office of Research Integrity that a certain set of papers appeared to contain made-up data. (ORI sent the case to the institution where the research was done. A committee to investigate did the shallowest possible review and decided I was wrong. I learned my lesson — don’t trust ORI — which I applied to the Chandra case.) After that allegation, I got stunningly unfair reviews from time to time, presumably from the people I accused. A small fraction of reviews (5%?) are so lazy they’re worthless. One reviewer of my long self-experimentation paper said it shouldn’t be published because it wasn’t science. The author (me) should go do some real science.

The main things I’ve learned about how to respond are: 1. When resubmitting the paper (revised in light of the reviews), go over every objection and how it was dealt with or why it was ignored. Making such a list isn’t very hard, it makes ignoring a criticism much easier (because you are explicit about it), and editors like it. This has become common. 2. When a review is unreasonable, complain. The theory-testing paper I wrote with Hal is one of my favorite papers and it wouldn’t have been published where it was if we hadn’t complained. Another paper of mine said that some data failed a chi-square test many times — suggesting that something was wrong. One of the reviewers seemed to not understand what a chi-square test was. I complained and got a new reviewer.

I’m curious: What have you learned about responding to reviewers?

In the current New Yorker, James Surowiecki writes:

The bailout of the auto industry, after all, was as unpopular as the bailout of the banks, even though it was much tougher on the companies (G.M. and Chrysler went bankrupt; shareholders were wiped out, and C.E.O.s pushed out), and even though the biggest beneficiaries of the deal were ordinary autoworkers. You might have expected a deal that helped workers keep their jobs to play well in a country spooked by ballooning unemployment. Yet most voters hated it.

Yes, rewarding failure doesn’t play well. The voters were right. The same money that was used to give a few giant companies a second (or third) chance could have been used to give many thousands of very small companies a first chance. It could have been used to help many thousands of people start new small businesses (often one-person businesses) or keep their new small business afloat. All those small businesses would have provided plenty of jobs. and they would have had a far more promising future, far more room for growth, than the Big Three, being both far more diverse and having not already failed. The many thousands of people who wanted to start small businesses were unable to get together and make themselves visible, so the failure of government to help them went unnoticed. Their diversity was economic strength but political weakness.

It’ isn’t surprising things happened as they did — the Big Three (not to mention Wall Street) were bailed out, small businesses were ignored — but it is an indication of how poorly our economy is managed in the most basic ways. I’m not even an economist and I understand this simple point. Bernanke and Summers do not.

It’s easy for me to understand because the same thing happens in science. Government support of research is a good idea, but the money is misspent, in the same way. Grant support goes to a few large projects — generally to people who have already failed (to do anything useful) — rather than to a large number of small projects that haven’t yet failed. The way to support innovation is to place many small bets not a few big ones. That’s one thing I learned from self-experimentation, which allowed me to place many small bets.

Today I got an email inviting me to contribute to a journal called SRX Neuroscience. The journal is “peer-reviewed open-access”.  The email continued: “There are many reasons to submit your work to SRX Neuroscience, including an efficient online submission process, no page limits or restrictions on large data sets, immediate publication upon acceptance, and free accessibility of articles without any barriers to access, which increases their visibility.”

I’d never heard of it. Its web page didn’t open. The website for SRX (short for Scholarly Research Exchange) was extremely vague: no names, no location. And no sign of how it was funded.

Finally I learned that SRX is run by Hindawi Publishing, in Egypt. From this excellent overview I learned its money comes from author fees, $500 or more per article. They are trying a new kind of editorship: 30 editors or more per journal. Each editor handles only two articles a year and receives a 50% discount when they themselves submit an article. (I wonder what referees get.) Meanwhile, BioMed Central, a better-known open-access publisher, is having trouble: They have been forced to raise their charges to libraries so high that Yale decided to cancel.

It seems very low-rent. But, as Clayton Christensen told in The Innovator’s Dilemma, this is often how important new things begin. In the beginning hydraulic shovels were only good for digging a ditch in your backyard. The makers of cable-powered shovels, whose products made the giant holes for skyscrapers, turned up their noses at such a low-prestige task. But the hydraulic shovels got better and better. Companies that made cable-powered shovels eventually went bankrupt.

A profile of James Patterson, the hyperprolific novelist, says this:

“I don’t believe in showing off,” Patterson says of his writing. “Showing off can get in the way of a good story.”

A few days ago, just before this profile appeared, I gave a talk about self-experimentation at EG (= Entertainment Gathering), a TED-like conference in Monterey. One reason my self-experimentation was effective, I said, was that I wasn’t trying to impress anyone. Whereas professional scientists doing professional science care a lot about impressing other people. I planned to say it like this but didn’t have enough time:

Years ago, I went to a dance concert put on by students at Berkeley High School. I really enjoyed it. I thought to myself: I like dance concerts. So I went to a dance concert by UC Berkeley students – college students. I enjoyed it, but not as much as the high school concert. Then I went to a dance concert by a famous dance company that all of you have heard of. I didn’t enjoy it at all. Why were the professionals much less enjoyable than the high school students? Because the professionals cared a whole lot about being impressive. That got in the way of being enjoyable. Scientists want to be impressive. They want to impress lots of people – granting agencies, journal editors,  reviewers, their colleagues, and prospective graduate students. All this desire to be impressive gets in the way of finding things out.

In particular, it makes self-experimentation impossible:

They can’t do self-experimentation because it isn’t impressive. Self-experimentation is free. Anyone can do it. It’s easy; it doesn’t require any rare or difficult skills. If you want to impress someone with your fancy car, self-experimentation is like riding a bike.

Because my self-experimentation was private, I was free to do whatever worked.

My broader point was that my self-experimentation was effective partly because I was an insider/outsider. I had the subject-matter knowledge of an insider, but the freedom of an outsider.

This article (”Ten statisticians and their impacts for psychologists”) impressed me. It’s a lot more accessible and basic than the usual academic article. However, my list — of the statisticians who’ve had the biggest effect on how I analyze data — is much different than his. From more to less influential:

1. John Tukey. From Exploratory Data Analysis I learned to plot my data and to transform it. A Berkeley statistics professor once told me this book wasn’t important!

2. John Chambers. Main person behind S. I use R (open-source S) all the time.

3. Ross Ihaka and Robert Gentleman. Originators of R. R is much better than S: Fewer bugs, more commands, better price.

4. William Cleveland. Inventor of loess (local regression). I use loess all the time to summarize scatterplots.

5. Ronald Fisher. I do ANOVAs.

6. William Gosset. I do t tests.

My data analysis is 90% graphs, 10% numerical summaries (e.g., means) and statistical tests (e.g., ANOVA). Whereas most statistics texts are about 1% graphs, 99% numerical summaries and statistical tests.