Seth’s blog

According to a recent New York article about Rupert Murdoch, Robert Thomson, one of Murdoch’s top editors,

thinks most [journalists] are liberals overly concerned with writing stories that will impress other liberal journalists and win prizes in journalism competitions.

Well, yes. Not everyone is a liberal, of course, but basically everyone wants to impress their colleagues. Scientists have an amusing spin on this: They call it “peer review.” The amusing part is that somehow no one else’s opinion should matter. (E.g., all journals must be peer-reviewed.) Scientists get away with this bizarre view of economics (thinking someone should pay you and get nothing in return) perhaps because it is indeed difficult to assess the quality of this or that bit of science if you’re not in the field and because science has produced huge benefits for the rest of us in the past.

As I said, this is just human nature. As far as I can tell, professors act this way — try to impress colleagues — in every academic department. In schools of education, the result is this:

Amy Treadwell . . . received her master’s degree in education from DePaul University, a small private university in Chicago. . . . But when she walked into her first job, teaching first graders on the city’s South Side, she discovered a major shortcoming: She had no idea how to teach children to read. “I was certified and stamped with a mark of approval, and I couldn’t teach them the one thing they most needed to know how to do,” she told me.

It’s no secret that many schools of education do a poor job of training their students to teach — which is nominally one of their main goals. I am just repeating what Veblen said long ago.

What’s new is this: One man, Doug Lemov, working mostly alone, has figured out how to make people better teachers. One man. Not an professor. Did he build on the work of others? No, he started from scratch. He’s made a list of about 50 techniques. They are teachable. He gives workshops about them. As far as I can tell from this magazine article, Lemov has done a better job of figuring out how to train teachers than all the education professors in the world put together. If you arrived on earth from outer space, and didn’t understand human nature, you’d think this couldn’t possibly be true, but apparently it is. It’s like something out of a comic book.

In 1977, John Tukey published a book called Exploratory Data Analysis. It introduced many new ways of analyzing data, all relatively simple. Most of the new ways involved plotting your data. A few involved transforming your data. Tukey’s broad point was that statisticians (taught by statistics professors) were missing a lot: Conventional statistics focussed too much on confirmatory data analysis (testing hypotheses) to the omission of exploratory data analysis — data analysis that might show you something new. Here are some tools to help you explore your data, Tukey was saying.

No question the new tools are useful. I have found great benefits from plotting and transforming my data. No question that conventional statistics textbooks place far too little emphasis on graphs and transformations. But I no longer agree with Tukey’s exploratory versus confirmatory distinction. The distinction that matters — at least to historians, if not to data analysts — is between low-status and high-status. A more accurate title of Tukey’s book would have been Low-Status Data Analysis. Exploratory data analysis already had a derogatory name: Descriptive data analysis. As in mere description. Graphs and transformations are low-status. They are low-status because graphs are common and transformations are easy. Anyone can make a graph or transform their data. I believe they were neglected for that reason. To show their high status, statistics professors focused their research and teaching on more difficult and esoteric stuff — like complicated regression. That the new stuff wasn’t terribly useful (compared to graphs and transformations) mattered little. Like all academics — like everyone — they cared enormously about showing high status. It was far more important to be impressive than to be useful. As Veblen showed, it might have helped that the new stuff wasn’t very useful. “Applied” science is lower status than “pure” science.

That most of what statistics professors have developed (and taught) is less useful than graphs and transformations strikes me as utterly clear. My explanation is that in statistics, just as in every other academic area I know about, desire to display status led to a lot of useless highly-visible work. (What Veblen called conspicuous waste.) Less visibly, it led to the best tools being neglected. Tukey saw the neglect –  underdevelopment and underteaching of graphs, for example — but perhaps misdiagnosed the cause. Here’s why Tukey’s exploratory versus confirmatory distinction was misleading: Because the tools that Tukey promoted for exploration also improve confirmation. They are neglected everywhere. For example:

1. Graphs improve confirmatory data analysis. If you do a t test (or compute a p value in any way) but don’t make an associated graph, there is room for improvement. A graph will show whether the assumptions of the computation are reasonable. Often they aren’t.

2. Transformations improve confirmatory data analysis. That a good transformation will make the assumptions of the test more reasonable many people know. What few people seem to know is that a good transformation will make the statistical test more sensitive. If a difference exists, the test will be more likely to detect it. This is like increasing your sample size at no extra cost.

3. Exploratory data analysis is sometimes thought of as going beyond the question you started with to find other structure in the data — to explore your data. (Tukey saw it this way.) But to answer the question you started with as well as possible you should find all the structure in the data. Suppose my question is whether X has an effect.  I should care whether Y and Z have an effect in order to (a) make my test of X more sensitive (by removing the effects of Y and Z) and (b) assess the generality of the effect of X (does it interact with Y or Z?).

Most statistics professors and their textbooks have neglected all uses of graphs and transformations, not just their exploratory uses. I used to think exploratory data analysis (and exploratory science more generally) needed different tools than confirmatory data analysis and confirmatory science. Now I don’t. A big simplification.

Exploration (generating new ideas) and confirmation (testing old ideas) are outputs of data analysis, not inputs. To explore your data and to test ideas you already have you should do exactly the same analysis. What’s good for one is good for the other.

Likewise, Freakonomics could have been titled Low-status Economics. That’s essentially what it was, the common theme. Levitt studied all sorts of things other economists thought were beneath them to study. That was Levitt’s real innovation — showing that these questions were neglected. Unsurprisingly, the general public, uninterested in the status of economists, found the work more interesting than high-status economics. I’m sensitive to this because my self-experimentation was extremely low-status. It was useful (low-status), cheap (low-status), small (low-status), and anyone could do it (extremely low status).

More Andrew Gelman comments. Robin Hanson comments.

Jonathan Cole used to be provost of Colombia University. He has written a book called The Great American University, in which, according to this review,

He lists their dazzling achievements, which in biology and medicine include findings on gene-splicing, recombinant DNA, retroviruses, cancer therapies, coch­lear implants, the fetal ultrasound scanner, the hepatitis B vaccine, prions, stem cells, organ transplantation and even a treatment for head lice. . . . In a chapter on the social sciences, he cites, among many others, such useful innovations as theories of human capital and social mobility, research in linguistics and even the use of prices to reduce traffic jams.

“Research in linguistics”? Yes, that sounds dazzling. I’m sure those “theories of human capital” have been v v “useful”. And who would have thought that if you raise the price of something (”use of prices to reduce traffic jams”) . . . people use less of it? Which was traffic engineering, not social science. Did the reviewer, an economics professor at Harvard named Claudia Goldin, write this with a straight face?

The “dazzling achievements” in biology and medicine are only slightly less unconvincing.”Gene splicing” and “recombinant DNA” research are different names for the same thing. Fetal ultrasound scanners may cause autism. Vaccines were not invented by an American university professor. The discovery of prions has had no obvious non-laboratory use — besides being questionable. Stem-cell research has yet to produce anything of use outside of labs. To be fair, gene splicing has been used to produce human insulin, which is better than the insulin previously available, but conspicuously absent from the list of accomplishments is prevention of diabetes — not to mention allergies, obesity, depression, arthritis, stroke, or any of the other lifestyle problems that a large fraction of Americans suffer from. Such achievements would be truly useful. Great American universities haven’t given us any of those . . but they have given us a treatment for head lice.

There’s a reason for the term ivory tower. Apparently Cole, conscious of the term, is trying to argue against it — but merely shows why it exists. (I’m assuming the review is accurate.) It reminds me of the time that top Chinese students, visiting top American colleges such as Harvard and Yale, found the American students ignorant and arrogant. The theme of Cole’s book is that American universities are in trouble and need more support. What useful stuff they’ve accomplished is central to his argument. When I was an undergrad, I read Thorstein Veblen’s bitter The Higher Learning in America, which said American universities were dysfunctional. He mentioned “committees for the sifting of sawdust.”

More “Graduate school in the humanities is a trap” (via Marginal Revolution).

• Fire Larry Summers Now — a blog by “Thorstein Veblen”. An especially good post. Greg Mankiw distances himself from reality when he calls Veblen “cowardly”. My view of Summers. Nassim Taleb says “short US Treasury bonds as long as Summers is in charge.”
• Chinese museums. I fondly remember the Tap Water Museum.
• The Journal of Serendipitous and Unexpected Results is now accepting manuscripts. Better name: Journal of Unexpected Results.
• Philip Weiss on web journalism: “You see someone’s mind in action rather than through 100 corporate filters.”

Thanks to Oskar Pearson and Dave Lull.

Science journalists, like other journalists, have a built-in problem: What they write affects the careers of the scientists they talk to. So those scientists are unlikely to be honest. No doubt most science journalists realize this but cannot say it, for fear of damaging their own careers. Dirty little secret is the phrase.

This is why, when Climategate happened, the many claims of climate scientists that the emails meant nothing themselves meant nothing. “The reason for the denial was the need for it,” Thorstein Veblen was fond of saying. What the climate scientists really thought they were unlikely to make public. The faux-horrified reactions of the few who made a living on the other side of the debate also meant nothing.

And this is why this reaction to Climategate, from Robert Detlefsen, an insurance industry group vice president, is meaningful: what he says will have no effect on his career. He is disinterested.  And he makes some good points:

• “The CRU e-mails show that a close-knit group of the world’s most influential climate scientists actively colluded to subvert the peer-review process [to prevent publication of disagreement]; manufactured pre-determined conclusions through the use of contrived analytic techniques; and discussed destroying data to avoid [FOIA] requests.”
• He quotes from the Wegman report, which I hadn’t heard of. The Wegman report is by a group of statisticians.  It says: “‘ independent studies’ may not be as independent as they might appear on the surface”. It also says that when climate scientists were asked to explain their work, “the sharing of research material, data and results was haphazardly and grudgingly done.”

He concludes that the science is less certain than has been claimed.