Comments on: The Parable of the Wii http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/ Self-Experimentation, Scientific Method, the Shangri-La Diet, etc. Thu, 18 Mar 2010 05:41:39 +0000 http://wordpress.org/?v=2.0.7 by: Dennis Whittle: Good Scientists are Like Wandering Ants | News from: The Huffington Post - Breaking News and Opinion http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-362318 Tue, 17 Nov 2009 17:44:01 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-362318 [...] That is from the iconclastic Seth Roberts, formerly a professor of psychology at Berkeley who now teaches at in Beijing at Tsinghua University. His overarching theme is that for science to advance it requires people to come up with and test novel hypotheses rather than tinkering at the margins of the currently accepted wisdom. In short, orthodoxy is often unproductive for scientists, and sometimes dangerous. His blog is full of unexpected hypotheses about how the world works, and he often tests these hypotheses on himself, enlisting his own readers as co-experimenters. [...] […] That is from the iconclastic Seth Roberts, formerly a professor of psychology at Berkeley who now teaches at in Beijing at Tsinghua University. His overarching theme is that for science to advance it requires people to come up with and test novel hypotheses rather than tinkering at the margins of the currently accepted wisdom. In short, orthodoxy is often unproductive for scientists, and sometimes dangerous. His blog is full of unexpected hypotheses about how the world works, and he often tests these hypotheses on himself, enlisting his own readers as co-experimenters. […]

]]> by: seth roberts on scientists as wandering ants « the pulchrifex papers http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-362041 Mon, 16 Nov 2009 20:08:26 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-362041 [...] seth roberts on scientists as wandering ants 2009 November 16 by Matt I’m competing for postdoc funding from the National Institutes of Health; my career path, if I follow it to its logical culmination, will be regularly punctuated with similar competition. So I’m keenly interested in claims like this: Scientists don’t like thinking of themselves as wandering ants. But that’s how they are most effective. This goes against human psychology because wandering (Nassim Taleb calls it “tinkering”) is low status and lonely. The payoff is too rare and too unclear. It isn’t supported by powerful institutions, such as research universities and medical schools. Imagine an ant who says “I know where food is!” This is a way to get many ants to follow him, to feel important, to have high status, to get support from his employer. That’s why he does it. But he doesn’t know. The effect on the rest of us, the potential beneficiaries of progress, is that instead of having a thousand ants wandering everywhere, we have a thousand ants following one ant who doesn’t know what he’s doing. (Full post.) [...] […] seth roberts on scientists as wandering ants 2009 November 16 by Matt I’m competing for postdoc funding from the National Institutes of Health; my career path, if I follow it to its logical culmination, will be regularly punctuated with similar competition. So I’m keenly interested in claims like this: Scientists don’t like thinking of themselves as wandering ants. But that’s how they are most effective. This goes against human psychology because wandering (Nassim Taleb calls it “tinkering”) is low status and lonely. The payoff is too rare and too unclear. It isn’t supported by powerful institutions, such as research universities and medical schools. Imagine an ant who says “I know where food is!” This is a way to get many ants to follow him, to feel important, to have high status, to get support from his employer. That’s why he does it. But he doesn’t know. The effect on the rest of us, the potential beneficiaries of progress, is that instead of having a thousand ants wandering everywhere, we have a thousand ants following one ant who doesn’t know what he’s doing. (Full post.) […]

]]> by: seth http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361950 Mon, 16 Nov 2009 10:40:10 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361950 Andrew, yes, there are exceptions to the broad statements I made. But most scientists (95%?) gather data in one way or another. I think physics and astronomy are the only sciences where there are a lot of pure theorists (such as Feynman), and even in those fields I think the data collectors outnumber them. I think you're an exception, too -- as far as I can tell, most statistics profs spend little of their time trying to answer substantive questions. Most statistics profs are most interested in developing new methods. Only a few statistics profs are also profs in a substantive area, such as political science. I say all this partly because I wonder what would be a "high-risk" line of research for a statistics professor. Maybe developing methods to do something unconventional, such as generate ideas? Andrew, yes, there are exceptions to the broad statements I made. But most scientists (95%?) gather data in one way or another. I think physics and astronomy are the only sciences where there are a lot of pure theorists (such as Feynman), and even in those fields I think the data collectors outnumber them. I think you’re an exception, too — as far as I can tell, most statistics profs spend little of their time trying to answer substantive questions. Most statistics profs are most interested in developing new methods. Only a few statistics profs are also profs in a substantive area, such as political science. I say all this partly because I wonder what would be a “high-risk” line of research for a statistics professor. Maybe developing methods to do something unconventional, such as generate ideas?

]]> by: Andrew Gelman http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361936 Mon, 16 Nov 2009 09:18:34 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361936 Seth: Psychologists are good at gathering data. But you don't need to gather data to be a scientist. You can analyze or build theories based on others' data. For example, Einstein, Feynmann, etc. were scientists even though they were not experimentalists. And they were empirical scientists too--they explained empirical facts and made testable predictions. Also, I completely disagree with your statement that statistics professors don't "focus on substantive issues." Take a look at my research articles! It's possible to be a statistics professor and do applied statistics. Seth: Psychologists are good at gathering data. But you don’t need to gather data to be a scientist. You can analyze or build theories based on others’ data. For example, Einstein, Feynmann, etc. were scientists even though they were not experimentalists. And they were empirical scientists too–they explained empirical facts and made testable predictions.

Also, I completely disagree with your statement that statistics professors don’t “focus on substantive issues.” Take a look at my research articles! It’s possible to be a statistics professor and do applied statistics.

]]> by: seth http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361849 Sun, 15 Nov 2009 23:18:42 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361849 Andrew, by science I meant empirical science, where the main goal is gathering data from which you learn how the world works. This isn't the main goal of statistics professors. They don't gather data nor focus on substantive issues. But I agree, you make a good point. Even within this subset of science I agree that one can pursue a path where progress is more predictable. In terms of the scientist = ant analogy, an ant has a choice of joining a trail of many ants to a food source or wandering around by itself to find a new food source. Likewise a scientist who gathers data has the choice of exploiting a known cause-effect relationship (e.g. doing variations on it, trying to explain it) or trying to find new cause-effect relationships. I don't know what a similar choice would be for statistics profs. Andrew, by science I meant empirical science, where the main goal is gathering data from which you learn how the world works. This isn’t the main goal of statistics professors. They don’t gather data nor focus on substantive issues. But I agree, you make a good point. Even within this subset of science I agree that one can pursue a path where progress is more predictable. In terms of the scientist = ant analogy, an ant has a choice of joining a trail of many ants to a food source or wandering around by itself to find a new food source. Likewise a scientist who gathers data has the choice of exploiting a known cause-effect relationship (e.g. doing variations on it, trying to explain it) or trying to find new cause-effect relationships. I don’t know what a similar choice would be for statistics profs.

]]> by: Andrew Gelman http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361788 Sun, 15 Nov 2009 20:54:51 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361788 Seth: As we've discussed many times (and I've blogged about too), I think I've followed a low-risk, low-return model for science, doing a lot of small projects, each of which is a sure thing (or, at least, something like 50% chance of success, where "success" means advancing the field in some way and publication in a top journal). In contrast, you've followed a high-risk, high-return model by spending 15 years doing self-experimentation. (Also, rather than writing 6 big books as I did, you wrote one little book--but your one little book was a bestseller.) I guess what I'm saying is that your statement about science, "It is much harder than expected, then it pays off in ways that defy understanding," describes how you do science, but not necessarily how others do science. Perhaps both types are necessary: we need the bold thinkers like yourself and also the more methodical people like me to fill in the gaps. Seth: As we’ve discussed many times (and I’ve blogged about too), I think I’ve followed a low-risk, low-return model for science, doing a lot of small projects, each of which is a sure thing (or, at least, something like 50% chance of success, where “success” means advancing the field in some way and publication in a top journal). In contrast, you’ve followed a high-risk, high-return model by spending 15 years doing self-experimentation. (Also, rather than writing 6 big books as I did, you wrote one little book–but your one little book was a bestseller.)

I guess what I’m saying is that your statement about science, “It is much harder than expected, then it pays off in ways that defy understanding,” describes how you do science, but not necessarily how others do science. Perhaps both types are necessary: we need the bold thinkers like yourself and also the more methodical people like me to fill in the gaps.

]]> by: Walter http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361785 Sun, 15 Nov 2009 20:43:31 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361785 When I worked in R/D the upper management wanted predictability in "findings" so we went our way in testing ideas so we ended up with a pipeline of hidden successes and then we metered it out in a "predictable" manner. We ended up predictable but slow in comparison to the rest of the competitor. Upper management patted themselves on the back for being able to manage and to be predictable. There is a lot of randomness in R/D and the quicker one built up a pile of "failures" the quicker one found the nugget(s). The philosophy I found to work the best is TREE-test randomly, evaluate and elect-and the team should consist of a bunch of people with cognitive diversity and experience diversity. When I worked in R/D the upper management wanted predictability in “findings” so we went our way in testing ideas so we ended up with a pipeline of hidden successes and then we metered it out in a “predictable” manner. We ended up predictable but slow in comparison to the rest of the competitor. Upper management patted themselves on the back for being able to manage and to be predictable.
There is a lot of randomness in R/D and the quicker one built up a pile of “failures” the quicker one found the nugget(s).
The philosophy I found to work the best is TREE-test randomly, evaluate and elect-and the team should consist of a bunch of people with cognitive diversity and experience diversity.

]]> by: Ashish http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361538 Sat, 14 Nov 2009 23:20:59 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361538 If you enjoy Dance Dance Revolution, definitely check out Dance Dance Immolation at Burning Man. If you enjoy Dance Dance Revolution, definitely check out Dance Dance Immolation at Burning Man.

]]> by: NotInChinanow http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361537 Sat, 14 Nov 2009 23:18:19 +0000 http://www.blog.sethroberts.net/2009/11/14/the-parable-of-the-wii/#comment-361537 I was and still am unprepared for how unsuccessful my experiments are. Although I don't have a true tally, the number is certainly less than 20%, and I think probably a fair and accurate count would probably put that number at around 5%. It seems like that number is similar to your success rate with self-experiments. While it's personally hard to deal with that level of failure, I think it's even harder at the institutional level. Most institutions couldn't tolerate employees who get such a low success rate for their experiments even if the resulting success brings much more valuable information. So on a personal level thats why I try to spend time on things that are much more likely to work because it's easier to defend your work that way. That's why I like your self-experiments because they allow for more risk that would probably kill the career of a normal experimenter. I was and still am unprepared for how unsuccessful my experiments are. Although I don’t have a true tally, the number is certainly less than 20%, and I think probably a fair and accurate count would probably put that number at around 5%. It seems like that number is similar to your success rate with self-experiments. While it’s personally hard to deal with that level of failure, I think it’s even harder at the institutional level. Most institutions couldn’t tolerate employees who get such a low success rate for their experiments even if the resulting success brings much more valuable information. So on a personal level thats why I try to spend time on things that are much more likely to work because it’s easier to defend your work that way. That’s why I like your self-experiments because they allow for more risk that would probably kill the career of a normal experimenter.

]]>