Comments on: Voodoo Correlations in Social Neuroscience http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/ Self-Experimentation, Scientific Method, the Shangri-La Diet, etc. Tue, 16 Mar 2010 07:17:04 +0000 http://wordpress.org/?v=2.0.7 by: seth http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-377443 Fri, 01 Jan 2010 14:13:34 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-377443 Justin, because I'm in China I can't get YouTube. Justin, because I’m in China I can’t get YouTube.

]]> by: Justin http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-377400 Fri, 01 Jan 2010 12:14:28 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-377400 Hi Seth, Any chance of some constructive feedback on this video on the above study? [youtube=http://www.youtube.com/watch?v=nMZvpVwfObE] Regards Justin Hi Seth,
Any chance of some constructive feedback on this video on the above study?
[youtube=http://www.youtube.com/watch?v=nMZvpVwfObE]
Regards

Justin

]]> by: ‘Voodoo Correlations in Social Neuroscience’ « The Amazing World of Psychiatry: A Psychiatry Blog http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-266121 Sat, 31 Jan 2009 01:08:54 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-266121 [...] Seths Blog article [...] […] Seths Blog article […]

]]> by: seth http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265809 Thu, 29 Jan 2009 16:56:06 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265809 Thanks, Matt. Thanks, Matt.

]]> by: Matt http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265793 Thu, 29 Jan 2009 16:03:42 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265793 So, we combine a significance threshold (i.e. p-values less than .005 or .001) with an extent threshold (i.e. there have to be at least 10 contiguous voxels that all have p-values less than the significance threshold). This is a standard procedure used throughout cognitive neuroscience for the past 15 years. So, we combine a significance threshold (i.e. p-values less than .005 or .001) with an extent threshold (i.e. there have to be at least 10 contiguous voxels that all have p-values less than the significance threshold). This is a standard procedure used throughout cognitive neuroscience for the past 15 years.

]]> by: Matt http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265780 Thu, 29 Jan 2009 15:04:59 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265780 Sorry for the typos (yes its the correlations that are inflated, but that's not what the test is testing - its testing for reliable non-zero relationships). Your Jordan analogy doesn't quite apply. That would assume that we are making claims about the average correlation in the brain but only reporting on a subset of voxels (and pretending they are all the voxels). We aren't making claims about how the brain as a whole or on average relates to personality - rather we are looking for which regions do correlate reliably and then providing descriptive statistics for those that do. To get the Jordan analogy right, the question would be "Are there certain days of the week when Jordan shoots a higher percentage than others?". We'd have him shoot 100 free throws each day of the week for say 10 weeks. So we'd have 1000 data points for each of the seven days of the week. We wouldn't care at all what his average across all days was, just how each day compares to each other. If his averages were 30% on mondays, 90% on fridays and 60% on all other days, we would say something interesting is happening on mondays and fridays, report that test and the descriptives that go along with it (e.g. 90%). Now if we reported that Jordan shoots 90% on average because we claimed that fridays were the only days were looking at, we'd be in trouble, but nobody does that. Our question isn't the average, but rather, when is there something different from average going on. Sorry for the typos (yes its the correlations that are inflated, but that’s not what the test is testing - its testing for reliable non-zero relationships). Your Jordan analogy doesn’t quite apply. That would assume that we are making claims about the average correlation in the brain but only reporting on a subset of voxels (and pretending they are all the voxels). We aren’t making claims about how the brain as a whole or on average relates to personality - rather we are looking for which regions do correlate reliably and then providing descriptive statistics for those that do.

To get the Jordan analogy right, the question would be “Are there certain days of the week when Jordan shoots a higher percentage than others?”. We’d have him shoot 100 free throws each day of the week for say 10 weeks. So we’d have 1000 data points for each of the seven days of the week. We wouldn’t care at all what his average across all days was, just how each day compares to each other. If his averages were 30% on mondays, 90% on fridays and 60% on all other days, we would say something interesting is happening on mondays and fridays, report that test and the descriptives that go along with it (e.g. 90%). Now if we reported that Jordan shoots 90% on average because we claimed that fridays were the only days were looking at, we’d be in trouble, but nobody does that. Our question isn’t the average, but rather, when is there something different from average going on.

]]> by: john http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265748 Thu, 29 Jan 2009 12:40:50 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265748 Seth, "Let’s say I select a subset of free throws where Michael Jordan missed. Then I compute his free throw percentage over only those free throws. It is 0%. To report that 0% as if it means something is . . . well, call it what you want. As far as I can tell, that is basically what you did." I may have missed something here, but isn't that what Matt is claiming that Vul has done? Isn't one of the strong arguements in the Lieberman paper that Ed Vul simply hand picked results from papers that would show the effect he wanted to show and ignored the others that didn't? In fact when Matt puts all the data in to the analysis there is no bias in the correlation coeffecient at all. Seth,

“Let’s say I select a subset of free throws where Michael Jordan missed. Then I compute his free throw percentage over only those free throws. It is 0%. To report that 0% as if it means something is . . . well, call it what you want. As far as I can tell, that is basically what you did.”

I may have missed something here, but isn’t that what Matt is claiming that Vul has done? Isn’t one of the strong arguements in the Lieberman paper that Ed Vul simply hand picked results from papers that would show the effect he wanted to show and ignored the others that didn’t? In fact when Matt puts all the data in to the analysis there is no bias in the correlation coeffecient at all.

]]> by: seth http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265697 Thu, 29 Jan 2009 08:47:24 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265697 Matt, could you post again the last part of your comment? It was cut off. What you call "selection bias" -- computing a correlation using only voxels selected by looking at the data and thereby inflating the correlation -- doesn't inflate "tests" it inflates correlations. Nor are "voxels" inflated (by "voxel" I guess you mean the correlation computed for just one voxel), it is the correlation computed over many voxels that is inflated. That's when you got into trouble -- by computing a number that might be grossly inflated. Let's say I select a subset of free throw attempts where Michael Jordan missed. Then I compute his free throw percentage over only those free throws. It is 0%. To report that 0% as if it means something is  . . . well, call it what you want. As far as I can tell, that is basically what you did. Matt, could you post again the last part of your comment? It was cut off.

What you call “selection bias” — computing a correlation using only voxels selected by looking at the data and thereby inflating the correlation — doesn’t inflate “tests” it inflates correlations.

Nor are “voxels” inflated (by “voxel” I guess you mean the correlation computed for just one voxel), it is the correlation computed over many voxels that is inflated. That’s when you got into trouble — by computing a number that might be grossly inflated.

Let’s say I select a subset of free throw attempts where Michael Jordan missed. Then I compute his free throw percentage over only those free throws. It is 0%. To report that 0% as if it means something is  . . . well, call it what you want. As far as I can tell, that is basically what you did.

]]> by: Matt http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265661 Thu, 29 Jan 2009 05:49:43 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265661 Actually, what I am denying is that there was any post-hoc selection of a subset of the data. Running multiple comparisons leads to inflated effect sizes but that has nothing to do with "post-hoc selection". And actually, the main point of their article was not that there is inflation but rather that this inflation is so great that the results should be considered worthless and likely spurious and also that the methods used to obtain the results are invalid and therefore the results themselves are invalid. These tests are run in order to identify regions where there are reliably non-zero correlations and they are a perfectly valid way of doing so. To report descriptive statistics is entirely valid as well. Since we seem to be talking past each other, let's consider one last example. Let's say I run my 40,000 independent tests on my 40,000 voxels. You would admit at this point there has been no "selection bias" inflating these tests, correct? You might have some large effects due to sampling fluxuations, but our Figure 1 shows that with normal fMRI sample sizes and appropriate correction for multiple comparisons, this is relatively rare (Vul's simulation was done assuming 10 subjects which is not representative of fMRI studies). Let's further assume that I submit my paper to the journal with a 200 page table that lists the p-value (along with descriptive statistics) for every voxel in the brain. Still no selection bias inflating these tests, correct? If you sorted this table by p-value we'd still be ok, right? Now the editor comes along and says "we can't have a 200 page table" so cut off everything with a p-value worse than ___ and add a note to indicate that all other voxels had p-values above that threshold. The voxels that remained would be no more inflated after this editorial decision than before - its just a matter of convention for displaying data. This is what we all do and there is no "non-independence error" as Vul claims. Actually, what I am denying is that there was any post-hoc selection of a subset of the data. Running multiple comparisons leads to inflated effect sizes but that has nothing to do with “post-hoc selection”. And actually, the main point of their article was not that there is inflation but rather that this inflation is so great that the results should be considered worthless and likely spurious and also that the methods used to obtain the results are invalid and therefore the results themselves are invalid. These tests are run in order to identify regions where there are reliably non-zero correlations and they are a perfectly valid way of doing so. To report descriptive statistics is entirely valid as well. Since we seem to be talking past each other, let’s consider one last example. Let’s say I run my 40,000 independent tests on my 40,000 voxels. You would admit at this point there has been no “selection bias” inflating these tests, correct? You might have some large effects due to sampling fluxuations, but our Figure 1 shows that with normal fMRI sample sizes and appropriate correction for multiple comparisons, this is relatively rare (Vul’s simulation was done assuming 10 subjects which is not representative of fMRI studies). Let’s further assume that I submit my paper to the journal with a 200 page table that lists the p-value (along with descriptive statistics) for every voxel in the brain. Still no selection bias inflating these tests, correct? If you sorted this table by p-value we’d still be ok, right? Now the editor comes along and says “we can’t have a 200 page table” so cut off everything with a p-value worse than ___ and add a note to indicate that all other voxels had p-values above that threshold. The voxels that remained would be no more inflated after this editorial decision than before - its just a matter of convention for displaying data. This is what we all do and there is no “non-independence error” as Vul claims.

]]> by: seth http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265657 Thu, 29 Jan 2009 05:22:33 +0000 http://www.blog.sethroberts.net/2008/12/28/voodoo-correlations-in-social-neuroscience/#comment-265657 Gee, Matt, you're still not denying that post-hoc selection of a subset inflates the correlations. Which -- correct me if I'm wrong -- was the main point of Vul et al. Along with the point that this inflation was not made clear in the published papers. I'm still curious: What correction for multiple tests did your research group use? Gee, Matt, you’re still not denying that post-hoc selection of a subset inflates the correlations. Which — correct me if I’m wrong — was the main point of Vul et al. Along with the point that this inflation was not made clear in the published papers.
I’m still curious: What correction for multiple tests did your research group use?

]]>