Comments on: Unfortunate Obituaries: The Case of David Freedman http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/ Self-Experimentation, Scientific Method, the Shangri-La Diet, etc. Thu, 11 Mar 2010 15:46:09 +0000 http://wordpress.org/?v=2.0.7 by: Seth’s blog » Blog Archive » How Much Should We Trust Clinical Trials? http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-320730 Fri, 03 Jul 2009 03:32:25 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-320730 [...] Kolata also quotes David Freedman, a Berkeley professor of statistics who knew the cost of everything and the value of nothing. Perhaps it starts in medical school. As I blogged, working scientists, who have a clue, don’t want to teach medical students how to do research. [...] […] Kolata also quotes David Freedman, a Berkeley professor of statistics who knew the cost of everything and the value of nothing. Perhaps it starts in medical school. As I blogged, working scientists, who have a clue, don’t want to teach medical students how to do research. […]

]]> by: Mike http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303355 Fri, 15 May 2009 15:51:49 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303355 Section 14 of the paper discusses estimation of bias and variance from adjustment methods, and makes explicit that they are being combined in the conventional way: by adding the squared bias to the variance to get mean squared error. Maybe mean squared error is not the right way to measure accuracy, but it's not like Freedman invented it just to make his argument! In fact, he's responding to arguments in favor of adjustment which also cast the discussion in this framework. For better or for worse, most of the common statistical tools (mean, variance, regression) are closely tied to MSE. At least, MSE has the virtue that the random and systematic errors can be separated additively, which makes it a lot easier to discuss their relative contributions. If, for example, we were primarily interested in mean absolute error, another reasonable candidate, we would have no nice way to decompose it into systematic and random parts, and "variance" (MSE of an unbiased estimate) would have no particular importance. So, if your criticism of Freedman boils down to "he used MSE without justifying it" - I agree he is guilty of that, along with most of the rest of the statistical profession. However, it doesn't seem to justify the harsh words in your original post. Section 14 of the paper discusses estimation of bias and variance from adjustment methods, and makes explicit that they are being combined in the conventional way: by adding the squared bias to the variance to get mean squared error.

Maybe mean squared error is not the right way to measure accuracy, but it’s not like Freedman invented it just to make his argument! In fact, he’s responding to arguments in favor of adjustment which also cast the discussion in this framework. For better or for worse, most of the common statistical tools (mean, variance, regression) are closely tied to MSE.

At least, MSE has the virtue that the random and systematic errors can be separated additively, which makes it a lot easier to discuss their relative contributions. If, for example, we were primarily interested in mean absolute error, another reasonable candidate, we would have no nice way to decompose it into systematic and random parts, and “variance” (MSE of an unbiased estimate) would have no particular importance.

So, if your criticism of Freedman boils down to “he used MSE without justifying it” - I agree he is guilty of that, along with most of the rest of the statistical profession. However, it doesn’t seem to justify the harsh words in your original post.

]]> by: seth http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303324 Fri, 15 May 2009 13:35:15 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303324 Mike, no I'm not claiming that. I'm claiming that it was a bad idea to omit the bias/variance distinction when discussing the "accuracy" of the census and the effects of adjustment. In a certain way I agree with you. It doesn't necessarily resemble not returning library books to claim the census shouldn't have been adjusted. If someone made clear the bias/variance difference and explained their relative weighting scheme -- why they chose to weight bias like this and variance like that -- that would be reasonable. I saw nothing like that in Freedman's writings. Mike, no I’m not claiming that. I’m claiming that it was a bad idea to omit the bias/variance distinction when discussing the “accuracy” of the census and the effects of adjustment.

In a certain way I agree with you. It doesn’t necessarily resemble not returning library books to claim the census shouldn’t have been adjusted. If someone made clear the bias/variance difference and explained their relative weighting scheme — why they chose to weight bias like this and variance like that — that would be reasonable. I saw nothing like that in Freedman’s writings.

]]> by: Mike http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303245 Fri, 15 May 2009 07:00:14 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303245 Seth, Mean squared error is indeed arbitrary, as is the notion of variance itself. That is not the main point. The main point is that pretty much however you would like to define "total error," it is possible to increase the total error while reducing bias (unless the only thing you care about is bias, in which case what you want to do is pick a really random sample of one person). Freedman argues that this actually happens, for a reasonable (and conventional, though arbitrarily so) measure of total error, when the usual techniques are applied to adjust the census. You are claiming, I think, that the bias-reduction benefits of census adjustment outweigh the costs of increasing other kinds of error. This isn't self evidently true or false. Freedman has given some reasons for believing it is false. What are your reasons for believing it is true? Seth,
Mean squared error is indeed arbitrary, as is the notion of variance itself. That is not the main point. The main point is that pretty much however you would like to define “total error,” it is possible to increase the total error while reducing bias (unless the only thing you care about is bias, in which case what you want to do is pick a really random sample of one person).

Freedman argues that this actually happens, for a reasonable (and conventional, though arbitrarily so) measure of total error, when the usual techniques are applied to adjust the census. You are claiming, I think, that the bias-reduction benefits of census adjustment outweigh the costs of increasing other kinds of error. This isn’t self evidently true or false. Freedman has given some reasons for believing it is false. What are your reasons for believing it is true?

]]> by: seth http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303220 Fri, 15 May 2009 04:57:49 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303220 Mike, your "mean squared error" is only one of an infinity of ways of combining bias and variance to get an overall measure of goodness/badness. Given that we value fairness (lack of bias) more than uniformity (lack of variance), it is hardly an obvious choice. Freedman's idea that his particular relative weighting of bias and variance was so sure -- on such a "firm statistical foundation" -- as to not need mentioning or discussion, that no reasonable person could disagree, really does resemble not returning library books in its self-centeredness. For more about why I thought Freedman's position resembled not returning library books, read the rest of the post. Mike, your “mean squared error” is only one of an infinity of ways of combining bias and variance to get an overall measure of goodness/badness. Given that we value fairness (lack of bias) more than uniformity (lack of variance), it is hardly an obvious choice. Freedman’s idea that his particular relative weighting of bias and variance was so sure — on such a “firm statistical foundation” — as to not need mentioning or discussion, that no reasonable person could disagree, really does resemble not returning library books in its self-centeredness. For more about why I thought Freedman’s position resembled not returning library books, read the rest of the post.

]]> by: Mike http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303156 Fri, 15 May 2009 00:32:03 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303156 Seth, You initially claimed that it was "ridiculous" to recommend against trying to remove bias. It is not ridiculous if the attempt to reduce the bias increases the variance so much that the mean squared error increases overall, which is exactly what the quote (and the paper) from Freedman says. I bring up mean squared error since that is the thing of which variance and (squared) bias are components. Freedman didn't say "mean squared error" since the average person would not have understood that. Instead he said "accuracy." The misleading quote from you is "He was against adjusting the census to remove bias caused by undercount. This was only slightly less ridiculous than not returning library books." You have offered no argument for this statement; Freedman on the other hand, offers a detailed argument for his position in the paper cited. Seth,
You initially claimed that it was “ridiculous” to recommend against trying to remove bias. It is not ridiculous if the attempt to reduce the bias increases the variance so much that the mean squared error increases overall, which is exactly what the quote (and the paper) from Freedman says. I bring up mean squared error since that is the thing of which variance and (squared) bias are components. Freedman didn’t say “mean squared error” since the average person would not have understood that. Instead he said “accuracy.” The misleading quote from you is “He was against adjusting the census to remove bias caused by undercount. This was only slightly less ridiculous than not returning library books.” You have offered no argument for this statement; Freedman on the other hand, offers a detailed argument for his position in the paper cited.

]]> by: seth http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303106 Thu, 14 May 2009 20:43:24 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303106 Mike, the average person doesn't know there are two components. Lumping them together under the name of "accuracy" glosses over the question of how to weight them. I don't see how it is misleading to say that adjustments will likely increase variance and reduce bias. Complicated, yes, misleading, no. Mike, the average person doesn’t know there are two components. Lumping them together under the name of “accuracy” glosses over the question of how to weight them. I don’t see how it is misleading to say that adjustments will likely increase variance and reduce bias. Complicated, yes, misleading, no.

]]> by: Mike http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303026 Thu, 14 May 2009 16:34:19 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-303026 Seth, As you say, adjustments will likely increase variance and reduce bias. A large increase in variance may not be worth a small reduction in bias, so it is not clear that adjustments are a good idea. To talk about "accuracy" is simply to talk about the net effect, which is what we care about. It seems much more misleading to focus on only one component at a time. Seth,
As you say, adjustments will likely increase variance and reduce bias. A large increase in variance may not be worth a small reduction in bias, so it is not clear that adjustments are a good idea. To talk about “accuracy” is simply to talk about the net effect, which is what we care about. It seems much more misleading to focus on only one component at a time.

]]> by: seth http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-296980 Wed, 29 Apr 2009 14:34:44 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-296980 "Whether adjustment will produce a more accurate count." As I said, to talk about "accuracy" is confusing. Accuracy has two components: variance and bias. Adjustment will very likely increase variance and reduce bias. The EK scheme was "equivalent" to a scheme where you "subtract random numbers"? Another confusing statement. Random numbers can be anything. “Whether adjustment will produce a more accurate count.” As I said, to talk about “accuracy” is confusing. Accuracy has two components: variance and bias. Adjustment will very likely increase variance and reduce bias.
The EK scheme was “equivalent” to a scheme where you “subtract random numbers”? Another confusing statement. Random numbers can be anything.

]]> by: Asad Zaman http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-296807 Wed, 29 Apr 2009 03:31:01 +0000 http://www.blog.sethroberts.net/2008/12/02/unfortunate-obituaries-the-case-of-david-freedman/#comment-296807 Technically you are right that there is a possibility of adjustment when the sample/resample undercounts are not perfectly correlated. Practically speaking, I do not know of any census adjustment schemes which allow for and adjust for such dependence. Erikson-Kadane et. al (EK). whom Freedman was arguing against, did assume independence in producing their adjustments. It is strange that the central issue: whether or not an adjustment will produce a more accurate count, is viewed as a distraction by you. I propose to subtract random numbers to adjust the census -- this is likely to improve the census, since it is known that there is an undercount. EK proposed a scheme which, as Freedman demonstrated, was essentially equivalent. He showed this in three different ways. A: Following exactly the same methodology, but using an alternative series to the one picked by EK (out of 18 equivalent series) produces seriously different adjustments. B: Changing some of the theoretical assumptions (in particular independence) required for the adjustments to more plausible ones, leads to seriously different numbers for adjustment. C: "Shrinkage," the Bayesian methodology for producing adjustments used by EK requires specification of a prior distribution. This is elegantly finessed via the empirical Bayes method actually used by EK, where prior assumptions are hidden at a deeper level. By changing these, one can arrive at substantially different numbers. In particular, someone who knows the game well can produce numbers which favor one state over another or to suit his own political preferences. Technically you are right that there is a possibility of adjustment when the sample/resample undercounts are not perfectly correlated.
Practically speaking, I do not know of any census adjustment schemes which allow for and adjust for such dependence. Erikson-Kadane et. al (EK). whom Freedman was arguing against, did assume independence in producing their adjustments.
It is strange that the central issue: whether or not an adjustment will produce a more accurate count, is viewed as a distraction by you.
I propose to subtract random numbers to adjust the census — this is likely to improve the census, since it is known that there is an undercount.
EK proposed a scheme which, as Freedman demonstrated, was essentially equivalent. He showed this in three different ways.
A: Following exactly the same methodology, but using an alternative series to the one picked by EK (out of 18 equivalent series) produces seriously different adjustments.
B: Changing some of the theoretical assumptions (in particular independence) required for the adjustments to more plausible ones, leads to seriously different numbers for adjustment.
C: “Shrinkage,” the Bayesian methodology for producing adjustments used by EK requires specification of a prior distribution. This is elegantly finessed via the empirical Bayes method actually used by EK, where prior assumptions are hidden at a deeper level. By changing these, one can arrive at substantially different numbers. In particular, someone who knows the game well can produce numbers which favor one state over another or to suit his own political preferences.

]]>