Seth’s blog

The Roberts-Schwartz correspondence continued. I replied to Schwartz:

“Dining establishments”? [His previous email stated: “Four restaurants simply cannot represent the variety of dining establishments in New York City”] I thought the survey was about sushi restaurants. Places where raw fish is available.

Quite apart from that, I am sorry to see such a fundamental error perpetuated in a science section. If you don’t believe me that the teenagers’ survey was far better than you said, you might consult a friend of mine, Andrew Gelman, a professor of statistics at Columbia.

John Tukey — the most influential statistician of the last half of the 20th century — really did say that a well-chosen of sample of 3 was worthwhile when it came to learning about sexual behavior. Which varies even more widely than sushi restaurants. A sample of 4 is better than a sample of 3.

Schwartz replied:

The survey included 4 restaurants and 10 stores.

The girls would not disclose the names of any of the restaurants, and only gave me the name of one store whose samples were not mislabeled. Their restaurants and stores might have been chosen with exquisite care and scientific validity, but without proof of that I could not say it in the article.

I wrote:

I realize the NY Times has an “answer every letter” policy and I am a little sorry to subject you to it. Except that this was a huge goof and you caused your subjects damage by vastly undervaluing their work. Yes, I knew the survey included 4 restaurants and 10 stores. That was clear.

As a reader I had no need to know the names of the places; I realized the girls were trying to reach broad conclusions. They were right not to give you the names because to do so might have obscured the larger point. It was on your side that the big failing occurred, as far as I can tell. Did you ask the girls about their sampling method? That was crucial info. Apparently The Times doesn’t correct errors of omission but that was a major error in your article: That info (how they sampled) wasn’t included.

He replied:

I could have been more clear on the subject of sample size, but I did not commit an error. Neither do my editors. That is why they asked me to write a letter to you instead of writing up a correction.

I don’t feel I have been “subjected to” anything, or that this is some kind of punishment. This is an interesting collision between the precise standards of someone with deep grounding in social science and statistical proof and someone who tries to write intelligible stories about science for a daily newspaper and a general interest audience. But I am not sorry that you wrote to me, even a little sorry.

i wrote:

“I did not commit an error.” Huh? What am I missing? Your article had two big errors:

1. An error of commission. You stated the study should be not taken seriously because the sample size was too small. For most purposes, especially those of NY Times readers, the sample size was large enough.

2. An error of omission. You failed to describe the sampling protocol — how those 10 stores and 4 restaurants were chosen. This was crucial info for knowing to what population the results should be generalized.

If you could explain why these aren’t errors, that would be a learning experience.

Did you ask the girls how they sampled?

His full reply:

We’re not getting anywhere here.

Not so. After complaining he didn’t have “proof” that the teenagers used a good sampling method, he won’t say if he asked them about their sampling method. That’s revealing.

Something similar happened with a surgeon I was referred to, Dr. Eileen Consorti, in Berkeley. I have a tiny hernia that I cannot detect but one day my primary-care doctor did. He referred me to Dr. Consorti, a general surgeon. She said I should have surgery for it. Why? I asked. Because it could get worse, she said. Eventually I asked: Why do you think it’s better to have surgery than not? Surgery is dangerous. (Not to mention expensive and time-consuming.) She said there were clinical trials that showed this. Just use google, you’ll find them, she said. I tried to find them. I looked and looked but failed to find any relevant evidence. My mom, who does medical searching for a living, was unable to find any completed clinical trials. One was in progress (which implied the answer to my question wasn’t known). I spoke to Dr. Consorti again. I can’t find any studies, I said, nor can my mom. Okay, we’ll find some and copy them for you, she said, you can come by the office and pick them up. She sounded completely sure the studies existed. I waited. Nothing from Dr. Consorti’s office. After a few weeks, I phoned her office and left a message. No reply. I waited a month, phoned again, and left another message. No reply.

Recently the New York Times published a fascinating article by John Schwartz in the science section about how two teenagers discovered that a lot of raw fish sold in New York is mislabeled. Unfortunately, the article contained two big mistakes: 1. The teenagers’ results were dismissed as unconvincing because the sample size (10 stores and 4 sushi restaurants) was, according to Schwartz, too small. For many purposes the sample was large enough, if their sampling method was good. 2. The sampling method wasn’t described. Without knowing how the stores and restaurants were chosen, it’s impossible to know to what population the results apply. This was like reviewing a car and not saying the price.

In an email to the Times I pointed out the first mistake:

Your article titled “Fish Tale Has DNA Hook” by John Schwartz, which appeared in your August 22, 2008 issue, has two serious errors:

1. The article states: “The sample size is too small to serve as an indictment of all New York fishmongers and restaurateurs.” To whom the results apply — whom they “indict” — depends on the sampling method used — how the teenagers decided what businesses to check. Sample size has almost nothing to do with it. This was the statistician John Tukey’s complaint about the Kinsey Report. The samples were large but the sampling method was terrible — so it didn’t matter that the samples were large.

2. The article states: “the results are unlikely to be a mere statistical fluke.” It’s unclear what this means. In particular, I have no idea what it would mean that the results are “a mere statistical fluke.” The error rate of the lab where the teenagers sent the fish to be identified is probably very low.

In retrospect the second error is “serious” only if incomprehensibility is serious. Maybe not. I should have pointed out the failure to describe the sampling protocol) but didn’t.

I got the following reply from Schwartz:

Thank you for your note about my article, “Fish Tale Has DNA Hook,” which appeared in the newspaper on Friday. You state that the story misstated the importance of sampling size as “an indictment of all New York fishmongers and restaurateurs.” Although you are certainly correct in stating that poor methodology can undercut work performed using even the largest samples, it is also ill advised to try to establish broad conclusions from a very small sample. The fact that mislabeling occurred one in four pieces of seafood from 14 restaurants and shops in no way allows us to conclude that 25 percent of fish sold in New York or in the United States is mislabeled. And that is all I was trying to say with the reference to sample size was that while the girls’ experiment shows that some mislabeling has occurred, their work cannot say how much of it goes on or whether any given restaurant or shop is mislabeling its products. Similarly, when I wrote that it is unlikely the findings are a “statistical fluke,” I merely meant that while it is possible that Kate and Louisa found the only 8 restaurants and shops in New York City that mislabel their products, that is not likely, and so the possibility that the practice is widespread should not be discounted. And, of course, I hope you can forgive the pun.

Thanks again for taking the time read the article and respond to it, and I hope that you will find more to like in other stories that I write.

Uh-oh. The email was as mistaken as the article, although it did clear up what “statistical fluke” meant. I wrote again:

Thanks for your reply. I’m sorry to say that you still have things more or less completely wrong.

“Their work cannot say how much of it goes on or whether any given restaurant or shop is mislabeling its products.” Wrong. [Except for the obvious point that the survey does not supply info about particular places.] I don’t know what sampling protocol they used — how they chose the restaurants and fish sellers. (This is another big problem with your article, that you didn’t state how they sampled.) Maybe they used a really good sampling protocol, one that gave each restaurant and fish seller an equal chance of being in the sample. If so, then their work can indeed “say how much [mislabeling] goes on.” They can give an estimate and put confidence intervals around that estimate. Just like the Gallup poll does.

Somewhere you got the idea that big samples are a lot better than small ones. Sometimes you do need a big sample — if you want to predict the outcome of a close election, for example. But for many things you don’t need a big sample to answer the big questions. And this is one of those cases. There is no need to know with any precision how much mislabeling goes on. If it’s above 50%, it’s a major scandal, if it’s 10-50% it’s a minor scandal, if it’s less than 10%, it’s not a scandal at all. And the study you described in your article probably puts the estimate firmly in the minor scandal category. In contrast to your “it’s cute but doesn’t really tell us anything” conclusion quite the opposite is probably true (if their sampling procedure was good): It probably tells us most of what we want to know. You’re making the same mistake Alfred Kinsey made: He thought a big sample was wonderful. As John Tukey told him, he was completely wrong. Tukey said he’d rather have a sample of 3, well-chosen.

Thanks for explaining what you meant by “statistical fluke.” You may not realize you are breaking new ground here. Scientists wonder all the time if their results are “a statistical fluke.” What they mean by this is that they’ve done an experiment and have two groups, A (treated) and B (untreated) and wonder if the measured difference between them — there is always some difference — could be due to chance, that is, is a statistical fluke. In your example of the mislabeled fish there are not two groups — this is why your usage is mysterious. I have never seen the phrase used the way you used it. And I think that the readers of the Times already realized, without your saying so, that it is exceptionally unlikely that these were the only fish sellers in New York that mislabeled fish.

Schwartz replied:

I understand your points, and certainly see the difference between a small-but-helpful sample and a large-but-useless sample. but four restaurants simply cannot represent the variety of dining establishments in New York City. Four restaurants, ten markets.

I also realize that you must think I am thickheaded to keep at this, but I will certainly keep in mind your points in the future and will try not make facile references to small and large samples when the principles are, as you state, more complicated than that.

To be continued. My original post about this article.

I got this in a Japanese supermarket:

Translation:

How to make soup curry:

Ingredients-
1 packet of soup mix
1 packet of flavorful oil
1 packet of spicy flavoring
80g (3 oz.) of chicken thigh meat cut into bite sized pieces
1/4 medium sized carrot
1/2 medium sized potato (cut in half)
400ml water

1. Boil water in a small pan.  Add chicken, potato, and carrot, cook until vegetables become soft, about 20 minutes on med-low heat.
2. Turn off the heat, add the soup mix and mix thoroughly, turn the heat back on and cook a little longer until the flavor penetrates the meat and vegetables.
3. Pour the flavorful oil onto a plate and pour the finished curry on top.
4. Add a desired amount of the spice flavoring.

-the spice flavoring is fairly spicy, so please use caution when adding
-please cook the chicken thoroughly before adding the soup mix
-to make a double portion, double the meat and vegetables, and increase water to 700ml.
-the black things in the soup are basil

To make a dish like the picture on the box: Add sauteed japanese eggplant, shimeji mushrooms, green peppers, and hard boiled egg to the dish. Use boned chicken meat.

How to eat: Using a spoon, scoop rice and add a small amount of curry to the spoon.  Please keep the curry and rice in separate dishes to prevent the rice from getting soggy.

Caution: Please use the entire contents of the packets after opening.  Cannot be preserved for later use.

I have bolded the interesting parts: 1. The use of please. 2. The explanation (”the spice flavoring is fairly spicy”). You won’t find them in the instructions on most American products. I became aware of this aspect of Japanese life when I read T. R. Reid’s wonderful book Confucius Lives Next Door: What Living in the East Teaches Us about Living in the West, which was based on six years Reid spent in Japan as a correspondent for the Washington Post. At one point Reid quoted from a sign in a park. The sign had a list of prohibitions: No littering, no music, and so on. But instead of saying, as an American sign would, “no littering”, the sign said something like: “So that others can enjoy the beauty of the park, please put your litter in the proper receptacle.”

A few years ago I taught a class called Psychology and the Real World in which students did some sort of off-campus work of their choosing. (An example of my teaching philosophy.) One student volunteered in a hospital. One day he told a story about being treated rudely by a nurse. I said, yeah, we live in a pretty rude culture. Japan is different, I said, and told the class about Reid’s Japanese park signs.

My student was impressed. He had a part-time job monitoring parking in front of a San Francisco hotel. People would often try to park in an area that needed to be kept clear and it was his job to get them to move. His method — pre-Reid — had been to go over to the offending car and say “sorry, you can’t park here.” Post-Reid, he was elaborately polite: “Please forgive me for disturbing you, but we need to keep this area clear so that taxis can pick up and drop off passengers. I’m sorry for inconveniencing you, but would you be kind enough to move your car?” Something like that. Pre-Reid, about half the time the driver would argue or cause some sort of difficulty. Post-Reid, there were no problems.

Thanks to Pearl Alexander.

I subscribed to Saveur for several years but never finished any of the long articles — which weren’t that long. This should have puzzled me, but it didn’t. A month ago, however, I got the audiobook of Secret Ingredients, the New Yorker anthology about food. I was surprised how many of the articles I didn’t want to listen to. 90%? Usually I like New Yorker anthologies and read most of the articles. I’m a more tolerant listener than reader which made the comparision even worse.

Here’s my explanation. Food writing is like downtown Edinburgh. Its main street has shops on one side, on the other side a park. What should have been the economically most lively street in the city is rendered half as effective as it might be by the fact that half of it isn’t businesses. As something to write about, food is similar. Just as a park is economically inert, food is psychologically inert. Like a park, food can be pleasant (to read about) but it doesn’t act. It isn’t alive. This is why those Saveur feature articles were hard to read, I realized. They resembled flat lists: We cooked and ate X, Y, and Z. It’s incredibly hard to make that sort of thing fun to read. The best article in Secret Ingredients was John McPhee’s profile of Euell Gibbons. It’s a mini-adventure story, with an interesting guy at the center. The food is . . . a condiment.

This is why I’m so impressed by the chapter “Waizhou, USA” in The Fortune Cookie Chronicles by Jennifer 8. Lee. Waizhou means “out of state” — in this case, away from New York City. It’s the story of a family who left New York to run a Chinese restaurant in a small Georgia town. It took a shocking turn that Lee didn’t expect. Police took the children away. The father was arrested. “The offices pointed to the burn scars from cooking oil on the parents’ arms and said that was evidence that the couple had a history of fighting.” This had horrible and ramifying effects. “Oh, I can’t eat there anymore,” said a lawyer, “that’s the DV [domestic violence] case.” The teenage daughter starts using the court system to punish her mother. The parents are arrested again. “They had violated court rules by driving near their children’s foster home. Because they had sold their restaurant they were considered a flight risk.” Eventually they get their children back, and go back to New York. It’s a whole slice of life I’d never read about before. Enormously emotional and unpredictable. The father enjoyed jail. “When I was in jail for two days, it was really relaxing,” he told Lee.

The Harvard Society of Fellows, I learned from this great post by Steve Levitt, drink expensive wine — like $60/bottle. Steve, who was a Fellow for 3 years, did a simple experiment that showed the other members couldn’t tell expensive wine from cheap wine. Although the other members had liked the idea of doing the experiment, they didn’t like the results:

There was a lot of anger when I revealed the results, especially the fact that I had included the same wine twice. One eminent scholar stormed out of the room stating that he had a cold — otherwise he would have detected my sleight of hand with certainty.

Stormed out of the room! Why were they so angry? I think they were embarrassed. And not just that. Steve doesn’t say it, but I think there had been lots of dinner table conversation about how great the wine was. Now all that conversation was revealed to be delusional. Noting the greatness of the wine was — to be crude about it — a way of noting the greatness of those assembled at the table. “We appreciate the finer things in life,” they were saying. “We deserve to be here.” Snobbery is reassuring. In a tiny voice, the results said, yes, you are here, congratulations, but the reason you are here is more complicated than “you deserved it”.