Actually, Kaplan's analysis doesn't really hold a candle to Lancet's peer review. He uses an undergraduate's knowledge of statistics to set up and demolish a straw-man rather than looking at the Lancet study for what it shows. Here are a few more-well-done third-party looks at that study:

• from The Economist
• from Z Magazine
• from The Picket Line

Actually, Kaplan is still the best - and 2/3 examples you cite obviously correspond to some new definition of "well done."

The Economist treatment was good, and got at most of the issues. A study with a range of 8,000 - 194,000 deaths is kind of like an election poll saying Bush will win between 5% and 98% of the vote. The Economist also notes that the study got Iraq's pre-war death rate wrong. And you can stop right there - on the basis of those 2 points alone, why should anyone take this study seriously?

The Economist's flaw was that it sidestepped the study's authors claiming figures of 100,000 to the media whith "high confidence" - a statement that, in light of their study's self-acknowledged limitations, can only be described as a lie. That's a significant omission. The authors' promotional dishonesty is central to the issue of professional misconduct in the service of their ideology. Which this certainly appears to be.

Z Magazine pretended to be critical, but its own author glosses over the amazing range figures and myriad flaws to praise it - perhaps because he, too, is a self-admitted activist who finds its point of view congenial? Lots of reasons found to believe it anyway, few questions about the reliability of poll-based methods in war, where violence patterns are not random. Or about issues around trying to do a survey that may not be random but does match the weighting and intensity of populations and fighting throughout the country. The study is only "good" because he never really goes after its methodology, and elides over the incongruity of the researchers' 100,000 claim and the lack of support for same with any great confidence. Not useful, not honest, not convincing.

The Picket Line doesn't even pretend much, and its author describes "folks, who probably, unlike myself, passed their introductory stats class with a grade that helped their average." Yeah, which is why we ought to take her seriously on a statistical health study, I guess.

She confesses ignorance of the subject, then follows with an idiotarian rant that blathers on about the evils of america and "cowardly" American methods of warfighting. Wow, I'm sure convinced now... and that's what "well-done" looks like to you, hmm?

Well, that tells me pretty much all I need to know. You and I are done with this argument. Anyone else?

Anyone else? OK, I’ll bite.

Much of the criticism of the wide confidence range (8,000 – 190,000) is cast in a way to imply a uniform distribution, when in fact the study would produce a non-uniform distribution (probably a chi-square, but it’s been a while since I took stats). As the Economist states:

It does not, however, mean, as some commentators have argued in response to this study, that figures of 8,000 or 194,000 are as likely as one of 98,000. Quite the contrary. The farther one goes from 98,000, the less likely the figure is.

There are a couple other salient points from the Economist article: the Lancet study includes the increase in the death rate of everyone, civilian and, well, non-civilian, and death from all causes, violent and non-violent. Subtracting the 40% “increase” in non-violent deaths gets you to 60,000, and further subtracting the deaths of combatants would get you – where? Perhaps 20 or 30 thousand, which is within spitting distance of the Iraq Body count number of 15,000. This number is commonly believed to undercount the number due to unreported deaths (and also challenged as inflated by reporting bias).

With regard to the methodology, it is fair to state that the Economist raises substantial issues with the sample size and the randomness of the Lancet study. It is not the Economist but Kaplan who points out that the study’s assessment of the pre-war Iraq death count is anomalously low. Perhaps the reporting bias against reporting pre-war deaths is more systematic than the Economist conjectures. In my mind the pre-war death rate anomaly is Kaplan’s strongest argument, and the best reason to be suspicious of the Lancet’s numbers.

Given all this uncertainty I’d say simply that disagreement between the Lancet study and the Iraq Body Count has not been demonstrated, despite the surface difference in the numbers.

The violent deaths of non-combatants would in a perfect world never serve as the object of “statistics” – they are all individual tragedies. But who killed them, who bears the guilt?

I’m compelled here to point out that in his last paragraph, Kaplan demonstrates that he as dissected the Lancet study simply to further the idea that we have effected in Iraq not liberation but murder of innocents. What all leave unexamined is this: that the proportion of violent deaths in the post war period caused by intra Iraqi political violence is substantial, and perhaps dominant. The Iraq war is perhaps best viewed as a civil war, a civil war that has been going on for much longer than we’ve been there, and that previous to our arrival had been particularly one sided. That Kaplan chooses not to address this obvious fact blunts the force of his final passive-aggressive invective. But I digress.

It's always embarassing to see scientists prostitute themselves like that. Getting real data is hard enough as it is without the need for this sort of spin.

>>The violent deaths of non-combatants would in a perfect world never serve as the object of “statistics” – they are all individual tragedies. But who killed them, who bears the guilt?

Indeed.

It’s a good article, until the end, when Kaplan says:

There is one group out there counting civilian casualties in a way that's tangible, specific, and very useful—a team of mainly British researchers, led by Hamit Dardagan and John Sloboda, called Iraq Body Count...

Unfortunately, Iraq Body Count’s methods are based is the methodology created by the infamous Professor who Couldn’t Count Straight, Marc Herold.

According to the Weekly Standard:

But far worse is a group that claims to be keeping an accurate running count of Iraqi civilian deaths but is, in fact, doing no such thing. The group is called the Iraq Body Count Project, and its main figure is Marc Herold, a professor of economics and women's studies at the University of New Hampshire. You may remember Herold from his similar project during the Afghanistan campaign. There, he produced a figure of almost 3,800 civilian casualties, and his methodology was immediately criticized by many for taking reports from unreliable media sources at face value and for double-counting some incidents. An independent analysis by the Los Angeles Times found 1,200 or fewer civilian casualties.

Unbowed, Herold turned his attention to the Iraq Body Count Project. The Project's website has a continuously updated "maximum" and "minimum" count. The problem is that the minimum is anything but. As the Project's methodology page explains, "The minimum can be zero if there is a report of 'zero deaths' from two of our sources. 'Unable to confirm any deaths' or similar wording (as in an official statement) does NOT amount to a report of zero, and will NOT lead to an entry of '0' in the minimum column." In other words, suppose the Iraqi Information Minister said, "Today the imperialist aggressors slaughtered 300 innocent Iraqi children." Reputable news outlets will report what the Minister said, while simultaneously reporting that they were unable to confirm it and that the Pentagon was unable to confirm it.

News sources from Time Magazine to the Guardian have questioned Herold’s inflated body counts.
Inflated body counts are used to inflict guilt about the ‘wages of war’. They’re rarely used to keep track of the wages of peace. Herold and co. have never done a body count of the hudreds of thousands murdered by the Islamist regime in the Sudan, and of course they have no intention of counting the casualties of the Saddam/UN oil for food alliance. Herold has never expressed any interest in counting the dead that resulted from the French-trained Hutu 'rebellion' in Rwanda.

The wages of war are death and the wages of international-community-controlled "peace" (genocide inflicted by out-of-control dictatorships) are a tragedy.

..umm...that should be "The wages of war and the wages of international-community-controlled "peace" (genocide inflicted by out-of-control dictatorships) are a tragedy."

I'd echo the point about the relative likelihood of the "tails" of that estimate versus the center. The overall point is that the estimate doesn't include zero -- the war/occupation has increased death rates compared to pre-war -- given assumptions that Kaplan rightly points out. But these are not political hacks, in my view, at least co-author Richard Garfield isn't. As I remember it, he arrived at low-ball estimates of the effect of sanctions once upon a time, so he is particularly well qualified to judge that baseline. (Indeed, be careful what to wish for in that respect -- the higher the pre-war mortality, the more deaths were caused by sanctions; supporters of stern measures against Iraq can't have that death rate every which way.)

That leaves objections to the true randomness of the sampling, and those may turn out to be valid. On-the-spot decisions like these are almost always made in surveys, and it's not surprising or necessarily damaging to the results that they happened here. Kaplan understates the effort that the study made to be random, in my view.

It seems to me the proper response to this study is to do a better one -- require better statistics from the U.S. and coalition military -- that's arguably their duty under the law of war. It isn't to say "debunked" and wash your hands of the matter.

Tim Lambert is following the issue at his blog, although he hasn't commented on Kaplan's piece yet.

Shannon Love at Chicago Boyz has been all over this since nearly 2 weeks ago.

lewy14 - like you, it's been a while since I was doing stats, but when I read the Economist report a few days ago the last sentence of the statement you quoted made me furrow my brow. I have a guess as to what the writer was trying to explain, but, whatever type of distribution we're talking about, the statement that "the farther one goes from [the center of a confidence interval], the less likely the figure is" is just incorrect.

That is, I think it's incorrect. Corrections welcome. (The problem with the blog discussions I've seen on this topic is that I haven't seen any people who seriously know what they're talking about participating. Please don't think that's intended as a diss on you, lew. I'm mean all of us "it's been a while" types.)

Contra Nephew, lewy14 and The Economist, and supporting Breen; a confidence interval is not a normal distribution. A confidence interval (CI) is a statement of, well, confidence in your estimate of a value; the standard 95% interval means that 95% of the time the real value will fall somewhere within the range you have estimated (assuming your sample, and the population as a whole, meet the assumptions of whatever technique was applied).

It does not mean that the real value is more likely to be near the center.

As an analogy, think of public opinion polls - a 4% margin of error means "the real value is x, plus or minus 4%"; not "the real value is x, plus or minus 4%, but really it's closer to x".

The large CI is the result of the small sample size and the variability between the samples. You can think of the CI as a 'penalty' that is applied to an estimated value; an estimate based on small or highly variable samples will have a large 'penalty' applied, an estimate based on large samples with little variation will have little 'penalty'. When you have applied a 95% CI, you are saying (or should be saying) "My real value is somewhere in here; but based on the data I have, I can't be any more specific than that." If the researchers had been able to collect a greater number of samples, then they probably would have been able to reduce the CI. But these data were probably hard enough to get as it is.

Nephew is correct in saying the result indicates that the rate of death has increased. But because of the very large confidence interval, it can't tell us how much.

[Caveats: haven't read the original study, and it's been about 7 years since I did statistical work on a regular basis.]

I don't know squat about statistics, but my impression is that this seems like a controversial enough topic that would be worthwhile to have a roundup of several sources rather than just an outright dismissal. Tim Lambert in particular seems to know what he's talking about (and has gotten around to addressing the Slate piece now), and while I'm not a regular reader of his, I've found him to be intellectually honest in the past.

...a controversial enough topic...

Isn't it, though? Unfortunately, most people discussing it seem to be more interested in trading insults than in clearly stating points or refuting them (for instance, who knew Michael Fumento was such a jackass?)

Generally, I agree with Tim Lambert on this: "The best estimate (not including Falluja) is 100,000 extra deaths, but the confidence interval is wide, so the estimate should be treated with caution."

However, I would restate it thus -
"The estimate of this study is 100,000 extra deaths, but the confidence interval is very, very wide - so wide that in my end of science (ecology) I think you'd have a lot of trouble getting data with similar results published - so the estimate should be treated with extreme caution."