Ah, Joe, if only you visited us over here at debunkers.org more often ...

Lancet isn't the first "leading" medical journal to do so by a long stretch and won't be the last.

Ah, Joe, if only you visited us over here at debunkers.org more often ...

Lancet isn't the first "leading" medical journal to do so by a long stretch and won't be the last.

The global warming farce is another example, based on the fake hocky stick fraud.

The global warming farce is another example, based on the fake hocky stick fraud.

Hey, what about gun violence as a "public haelth" problem? Or "400,000 US deaths" annually from second-hand smoke? I don't trust doctors or medical journals whenever there's a public-policy suggestion behind what they're doing; they seem particularly bad about stretching the truth.

Thanks, Joe, for reprinting my ChicagoBoyz comments.

I pulled together my thoughts on the Roberts et al. Lancet paper last night, and posted them as a "pretend" peer-review: as what I would have written to the Lancet editor in October 2004, had I been asked to review the manuscript.

That comment is timestamped 28/3 13:48 on the 'Lancet Links' post at Aussie Tim Lambert's Roberts-boosting site Deltoid. (This link ought to take you right to that comment; it worked on my Mac, but not on this Windows machine--good luck)

As you will see, IANAS (I am not a statistician), and am proceeding on a common-sense-based basis. Strident Roberts advocate dsquared has claimed that common sense is a poor guide to matters statistical. While, overall, I strongly disagree, dsquared's point does have some merit, and was a reminder to me to try to tread humbly and lightly in these matters.

In that regard, based on another comment by dsquared, I have provisionally withdrawn my claim #2, that the exclusion of Fallujah was a severe design flaw in Roberts' original study design. Statistically-sophisticated advocates from each side (and neutral, disinterested parties, if they exist...) will have to shed more light on that.

If you visit the comments at Deltoid or ChicagoBoyz, please note that, although these sites are on different sides of the Roberts controversy, the levels of snark and acrimony are high at each. I've tried to keep on-topic and stay away from the personal in my remarks, and would hope that WoC commenters would also hold to that practice as "guest commenters."

Too bad AMac doesn't have a blog.

Funny you should mention that...

Given the enormous limitations of US rules of engagement, Lancet could only come up with anything close to 100,000 Iraqi deaths by fabrication.

Micahel, I disagree. Fabrication is what we might reluctantly have to discuss, once all other possiblities are ruled out. I'm not there; not even close.

Roberts' estimate was for 100,000 deaths of all causes, not counting Fallujah, in excess of what would have taken place if trends had not been disturbed by the invasion. They say they are 95% sure that the actual number is between 8,000 and 194,000.

The big problems aren't with the estimate, they are with the meaning of the estimate, and with the way the authors, the Lancet, and the wider scientific, public health, journalistic, and blogospheric communities have interpreted these findings.

Funny you should mention that...

Why's that?

Analyzing the details and statistcal arcana is all well and good, but does it really matter? Even if we assume there's nothing wrong with the data they used, the thing is still useless. As AMac said, all it tells us is that they're 95% confident that the number of deaths is between 8,000 and 194,000. As Fred Kaplan said months ago: "This isn't an estimate. It's a dartboard." It's like saying I'm 95% confident (before the election) that Bush would win between 30% and 70% of the vote. It's meanignless. You can pick whatever number you want between those two points and it's impossible to argue with because nobody actually knows.

Amac has been very fair on a number of issues where we disagreed, but the following facts do seem to be established, and I am surprised that nobody except me appears to regard them as important:

1. It would be very odd to get a sample in which the death rate increased by 50% if the death rate had actually fallen.

2. It would be quite odd to get a sample in which the death rate had increased by 50%, if the death rate had actually only risen by a small amount corresponding to the 8,000 excess deaths figure.

3. It would be quite odd to get a sample in which violence had gone from an insignificant cause of death to a very significant one, if the increase in violence was not in fact the major cause of the increase in the death rate.

I don't think anyone has a credible argument to suggest that the Lancet study does not provide fairly convincing evidence for all three of these propositions (the arguments at the Chicago Boyz site are now all about presentational matters rather than factual ones). And taken together, they do appear to substantially support the proposition "the invasion of Iraq has made things substantially worse for the Iraqi population (in so far as this can be measured by the death rate), not better".

I'd also add a fourth comment:

4. Everyone seems to agree that the vast majority of the deaths in Fallujah were, in fact, attributable to violence and most likely to violence by coalition forces. It seems unlikely to me that the death toll in Fallujah (plus that in Najaf, Samarra and Ramadi) will, in the final analysis, be so small compared to the total that it can safely be ignored.

I don't believe that any of my 1-4 require any statistical training at all to understand, at least in qualitative terms. And taken together, they imply something really quite serious indeed.

Good points, dsquared, and thanks for putting them up for the readership here.

A tip 'o the hat to you, because you are a trained statistician, and willing to put yourself in the spotlight, blogging on controversial matters under your real name (dsquared is an easily-decoded pseudonym). You have raised telling points on the Lancet paper, and moved the debate forward, especially regarding your specialist (statistical) expertise.

But you are not in my "circle of trust." I don't mean this as an insult, but to provide context to other readers. With the Lancet paper and in other matters, I have witnessed you blogging and commenting in an extremely partisan, adversarial fashion. You must feel--correctly--that you spend a lot of effort pointing out the logical fallacies in your adversaries' posts. But--in my estimation--you are willing to use many of the same tactics to advance the positions you favor.

In particular, when mulling over your comments, I have found myself thinking there is key information, crucial to evaluation of the points dsquared is making, that he is passing over in silence.

Joe has offered to let me post further thoughts on the Lancet study at WoC, and I will try and expand on this general point in a few days. No time now. And I really don't mean this as a personal affront to dsquared, but as a reflection on a style of discussion that has become normal in large swaths of the blogosphere. The contentious debate on the Roberts Lancet article could be a case study.

Brief notes on dsquared's four points:

1. It would be very odd to get a sample in which the death rate increased by 50% if the death rate had actually fallen.
2. It would be quite odd to get a sample in which the death rate had increased by 50%, if the death rate had actually only risen by a small amount corresponding to the 8,000 excess deaths figure.

Yes. But we aren't limited to descriptors like "odd." Roberts carried out a study of a certain size and certain design, and that gave it a certain "power." Roberts best-estimate is that, excluding Fallujah, 98,000 more Iraqis died in the 17.8 months post-invasion than would have otherwise. They are 95% confident that the real figure (known only to God) is at least 8,000, but not more than 194,000. As you know, I have seen no credible evidence that Roberts didn't do their cluster survey as they described, or that their statistical treatment was incorrect.

3. It would be quite odd to get a sample in which violence had gone from an insignificant cause of death to a very significant one, if the increase in violence was not in fact the major cause of the increase in the death rate.

Let me contrast dsquared's careful phrasing with some numbers plucked from the paper's Figure 2. Excluding Fallujah (since Roberts had to exclude it from their statistical analysis), the survey recorded 90 deaths (cf. 46? prewar). 21 (23%) of these 90 deaths (not excess deaths, total deaths) were due to violence (no violent deaths recorded in the prewar period). Although Table 2 doesn't say, Roberts emailed a blogger to say that 9 of these 21 deaths were due to Coalition action, and 12 were not.

Thus, Roberts' statistical estimate of the number of Iraqis killed by Coalition violence would be an extrapolation from those 9. Their statistical estimate of Iraqis killed by other violent acts would be an extrapolation from those 12. Common sense says these cannot be precise estimates. The problem is that the authors do not provide these figures, or the 95% confidence intervals that would accompany them. In the absence of these proper statistical treatments, it's open season for plausible assertions. That said, I agree with dsquared that, probably, the major cause of the increase in the death rate (not the major cause of death) as calculated by Roberts was due to the upsurge in violence.

4. Everyone seems to agree that the vast majority of the deaths in Fallujah were, in fact, attributable to violence and most likely to violence by coalition forces. It seems unlikely to me that the death toll in Fallujah (plus that in Najaf, Samarra and Ramadi) will, in the final analysis, be so small compared to the total that it can safely be ignored.

Yes. Perhaps if Roberts had run a more powerful study, they would have some insight to offer on this matter. By powerful I mean larger, more expensive, more time-consuming, and more dangerous. These aren't fair demands to make of authors. Less politely, I also mean better-designed (more on that in a few days), and not needing to be rushed to completion in time to, hopefully, affect the US Presidential election in Kerry's favor. That was a self-imposed constraint.

Amac, Shannon Love accused the researchers and the Lancet of fraud and treason. So why did you single out dsquared for criticism for being "strident"?

Tim Lambert (#13),

Good catch. I singled out dsquared because I thought it was a fair and relevant characterization to make in front of this readership in discussing a point he raised with me. As you will have noted, I expanded on my thoughts in post #12 above.

I don't think a catalog of "blog commenters I think are strident, or worse" will make WoC's Top-20-Posts list.

Some bloggers are very concerned about the conduct of those commenting from the other side, showing rather less concern about the behavior of their own team. I can refer you to some threads that illustrate this point, if you wish.

Because Shannon Love has been an important blogospheric critic of Roberts, I will share my opinion (FWIW) with readers here.

I think that Shannon Love has raised valid concerns about the Roberts study. Unfortunately, he or she has adopted a strident tone in some of her ChicagoBoyz posts on the subject. It would speak very well of him/her if s/he would go through the most serious charges of misconduct that s/he has leveled against Roberts et al., and support each with persuasive evidence, or retract it.

AMac has a valid point. Given the problems with the Falluja cluster, the study team was wrong to conclude that "air strikes from coalition forces accounted for most violent deaths". On that point, since the with-Falluja and without-Falluja results were contradictory, the team should have reached no conclusion.

But (or rather, BUT), that's it. Other than that half sentence in the "interpretation" section, the study has stood up to scrutiny. The critics have gone after the methodology and been refuted; they've gone after the findings and phrasing, and -- with the exception noted above -- been refuted. Now AMac's resorting to alleged defects in extraneous material -- "editorial comments" and "invited comments" -- to show that "what [the Lancet] publishes is not to be trusted".

A more reasonable conclusion would be, given the critics vast efforts and meagre achievements, that this particular study withstands hostile scrutiny very well indeed.

Oh, BTW, AMac. You've engaged on this topic long enough and thoughtfully enough to perceive that dsquared's and Tim Lambert's contributions have been careful and competent and that Shannon Love's haven't. Do you ever wonder where you might end up if you drew your "circle of trust" with more objective criteria?

Just to clarify, I never asserted (and never intended to imply) that the actual data produced by the study was fabricated by the researchers themselves. I do think it possible that the study's data was subverted by individuals or groups separately and without the knowledge of the researchers. Individuals may have intentionally exaggerated and, especially in Falluja, interviewees might have felt pressure from insurgents, whether actual or anticipated, to exaggerate.

Of course, I have no firm evidence either way and do not insist on it. I brought the matter up largely to empathize the very difficult conditions under which the study was conducted that raise legitimate questions about its accuracy.

I do think the study was carried out and that it did produce the data reported. As, Amac has pointed out, someone seeking to fabricate a study could do a much more plausible job.

Tim Lambert,

"Shannon Love accused the researchers and the Lancet of fraud and treason."

Actually, I don't think I used the words fraud or treason. Fraud would imply outright fabrication of some or all of the study which I don't think occurred. They do repeatedly make statements unsupported by their data which might be construed as fraud I suppose.

I would typify their actions as corruption. They seek to hijack the prestige and reputation for objectivity of a major scientific publication in order to advance their own political agenda.

I don't think their actions rise to the level of treason but they had to know that the study represented a propaganda boon to the insurgents and jihadist whether it will eventually be verified or not. It will assist them in recruitment and raising of money. By having such a positive effect for the insurgent/jihadist, the study will cause the war to last longer and to cause the deaths of more innocents.

The study's authors knew or should have known that this effect would occur. They either decided to assist the enemy as a necessary cost of doing the study or they showed a studied indifference to the easily predictable consequences of their work. Either way, it doesn't reflect well on them.

When I looked at the study back in the fall, I was struck by the very low pre-war crude mortality rate. (5 per 1000 per year) In 2000 and 2001, we were being told that sanctions were killing 5000 Iraqis per month. If both statements were true, that means that sanctions were responsible for about half of the deaths in Iraq. Also, it would imply that, without sanctions, Iraq's crude mortality rate might have been around 2.5. Compare with Iran, at 5.0, or Jordan, at 4.3 (Economist World in Figures, 2003). Are these figures in any way surprising, or has the 5000 per month figure been discredited? Also, do the authors discuss how certain interviewees were? Asking, "did Granny die after the start of the war" should elicit a confident response, but "did Granny die in the preceding 18 months" might get an "I'm not quite sure. It was a year or two before, but I would have to check some documentary evidence to be 100% sure". Would this count in the prewar mortality rate?

Mark Childerson:

has the 5000 per month figure been discredited?"

Yes.

do the authors discuss how certain interviewees were?

Yes; p. 6:

We believe it unlikely that recall bias existed in the reporting of non-infant deaths, because of the certainty and precision with which these deaths were reported, and the importance of burial ceremonies in the Iraqi culture.

I don't really object to being described as "strident", to be honest (I do object to being described as "dishonest", however, and I think I will need to have a particularly strident word with Jonathan Gewirtz over that epithet). Nor am I particularly concerned with anyone's theories about the authors' motivations; I'm in general of the opinion that lots of people dying is more important than "media bias" against the Republican party.

Personally, I think Richard Horton is right that the fact that people are dying from possibly preventable causes in Iraq is a proper object of study for doctors, that having studied it they are obliged by the Hippocratic Oath to do something about it, and that if "doing something about it" involves intervening in a US election, so be it. Medecins Sans Frontieres rushed out a study on deaths in Western Darfur in the hope of embarrasing the government of France at the United Nations ahead of a key debate and I wish they'd had more success. However, this personal opinion of mine is clearly irrelevant to the matter at issue, which is the fact of a significant (by which I mean, both statistically and practically significant) increase in the death rate in Iraq as a result of the invasion.

It appears that Shannon is moving away from accusing the authors of falsifying their study and providing propaganda for Fascists, for which I suppose thanks, although a real mensch would provide "clarifications" to this on the Chicago Boyz website rather than buried in a comments thread. However, I have a few issues with Shannon's disclaimer in post #17:

Shannon certainly seemed to claim in "A Lie in A Labcoat" that the Fallujah cluster was not selected at random (the relevant paragraph is the ninth one, beginning "The fact that a Falluja cluster even got studied raises red flags"). This is a very specific and very serious accusation and I would like some clarity as to whether Shannon is standing behind it.

In the next paragraph, Shannon makes the claim "The Falluja data was selectively excluded merely because it was extreme. This is a textbook example of bad science". I think we've established that it's not, so this claim also ought to be dropped; we've also established that the statement "The honest approach would have been to either completely include or exclude (preferably the later) the Falluja cluster completely" does not describe good statistical practice as set out in Kruskal's article.

In "Fisking Fallujah", Shannon claims that "GPS couldn't be used to randomly select the cluster based on demographics, unlike the other clusters, so they did it manually". This isn't true; the cluster was selected at random from a map grid; the researchers did not carry GPS units but they still did their level best to preserve the randomness of the selection. To claim that "they did it manually" is another false claim.

And of course, Tim is entirely correct to say that the notorious passage "Either through intention or willful disregard, the researchers and publisher acted as a propaganda tool for the Fascist elements in Iraq. Given the degree to which they carefully spun their results, I conclude the effect was intended. " is an accusation of treason. People who provide propaganda for the enemy during wartime get executed.

Furthermore, Shannon's second post on the subject refers to the use of cluster sampling as "Scientific Malpractice". It clearly isn't. Even if the long-promised defence of this critique arrives (with pictures!) and Shannon manages to prove that the cluster sampling critique isn't complete balderdash (my current opinion is that it is), it's been established by numerous epidemiologists that cluster sampling was a sensible approach in the circumstances and certainly does not constitute malpractice.

In general, I would agree with Robert MacDougall that if all the Lancet quibblers (I think it is no longer fair to call them "denialists", since nobody appears to be denying the key finding any more, for which, thanks) can come up with is claims that the article was easy to misinterpret rather than that it made false factual statements, then perhaps the right conclusion to draw is not so much "We Can Never Believe This Journal Again" as "Perhaps people who don't really know all that much about statistics shouldn't be so quick to assume they understand journal articles in statistics". This would go for hacks of both sides, obviously, though as I've mentioned on CT, I find it quite hard to fight a battle against left-wing hacks screaming "100,000 dead!" when the other side is trying to deny the study outright.

Finally, on the question of "circles of trust", me and Tim might be "strident" in insisting on scrupulous standards, but a quick glance at the amount of red ink and strikethrough text in the Chicago Boyz posts reveals that the alternative is to make serious accusations based on schoolboy calculation errors. Which is also pretty strident, and if we had decided that we couldn't be bothered doing it, I suspect that at least three flat out falsehoods would have remained on the Chicago Boyz website, and presumably propagated throughout the blogosphere (there are also a few other basic errors in comments threads, such as Shannon's assertion that " Everybody who has studied armed conflict at all knows that you will see a spike in deaths. Did we need the study to tell us this?", when a glance at the bar chart in the study shows that there was, in fact, no "spike", but these are probably less serious).

The educational system needs more strident people, not fewer, and when people claim that "dsquare's arrogant assertions of superior knowledge and his belittling tone toward opponents are essentially dishonest", after I've corrected their maths for them, gratis, it is an insult of the kind that is not easily forgotten.

dsquared (#21):

Thanks for the long and thoughtful comment. You nicely set out your positions for the reader to consider.

I do have one quibble.

In general, I would agree with Robert MacDougall that if all the Lancet quibblers (I think it is no longer fair to call them "denialists", since nobody appears to be denying the key finding any more, for which, thanks) can come up with is claims that the article was easy to misinterpret rather than that it made false factual statements, then perhaps the right conclusion to draw is not so much "We Can Never Believe This Journal Again" as "Perhaps people who don't really know all that much about statistics shouldn't be so quick to assume they understand journal articles in statistics".

Or perhaps more than a quibble. This is a slippery paragraph. The members of the vast denialist conspiracy are individuals, each with their own synaptic pathways, thoughts, ideas. I don't engage in the hobby of blogging to attack every defense of Roberts, or to defend every attack. Instead, my objective is to present a reasoned view of a contentious story, support it with evidence, and open the floor to reasonable dissenters. Blog readers can then explore further, and make their own judgements.

It's your prerogative to declare that you have rebutted every criticism of substance; that a consensus has, belatedly, formed around your point of view; and that it's time for a victory lap.

Alas, I can still write:

• The statistical analysis Roberts performed seems inadequate, given their failure to disaggregate mortality statistics by cause of death.

• The Roberts paper was written and edited in such a way that many or most readers are likely to be misled on the significance of some key assetions.

• That such misinterpretation is commonplace is emphasized by the Lancet Editorial and the Invited Comment that accompanied the paper.

I won't provide links this time, as they are readily available to the curious reader who scans this and the accompanying post on the subject.

For a standard against which to measure Roberts' performance, I would suggest what The Lancet itself advises its authors:

Whatever you have written, remember that it is the general reader whom you are trying to reach. One way to find out if you have succeeded is to show your draft to colleagues in other specialties. If they do not understand, neither, very probably, will The Lancet's staff or readers. [bold in original]

As far as "Perhaps people who don't really know all that much about statistics shouldn't be so quick to assume they understand journal articles in statistics": I think we've each provided the readership with sufficient food for thought on our opposing views of that contention.

One bracing victory lap later ...

To be frank, some of your presentational points have merit (though not as much as you think; "the general reader" here means "British MDs who are meant to be able to assess statistical material on public health and drug trial results"). I made a fair few criticisms of the Lancet's handling of the paper myself in the early days, before I became outraged at the way in which the study was being treated by denialists like Her Britannic Majesty's Foreign and Commonwealth Secretary, Mr Jack Straw.

The Lancet editors and the authors can defend themselves on all the points you're making; I think that they have a good hand to play. Personally, myself, I am concerned with one particular point, on which subject I think I have established that there are no critiques indexed by Google or Technorati which have merit (as far as I can see, you accept this yourself, don't you?);

that it should be accepted that there is good reason to believe that the invasion of Iraq resulted in a substantial increase in the death rate

I really can't stop shouting about this. The death rate of a populous country rose by 50%! What the hell!?! How can this be a source of indifference? Whatever one thinks about the Iraq War, this appears to be a fact about its consequences, and it is an important fact. It is currently being either denied or ignored and I consider that to be dangerously silly.

Not that you're a hack, dsquared, far from it; but what's the difference between 'left-wing hacks screaming "100,000 dead!"' and your own, as you describe it, 'shouting' that the death rate 'rose by 50%!'?

I'm happy to be corrected on this, but aren't both claims different ways of saying the same thing, based on exactly the same survey data, the same calculations and margins of error? The 100K figure follows from acceptance of a 50% rise in general mortality in Iraq over 18 months; if the '100K' figure shouldn't be proclaimed so baldly, then neither, perhaps, should the '50%'.

Your earlier and more parsimonious observation that the error margins for excess deaths don't include zero are I think more on the mark.

but what's the difference between 'left-wing hacks screaming "100,000 dead!"' and your own, as you describe it, 'shouting' that the death rate 'rose by 50%!'?

Probably a fair point. But when you talk about "death rates" (or even better, risk ratios) you're unlikely to forget that you're talking about a sample frequency and unlikely to confuse it with a body count.

dsquared (#23)

> "the general reader" here means "British MDs who are meant to be able to assess statistical material on public health and drug trial results".

We've been around the dance floor on this one a couple of times. Either you have a star-struck view of MDs, or you travel in better circles than I do. Maybe both. I'll leave that to the readers.

that it should be accepted that there is good reason to believe that the invasion of Iraq resulted in a substantial increase in the death rate.

Do I believe that this is what Roberts data and analysis shows? Yes. Do I believe it to be true? I believe it's cause for concern, and that it may well be true. What, exactly, is the "it"? Can one, as you do, justify "50%" from 98,000 CI 8,000-194,000 ? No.

The excess death findings are crucially dependent on a correct appreciation of the prewar circumstances, and I am uncertain that Roberts got this right.

• The low power of the study is a big problem.
• Do the infant-mortality figures Roberts quote reflect the actual 2002 circumstances? I'm unsure.
• Was Iraq as non-violent as the 1 prewar death-by-violence figure suggests?
• Are there systematic errors in the study methods that would bias results in one direction? Some are plausible,but I don't know.

There are four assertions implicit in Roberts work:

• Iraq was a pretty healthy place beforehand. Deaths due to diseases have greatly increased.

• Iraq was a pretty low-accident place beforehand. Fatal accidents have greatly increased.

• Iraq was a pretty non-violent place beforehand. Deaths due to crime and other sources have greatly increased.

• Iraq had no Coalition-caused deaths beforehand, and has suffered many now, many in Fallujah, over half women and children settle this discussion on Iraq any more than they would have for the US Civil War, World War 2, or any other conflict.

let me try that fourth point on, again:

• Iraq had no Coalition-caused deaths beforehand, and has suffered many now, many in Fallujah, over half women and children under 15.

I don't see deaths from heart attacks and deaths from roving Coalition soldiers as identical. Since this is a statistical survey of mortality, what are the best-estimates and confidence intervals of these, or other reasonable, categories?

Once these questions are addressed, there are weighty moral matters to consider, as you allude to.

I haven't addressed them here. They are beyond the scope of this part of the discussion. Whatever conversations and decisions that our societies come to will be advanced by good data and good science. Mortality changes don't settle this discussion on Iraq any more than they would have for the US Civil War, World War 2, or any other conflict.

The low power of the study is a big problem.

Is it? Says who? On what basis? I am not aware of anyone in a position to judge who thought that 7800 individuals was too few individuals, or that 33 clusters was too few clusters.

Do the infant-mortality figures Roberts quote reflect the actual 2002 circumstances? I'm unsure.

Was Iraq as non-violent as the 1 prewar death-by-violence figure suggests?

Well the answer to this is almost certainly not; it was either more violent or less violent. The question here ought to be; is there any reason to believe that the quoted confidence intervals are wrong (I've not seen any)?

In any case, let's get out the pocket calculator and the back of an envelope and do a bit of the old senstivity analysis. If the sample had found another 2 deaths prewar, bringing the violent death rate up to 34.6k/100k, then this would have moved the total prewar death rate from 5.3/1000 to 5.5/1000 [1]. If we double the number of infant mortalities as well, we can get it up to 6.1/1000. This is still quite a long way from the ex - Fallujah postwar crude death rate of 7.9/1000 and in order to get to it I have assumed the world's third highest murder rate and an infant mortality rate of 59/1000 live births, implying that the infant mortality rate fell after the invasion.

Are there systematic errors in the study methods that would bias results in one direction? Some are plausible,but I don't know.

Which ones do you regard as plausible, and (importantly) are they all in the direction of making the excess deaths figure smaller? You'll forgive a pre-emptive snarl at people who start building up recall bias into a "fatal flaw" while ignoring the fact that, say, the study did not adjust for the fact that sometimes, the nearest "household" to a sampled GPS cluster was a burnt-out dwelling with nobody living there; they just went to the next nearest.

I don't see deaths from heart attacks and deaths from roving Coalition soldiers as identical.

I do, and I would submit that so does science. It really isn't for doctors to start making judgements about moral culpability. The key death rates/excess deaths number is based on an objective criterion; is your relative consuming oxygen or not? Although I would prefer the debate to be carried out in terms of risk ratios, because it focuses attention more closely on the nature of the data, that doesn't mean I don't believe the excess deaths number; as an anoynymous commentator above points out, anyone who believes in sampling theory, believes that the survey was carried out in a reasonably competent manner and believes in multiplication is pretty much obliged to take the central 100,000 "excess deaths" as his central estimate if he starts talking about excess deaths at all. The granular data is not, in my opinion, a fit subject for extrapolation in this manner.

Mark Mulholland points out that at the low end of the confidence interval for the crude postwar death rate, Iraq is twice as dangerous as the worst eyar in Northern Ireland. At the centre of the confidence interval, it is ten times as
dangerous. Since the British Army never called in airstrikes in Northern Ireland (not even with "precision" bombs), I would guess that Iraq is much closer to ten times worse than twice as bad.
¹ These are hand calculations carried out by me. The reported prewar death rate is 5.0/1000; I don't quite know why. I multiplied the number of deaths by 12/14 and divided by 7438.

dsquared (#28),

Good points. I am out of time for today. Some of your points have quick answers, many others don't. I don't understand why you have rejected disaggregation of mortality data by cause-of-death as an important and even necessary component of analysis. Seems to me that back of the envelope calculations imply that such best-estimates and CIs can be computed.

dsquared,

I think I will accept your suggestion that I answer your questions in more detail in a post at Chicago Boyz.

On a matter less technical, let me say I find it amusing but not surprising that you evince outrage that I would dare question the integrity of Les Roberts et al but that you yourself fell free to make casual accusations of mass murder and war crimes against U.S. service members based on no better evidence.

I think my assertion that the study on helps the cause of the Fascist is inarguable. There is nothing in the study that reflects poorly on them and the study's authors go out of their way to lay deaths at the feet of the Coalition.

The only real question is whether Roberts et al helped the Fascist intentionally or whether they were simple indifferent to such a negative effect. I would like to be able to say for certain that they showed mere willful disregard but unfortunately there does exist a considerable fraction of the Left that believes that America must fail in Iraq no matter what the cost to the people of Iraq. There is precedent in this type of thinking in the opposition to the Vietnam war where many viewed an American success as a greater evil than a communist one.

As to whether I obliquely accused Roberts et al of treason? Well, treason isn't what it used to be. In the 70's, John Kerry repeatedly made unsubstantiated allegations that US service members in Vietnam routinely committed war crimes with the knowledge and approval of "all levels of command." He even made those allegations under oath before congress. His allegations were repeated verbatim in Communist propaganda in Vietnam and throughout the world. Yet apparently nearly half the US electorate did not consider his actions treasonous enough to disqualify him for the Presidency. If Kerry's actions weren't treasonous, I don't think we can accuse Roberts of it.

When I read both the study and Roberts related comments in other forums I get the distinct impression that he regards the Fascist in Iraq as something analogous to a mindless force of nature. For Roberts, only the Coalition is culpable so I doubt he spent much time contemplating the possible negative consequences of his work.

Shannon writes:

"On a matter less technical, let me say I find it amusing but not surprising that you evince outrage that I would dare question the integrity of Les Roberts et al but that you yourself fell free to make casual accusations of mass murder and war crimes against U.S. service members based on no better evidence."

This is rather disingenuous. Dsquared is NOT accusing any one of mass murder or war crimes (where does he get this? Find a quote. I dare ya) and neither are Roberts et al. Look, I will reiterate the central point of the study: the death rate in Irag has grown and by a very large margin. This conclusion is based upon what dsquared (and myself, and Lancet's board, and....the list goes on) believes is a statistically and scientifically sound study. How this translates into "mass murders" and "war crimes" is beyond me. What union do you belong to? They must give you an award for stram man construction work.

And your points about Kerry and treason are risible. Why didn't anyone in Nixon's administration bring Treason charges against Kerry at the time? You dig a hole, and you like it.

you yourself fell free to make casual accusations of mass murder and war crimes against U.S. service members based on no better evidence.

That vile accusation is a complete lie. I have never said anything that could be remotely interpreted in anything like this way. I'm no longer inclined to mince words. You're a liar, and you are telling what are transparently obviously lies. How do you expect to be taken seriously when you march into a good-natured and intelligent debate and say something like that. I would be more concerned by this if I thought that there was the slightest chance of anyone believing you.

Go on, substantiate this charge. You've got the whole internet and twenty four hours. You dirty liar.

dsquared and Amac,

I am far from a statistician. I only passed my one graduate statistics course because the professor was a good-natured, hard-drinking Irishman who pretty much led the class by the nose through the problems, excercises and definitions he presented us. And I will certainly not join Shannon in making any accusations of anything against anyone. I do have a question though. What are the policy implications of all this? Let's assume dsquared and Lancet's are right. Does that mean the war in Iraq was useless or immoral? I, for one, refuse to believe that it's just coincidence that Ghaddafi gave up his WMD program a week after we dragged Saddam out of his hidey-hole or that after 1/4 century in power, Hosni Mubarak picks now to pretend to have a democracy. Walid Jumblatt, from what I've gathered, is no fan of Bush yet maintains that the "cedar revolution" in Lebanon is a direct result of the invasion of Iraq. Are we statistics ignoramuses (my term, I'm not accusing anyone of name-calling) to understand that if dsquared is right the war was not worthwhile, but if Amac is right it was? If not, what is the point of the whole debate?

Fred,

I'd hate to be an iraqi reading this and finding that people might consider a probable excess of 100,000 of my people worth it because the lebanese decided to get rid of the syrian troops.

In any case, the answer to your question is political, and obviously subjective. It has little to do with whether the study is sound and solid.

erratum
it should have said:
a probable excess of 100,000 of my people dead

Darwin... first, note the point above about those mortality statistics. By badly underestimating the baseline, you get "excess" figures that are wildly higher than they'd be otherwise. And that's just the start in terms of the way this report was compiled and presented.

Second, let's try to keep the projection in the movie theatres where it belongs. You didn't live under Saddam, you pretty clearly have no idea what that entails, and so in a perfect mirror image of the study, you're not in a good position to be making evaluations because your baseline is a shambles.

I give you Jeff of And I Am Not Lying, For Real:

"Tash and I are both terrifically loudmouthed critics of the Bush administration, she from an Australian perspective and me from a disgruntled American's p.o.v. We are also both white, young, healthy, and from lands that enjoy a vast degree of privilege, thousands of miles away from true suffering."

First point of light - he recognizes this.

"We knew we didn't have the full story so we went out to the Iraqi Out-Of Country Voting poll on Sunday to get another truth and see ground-level democracy for ourselves."

What. A. Concept. Back to Jeff:

"You may think that you have felt dumb before, but let me tell you something: until you have stood in front of a man who knows real pain and told him that you are against your country's alleviation of his country's state-sponsored murderous suffering, you have not felt truly, deeply, like a total f---g moron."

You might consider reading the rest.

Joe,

1. Whether you agree or not with The Lancet study, the discussion on its validity depending on the political consequences of the war is still irrelevant. That said, you seem unaffected by the fact that its been clearly established (even AMac had to agree) that the Lancet study is a sound one.

2. Your anecdotal evidence of pain and happiness in Iraq is as inconsequential as obviously incomplete: I doubt that the families of the thousand of dead are too happy with 'ground-level democracy' obtained over their pain.

3. With respect to:
You didn't live under Saddam, you pretty clearly have no idea what that entails, and so in a perfect mirror image of the study, you're not in a good position to be making evaluations because your baseline is a shambles

I didn't live under Saddam. I did live under General Videla '76-'83. I was jailed once, beaten up twice and a member of my family was kidnapped and is since the a "desaparecido". All this is still irrelevant to the discussion, other than serves to make a fool of you for opening your mouth too wide.

Fred:

What are the policy implications of all this? Let's assume dsquared and Lancet's are right. Does that mean the war in Iraq was useless or immoral?

I can't answer for dsquared or AMac, but the study authors and the Lancet have already answered for themselves.

From the study:

Our results . . . should lead to changes to reduce noncombatant deaths from air strikes.

and

. . . with modest funds, 4 weeks, and seven Iraqi team members willing to risk their lives, a useful measure of civilian deaths could be obtained. There seems to be little excuse for occupying forces to not be able to provide more precise tallies. In view of the political importance of this conflict, these results should be confirmed by an independent body such as the ICRC, Epicentre, or WHO. In the interim, civility and enlightened self-interest demand a re-evaluation of the consequences of weaponry now used by coalition forces in populated areas.

Lancet editor Richard Horton:

For the occupiers, winning the peace now demands a thorough reappraisal of strategy and tactics to prevent further unnecessary human casualties.

An invited commentator on the study, Sheila Bird, wants "an alliance between military intelligence and public health sciences" to improve estimation of air strike casualties. And another, Bushra Ibrahim al Rubayi, wants "the international community and the medical establishment in the UK" to protest the "continuous and preventable loss of innocent lives in Iraq".

I've no time for anything but this snippet, but would be gratified to see the discussion nudged back to civility...if there's much left to say.

Darwin (#37),

>(even AMac had to agree)
AMac didn't have to agree to anything, any more than anyone else. If I felt my criticisms had been shown to be invalid, I would say as much. If you read my contributions to this lengthy thread, you may see that I try and acknowledge good arguments as such, and try to let them affect my judgement. Some things I'm letting hang; I don't have a response formulated, and may not for some time. Or may set out in an orthogonal direction. That's the way of these things.

What I chose to do is to limit my criticisms to where I might be able to add to the conversation. Since I have little knowledge of the Roberts team's conduct in Iraq, I've little to say about that. Etc.

Hope this helps.

I am sorry to hear about your experiences under the junta, and especially of the death of your relative. This is truly terrible. You are right, this affords no particular insight into the technicalities of the Roberts paper. But it may give you a unique perspective on Iraq.

My experience of Joe Katzman is that he is far from a fool. Perhaps one outright argument is enough for this thread?

Ugh, even short notes grow long. Brevity is what I need.

Darwin,

I find the loss of several hundred thousand Americans in WW II to save Europe from the Nazis and Asia from the Japanese to be worth it. And yes, my comment was certainly political and has zero bearing on the soundness of the study. My point is that sound or unsound, the study has political implications. I was simply trying to ascertain what those implications might be.

Robert,

Thanks for the summary of the study. That's the kind of thing I was asking about.

dsquared,

"That vile accusation is a complete lie."

Well, the Lancet study accuses the US of a studied indifference to civilian casualties. (See the concluding paragraph) which is a war crime in anybody's book. You are a passionate defender of the study so I rather concluded that you shared Roberts' outlook.

Personally, I don't see how the US could inflict the hundreds of thousands of civilian deaths via airstrikes that the study claims without quickly crossing over the line into war crimes. According to the study, US helicopter attacks must be killing hundreds of women and children everyday. I fail to see how they could be doing so in a responsible and lawful manner. If you believe the study's view of the death rate in Iraq is accurate then logically you would have to support the contention that the behavior of the US military is unlawful.

More specifically however, I had in mind some rather intemperate remarks I recall you having posted on Crooked Timber. Perhaps I misremembered and if so I apologize in advance. I will see if I can track down what I remember.

Personally, I don't see how the US could inflict the hundreds of thousands of civilian deaths via airstrikes that the study claims without quickly crossing over the line into war crimes.

So, if the study proves to be right (and in the mind of most observers that is the case) Shannon Love will be accusing U.S. service members of mass murder and war crimes.

TREASON!!!

Can someone supporting the Lancet report explain the usefulness of a claim of casualties in the range of [8000, 198000]? How many orders of magnitude is that anyway?

I can accept that all the methodoglies employed are perfectly correct. What I don't understand is how this data is useful. If a serious decision, say support member loading for a high rise, were to be based on similar data, the advocate would likely be shown the door.

Can someone explain to me how such data is worthy of publication? Is this the level of proof required for the medical studies published by Lancet? If so, I'll likely have to decline any medical procedures advocated by them!

Maybe I'm just confused. Someone help me out here!

Can someone explain to me how such data is worthy of publication? Is this the level of proof required for the medical studies published by Lancet? If so, I'll likely have to decline any medical procedures advocated by them!

If ignorance-based skepticism is your thing, then maybe you should just reject cholesterol-lowering remedies for vascular disease, cancer treatment, and, in general any modern medicine therapies. Your family might not appreciate your principled stand though.

You can bore yourself to death with the multitude of answer to your questions anywhere here, or at
http://www.crookedtimber.org
or at Tim Lambert's site:
http://cgi.cse.unsw.edu.au/~lambert/cgi-bin/blog/

Well, the Lancet study accuses the US of a studied indifference to civilian casualties. (See the concluding paragraph) which is a war crime in anybody's book. You are a passionate defender of the study so I rather concluded that you shared Roberts' outlook.

Which paragraph reads, in its entirety:

US General Tommy Franks is widely quoted as saying “we don’t do body counts”. The Geneva Conventions have clear guidance about the responsibilities of occupying armies to the civilian population they control. The fact that more than half the deaths reportedly caused by the occupying forces were women and children is cause for concern. In particular, Convention IV, Article 27 states that protected persons “. . . shall be at all times humanely treated, and shall be protected especially against acts of violence . . .”. It seems difficult to understand how a military force could monitor the extent to which civilians are protected against violence without systematically doing body counts or at least looking at the kinds of casualties they induce. This survey shows that with modest funds, 4 weeks, and seven Iraqi team members willing to risk their lives, a useful measure of civilian deaths could be obtained. There seems to be little excuse for occupying forces to not be able to provide more precise tallies. In view of the political importance of this conflict, these results should be confirmed by an independent body such as the ICRC, Epicentre, or WHO. In the interim, civility and enlightened self-interest demand a re-evaluation of the consequences of weaponry now used by coalition forces in populated areas." (footnotes removed from original)

I think we have established, after six months and God knows how many thousands of words, that I have read the study. So there is no point trying to fool me about what it says.

Furthermore, there is the little matter of mass murder. If I could refresh your memory:

you yourself fell free to make casual accusations of mass murder and war crimes against U.S. service members based on no better evidence.

I count three specific claims about what I have said: 1) accusations of mass murder 2) accusations of war crimes 3) that these accusations were made against US service members (rather than coalition forces commanders who were not US service members).

When challenged, you have attempted (amateurishly) to substantiate 2), by making exaggerated claims about a paragraph of the Lancet study and claiming without evidence that I endorse your exaggeration. That leaves 1) and 3), both of which are very serious slurs on my integrity which I cannot allow to stand unchallenged.

You've got twelve hours left, and then I'll be wanting my apology.

By the way, I would hate to lose my reputation for mastery of detail, so I would point out that the fact that I am now having to defend myself against Shannon's baseless slurs does not mean I have forgotten his slurs against other people. In particular, as I pointed out earlier ...

Shannon certainly seemed to claim in "A Lie in A Labcoat" that the Fallujah cluster was not selected at random (the relevant paragraph is the ninth one, beginning "The fact that a Falluja cluster even got studied raises red flags"). This is a very specific and very serious accusation and I would like some clarity as to whether Shannon is standing behind it.

Setting a deadline seemed to work quite well on the previous baseless accusation, but I don't want to overuse the tactic. So, Shannon, at your leisure ...

GIGO. Only fools accept the conclusions given.

The sampling error in this Lancet study is large enough to drive a NASA rocket tractor through. It is quite easy to use precise statistical methods to achieve a patently ridiculous result.

But this is simply absurd. We have the mass graves from Saddam's atrocities. Where are the mass graves from coalition air attacks? There are none. Second hand and third hand reports from non-randomly selected informants with potentially ulterior motives to provide false information. My first year epidemiology students would be in trouble for committing these errors.

Statistics is not epidemiology. You do not consult a statistician with no epidemiologic experience in such cases. The faulty assumptions made will kill you.

So Gunter Haas, where do you teach epidemiology? And under what name?

McDougall, judging from your comments above, you would not qualify for admission. Thanks for the inquiry.

The author of post #47 is visibly either a troll or someone whose first language is not English. In either case, I'm personally not planning on replying to them; anyone else who wants to can do what they like.

If ignorance-based skepticism is your thing, then maybe you should just reject cholesterol-lowering remedies for vascular disease, cancer treatment, and, in general any modern medicine therapies. Your family might not appreciate your principled stand though.

Sorry. Perhaps I should be more precise. Would a drug company undertake the development of a new drug based on a CI this wide? Would the FDA approve it?

And if a drug shown to have a positive result with a CI this wide were available, would it be worth it? The cost of the drug at the lower end of effectiveness would likely not be practical. What if the various side effects were only known with similar precision? Should I then be happy to feed this drug to my family?

These are all fundamental issues. Even granting that the survey had no flaws and the methods were perfect, the Lancet study can only assert with certainty that 8000 more deaths have occured post invasion than would have occurred before. There is no basis to assert the 100,000 (or is it 98,000) figure. That assumes data or analysis not in evidence.

Back to the highrise analogy... Do I need to design the bulding to support 8000 KG/M2 or 198,000 KG/M2? A engineering study of this precision would force a building to support the larger figure. This is obviously a nonsensical approach. No one could afford to build a highrise based on data like this.

A study of this precision is meaningless in the context of actually making an informed decision, about anything. The most that can concluded is that a positive response was noted and further study is required to characterize it further. This is not a conclusion to be shouted from the mountaintops.

Would a drug company undertake the development of a new drug based on a CI this wide? Would the FDA approve it?

How about a more directly relevant analogy. Do military commanders take action based on much worse evidence? Did we ourselves have information about Iraq as detailed and rigorous as this survey when we decided to invade? Or, if you find that a bit near the knuckle, would a government take immediate action to respond to evidence of an epidemic on the basis of a survey like this?

The most that can concluded is that a positive response was noted and further study is required to characterize it further.

I would say that is the least that can be noted, but there you go:

This is not a conclusion to be shouted from the mountaintops.

Yes it is, when those mountaintops are occupied by an army commanded by someone who says "we don't do bodycounts".

How about a more directly relevant analogy. Do military commanders take action based on much worse evidence? Did we ourselves have information about Iraq as detailed and rigorous as this survey when we decided to invade?

It's like anything else. The risks and rewards are considered. I'm guessing that we will never agree on the benefits of the invasion versus the risks of not invading. It's interesting to me that you suggest that military decisions, including the whole "fog of war thing",be taken with the same rigor as medical practice. It's also interesting that you apparently don't mind when a medical journal publishes articles whose purpose isn't the furtherance of medical practice, and stretches to make claims that their own data may, but does not with certainty support, e.g. 98,000 extra deaths.

would a government take immediate action to respond to evidence of an epidemic on the basis of a survey like this?

Again this is a simple cost/benefit tradeoff. A small positive response may require immediate action if the costs of the action is equally small. Further study may be required if the bottom of CI didn't support the costs but the upper end does.

I would say that is the least that can be noted

You have a point. This depends on your objectives. As an engineer, I've been trained to be conservative in my claims in the interest of reliability and public safety. That's why I used the "most" phraseology. It refers to the highest figure that we can assume with 100% certainty without overstatement. If your goals are not to be conservative or reliable, but to make political hay perhaps, then you may wish to use the "least" phraseology with respect to the lower limit of the CI.

Yes it is, when those mountaintops are occupied by an army commanded by someone who says "we don't do bodycounts".

Perhaps so. But, you shouldn't hide behind dubious scientific claims to do so. Though 98,000 figure might be asumed to be more reliable than the 198,000 upper limit of the CI, it's not clear why it is asserted as THE number of deaths. Why not claim that higher figure? On what basis can the 98,000 figure be asserted? The 8000 lower CI limit is the most that can be asserted with 100% certainty, beyond that there is no reliable clue.

Err.. make that 95% certainty.

We have the mass graves from Saddam's atrocities. Where are the mass graves from coalition air attacks? There are none. Second hand and third hand reports from non-randomly selected informants with potentially ulterior motives to provide false information. My first year epidemiology students would be in trouble for committing these errors.

why would the scattered victims of air strikes or machine-guns be buried in mass graves? Why would the family of Mohammed bury him with Abdul? Is that your only standard for innocent civilian deaths?

You state, correctly, that iraqis were victims of saddam's atrocities? Why would they now be 'informants' with 'ulterior motives'? Are they nuts? Or are you saying that The Lancet researchers only interviewed baathists? If so, prove your ridiculous claim.

Your first year epidemiology students would be in trouble if they keep learning to analyse statistical data based on their political beliefs.

That's no substantiation, ladies & gentlemen. I hope I'm not going to have to make do with Shannon Love's grudging "apologize in advance", but if I do, I daresay I'll get over it.

dsquared,

Pointing out flaws, fallacies, and gaps at both a scientific methodology level and the level of common sense is legit.

The budren of substantiation does not fall to your opponents. It falls to the one making the claims, and that's the Lancet. Also you, as one of those who believe the Lancet's study was something more than a flawed and ideologically driven study, sold dishonestly.

Nice try on shifting the burden of proof, though.

Nice try on shifting the burden of proof, though

The study was published in one of the most respectable journals, reviewed by at least 3 experts that found it sound and solid and you think the burden of proof on disproving claims that it is scientifically dubious and its authors dishones falls on them?.

Twisted, dude.

The truth is that so far all claims of flaws, fallacies, and gaps have been proved wrong, trivial or overstated.

Joe, I was actually asking for substantiation of the specific claim that Shannon had made about me; that I had accused US forces of war crimes and mass murder (post #30 above). I don't think it's unfair to suggest that the burden of proof lies on the accuser in that case.

Sorry, d^2, I misread. You are correct. Just as the burden of proof lies on the Lancet for its claims regarding Operation Iraqi Freedom, the burden also lies on Shannon for his claims regarding you.

Unlike scientific study debates, explicitly political accusations can always come down to perception and opinion wherein lie irreconcilable differences. But even then, one must explain with some persuasiveness what these perceptions/ opinions are based on, or risk losing the non-partisan portion of one's audience (and often, some of the partisans as well).

Re: "we don't do body counts"

The US military used to do body counts. It no longer does so, partly because body counts tended to be wildly inaccurate and partly because the practice of counting the bodies of dead enemy soldiers created perverse incentives for American troops. Tommy Franks, who made the quoted statement, served in Vietnam, where body counting became disreputable. Such a statement from an officer of his age and experience is unremarkable. All of this is widely known.

I don't know if the US military ever did civilian body counts, but given its bad experiences with body counts in Vietnam it is not surprising that it does not do so now. The Lancet paper, and some of the commenters who cite General Franks in various comment threads, seem to suggest that the US military is being evasive or is indifferent to civilian deaths. This suggestion is ludicrous in light of the US military's recent history, which features tactical innovations to minimize civilian casualties in urban combat and systematic efforts to correct many of its past mistakes.

Of course, if we did count civilian deaths we would be criticized for that, too.

Shannon, your posts are all heat and no light. You should just shut up before you embarass yourself further.