When The Lancet online-published Mortality before and after the 2003 invasion of Iraq: cluster sample survey a few days before the U.S. Presidential election, a firestorm of controversy resulted. Was this a case of science done right, afflicting the comfortable — in this case, George Bush? Or was it hack science in the service of a political cause and candidate?
The controversy died down without resolution, as these things often do, and people moved on to other matters — the Red State/Blue State divide, the Boxing Day Tsunami, the Jan. 30 elections. Then on March 18, Instapundit highlighted a post from Oh, That Liberal Media on The Debunked Lancet Study. And the whole thing picked up, right where it had left off.
Amidst the charges and countercharges flying in the press and the blogosphere, how do citizens approach this sort of Battle of the Experts?
That’s a subject that has been of some interest here at WoC, as it has related to AIDS and climate change, and other issues. It’s a key question that Dan Darling struggles with as he evaluates open-source and classified intelligence on Al-Qaeda.
Earlier today, Joe shared some of my reflections on this issue as it relates to that Lancet paper by Les Roberts et al. This post is an extension and a summary, originally posted as a comment under ‘Lancet Links’ (28/3 13:48) at Tim Lambert’s blog Deltoid.
My background is important. I’m a biologist by training, with training in “simple” statistics comparable to my peers’. To a nonscientist, familiarity with Gaussian distributions, means, standard deviations, Student’s t-tests and the like sounds impressive. But it isn’t to a real statistician, much less a practicing epidemiologist. For the subject matter of Roberts' study, I am not an expert.
My comments on the Roberts controversy follow. I have rejected the most damning criticisms of the paper: that the data are forged or manipulated, that the cluster survey method used is fundamentally inapplicable, that the statistical summary of excess casualties is inflated or invalid. I have also rejected important claims of Roberts defenders -- that the paper is a paragon of excellent science, yielding valuable information obtained under trying wartime conditions.
I have written this critique by imagining that I had been asked by The Lancet’s editor to peer-review the manuscript. Actually, there’s a subtle distinction: "If I had been a reviewer, and had known what I know now as a result of online discussions." My actual review would have been about three times as long as this, because I would have cited specifics. Those can be picked up in the posts and comments cited earlier. Or, prospect for content-rich but very partisan posts on the pro-Roberts side at Deltoid and Crooked Timber. Content-rich but very partisan posts on the anti-Roberts side are found at ChicagoBoyz.
Lancet Editor,
I recommend that this article be accepted for publication, provided the authors extensively rewrite certain parts of it. Under the circumstances, to require additional fieldwork is to effectively reject the paper; I have not done so here. If the authors are unwilling to satisfactorily address the points raised here, I would urge that The Lancet decline publication.
1. Readers must be given a more complete view of the data. As this article is sure to generate controversy, Roberts should collate the survey results by cluster, and submit the results as Supplementary Online Material, as an Excel spreadsheet. It should be accompanied by an online supplement to the Methods, describing the relevant intermediate calculations that led to the best-estimate of excess deaths and its 95% confidence interval. In addition, within the paper itself, the authors must include additional summary information, in particular the number of interviews attempted and completed, and the numbers of individuals covered. Fallujah-cluster data must be provided separately from data for the other 32 clusters.
2. [The authors’ statistical approach, as designed, was not able to accommodate the mortality survey results from Fallujah. So many people died from violence there that this cluster couldn’t be incorporated int the overall analysis. Was this “statistical outlier” an example of “Wild Data” that can’t be anticipated, as pro-Roberts statistician dsquared suggested (timestamp 3/28 05:45)? Or, as Roberts skeptic Heiko Gerhauser claimed (timestamp 28/3/2005 21:40), was a simple alternative available to the surveyers, but neglected? Since I don’t know, I provisionally withdrew this criticism. See this classic 1960 essay by William Kruskal for plain-spoken background.]
3. The Summary and certain paragraphs of the Discussion must be rewritten. Conclusions drawn from the statistical analysis of the survey must not be conflated or juxtaposed with conclusions drawn from treatments of raw numbers. By definition, conclusions supported by statistical analyses are more likely to be widely correct. As written, the MS is misleading on key points. All assertions based on non-statistical summaries that include data from the Fallujah cluster must be explicitly labeled as such.
4. The authors comment extensively on the implications of their findings for Coalition military policy without providing statistical analysis for the excess deaths calculated to have resulted from Coalition action. The authors have implicitly disaggregated their data in raising certain issues in the Discussion. Therefore, they must provide estimates of excess deaths and 95% CIs for the subsets they choose to discuss, or explicitly justify why they have not done so. The Lancet Editor should urge the authors to perform and present such additional analyses in any case, as the public-policy implications of the study’s results are critically dependent on the causes of the excess mortality. I would suggest subsets of (A) Diseases, incl. Infant Mortality; (B) Accidents; (C ) Violence not caused by Coalition; and (D) Violence caused by Coalition forces. Other approaches to disaggregation would also be acceptable.
5. The authors must alert readers to the sensitivity of excess-death calculations to initial circumstances. For the total figure and for subsets A, B, and C suggested immediately above (not an issue for subset D), excess deaths are the subtraction of a pre-war estimate from a post-war estimate, where each estimate has a large uncertainty. Figures so calculated can have particularly large uncertainties associated with them. There is little expert consensus on certain pre-war mortality rates in Iraq, preventing a robust check of study-generated pre-war estimates with independently-derived figures.
In conclusion, if I had been a reviewer whose advice had been followed, the Roberts study would have been published in the Lancet, perhaps on the same schedule. Working from the same data set, the message imparted by such an article to non-statisticians would have been very different from that delivered by the actual paper.
For those who argue that the Roberts article, as published, was not easily misinterpreted, here is the opening of the Invited Comment that was published alongside it (Bushra Ibrahim Al-Rubeyi, "Mortality before and after the invasion of Iraq in 2003," Lancet v. 364, # 9448, p. 1834, 20 Nov 2004):
In this week's Lancet, Les Roberts and colleagues show that the death toll from the invasion and occupation of Iraq is about 98000 civilians, and it might be considerably higher. The deaths are mostly related to air strikes.
To conclude: this non-expert’s analysis took a lot of time.
Does it display common-sense--and how important is that, anyway?
Do the links supply the necessary background? Is it properly recounted in the post?
Can the reader link the strongest arguments against what the writer’s arguing?
Is it good enough to help a “lay” person to come to a conclusion?
The readers make the call.
Welcome to the team, AMac!
A note on point 5: ih his comment reply over at the blog Deltoid statistics expert dsquared offered this remark:
An example of how a non-expert can unwittingly go astray; my thanks to dsquared for the correction. I still suspect that this is one reason why Roberts' best-estimate of excess deaths (98,000) has a 95% confidence interval that is so broad (8,000 - 194,000).
Urgh... because of a new bug in Movable Type, it's important NOT to put any links in the first 8 words or so.
Otherwise, it wrecks out main page.
From Crooked Timber:
The Chronicle of Higher Education has an excellent article following up on the Lancet study. That study is still basically unchallenged, by the way; however many epidemiologists you ask, they’re all going to give the same answer, that it was good science. Link here".
Too beyond me, unless I take quite a lot of time, which I don't have. But if you are very interested, take the time. You've heard the establishment view, now hear the democratic response.
Small world, i was just having an email conversation about this issue.
The statistical analysis seems to be sound and peer reviewed. How the results have been spun by the authors and media is a different story.
It is deeply concerning that this is far from a neutral party conducting this research. We dont take what DNC or RNC pollsters tell us at face value obviously. I think we have learned in the last few years just how dangerous blindly trusting polls can be. It nearly turned a presidential election last fall, or at least threw the media into a tizzy.
That being said, if there is an error it is in the polling methodology, and that area is rife with potential problems in this case. Just one small example of many: how many families would fess up to their Fedayeen-Al Saddam member brother getting shot up by GIs? So much easier to say he was an innocent civilian and produce the death cert if asked. How can we possibly even approach accounting for that type of behavior?
I am also suspect of the data from the years preceeding the war. Were we not informed reliably by every aid orginization in the world of the hundreds of thousands sanctions were killing every year? Its simply difficult to have faith in the numbers being bandied about by those with an axe to grind, and these authors have a serious axe (go read the paper yourselves).
"The deaths are mostly related to air strikes."
It is particularly hard to fathom how they can declare this without so much as a caveat. I recommend everyone look at the report. They slip the Fallujah data in and out of their analysis as they see fit.
Take a look at the data and see if it look anamalous to you.
- In the preinvasion data, there is 1 violent death reported out of the 7438 individuals. Does that sound accurate of the Hussein's Iraq? That translates out to 3250 violent deaths (.0134%) in all of Iraq, including crime and the regime. I would be surprised if American violent death levels were so low.
-13 men and 4 children were reported killed in violence outside Fallujah. But only 2 women and 2 elderly. If airstrikes were to blame why the disconnect? I can see children being less likely to survive, but why not the elderly, and why do men seem to attract bombs, and if indeed they were not combatants why were they 6.5x more likely to die than their mates/parents?
Why were children twice as likely to die violently as their mothers?
-childrens mortality jumped up 87%, while the elderly only 16%, and women only 31%. That seems odd to me.
-The Falluja data suggests most of the city was erradicated. Something in the neighborhood of 57,000 dead if the falluja data is accurate. Thats a lot of bodies to account for.
Hmm, "good science," you say? Kaplan:
Alastair Mackay:
1. Readers must be given a more complete view of the data.
Desirable but expensive. A rule of thumb is that it takes about three times as long to write your programs in a form suitable for outside use and reuse as it does to just do the job the quickest way you can. To quote from Kevin Donoghue at deltoid:
If we can reach a social decision to go for more replicability in exchange for higher costs per study, that might be a good thing on balance. But you should be clear that what you're demanding here is far above current standard practice.
3. The Summary and certain paragraphs of the Discussion must be rewritten.
This picks up in more moderate form Shannon Love's claim that Roberts et al. "confuse the issue as much as humanly possible" by "weav[ing] the Falluja data though the paper like a poison skein" (or, in your earlier words, "segueing, unannounced, from EFC analysis to IFC raw numbers").
With one exception, this criticism is baseless. Throughout the main text of the paper, it's always clear which numbers are with - Falluga and which are without. Without pretending to be a literary gem, the paper is in fact a reasonably clear piece of technical writing. As appropriate, references to "reported" or "attributed" happenings include Falluja reports unless noted otherwise.
The exception is the second sentence of the "interpretation" paragraph of the summary: "Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths." It's not clear whether this refers to data or extrapolated estimates. If it refers to data, it's accurate; if it refers to estimates (which in context would be more appropriate), then the part relating to air strikes is unwarranted (because contradicted by the without - Falluja results).
So yes, one sentence in the study is misleading (on one interpretation; or ill-founded, on another) and should have been rewritten. Maybe if you keep trying you'll eventually find a second bad sentence but you haven't so far.
4. The authors comment extensively on the implications of their findings for Coalition military policy without providing statistical analysis for the excess deaths calculated to have resulted from Coalition action.
The first part of this is false: the comments on implications for Coalition policy are few and brief: the last sentence of the summary:
and the last paragraph of the paper, especially:
Similar to point 1, the demand for statistical analysis of findings used to support recommendations -- in this case, violent deaths caused by Coalition forces, and deaths from Coalition air strikes -- is desirable. Unlike point 1, it seems likely that it would also be reasonably affordable. So this seems a reasonable request.
5. The authors must alert readers to the sensitivity of excess-death calculations to initial circumstances.
This seems to be based on a misunderstanding, especially given the following remarks:
As a matter of standard practice, the excess deaths calculation doesn't assume that the pre-war death rate is known with certainty. The confidence intervals reported for excess deaths take account of both the pre- and post - war uncertainties; that's part of the reason why they're so wide.
In summary, you make a reasonable case for (1) revising one sentence, and (2) reporting two more pairs of confidence intervals (deaths from Coalition, and deaths from Coalition air strikes, with and without Falluja). If that's all you can reasonably conclude, then you should also be happy to agree:
Mark (#5):
I wouldn't call the lead author neutral, given his interview with the Chronicle of Higher Education and the study release's timing. But that doesn't invalidate the survey. This is a Science version of the Jeff Jarvis/Buzzmachine conversation about Journalism. If some thoughtful people are saying, "be open with yer biases!," it can't be beyond-the-Pale to have biases.
I think it's important to speak very carefully and to document facts very carefully before thinking of making a public accusation of fraud or misconduct. These would be very serious charges, and we owe it to people in the public eye to be scrupulously fair. When it comes to Les Roberts and his colleagues, I have seen no credible evidence of fraud or misconduct.
It might be possible to build a circumstantial case centered on funny-looking data. As a non-expert, I am not capable of making that judgement. In fields where I am an expert, I've seen cases where what seemed to be evidence of misconduct--wasn't.
Possible undercounts or overcounts due to dishonest or incorrect answers to surveyers' questions aren't evidence of misconduct.
When the raw numbers are small, as here, funny-looking distributions don't mean much.
By all means, let's discuss the methods, the data, the interpretation, the presentation. But if commenters feel they must charge fraud, I ask that they refrain from doing so on this thread.
I dont believe i ever suggested fraud. I did note that when a party is clearly agenda driven their results should be scrutinized all the more. I dont think partisan polling firms necessarily fabricate their data, but they certainly have consious and unconcious biases that have been shown in many, many cases to affect their work.
These researches had an agenda and they made no secret of it (to their credit perhaps). There's a well known truism in science that when you go looking for something you're likely to find it.
Robert McDougall (#7):
You misread my first point,
1. Readers must be given a more complete view of the data.
On point 3, the mistake in the Summary is significant. Refer back to the Invited Comment I quoted from for an example of the effects of not writing clearly.
The second-to-last paragraph of the Discussion is about "Iraqis reportedly killed by US forces...28 of 61 killings (46%) attributed to US forces..." Do the authors make clear that this is mainly about Fallujah, or must the reader return to Table 2 to figure that out? Did all readers do so?
On point 4, I count the final four paragraphs of the Discussion as concerning the Coalition. The first two of these are arguable, the final two aren't.
On point 5, see my exchange with dsquared, quoted in comment #2.
As for your summary, I accept that's how you see things, but would not present it here as my own view.
Joe Katzman:
From 1985-90, the years leading up to the 1991 Gulf War, the [infant mortality] rate declined to 6.8 per 1,000. After '91, the numbers are murkier, but clearly they went up.
. . . as a result of sanctions; and then, as a result of Oil for Food, they went down.
Robert McDougall (#11):
Thanks for the citations concerning the pre-invasion infant mortality rate in Iraq. Readers clicking on the first will see that CASI is the Committee Against Sanctions in Iraq, and be able to view the authors' openly-proclaimed stance about how the civilized world should have treated with Saddam's Iraq. E.g. "I am absolutely convinced that when history is written with truth, then this embargo on Iraq will go down with the firebombing of Dresden, with the Holocaust and with Hiroshima." Valid source for statistics--maybe. Impartial? No. Reliable? For me, that would have to be proven.
If readers follow the second link, to Chris Lightfoot's blog, they can contrast his point of view to that of commenter Heiko Gerhauser.
These sources, and the UN/UNICEF figures they largely depend on, leave me with a lot of doubt as to the (known-only-to-God) actual rate of Iraqi infant mortality on the cusp of the invasion.
When I looked at the study back in the fall, I was struck by the very low pre-war crude mortality rate. (5 per 1000 per year) In 2000 and 2001, we were being told that sanctions were killing 5000 Iraqis per month. If both statements were true, that means that sanctions were responsible for about half of the deaths in Iraq. Also, it would imply that, without sanctions, Iraq's crude mortality rate might have been around 2.5. Compare with Iran, at 5.0, or Jordan, at 4.3 (Economist World in Figures, 2003). Are these figures in any way surprising, or has the 5000 per month figure been discredited? Also, do the authors discuss how certain interviewees were? Asking, "did Granny die after the start of the war" should elicit a confident response, but "did Granny die in the preceding 18 months" might get an "I'm not quite sure. It was a year or two before, but I would have to check some documentary evidence to be 100% sure". Would this count in the prewar mortality rate?
In response to Robert McDougall
bq._"as a result of sanctions; and then, as a result of Oil for Food, they went down"_.
That's not entirely true. According to the much cited UNICEF Child and Infant Mortality Survey of 1999, the infant and child mortality rates rose from 47 and 56 in 1989 up to 108 and 131 respectively in 1999. This was in the 15 governorates where oil-for-food aid distribution was completely controlled by Saddam Hussein.
However, the story is quite different in the three governorates where oil-for-food aid was completely controlled by the Kurdistan Regional Government at the time. The UNICEF Survey reports infant and child mortality rates falling from 64 and 80 in 1989 down to 59 and 72 respectively in 1999.
AMac:
You misread my first point . . .
Yes, I did. You call for the authors to provide both the data and a description of their calculations. It seems to me there'd be a better return to effort if they published the Makefiles and scripts than just a description (which is inevitably either imprecise or unbearably turgid), but that's what you specified.
Still this does not much change the case. You're still demanding a burdensome addition to normal requirements. And (to bring in a new point), I don't know what the conventions are in epidemiology, but if they're similar to those in economics, then it's understood that authors own their data, and they're under no obligation to share it with other researchers. If they do, that's praiseworthy, but no one has a right to demand it; for a journal to demand its release as a condition of publication of some individual article would be considered highly unreasonable. I don't much admire this state of affairs, but that's how it is.
On point 3, the mistake in the Summary is significant. Refer back to the Invited Comment I quoted from for an example of the effects of not writing clearly.
It's either a mistake or misleading writing but either way I agree it's significant.
The second-to-last paragraph of the Discussion is about "Iraqis reportedly killed by US forces...28 of 61 killings (46%) attributed to US forces..." Do the authors make clear that this is mainly about Fallujah, or must the reader return to Table 2 to figure that out? Did all readers do so?
The second - to - last paragraph of the discussion is also the second - to - last paragraph of the paper. And yes, it's abundantly clear to anyone who's read the paper through to that point that most of these deaths are in Falluja. If your demand is that this should be clear to someone who's jumped to this paragraph from the initial summary, or someone who's forgotten the discussion of the special characteristics of the Falluja cluster between the left column of page 7 and the right, then those aren't reasonable demands to place upon authors.
On point 4, I count the final four paragraphs of the Discussion as concerning the Coalition. The first two of these are arguable, the final two aren't.
Also inarguable is that you've illicitly changed the subject from "commenting on implications for Coalition military policy" to "concerning the Coalition".
On point 5, see my exchange with dsquared, quoted in comment #2.
Pleased to see you've withdrawn it (I admit, I began writing comment #7 before comment #2 was up).
As for your summary, I accept that's how you see things, but would not present it here as my own view.
No, I'm afraid not. But as you're better than the crowd you're running with in matters of process, so I hope you'll improve in outcomes also.
Two things stand out.
Shortly before the war, Saddam essentially opened up his prisons and let EVERY criminal out.
Do you count the criminal violence inflicted by these people as related to the invasion or not? You can make arguments either way, but it's a political decision, it's not going to be neutral and scientific.
Secondly, if there actually WERE around 100K dead, where are all the bodies in GI's photos, and that of the media? Were they killed and buried in secret? We've had nothing of that scale shown, and Al Jazeera which is hostile to the US surely would have shown it.
For the latter alone, I'd expect to see both lots of funerals shown, and many many claimants for death benefits against the US.
Jim Rockford (#16):
On your second point: Roberts don't claim that 100,000 Iraqis were killed by Coalition forces. They claim that, with 95% confidence, more than 8,000 but fewer than 194,000--best estimate 98,000--Iraqis died in the 17.8 months post-invasion than would have, from all causes, under pre-invasion conditions.
It's a mouthful, but that's what they say.
They don't give a best-estimate or a 95% CI for deaths caused by the Coalition. (See the post's point 4). My cocktail-napkin figure based on their raw data is 26,000 for the best-estimate.
For the mistake you made to be so common--and it is; The Lancet's own invited Commenter claimed something similar--there is something wrong with the writing produced by this Author/Editor team, or with the reading skills of a lot of people, reporters especially.
I don't understand why the pro war lobby is defending itself against this study. It is like holocaust deniers who after a lot of massaging of numbers "proof" that it were not 6 but "only" 3 million death. You just show that an awful number of Iraqi's have died because the US invaded Iraq.
AMac: Re comment #12:
Readers clicking on the first will see that CASI is the Committee Against Sanctions in Iraq, and be able to view the authors' openly-proclaimed stance about how the civilized world should have treated with Saddam's Iraq.
They'll also be able to see that the citation is specifically to a presentation by Richard Garfield, whose competence and objectivity have been acknowledged by both sides of the sanctions debate (and who, in particular, has been prominent in debunking the 5000 excess deaths per month claim mentioned by Mark Childerson). If CASI have the gumption to give him a podium, good luck to them.
Thank you all.
As I stood beneath 'suicide cliff' on Okinawa I saw the bodies of people who died because they feared their treatment by the invader. That was us.
If 100,000 died in Iraq, if 10,000 died, in Iraq, or 1 why did we invade? In Okinawa, I understand. In Iraq, we had other options.
When I recall the little childrens' skeletons, my whole soul revolts.
You're right, Ruth, we did have other options. We could have told the UN to go to hell in '91 and finished the job. We could have continued the Chamberlain imitation of the Clinton years and hoped for the best. We could have clicked our heels three times and wished real hard for a perfect world.
And most of the little children's skeletons were the result of not taking option 1, and taking option 2 for far too long. How many more before you and you kind actually did something?
Robert McDougal (#15):
My %$&* machine crashed as I hit "post," so this is reconstructed, grrr. Also grown too long, so I'll duck out after hitting "post" again, this time with fingers crossed...
On Point 1, in 2005, it is normal to ask authors to submit supporting data. From The Lancet’s Instructions to Authors:
The reason this wasn’t standard practice for the past hundred years is that journals were under immense pressure to keep page counts down, i.e. to publish as many articles as possible in an issue of limited size and weight. Journals, including The Lancet, have recognized that the Web greatly eases the sharing of supplementary material.
Thus, my would-be demand that Roberts give an account by cluster of the number of interviews conducted, the number of individuals described, and the cause-of-death of each mortality enumerated is not burdensome, or unreasonable. If statisticians think that a closer description of Roberts’ procedures would be turgid rather than helpful in evaluating their work, then obviously I withdraw that part of the suggestion.
On Point 3, we agree that the misleading or mistaken sentence in the Summary is significant and should be corrected.
As far as the clarity of the Discussion (still Point 3), you say, "it's abundantly clear to anyone who's read the paper through to that point that most of these deaths are in Falluja. If your demand is that this should be clear to someone who's…" No, my demand is that the writing be clear, period. Paragraph by paragraph, then:
Readers who doubt my typing skills can click on the link at the beginning of the post and read Roberts’ Discussion for themselves.
"Also inarguable is that you've illicitly changed the subject from ‘commenting on implications for Coalition military policy’ to ‘concerning the Coalition’.
Next draft, if there is one, I’ll change the phrasing to "commenting on Coalition military policies and actions for 3 paragraphs, then concluding in the final paragraph with a series of criticisms of Coalition military policies."
You didn’t comment on Point 4, the demand for a statistical treatment of disaggregated data. I hold that this is essential, for the already reasons given.
"Pleased to see you've withdrawn [Point 5]"
dsquared’s corrections would make little numerical difference, I suspect. But on reflection, if the authors showed the best-estimates and 95% CIs for the disaggregated data (or posted the data set to allow others to calculate them), then the tale would tell itself. I would be explicit about this source of uncertainty, but if the statistical information was presented, I wouldn’t demand it of Roberts.
Re. your comment #19 on Richard Garfield, a useful clarification. (I can't open 1 MB files on my dialup connection.)
"But as you're better than the crowd you're running with in matters of process, so I hope you'll improve in outcomes also."
Thank you for the kind words.
VD:
Good point. Parenthetically, there is a shift in the argument, between Daponte / Kaplan who base it on overall mortality estimates for 1985-90, to Heiko Gerhauser who bases it on infant mortality estimates for 1995-99. The estimates you cite don't affirm or deny any Oil for Food effect, since they don't don't include a post-sanctions pre-OfF rate. Garfield (in the presentation mentioned above, p. 45) cites results indicating that "it’s only in 1999 that nutritional improvements are starting to be recognised." This works against Gerhauser's version of the argument but has little bearing on Daponte's version.
In response to Robert McDougall
My citation does not include the post-sanctions, pre-oip data, but it is included in the UNICEF survey to which I linked. According to that survey, infant and child mortality rates rose from 47 and 56 in 1989 up to 79 and 92 respectively in 1994. The oil-for-food program begins in earnest in 1996 and had no apparent effect on the 15 governorates under Saddan Hussein's administration.
On the other hand, infant and child mortality rates rose from 64 and 80 in 1989 up to 71 and 89 respectively in 1994 for the three autonomous governorates of Iraqi Kurdistan. These numbers not only fell from the 1994 levels under the administration of the Kurdistan Regional Government, but they fell below the 1989 levels.
AMac:
On your point 1, if you're now asking only for data, then agreed that's not burdensome in terms of effort. But there still remains the issue of the authors' proprietory interest in the data. It's good that the Lancet is encouraging and assisting authors to publish their data but that's a far different thing from requiring it (also, requiring publication would be a far different thing from requiring submission in confidence).
On your alleged examples of lack of clarity, the only one that might leave a reader briefly uncertain is the first. She would see at once that the IraqBodyCount estimates must be compared with estimates, and that it would make sense that these should be estimates of Coalition - inflicted casualties, and conjecture that the "third and tenth" refer respectively to the without Falluja and with Falluja estimates; but she might reasonably be unsure of the whole chain of inferences until she'd done the back of the envelope calculations to verify the conjecture.
But, no big deal. This is a side issue in the paper; the natural conjectures are in fact correct; the context doesn't call for numerical precision. There would be few quantitative papers, I guess, in which the derivation of every number alluded to in every side discussion was immediately and certainly apparent. Word limits have something to do with this.
Your complaint about "seguing away" from estimates to survey data is unreasonable. That data discussion and estimate discussion are interwoven is natural and harmless; that one sentence in the Summary aside, it's always clear which is which. For example, as discussed above, it's obvious that Discussion para. 12 must be alluding to estimates. Raw data wouldn't make sense there -- the Iraqbodycount totals could not be compared directly to the raw survey data. Likewise, paras 13 and 14 obviously present raw data -- the "three of 61 incidents" and "28 of 61 killings" could not be references to all-Iraq estimates.
This is just Not A Problem.
You didn’t comment on Point 4 . . .
See comment #7.
I don't see what you're getting at with your point 5, less than ever now that you've linked the call for discussion of the implications of subtraction with your call for CIs on disaggregate estimates. The Lancet readership, non-specialist statisticians as they are, presumably don't need to be reminded that when you take small differences between large numbers, the uncertainties in the differences may be large relative to the means. If on the other hand you want validation of the reported CIs, guesstimating a bootstrapped non-parametric confidence interval is not exactly a "tale that tells itself".
VD:
According to that survey, infant and child mortality rates rose from 47 and 56 in 1989 up to 79 and 92 respectively in 1994.
No, from 47 and 56 in 1984-89 to 79 and 92 in 1989-94 (and then to 108 and 131 in 1994-99). 1984-89 is pre-sanctions, 1989-94 is partly pre- and partly post-sanctions but all pre-OfF, 1994-99 is all post-sanctions, partly pre- and partly post-OfF.
The oil-for-food program begins in earnest in 1996 and had no apparent effect on the 15 governorates under Saddan Hussein's administration.
That is, Iraq-wide, the "no apparent effect" is a one-year story. [Garfield's numbers are all-Iraq not just the fifteen Saddam-ruled governorates; appropriately, for the purpose at hand -- if that's still the merits or otherwise of the Roberts study's pre-invasion mortality.]
I would be against releasing the data a) to the general public and b) in Excel. I would be in favour of the data being made available by the reasearchers to anyone who asks for it and looks like they know what to do with it.
The reason for this is that while Excel is a fine program, you cannot calculate meaningful standard errors for a survey of this kind in Excel (or at least, not in any straightforward manner). The confidence intervals don't match up to any common distribution, mainly because of the design effects. As far as I can tell, they need to be bootstrapped, and I have learned the hard way in my life that you can go seriously wrong by trying to roll your own bootstrap program. The JHU team analysed the data with a specialised program called EpiInfo (it sez 'ere) and it doesn't seem at all unreasonable to me to assume that they did so because they needed specialised calculations.
My guess would be that to release the raw data to the public in Excel would result in an absolute proliferation of bad analyses (I'd guess that as many of them would be coming from the anti-invasion side as the pro-; I suspect that there is a parallel universe in which I am defending this study against hordes of online critics who think it outrageous that Fallujah should have been excluded from the extrapolated number). While the authors should stand behind their paper with respect to criticism, it's not reasonable to expect them to spend every minute of the day dealing with half-baked calculations. Note above that there are plenty of people who don't understand why it's perfectly reasonable to assume that you're going to find only 1 death by violence in a 14 month period in a sample of 7800 people; this would correspond to 12.8 per 100K people, about double the USA's murder rate.
Likewise, I don't think it's reasonable to say that the team should "[describe] the relevant intermediate calculations that led to the best-estimate of excess deaths and its 95% confidence interval". This would be akin to requiring that papers in astronomy journals include instructions on building a radiotelescope. The study says that the team used EpiInfo, a generally available computer program and one which will have been through its own peer review. It is really quite likely that the authors wouldn't be able to give you the exact details of how EpiInfo carries out the calculations, although they probably would be able to if they had a month and a stack of textbooks. Knowing how to use your tools is part of applied econometrics, and I personally am always much more suspicious of people who appear to be rolling their own.
As far as I can tell, every time the Lancet team been asked for data they've provided it, and that surely ought to be enough.
Erratum: in the interests of not wanting to portray myself as knowing more than I do, I'm not sure that it is because of the design effects that the CIs need to be bootstrapped.
#20 from Ruth on March 30, 2005 03:48 AM
As I stood beneath 'suicide cliff' on Okinawa I saw the bodies of people who died because they feared their treatment by the invader.
Ruth:
Source, please? I was under the impression that they died because their cultural programming told them that surrender was untenable and that the white devils were monsters. Fears can feed fears, it's true.
When I recall the little childrens' skeletons, my whole soul revolts.
Did the little children of Okinawa die by their own actions, or did their parents dash them against the rocks? If the latter, who was the proximate cause of their deaths--Franklin Delano Roosevelt?
But your main point is not about Okinawa. Okay. When I look at the circumstances surrounding Saddam's Iraq:
- I see pictures of mass graves with babies in them whose deaths apparently significantly predate the Coalition invasion.
- I hear and see evidence of peaceable people-- some raped; some with limbs and tongues amputated, with Xs branded on their foreheads; some with their heads or extremities crushed in vises while they suffered and died.
- I don't hear from anyone that the above things were done by Coalition forces. At the risk of sounding silly: I don't hear that these things made the people subjected to them happy--no cultural programming seems to have made that the case. This is another difference from Okinawa, it seems to me.
My soul's revulsion is for every innocent who suffers.
- The calculus of optimal suffering escapes me; every murder is, as I think Raymond Chandler said, "an act of infinite cruelty."
- I don't know that some of the above suffering is worse that "a clean death", and I don't know that it isn't; and I don't know how to know for sure.
- I don't think that deaths in war are somehow sanctified as non-murders. I think that innocence is hard to quantify, and that "every man's death diminishes me"--but not equally.
Your opinion probably differs in its basis, somehow. You appear to be sure of something that I am not sure of. I wish I knew how you arrived at that conclusion.
Sincerely,
Nortius Maximus
Robert McDougall (#25):
On lack of clarity/misleading statements in the paper: I have now typed up the most troublesome paragraphs and inserted my own thoughts, explaining to the reader why I would find this writing unacceptable, if I were a referee. I pointed out the the gross misinterpretations of the paper started in the pages of the Lancet itself; I call that strong evidence that a big problem exists.
You disagree. We'll have to leave it to the readers to weigh your take against my four indented paragraphs in comment #22.
Thanks for highlighting your response to my point 4 in comment #7. I missed it.
Re. my point 5: As I reflected in comment #22, this is really subsidiary to point 4: Roberts' failure to perform 95% CIs on the data disaggregated by cause-of-death. If they did that, it wouldn't matter.
dsquared (#27):
You make a good point about data, but I don't think it's persuasive. The point of Science is to allow qualified people to reproduce your work; worries that nuts will waste your time have to be secondary.
As far Excel, I meant to refer to "a format for tabular data readily imported by numerical-analysis programs," nothing more.
As far as data in the paper, Table 2 doesn't contain all the relevant numbers needed to get an oveall view of the findings. In particular, this means numbers disaggregated so that the data with/without the excluded outlier can be seen, and there is accounting by cause-of-death. Violence-by-coalition isn't listed separately, a major omission given the interpretations offered by the authors (final 4 paragraphs of Discussion) and, predictably, by others.
As noted earlier, if statisticians wouldn't find further descriptions of Roberts' processes helpful, then I withdraw that suggestion.
If you know of statisticians who have analyzed the Roberts data, disaggregated by cause-of-death, I would be grateful for a link, thanks.
If I ever figure out how to modify material without crashing the site, I will update the post to reflect my responses to these criticisms.
AMac:
On lack of clarity/misleading statements in the paper: . . . I pointed out the the gross misinterpretations of the paper started in the pages of the Lancet itself; I call that strong evidence that a big problem exists.
OK. Above, and earlier, you've pointed to Horton's and Rubeyi's comments. First up, you have one clear example of misinterpretation from Rubeyi:
Also one likely misinterpretation by Horton:
Rubayi's clearly, and Horton's likely, making a claim about the population that they're only entitled to make about the sample. In each case, this derives from the admittedly bad second half of the second sentence in Roberts et al.'s Interpretation paragraph.
From Horton you also quote:
This may raise some doubt that Horton's saying things about the whole population that he's only entitled to say about the sample. But with a more complete quotation from Horton, it becomes clear that he's talking about the sample:
This is not "gross misinterpretation" but perfectly accurate.
Finally, from Horton you also quote:
You may or may not consider this a good policy recommendation, but in any case a policy recommendation is what it is, not an interpretation of the paper.
Altogether then, your valid instances of "gross misinterpretations" "in the pages of the Lancet itself" all come down to the same single sentence.
As far as I'm aware, the only other misinterpretation you've complained of is the tendency for outside commentators to misinterpret the study's findings about excess deaths as findings about killings by Coalition forces. I don't see anything in the paper itself that supports that misinterpretation; it's the kind of distorting simplification that's inevitable with second- and third - hand reports.
So again, your claims about "gross misinterpretations" and "strong evidence" for a "big problem" of "unacceptable writing" all come down to one half of one sentence in the paper. Outside that half sentence, you've shown no evidence of any problem.
Robert McDougall (#31):
Thanks once again for thoughtful and carefully-documented comments.
Rather than repeat myself, I'll say that you've done a good job of explaining a stance that I don't agree with.
My comment #22 already detailed problems in the Roberts discussion. If, in a statistics-based survey paper, the authors extensively discuss subsets of the data, they should present the appropriate statistical treatment of those subsets, or explain why they have not done so.
The Lancet's editor pushed the margin in his description of the Roberts study--one that is promoted, recall, as being particularly powerful by virture of its statistical approach. The invited commenter crossed any such line.
Does the Roberts study justify the bitter critiques offered up by these authors to the US and UK? Interested readers will have to read the contested extracts in their original contexts to decide. Richard Horton's editorial and Bushra Ibrahim Al-Rubeyi's comment are both online (free registration may be required).
On the point that #1 is an excessive burden.
1) In the experimental hard sciences I was required (and do require) that the raw data be written up as a publish_able_ Appendix. Not the fancy grammatical constructs - just an itemization of the necessary information. There's always a joker at the poster session or someone who hears the speech that feels the fundamental analysis has issues of one sort or another. Yes, this requires crazy outliers with associated crazy comments "Local tornado killed power for 4.7 hours on this and so day." But when the raw data is presented this way along with the actual paper, you actually get informed, competent peer review. (As opposed to comments like 'Everyone knows this phenomena is linear, WTF are you trying to publish a curve?')
2) This is one area where Journals and the newsmedia have an edge over blogs. If they push for substantially stricter sourcing, footnoting, and openness, then they'll be rewarded. Or not.
AMac:
If, in a statistics-based survey paper, the authors extensively discuss subsets of the data, they should present the appropriate statistical treatment of those subsets, or explain why they have not done so.
No. Data description is good, and there's no obligation to match each descriptive remark on a data statistic with an estimate of a corresponding population parameter. It's only when inferences are drawn that statistical inference machinery must be employed.
A common situation arose with the Lancet study: the authors tested for one thing, and found that thing but also signs of something else besides. They designed their study to detect changes in the overall mortality rate ("Methods", para. 1). They found a statistically significant change, but along the way they also noted in the raw reports a striking number of deaths from air strikes. They could then have done one of three things: (A) extended the study to produce point estimates and confidence intervals for the all-Iraq air strike death rate, (B) noted the raw data and urged further research and, in the interim, action, ( C ) noted the raw data and urged further research. Wisely, I think, they chose (B); unfortunately they then stumbled, in the writing of the notorious half sentence.
This isn't just a paper about sexual dimorphism in octopus or the geology of the moons of Saturn, its subject matter is human death and human suffering. That doesn't relieve the authors of any scientific obligations but it imposes some non-scientific ones. To put those in perspective, the point estimate for the air strike death rate, excluding Falluja, would come to about ten per cent of the total pre-invasion death rate.
If that rate had applied to people you cared about the way you care about your own fellow citizens, would you be satisfied with a simple recommendation for "further study"? With an implicit message that until and unless such further study was performed, nothing need be done? If not, then perhaps you'll agree that option ( C ) should be ruled out. (A) is akin to your own point 4. Between (A) and (B) is a hard choice; there are reasonable arguments either way, though I suspect the authors were wise to keep the focus of their statistical inference on the overall mortality rate. At any rate, having chosen (B) they then had to balance between underplaying and overplaying the air strike death reports. A reasonable strategy, with one significant defect in execution.
The postwar mortality rate of 7.9 is below the World mortality rate of 8.81. Based on the Lancet survey, I would be better off in Iraq then 115 other countries. This defies common sense.
A better explanation is that the "Oil for Food" program was a motivation for both Saddam and the Iraqi people to under report death. More people = more rations = more money for Saddam to buy guns
Mortality rate rankings.
Mulling it over, I am going to try and perform a crude arithmetical calculation of expected mortality for under-59 individuals in pre-war Iraq, and compare it to the numbers Roberts gives in Table 2. There are some fairly complete numbers here for Iraq and neighboring countries.
This will be crude for a number of reasons, and may not work. If it does, I don't know what the results will be. Even then, it will be hard to "show my work" in this thread (I don't do html tables). Well, we'll see. G'night.
1) that "widespread Iraqi casualties" in graf 13 is immediately followed by "household interview data" suggests quite plainly that the raw data (cum-Fallujah) is being discussed,
2) the word "reportedly" in grafs 14 and 15 indicates that raw data (aka, the data that was reported by the survey participants) is being discussed.
Otherwise, I echo Robert McDougall's comments.
Soldier's Dad:
The postwar mortality rate of 7.9 is below the World mortality rate of 8.81. Based on the Lancet survey, I would be better off in Iraq then 115 other countries. This defies common sense.
This is a common fallacy which has tripped up quite a few commentators on the survey. You have to remember that Iraq is a very young country; it has a lot more of its population in younger age brackets than mosy countries outside Africa. This is what drives the crude mortality rate; even somewhere as violent as Iraq, the most common reason people die is old age.
Amac: care advised. The "2002" numbers in that WHO table are extrapolated numbers; they're most likely based on the same fieldwork as the UNICEF infant mortality numbers, carried out in 1998.
dsquared (#38):
Thanks for the caution. While I've scanned many comments on the subject, I frankly haven't followed the controversies over which infant mortality figures and other mortality figures are the best representations for Iraq during the study period, or how well they stand up to critical evaluation. Any links from you, or others, would be appreciated.
The calculations did work, but aren't a slam dunk in any way. I will try and say something on the subject today or, realistically, probably tomorrow.
From the head post:
hack science in the service of a political cause and candidate?
From AMac's post #32:
the bitter critiques offered up by these authors to the US and UK
Examples of being led astray by U.S. narcissism.
As Roberts acknowledged, the authors aimed to have the paper published during the U.S. presidential election campaign; various U.S. oriented commentators assume the purpose must have been to defeat President Bush. Roberts:
That AMac should find the Lancet commentators' remarks characterized by "bitter critiques" of the U.S. and U.K. says more about his concerns than theirs. Taking the three comentators together, the major concerns in evidence are -- not surprisingly, for a bunch of researchers and medical professionals -- for better knowledge and the physical wellbeing of Iraqis. Bird's comment consists mostly of a constructive suggestions for better military - civilian cooperation in information gathering; Horton's careful to avoid rancour in such criticisms as he does offer; Rubayi's does indeed betray some bitterness, arising out of frustration with apparent Western indifference to Iraqi well being, and leading to a call for constructive action:
Even for AMac, more so for the more extreme critics, this is all about the U.S. They have some difficulty understanding that for other participants, it might be primarily about Iraqis.
Robert McDougall (#40):
As I perhaps have indicated, I'm not commenting here on the subjects you raise. Not because I don't have thoughts on these matters, but because:
I appreciate what I take to be your efforts to hold your anger in check.
AMac:
Anger? No, two motivations:
(1) Methodology. People tend to underestimate how much other peoples' preoccupations differ from theirs, or to make sufficient allowance for it in interpreting other peoples' utterances. People commenting on blogs like this tend to be especially interested in politics. People doing epidemiological research tend to be especially interested in public health. The blog commentators tend to overestimate the extent to which the researchers are motived by political concerns.
Roberts' remark quoted in comment #40 is an example of the same tendency working in the opposite direction. In thinking that the study would "[force] both candidates . . . to pledge to protect civilian lives in Iraq", he overestimated the extent to which the political players and the electorate at large shared his and his colleagues public health concerns.
(2) Perspective. It struck me that there was some indecency in the commentators' (myself included) focusing so much on whether every care had been taken before reaching conclusions that might be politically uncomfortable to Westerners, and so little on whether every care had been taken to preserve Iraqi life and safety; that the debate was being driven far more by concerns that no American feelings should be needlessly hurt, than that no Iraqis should be needlessly dead. Of course everyone has a right to focus on his own concerns, but it's healthy to remember the broader perspective.
Robert McDougall (#40, #42):
I don't know which, if any, of your comments you regard as indecent because they focus on whether every care had been taken before reaching conclusions that might be politically uncomfortable to Westerners (&c). Since that has not been a factor that I've considered in my writing on the Lancet paper, I can't apologize on that score.
Perhaps your basis for assigning such motives to your correspondents is less well-founded than you imagine.
On that we can agree. I will once again point readers towards Norm Geras' writing . One need not be persuaded that it represents the wisest counsel on the subject to acknowledge that a moral case in favor of the Coalition's actions exists.
Declamations on the narcissism and narrow-mindedness of those who disagree with your politics may, indeed, enlighten readers. But not necessarily in the way that you would wish.
Geras' five-part series on Iraq, mysteriously absented from #43.
AMac:
it's healthy to remember the broader perspective.
On that we can agree. I will once again point readers towards Norm Geras' writing . One need not be persuaded that it represents the wisest counsel on the subject to acknowledge that a moral case in favor of the Coalition's actions exists.
Well, no, I don't call that "broader perspective", I call it another form of self-absorption.
I don't object to people stoushing about the pros and cons of the invasion -- I've enjoyed a round or three myself -- and of course in any such stoush antiwar people are free to use the higher postwar death toll as an argument against the invasion, and prowar to argue that it was still worthwhile overall. That said, using the study as a jumping off point for Yet Another Round of invasion polemics is the opposite of what I'm urging. You might note;
Rather than discuss whether the United States did right or wrong in March 2003, they discuss what is to be done now for the physical security of Iraqis. In particular, they urge (and, FWIW, I second) more diligence in gathering information on public health and safety, and, in the interim, more caution toward collateral damage in air strikes. I'd add, without any reference to epidemiological research whatsoever, no brainer moves like equipping checkpoint teams with Arab language signage.
As if, if the population in question happened to contain our brothers and sisters and children, we might not be spending all our time discussing whether the point estimate might be too high and whether the authors were always quite clear in distinguishing population parameters from survey statistics and whether the they might have some agenda relating to the politics of the occupying country.
Robert McDougall:
Perhaps you have withdrawn the harsh and unsubstantiated accusations you made towards some commentators in #42. Since much 'dialogue' in this thread (all, #40 on) has been between you and me, it's reasonable for me to take your charges personally.
Or, maybe you have moved to a new line of argument (#45), but will return to the old as circumstances warrant.
It doesn't matter.
To the extent you have succeeded in defining the subject matter that's up for discussion here, this thread turns into yet another wide-ranging cacophony, people arguing heatedly, past one another. There are already scores of such threads at dozens of sites. As you know.
You claim that the only moral stance is to accept the Roberts study's findings and interpretations--as you see it--and move on. I disagree. That said, I concur with some of your suggestions on policy. But to repeat myself from comment #41:
AMac:
Perhaps you have withdrawn . . . Or, maybe you have moved to a new line of argument (#45)
I don't know why you suppose that. Comments #45, #42, and #40 all take the same basic position.
harsh and unsubstantiated accusations . . . in #42. . . it's reasonable for me to take your charges personally.
If you'd made that complaint about comment #40, I should concede that you had just cause. In particular, it would be reasonable for you to take "narcissism" personally, and applied to you personally it would be "harsh and unsubstantiated". That I withdraw, and substitute "self-regard".
Concerning claims in the three comments, three might usefully be distinguished:
Concerning (1), that is a "charge" if you like, but neither harsh (since the failing is common) nor unsubstantiated (since comment #40 substantiates it).
Concerning (2), I don't consider it a "charge" at all. Certainly if a high profile study alleged harm done by Australian troops, I should feel concern that they not be unjustly accused of wrongdoing, even if there was no such accusation in the study itself. Since such concerns are right and natural, it hardly seems necessary to show that they are in fact prevalent. But if you want instances, I would point you, for starters, to Shannon Love's comments #30 and #41 to the earlier post (of course, they might also be used to illustrate why "reasonable concern" does not imply "reasonable reaction"). As for yourself, when you object to the authors' discussing the raw data on Coalition killings without any corresponding statistical inference, I take that to arise largely from a concern that "weighty moral matters" should not be raised against the U.S. military without due cause, and of course that's in no way discreditable.
But this gets us to point (3). This is not a charge against any individual, since no individual is tasked with keeping the blogosphere in balance. But I stand by
as a harsh but accurate description of the resultant collective behaviour; as how a disinterested (non-Iraqi non-Western) observer would see the overall debate. I don't expect you to agree, of course (and of course you're right that its accuracy would not make a profitable topic for debate), but I emphatically don't regret or retract it.
Robert McDougall,
You have put forward a series of theories:
Most of these beliefs are based on the fantastic presumption that you can read the hearts and minds of strangers who disagree with you.
Now allow me my turn at mind reading. We are having this exchange because you deeply misunderstand the process that is "science."
Roberts did a certain piece of work, and wrote it up a certain way, and presented it in the Lancet, a journal of medical science. By doing so, they opened themselves to scientific criticism. This is what I focused on in this thread. Worse than being wrong, your speculations are irrelevant. By all means, note that you think I'm biased, and keep that in mind as you evaluate my arguments on their merits. You started off in this spirit, then ran off the tracks.
While it's far from certain, good policy may follow from good science. Good policy will not result from assessments of fact that are performed or analyzed incorrectly, or that are misinterpreted by people who learn about them.
That's a simple motive for investigating and writing about the extent to which Roberts' conclusions are justified by their work.
Questions for you, Robert McDougall:
Answer, or not, at leisure, as I expect this will be my final contribution to the thread.
AMac:
I wonder why no one here is asking the jihadis to be more careful in their choice of targets.
I suppose it is not statistically justified.
I wonder why no one here is asking the jihadis to be more careful in their choice of targets.
I suppose it is not statistically justified.
M. Simon:
I wonder why no one here is asking the jihadis to be more careful in their choice of targets.
If you have any means of communication with them, and you're in any constituency of theirs, I would encourage you to ask them forthwith.