The invited comment (free reg. req'd) of BI Al-Rubeyi (Lancet v. 364, # 9448, p. 1834, 20 Nov 2004) that accompanied the paper summarized its findings as showing

that the death toll from the invasion and occupation of Iraq is about 98000 civilians, and it might be considerably higher. The deaths are mostly related to air strikes.

The study's defenders have allowed that this interpretation isn't justified. But what, exactly, is the take-away message of Mortality before and after the 2003 invasion of Iraq (henceforth, "Roberts")? This is one instance of an increasingly common problem: as none of us can be experts in a multitude of fields, how do we evaluate the assertions that credentialed experts make? Surely it affects the moral calculus of those of us who supported this war if Coalition actions have, directly and indirectly, resulted in the deaths of 100,000 Iraqi civilians who would, otherwise, have lived.

As I've said before at WoC, I have no experience with the sophisticated programs used in modern surveying techniques. However, I am familiar with the application of basic statistical procedures. In this post, I will try to show how I came to the conclusion that some of Roberts' conclusions aren’t supported by the evidence they presented. Others — importantly, the finding that tens of thousands of Iraqis died due to violence - are.

This entry was drafted prior to the May 12 release of the Iraq Living Conditions Survey 2004 (ILCS), available here. This Iraqi government/UN household survey is relevant to the evaluation of Roberts' paper. Since reading the summaries, I've inserted some comments, and removed one section for re-write. See this Volokh Conspiracy post by Jim Lindgren for a balanced view of the new information.

A Discussion of the Roberts Mortality Data

Roberts studied excess deaths. In arithmetical terms (not in moral terms!), this is similar to how we might look at a company's finances. It has revenues, and it has expenses. In the simplest case, the profits ("excess revenues") for a given time period are the expenses subtracted from the revenues. If annual revenues and expenses are known exactly, estimates of annual profitability can be quite precise.

On the other hand, our knowledge might be very hazy. Perhaps we are 95% sure that a company had revenues between $10,000 and $30,000, with a best-guess of $20,000. Likewise, perhaps we are 95% sure that expenses were between $5,000 and $25,000, with a best-guess of $15,000. Our back-‘o-the-envelope estimate would be that this company had profits of $5,000. But we would know that the company might have done much better than that--or much worse. This would be especially true if expenses were not connected to revenues (i.e., if they were independent variables), which can indeed happen.

The Roberts survey estimated rates of death in Iraq before the invasion, and then during and after it. Their excess-death figure of 98,000 is extrapolated from (during and after) minus (before)-—excluding the extremely violent Fallujah findings from the analysis. How confident are they in presenting this figure? Not very; they are 95% sure that the actual total lies between 8,000 and 194,000.

The mortality data that Roberts display in their Table 2 have a quality worth exploring. Most of the categories of death are fairly low-frequency events. For example, of the surveyed 2,726 children under 15 in the 14.6 months prior to the invasion, 1 died of accidental causes. A repeat survey of another group of 2,726 Iraqi children might well have turned up 0 accidental deaths, or 2, or 3. The poll of 3,084 children in the 17.8 months following the invasion identified 2 deaths-—an approximate doubling of the accidental-death rate in this group. However, common sense tells us that subtracting one small, uncertain number from another small, uncertain number will yield a very imprecise estimate.

However, one category stands out: Violence. Violent deaths were rare in the pre-invasion period, and common post-invasion. On a common-sense basis, we expect that this distressing circumstance--subtracting a small, uncertain number from a much bigger number--will result in a more precise estimate.

These intuitive assumptions turn out to be correct.

Modeling Roberts' survey to better understand their conclusions

While it’s beyond my abilities to follow Roberts’ analytical methods, I could explore a simpler question (following Roberts, I excluded Fallujah from the statistical analysis).

Assume, for the sake of argument, that Roberts’ survey was exactly correct in its estimates of Iraqi mortality. Under these circumstances, if Roberts were to apply their methods in repeated surveys of Iraq, what would the results of this set of surveys be?

This is an inversion of the real-world circumstances, where the surveyors uses samples to provide estimates of unknown actual mortality rates. But it is simpler to analyze, and provides insights into Roberts’ findings and claims.

I used the random-number generator function in the Excel spreadsheet program to conduct ten of these pseudo-surveys. Here is a thumbnail description of the methods:

• I subtracted the deaths of the Fallujah cluster from Roberts' Table 2 to obtain the mortality data set for the 32 remaining clusters. This is the data set that Roberts used for their statistical analysis. One Pre-invasion and one Post-invasion death couldn't be unambiguously assigned, but that doesn't markedly affect the overall conclusions.

• I "aggregated" causes of death into three meaningful categories: Accident-related deaths, Disease-related deaths, and Violent deaths. For each, Roberts' four gender and age groupings (Children <15, Men 15-59, Women 15-59, and Elderly >=60) were kept. This yielded 24 numbers, 12 (3 × 4) pre-invasion, and 12 for the post-invasion period. That data is here:

Roberts' Table 2, Without Fallujah Deaths

• The number-of-deaths for each "cell" was converted into a mortality rate, expressed as deaths per 1000 people per year.

• I compared the results of this arithmetic with Roberts' central estimate of 98,000 excess deaths. By multiplying the Excess death rate (the Post-invasion rate less the Pre-invasion rate) by a factor to account for the sample size compared to Iraq's population, this process yielded an estimate of 106,000 excess deaths—-8% too high. That's acceptably close for these purposes.

• By the terms of this exercise, the derived rates for each of the 24 cells were taken as exactly correct for Iraq (excepting Fallujah). Therefore, a series of random-number generations followed by "IF-THEN" calls could be repeatedly used to simulate the results of the re-application of Roberts’ survey methods to additional samples of the Iraqi population. The point of the simulation was to estimate the extent of survey-to-survey variation that is due to chance alone.

• Each pseudo-survey comprised 24 records of simulated deaths, 12 Pre-invasion matched with 12 Post-invasion. I ran the process ten times, then calculated means and standard deviations for the set of 10 pseudosurveys. The "standard deviation" is the "68% confidence interval," exactly half of the 95% CI for normally distributed data.

• The mean of the total number of excess deaths for these ten pseudosurveys was 107,000. Since Roberts' best-estimate was 98,000, I adjusted all pseudosurvey data downward by 10%. This adjustment was not necessary, but helps in graphical comparisons of the data.

Below, Roberts' findings are compared to the average of the pseudosurveys. Each of the ten pseudosurveys is then presented separately. All error bars are standard deviations (68% CI), one-half as big as the 95% CI interval that Roberts reported for their Total Excess Deaths estimate.

Excess Deaths, Actual Survey and Ten Pseudosurveys

The 68% CI of Roberts' data closely matches the 68% CI of the pseudosurvey data. This is a consequence of setting up the pseudosurveys to mirror Roberts' results. The match supports the idea that the variability of the simulated data is representative of the uncertainty with which Roberts' published data represents the actual (known-only-to-God) mortality rates in Iraq.

These pseudosurveys highlight the unreliability of the estimates of excess deaths due to Accident or Disease. In two trials, there were no excess deaths due to accidental causes; in a third, there were 80,000 recorded. Two other trials recorded over 40,000 lives "saved" by decreases in disease rates. This is not surprising--it corresponds to the problem of deducing the profits of the hypothetical company with poorly-known revenues and poorly-known expenses.

The pseudosurveys also emphasize that, to the extent Roberts' survey was properly carried out, the severe upsurge in violence they reported would be replicated by repeated administrations of the same survey methods. If Roberts had chosen to present the disaggregated results of their actual survey, they would have claimed that about 59,000 more Iraqis (excluding Fallujah) died from violence in the 17.8 months Post-invasion, than would have died if Pre-invasion conditions had continued. If the 95% CI is taken to be twice the standard deviation, it would be about 33,000 to 85,000.

Please email me at AMac-at-windsofchange.net if you would like to examine the Excel workbook that sets up the pseudosurveys, and contains the data described here.

Blogger Tim Lambert, one of Roberts' most ardent and able defenders, arrives at conclusions different from mine. He wrote "[The estimate for war-related deaths] from the Lancet study is 33,000 (the rest of the excess deaths are from increase in disease, accidents, and murders)." Presumably, he has access to data that goes well beyond the information presented in the paper itself, where "violence" was not disaggregated.

As Roberts has done in other places, he put together a team that estimated the immediate death toll of war. This is information that has moral importance, and could have practical significance as well. But the paper has met with as much skepticism as acceptance. There are some sound reasons for questioning some of Roberts' methods, and for reserving judgement on the validity of their claims.

• The survey was done in September, and written and reviewed so quickly that it could be online-published in late October, in time to influence the US Presidential election. This schedule was apparently a condition of publication.

• Roberts wrote their paper in ways that made it likely that most readers would misinterpret key portions of their findings, to the discredit of the US military and its allies. While this point is hotly contested by Roberts' partisans, I think it has held up. See this earlier post" and its comments and links for the arguments, pro and con.

• To be useful from a policy point of view, Roberts’ findings needed to be disaggregated. The pseudosurvey experiment strongly suggests that the ~16,000 excess deaths due to disease that Roberts implicitly folded into their "98,000" total are a statistical fluke, and that the ~23,000 excess deaths due to accidents are a highly inaccurate guess.

• Since the survey results do appear to show a "solid" surge in violence (about 59,000 extra deaths), why didn't the paper state as much? I don't claim mind-reading powers. (Roberts gives his account of the paper's creation in this Chronicle of Higher Education interview. ) But one can derive a sense of the attractiveness of the "Americans Killed 100,000" idea to the Anti-War Left by seeing the results of this Google search: 100,000 dead Iraq Lancet. It is exemplified by acclaimed Mideast expert and blogger Professor Juan Cole in his late-October 2004 post US Has Killed 100,000 in Iraq: The Lancet:

The Lancet, a respected British medical journal, reports that the US and coalition forces (but mainly the US Air Force) has killed 100,000 Iraqi civilians since the fall of Saddam on April 9, 2003.

Sources of Violent Deaths observed by Roberts

Throughout the preceding analysis, I've made the explicit assumption that the Roberts survey correctly reports the circumstances of the surveyed clusters. There is no way to know that this is so, any more than with any other survey, but it is a reasonable starting assumption.

On reflection, I think there are reasons to be skeptical of the causes of violent deaths, as the Roberts paper summarizes the team's surveyors' records of the families' answers.

• In the case of military action, many survivors will simply not know who was directly responsible for the death of a family member. Is a US rocket readily distinguished from Iraqi artillery, or jihadi mortar fire?

• There are good reasons why an Iraqi family would be inclined to blame Americans for deaths caused by others--especially if the guilty parties are nearby, and ruthless. Along the same lines, and as noted in the Roberts paper, Coalition forces offered compensation for some wrongful deaths. Ba'athists and jihadis do not.

• The temptation to blame "The Other" for misfortune exists for all people. Iraqis are probably no more or less prone to this phenomenon than are other groups.

In his recent interview with the UK's Socialist Worker newspaper, Roberts made a series of statements that led me to wonder about his team's biases, as well. Among his comments:

"Most of the [post-invasion] deaths were violent and most of those deaths were caused by the coalition forces."

"Americans are so hated that I couldn’t go around talking to people."

"Most of them [workers from non-governmental organizations, my colleagues and my driver] hate the Americans, most want the coalition troops gone."

"... I think that in my country, in particular among the leaders of my country, there is a grossly inadequate understanding of what a horrible thing war is, and all the misery and suffering that goes with it. My country went to war much too flippantly. Our data strongly supports that. I went to Iraq hoping I’d find fewer deaths. It certainly never occurred to me that I’d find more deaths caused by coalition forces than by non-coalition forces."

"I’m convinced that the war has been a dismal failure. People in my country might not know that for years to come. But we’ve sown the seeds of hatred to an enormous extent."

In contrast to the amiable persona Roberts projected to his sympathetic Chronicle interviewer, Roberts comes across here as committed to exposing the American government's moral culpability in invading Iraq. More than that, Roberts' contention that Americans are passionately hated by the Iraqis he met and worked with ought to raise a red flag. It was those same Iraqis, acting as interviewers and team managers, who recorded and conveyed the surveyed families' impressions of the identities of those who killed their close relatives.

In other circumstances--say, a Coke-sponsored survey on cola preferences, run by Coca-Cola interns, with the surveyers' ranks filled by the Coke Fan Club--issues related to unintentional, even unconscious, biases would not be ignored. The Roberts authors have not, to my knowledge, addressed the question of how their team's clear biases were kept from influencing the assignments of responsibility for violent deaths due to war, or due to unknown causes.

In one sense, it hardly matters if an excess death in postwar Iraq is due to a helicopter gunship, an armed robbery, or lousy medical care--the person is still dead, family and friends still grieve, the community still suffers. In a policy sense, however, the distinction is essential. One aspect of this point is made in the Commentary by Sheila Bird (free reg. req'd) that accompanied Roberts' paper (Lancet v. 364, # 9448, p. 1831, 20 Nov 2004). Bird argued that military/public-health cooperation was essential to accurately determine the number of collateral deaths due to air strikes.

Sadly, the manner in which Roberts presented their findings make the collaborations that Bird advocates less likely in the future. U.S. commanders might suspect that actors like the Roberts team would pitch their findings so that they would put the military in the least-favorable light possible. And they would have reason for these apprehensions.

In the course of the discussions of this paper, one Roberts partisan declared that the proximate cause of an Iraqi's death is immaterial to the question of moral culpability: absent the Coalition invasion, there would be no excess deaths. This is one of a number of moral stances that one could take. But arguments to buttress this contention shouldn't rely on presenting information so that it is less clear to the intended audience. Instead, the scientific ideal has been to use the methods of science and scientific communication to, before all else, inform. Starting from a common description of reality, different people can then take away different interpretations. In such a case, a survey's results would properly enter the public-policy arena, and affect individuals' views on the invasion and its aftermath.

Physicist Leo Szilard is quoted as saying "A scientist's aim in a discussion with his colleagues is not to persuade, but to clarify." With the American Presidential election looming, it does not seem to me that Roberts' team--or The Lancet--were content to clarify. This goal was slighted, in favor of persuasion.

Roberts and his colleagues conducted their study at considerable risk to themselves, and they provided information where none existed before. These are commendable. So it's particularly regrettable that they ended up joining the trend towards practicing science as an extension of politics. This isn't new, and it's hardly confined to the Left. That doesn't make this joint effort by the Roberts team and the Lancet's editors any more welcome.