More than 600,000 killed in Iraq?

So says a recent Johns Hopkins study published in the Lancet. Is it true? That's a huge number: somewhat more, as a share of population than the American Civil War, which took longer, and was fought with mass conscripted armies and 19th c. medicine and sanitation.

It's important to point out that this isn't an estimate of civilian dead. The study makes clear that there was no effort to exclude combatants from the death toll, and the study indicates that military age males account for a majority of violent deaths. The New York Times incorrectly described it as an estimate of civilian deaths, and I have not yet seen them publish a correction.

The Lancet's editor does seem to be to be a strong advocate for a particular political point of view with an interest in seeing the invasion of Iraq proved to have been a Very Bad Thing. That may have affected the timing of the study's release, and how vigorously it was reviewed before publication. Ultimately, however, the conclusions will rise and fall on the quality of the study itself.

The vast discrepancy between the estimate and the official figures raises concern, but not necessarily a fatal one. It is possible that local tallies of violent deaths at hospitals and mortuaries are either not making it in to the central government or being suppressed. That they are not being recorded at all seems an insufficient explanation: about 90% of the households in the survey that reported a death were able to provide a death certificate when asked.

More troubling is the discrepancy between the mid 2004 UNDP survey and this one. It used similar cluster sampling methodology to the Johns Hopkins study, but estimated 25,000 killed in the year following the invasion, compared to about 90,000 estimated by the Johns Hopkins study in the same period. It would appear that the UNDP figure was not intended to capture ordinary criminal murders, but that's hardly enough to explain the discrepancy: Insurgency, counter insurgency and sectarian violence seems to account for the lion's share of violent deaths.

Why might the John Hopkins study be wrong? To start with, it relies on dividing the country into equal size clusters, and sampling a group of households in each cluster. Accuracy depends on reasonably accurate estimates of population by region. One challenge is that accurate population estimates are hard to find in Iraq. The last nationwide census was 1993. Further, there's reason to question the accuracy of the Saddam era censuses: the ruling party had strong motives for inflating the number of Sunnis at the expense of other groups. The problem is compounded by the study's use of a two year old estimate.

If the population is overestimated in the more dangerous parts of Iraq, and underestimated in the less, then the more dangerous parts will be overrepresented in the sample. There's good reason to think that this was the case. The Sunni triangle would have benefited from any Saddam era selective distortion, and is generally considered to be one of the more violent parts of Iraq, and indeed this reflected is in the Johns Hopkins study. Further, you would expect the more violent regions to lose population share relative to the rest of Iraq since 2004, both because of flight and higher death rates.

Another problem might be nonrandom sampling of households. As I understand the protocol, the first household in the cluster was randomly chosen, and then the team would go to the nearest neighboring household, and so on. But in a lot of communities, a household will have two neighbors that are equally close. Ideally, the team should flip a coin, but that might not be what happens. "Asking about people killed in the fighting, are you? Then you should visit Widow Tikriti. She's right next door".

How can one confirm or refute the study? I have two suggestions. The first would involve asking local hospitals and mortuaries about the ratio of violent to nonviolent deaths. The other would be looking at demographics. If the study is correct, the violent death rate among adolescent and adult males has been very high since the invasion: 5% among 15-29 year olds, 9.5% among 30-44 year olds, and 7% among 45-59 year olds. Female violent death in the same cohort has been a very small fraction of that. The impact should be very visible in the male-female ratios in the relevant age groups. In doing such a study, one should keep in mind that sex ratios in the 44-59 age groups were already unbalanced by about that amount because of the deaths from Saddam's wars and massacres.

Update: More commentary at Asymmetrical Information, with more elsewhere on the site. And more here and here.