How Bad Is It in Iraq? (II)

ORB, a British polling group, has just released the results of a poll it took in Iraq in August that suggests that over a million Iraqis have died violently since the US invasion. However, it’s important to understand the level of uncertainty involved in the estimate. It is very, very hard to quantify what’s happening in a place as chaotic as Iraq.

One measure of the level of uncertainty is that the weighted estimate of violent death is about 50% higher than the raw figures would suggest. ORB asked Iraqis a number of questions, including whether one or more people living under their roof had been killed violently since the invasion. If the sample is randomly chosen, the margin of error should be fairly predictable.

One of the key challenges is a random sample. You can pick phone numbers at random: this works well if almost everyone has a phone, and is equally likely to answer it. If not, you can adjust for the difference between those that answer phones and those that don’t, if you can.

Or you can send out polling teams in a way that insures that everyone has about the same chance of being polled. This requires that you have a good idea of who is living where. This can work well under quite bad conditions. If a large share of the population is living in squalid refugee camps, and you have a good idea how many are living in each camp because a relief agency is handing out rations, and the residents have been thoroughly randomized by panicked flight from murderous militias, you can get a good random sample this way, at least for the people in the camps.

Iraq fits neither condition. Having completed their initial survey, ORB concluded they needed to massage the numbers. Did they undersample Baghdad? If so, they should adjust those numbers, assuming they have a good estimate of the current total population, and whether their sample was proportionate to the current population of the neighborhoods they sampled.

There’s a dilemma here. If the government is functioning adequately, you don’t estimate violent deaths by survey: you ask it to tabulate the corpses in the morgues, or the ration card holders that have stopped eating. On the other hand, if it has lost count of the corpses, then they are probably even less capable of telling you the current population of a specific neighborhood or region net of massive refugee flows.

Massaging polling numbers is a tricky business. Your poll may capture a surprisingly low number of Sunni Arab adult males. This might be sampling error, best dealt with by re-weighting the sample. On the other hand, it may reflect the fact that a large number of the expected Sunni Arab adult males are, in fact, dead. If that’s the case, re-weighting the sample will only distort the truth.

Something along those lines seems to be reflected in the recent ORB poll. Looking at the raw, un-weighted responses, 7% of Iraqi Kurds have lost a household member to violence since the invasion, and 6% of those that identify themselves as Shiites. Many Iraqis refused to claim a particular sect for the pollsters, and simply called themselves Muslims. 9% of these lost a household member. This group included some Kurds, and backing those out at the Kurdish death rate suggests that Arabs that identified themselves as “Muslim” rather than a particular sect had a household violent death rate of about 10%. This group includes both Sunni and Shiite Iraqi Arabs.

The group that identified themselves as “Sunni” had a 32% household violent death rate in the un-weighted ORB poll. However, most Kurds are Sunni, but reported much less violence. Backing them out at the average rate of reported violence for Kurds suggests that Sunni Arabs reported household violent death since the invasion at about 38%.
If the poll results are remotely in the right ballpark, Arabs that identify themselves as Sunni are suffering much worse losses than other groups in Iraq: the sort of casualties that knocked Russia and the Central Powers out of World War I, and drove the French army to the edge of mutiny.

Since the poll makes no effort to isolate civilian casualties, one plausible explanation of at least part of the disparity is that Sunni insurgents are being killed in large numbers.

While this poll will be taken by some as confirmation of Burnham et al (2006) published in the Lancet, they do contradict each other in one important sense. According to Burnham et al, Baghdad’s level of violence was about average for Iraq. According to ORB, Baghdad is about twice as deadly as Iraq as a whole.