Data Bomb
By Neil Munro and Carl M. Cannon, National Journal
? National Journal Group Inc.
Friday, Jan. 4, 2008
Three weeks before the 2006 midterm elections gave Democrats control of Congress, a shocking study reported on the number of Iraqis who had died in the ongoing war. It bolstered criticism of President Bush and heightened the waves of dread -- here and around the world -- about the U.S. occupation of Iraq.
Published by The Lancet, a venerable British medical journal, the study [PDF] used previously accepted methods for calculating death rates to estimate the number of "excess" Iraqi deaths after the 2003 invasion at 426,369 to 793,663; the study said the most likely figure was near the middle of that range: 654,965. Almost 92 percent of the dead, the study asserted, were killed by bullets, bombs, or U.S. air strikes. This stunning toll was more than 10 times the number of deaths estimated by the Iraqi or U.S. governments, or by any human-rights group.
In December 2005, Bush had used a figure of 30,000 civilian deaths in Iraq. Iraq's health ministry calculated that, based on death certificates, 50,000 Iraqis had died in the war through June 2006. A cautiously compiled database of media reports by a London-based anti-war group called Iraq Body Count confirmed at least 45,000 war dead during the same time period. These were all horrific numbers -- but the death count in The Lancet's study differed by an order of magnitude.
Queried in the Rose Garden on October 11, the day the Lancet article came out, Bush dismissed it. "I don't consider it a credible report," he replied. The Pentagon and top British government officials also rejected the study's findings.
Such skepticism would not prove to be the rule.
CBS News called the report a "new and stunning measure of the havoc the American invasion unleashed in Iraq." CNN began its report this way: "War has wiped out about 655,000 Iraqis, or more than 500 people a day, since the U.S.-led invasion, a new study reports." Within a week, the study had been featured in 25 news shows and 188 articles in U.S. newspapers and magazines, including The New York Times, The Washington Post, and the Los Angeles Times.
Editorials in many major newspapers cited the Lancet article as further evidence that the invasion of Iraq was a bad idea, and the liberal blogosphere ridiculed Bush for his response. Prominent mainstream media outlets quoted various academics who vouched for the study's methodology, including some who said they had reviewed the data before publication.
Within a few weeks a backlash rose, although the contrarian view of the study generated far less press attention than the Lancet article. In the ensuing year, numerous skeptics have identified various weaknesses with the study's methodology and conclusions. Political blogs and academic journals have registered and responded to the objections in a debate that has been simultaneously arcane and predictable. The arguments are arcane because that is the nature of statistical analysis. They are predictable because that is the nature of today's polarized political discourse, with liberals defending the Lancet study and conservatives contesting it.
How to explain the enormous discrepancy between The Lancet's estimation of Iraqi war deaths and those from studies that used other methodologies? For starters, the authors of the Lancet study followed a model that ensured that even minor components of the data, when extrapolated over the whole population, would yield huge differences in the death toll. Skeptical commentators have highlighted questionable assumptions, implausible data, and ideological leanings among the authors, Gilbert Burnham, Riyadh Lafta, and Les Roberts.
Some critics go so far as to suggest that the field research on which the study is based may have been performed improperly -- or not at all. The key person involved in collecting the data -- Lafta, the researcher who assembled the survey teams, deployed them throughout Iraq, and assembled the results -- has refused to answer questions about his methods.
Some of these questions could be resolved if other researchers had access to the surveyors' original field reports and response forms. The authors have released files of collated survey results but not the original survey reports, citing security concerns and the fact that some information was not recorded or preserved in the first place. This was a legitimate problem, and it underscored the difficulty of conducting research in a war zone.
Each death recorded by the Hopkins surveyors in 2006 extrapolated to 2,000 deaths in the Iraqi population.
Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros's Open Society Institute.
Origins Of The Survey
Since the beginning of the war, the media have meticulously tracked and documented the number of American soldiers killed in Iraq -- which reached 3,904 on January 1 -- particularly as the total approached and then surpassed (in December 2006) the 2,973 people killed in the 9/11 terrorist attacks. But determining the number of Iraqis who have died is much more difficult, as is determining how many of the dead were insurgents and how many were innocent civilians. With Iraq's central government barely functioning, health services overwhelmed, and political agendas coloring all agencies, no reliable statistics exist so far.
The Lancet study was based on techniques developed by public health experts to determine rates of illness and death from epidemics and famines in large populations. This "cluster" sampling is a relatively new methodology that attempts to replicate the logic of public opinion polling in Third World locales that lack a telecommunications infrastructure.
Following this method, questioners undertake a house-to-house survey in certain areas and then extrapolate the results from that statistical sample to the entire national population. According to this study's design, teams of Iraqi questioners would visit approximately 47 randomly chosen clusters of homes throughout the country and ask a series of census-style questions at 40 contiguous households in each cluster: How many people live in your household? How many lived here on January 1, 2002? In that time, how many were born -- and how many died?
In 2004, several of the same authors had done a preliminary Iraq study using this method. Also published in The Lancet (and also deliberately timed, by the authors' admission, to appear just before a U.S. election), that article reported at least 98,000 "excess" Iraqi deaths. Perhaps because that estimate contrasted sharply with the observations of embedded reporters, human-rights activists, and others on the ground in Iraq, the media gave it limited coverage.
The Authors
The origins of the Lancet studies can be traced to 1993, when two officials from the U.S. Centers for Disease Control and Prevention traveled to Bosnia-Herzegovina to view the devastation caused by the Balkan war. Only nine years after Sarajevo had triumphantly hosted the Winter Olympics, the once-lovely city was making the tragic transition from a cosmopolitan regional oasis to a hellhole identified by a chilling new phrase: "ethnic cleansing." The terrorized Bosnian populace related tales of brutality so appalling that the visiting Americans dismissed them as absurd rumors: Croatian guerrillas were buying castration devices from the Germans to use on Bosnian men; Serbian snipers were shooting children in the legs and using them as "bait" to bring their parents within range.
In pursuit of an accurate picture, the U.S. health officials toured a hospital in Sarajevo. In the surgical ward, they saw many children in post-operative recovery -- from bullet wounds in their legs. The "absurd" urban myths, apparently, had some truth to them. In the face of such exceptional horror, one of the Americans -- Les Roberts -- experienced an epiphany. First, he realized that in a sectarian civil war, the unthinkable is not only possible, it is commonplace. Second, the tribulations of children trapped in war zones are especially horrifying. Third, a public official who has seen such suffering has a moral duty to try to stop it.
"I think that's when I fully understood the need to step beyond peer-review journals and statistical analyses if you are going to do effective public health work in times of war," Roberts explained in a recent interview with a Belgian-based publication. This determination to become an advocate would lead him to Rwanda and the Congo, where in 2001 he was involved in studies that produced jaw-dropping estimates of more than 3 million dead in that nation's civil war. Roberts also went back to the Balkans -- this time to Kosovo -- and ultimately, when war came to Iraq in 2003, he traveled to Baghdad.
By then, Roberts was a researcher at the Johns Hopkins University's Bloomberg School of Public Health. He broached the idea of a postwar mortality study in Iraq with Gilbert Burnham, co-director of the school's Center for Refugee and Disaster Response. The two men approached Richard Garfield, a Columbia University epidemiologist who signed on and put them in touch with an Iraqi scientist he knew, Riyadh Lafta, to recruit and oversee researchers who could conduct field surveys in Iraq.
A car bomb attack in Sadr City that killed at least 60 people appears to have been counted by the researchers, even though it happened a day after the survey was to end, critics say.
Lafta had been a child-health official in Saddam Hussein's ministry of health when the ministry was trying to end the international sanctions against Iraq by asserting that many Iraqis were dying from hunger, disease, or cancer caused by spent U.S. depleted-uranium shells remaining from the 1991 Persian Gulf War. In 2000, Lafta authored at least two brief articles contending that U.N. sanctions had caused many deaths by starvation among Iraqi children. In one article, he identified malnutrition as the main contributor to 53 percent of deaths among hospitalized children younger than 2, during a 1997 survey carried out at Saddam Central Teaching Hospital. The article cited no health data from before the sanctions, yet it asserted, "We can conclude from results that the most important and widespread underlying cause of the deterioration of child-health standards in Iraq is the long-term impact of the nonhumanized economic sanction imposed through United Nations resolutions." The article was published in 2000 by the Iraqi Journal of Community Medicine. Roberts told National Journal he had not read Lafta's articles, and Burnham said he did not have a copy of the articles.
Lafta is now at Mustansiriya University in Baghdad, where he briefly served as dean of the medical college in 2003.
Lafta and his surveyors often worked under brutal political pressure. In January 2007, a Sunni suicide bomber killed more than 70 students at the university, partly because it is perceived as being under the control of Moktada al-Sadr, the Shiite religious leader whose Mahdi Army militia crippled Sunni insurgent groups in Baghdad during 2006. Until this fall, Sadr's party and his Mahdi Army also controlled the health ministry, which employed some of Lafta's researchers.
Dramatic Findings
In his first study of Iraqi war deaths, in September 2004, Lafta sent six Iraqi questioners to 33 clusters of homes throughout the country to ask how many people in each household had died since January 1, 2002. The researchers reported that 808 of the 998 identified households participated in the survey, and then extrapolated the number of deaths reported to the entire population of 24.4 million Iraqis. "Making conservative assumptions, we think that about 100,000 excess deaths or more have happened since the 2003 invasion of Iraq," concluded the authors -- Roberts, Lafta, Garfield, Jamal Khudhairi, and Burnham. That was when the war was just 19 months old.
"Violence accounted for most of the excess deaths, and air strikes from coalition forces accounted for most violent deaths," the report said. According to subsequent explanations by the authors, the total included 57,600 dead from violence, 24,000 dead from wartime accidents, and 13,600 dead from disease. The accidental deaths included 15,000 Iraqis killed by U.S. vehicles in road incidents -- extrapolated from five death reports.
Little is known about Lafta's decision-making in amassing the data for the Lancet surveys. Roberts provided some information, however, about Lafta's 2004 survey of casualties in Falluja. At the time, al-Sadr was publicly supporting the anti-American Sunni radicals who controlled the city. In September, Roberts said, he pleaded with "his Muslim friend Lafta not to go" into Falluja, according to an interview with a magazine published by Johns Hopkins. Roberts told the interviewer that Lafta replied, "God has picked these clusters. If God wants me, he will take me. I must go." Roberts also said of Lafta, "I know no one [who] perceives themselves so humbly to be a tool of God's destiny.... He sees his science as synonymous with service to God."
In Falluja, Lafta recorded 52 deaths in 29 households, which amounted to 71 percent of the violent deaths recorded by the first Lancet survey. If representative, Lafta's sample translated into 50,000 to 70,000 dead in Falluja by September 2004 -- two months before the start of the second major American military operation to restore order. Falluja's prewar population was estimated to be 250,000, although U.S. officials said that the vast majority of residents had fled before the battles began. Lafta's Falluja death estimate was so far off the chart that his colleagues dropped it from the study, the authors said.
The 2006 study, known as Lancet II, was somewhat larger, involving 47 clusters and using similar survey techniques. In all, 302 violent deaths reported in those 1,849 households became the basis for estimating that 601,000 Iraqis had died violently from the start of the war through June 2006.
Even though the second study was even further out of line with other sources' estimates than the first, it got tremendous attention -- probably because its findings fit an emerging narrative: Iraq was a horrific mess. The February 2006 bombing of Samarra's Golden Mosque, in particular, had sent the country spiraling toward sectarian warfare.
Democrats who had opposed Bush's Iraq campaign embraced the report. Sen. Edward Kennedy, D-Mass., for example, issued a statement saying that the "new study is a chilling and somber reminder of the unacceptably high human cost of this war.... We must not stay on the same failed course any longer." Such remarks, amplified by myriad articles, broadcasts, and blogs, helped to cement Americans' increasingly negative perceptions of the war. "For those who wanted to believe it, it gave them a new number to circulate, [and] it was a defining moment" in attitudes toward the war, said pollster John Zogby, who commended the report in a CNN interview.
The Lancet II article was also publicized widely overseas, especially in the Middle East. One Al Jazeera pundit said that the study revealed "what is surely the greatest crime in human history." A Pakistani columnist declared, "According to [the] highly reputed Lancet, an English science and medical journal, 650,000 Iraqis have been killed since the American invasion ... to fulfill the imperial lust of Washington and its cohorts."
Muslim commentators in the United States have been only slightly more restrained. "The Arab masses and the Muslims understand what's at stake here; they know what the U.S. is doing; they can see the casualties and suffering," Osama Siblani, the publisher of the Michigan-based Arab American News, said in an interview. The United States' destructive policies in the Middle East "are creating a fertile ground for Osama [bin Laden] to come in and recruit," he said, describing the elected Iraqi government as a "puppet" that should be removed from power.
In the Middle East, both Sunni and Shiite Islamist groups have used the study to bolster their claims that the West is waging a war against Islam. In an October 30, 2007, debate on Al Jazeera, for example, an Egyptian cleric, Sheik Ibrahim al-Khouli, slammed a Syrian author's criticism of fundamentalist Islam. The United States and Europe had "fought in Iraq and destroyed it," he said. They "killed one and a half million people ... [and] killed a million Iraqi children during the [1990s sanctions] siege; left traces of enriched uranium from the weapons that were used [in 1991]; and destroyed the environment for the next 35 billion years, according to American estimates."
The John Hopkins researcher, Les Roberts, began the studies by smuggling himself into Iraq with $20,000 stuffed in his money belt and shoes.
The study had such a significant impact partly because of where it appeared. The Lancet, founded in 1823, is one of the world's most-cited medical journals, credited with publishing articles that established the principles of antiseptics in 1867 and documented the dangers of thalidomide in 1961. Although few mainstream journalists ever plow through the journal's articles, news outlets typically refer to it as "the respected Lancet." In recent years, however, the journal's reputation has suffered from charges of politicization and a few prominent instances of scientific fraud.
Also driving the press attention was the study's association with Johns Hopkins University, whose School of Public Health was the first and is now the largest such institution in the world. Faculty members participated in the study, and the school's review board conducted an ethical review of the research plan. The Arab American's Siblani said that the university connection was one reason he put the study on the front page of his newspaper.
Potential Problems
Both Lancet studies of Iraqi war deaths rest on the data provided by Lafta, who operated with little American supervision and has rarely appeared in public or been interviewed about his role. In May, Lafta and Roberts presented their study to an off-the-record meeting of experts in Geneva, but other attendees declined to describe Lafta's remarks. Despite multiple requests sent via e-mails and through Burnham and Roberts, Lafta declined to communicate with National Journal or to send copies of his articles about Iraqi deaths during Saddam's regime.
When asked questions about the reliability of their Iraqi partner, the studies' American authors defend Lafta as a nice guy and a good researcher.
"I've known him for years," Garfield told NJ. "I used to work with his boss in 2003, studying how Saddam had pilfered cash [intended] for the health care system. He's thoughtful, careful, and we became friends."
John Tirman, a political scientist at the Massachusetts Institute of Technology, described Lafta as "a medical doctor, a professor of medicine. Those factors were a sufficient level of credibility. I never asked [Lafta] about his political views." Tirman commissioned the Lancet II survey with $46,000 from George Soros's Open Society Institute and additional support from other funders.
Lancet Editor Richard Horton shares this fundamental faith in scientists. He told NJ that scientists, including Lafta, can be trusted because "science is a global culture that operates by a set of norms and standards that are truly international, that do not vary by culture or religion. That's one of the beautiful aspects of science -- it unifies cultures, not divides them."
Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent -- and repeatable -- to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?
"The authors refuse to provide anyone with the underlying data," said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at Harvard University. Some critics have wondered whether the Iraqi researchers engaged in a practice known as "curb-stoning," sitting on a curb and filling out the forms to reach a desired result. Another possibility is that the teams went primarily into neighborhoods controlled by anti-American militias and were steered to homes that would provide information about the "crimes" committed by the Americans.
Fritz Scheuren, vice president for statistics at the National Opinion Research Center and a past president of the American Statistical Association, said, "They failed to do any of the [routine] things to prevent fabrication." The weakest part of the Lancet surveys is their reliance on an unsupervised Iraqi survey team, contended Scheuren, who has recently trained survey workers in Iraq.
When the study came out in October 2006, President Bush said it wasn't credible.
The research is "a field study in unstable conditions," Columbia University's Garfield, one of the authors of the preliminary 2004 study, told National Journal in October. "You know that it's imperfect, but ... I'll say this: It's much easier to discredit than to go into a place like this and try and find answers. None of these harpies are dodging bullets."
Perhaps. But overall, the possible shortcomings of the Lancet studies persist, in three broad categories.
Design And Implementation
Critics say that the surveys used too few clusters, and too few people, to do the job properly.
# Sample size. The design for Lancet II committed eight surveyors to visit 50 regional clusters (the number ended up being 47) with each cluster consisting of 40 households. By contrast, in a 2004 survey, the United Nations Development Program used many more questioners to visit 2,200 clusters of 10 houses each. This gave the U.N. investigators greater geographical variety and 10 times as many interviews, and produced a figure of about 24,000 excess deaths -- one-quarter the number in the first Lancet study. The Lancet II sample is so small that each violent death recorded translated to 2,000 dead Iraqis overall. The question arises whether the chosen clusters were enough to be truly representative of the entire Iraqi population and therefore a valid data set for extrapolating to nationwide totals.
# "Main street" bias? According to the Lancet II article, surveyors randomly selected a main street within a randomly picked district; "a residential street was then randomly selected from a list of residential streets crossing the main street." This method pulled the survey teams away from side streets and toward main streets, where car bombs can kill the most people, thus boosting the apparent death rate, according to a critique of the study by Michael Spagat, an economics professor at the Royal Holloway, University of London, and Sean Gourley and Neil Johnson of the physics department at Oxford University.
Burnham responds that The Lancet's description of how the researchers picked sites was an editing error, and that the method used eliminated main-street bias.
# Oversight. To undertake the first Lancet study, Roberts went into Iraq concealed on the floor of an SUV with $20,000 in cash stuffed into his money belt and shoes. Daring stuff, to be sure, but just eight days after arriving, Roberts witnessed the police detaining two surveyors who had questioned the governor's household in a Sadr-dominated town. Roberts subsequently remained in a hotel until the survey was completed. Thus, most of the oversight for Lancet I -- and all of it for Lancet II -- was done long-distance. For this reason, although he defends the methodology, Garfield took his name off Lancet II. "The study in 2006 suffered because Les was running for Congress and wasn't directly supervising the work as he had done in 2004," Garfield told NJ.
Black-Box Data
With the original data unavailable, other scholars cannot verify the findings, a key test of scientific rigor.
# Response rate. The surveyors said that 1.7 percent of households -- fewer than one in 50 -- were unoccupied or uncooperative, even though questioners visited each house only once on one day; that answers were taken only from the household's husband or wife, not from in-laws or adult children; and that householders had reason to fear that their participation would expose them to threats from armed groups.
To Kane, the study's reported response rate of more than 98 percent "makes no sense," if only because many male heads of households would be at work or elsewhere during the day and Iraqi women would likely refuse to participate. On the other hand, Kieran J. Healy, a sociologist at the University of Arizona, found that in four previous unrelated surveys, the polling response in Iraq was typically in the 90 percent range.
The Lancet II questioners had enough time to accomplish the surveys properly, Burnham said.
# Lack of supporting data. The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather. For example, D3 Systems, a polling firm based in Vienna, Va., that has begun working in Iraq, tries to prevent chicanery among its 100-plus Iraqi surveyors by requiring them to ask respondents for such basic demographic data as ages and birthdates. This anti-fraud measure works because particular numbers tend to appear more often in surveys based on fake interviews and data -- or "curb-stoning -- than they would in truly random surveys, said Matthew Warshaw, the Iraq director for D3. Curb-stoning surveyors might report the ages of many people to be 30 or 40, for example, rather than 32 or 38. This type of fabrication is called "data-heaping," Warshaw said, because once the data are transferred to spreadsheets, managers can easily see the heaps of faked numbers.
# Death certificates. The survey teams said they confirmed most deaths by examining government-issued death certificates, but they took no photographs of those certificates. "Confirmation of deaths through death certificates is a linchpin for their story," Spagat told NJ. "But they didn't record (or won't provide) information about these death certificates that would make them traceable."
Under pressure from critics, the authors did release a disk of the surveyors' collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single province of Nineveh. That means the surveyors reported zero missing or lost certificates for 180 violent deaths in 15 provinces outside Nineveh. The odds against such perfection are at least 10,000 to 1, Spagat told NJ. Also, surveyors recorded another 70 violent deaths and 13 nonviolent deaths without explaining the presence or absence of certificates in the database. In a subsequent MIT lecture, Burnham said that the surveyors sometimes forgot to ask for the certificates.
# Suspicious cluster. Lafta's team reported 24 car bomb deaths in early July, as well as one nonviolent death, in "Cluster 33" in Baghdad. The authors do not say where the cluster was, but the only major car bomb in the city during that period, according to Iraq Body Count's database, was in Sadr City. It was detonated in a marketplace on July 1, likely by Al Qaeda, and killed at least 60 people, according to press reports.
The authors should not have included the July data in their report because the survey was scheduled to end on June 30, according to Debarati Guha-Sapir, director of the World Health Organization's Collaborating Center for Research on the Epidemiology of Disasters at the University of Louvain in Belgium. Because of the study's methodology, those 24 deaths ultimately added 48,000 to the national death toll and tripled the authors' estimate for total car bomb deaths to 76,000. That figure is 15 times the 5,046 car bomb killings that Iraq Body Count recorded up to August 2006.
According to a data table reviewed by Spagat and Kane, the team recorded the violent deaths as taking place in early July and did not explain why they failed to see death certificates for any of the 24 victims. The surveyors did remember, however, to ask for the death certificate of the one person who had died peacefully in that cluster.
The Cluster 33 data is curious for other reasons as well. The 24 Iraqis who died violently were neatly divided among 18 houses -- 12 houses reported one death, and six houses reported two deaths, according to the authors' data. This means, Spagat said, that the survey team found a line of 40 households that neatly shared almost half of the deaths suffered when a marketplace bomb exploded among a crowd of people drawn from throughout the broader neighborhood.
The data also bolster Spagat's criticism that the surveyors selected too many clusters in places where bomb explosions and gunfights were most common.
Ideological Bias
Virtually everyone connected with the study has been an outspoken opponent of U.S. actions in Iraq. (So are several of the study's biggest critics, such as Iraq Body Count.) Whether this affected the authors' scientific judgments and led them to turn a blind eye to flaws is up for debate.
# Follow the money. Lancet II was commissioned and financed by Tirman, the executive director of the Center for International Studies at MIT. (His most recent book is 100 Ways America Is Screwing Up the World.) After Lancet I was published, Tirman commissioned Burnham to do the second study, and sent him $50,000. When asked where Tirman got the money, Burnham told NJ: "I have no idea."
In fact, the funding came from the Open Society Institute created by Soros, a top Democratic donor, and from three other foundations, according to Tirman. The money was channeled through Tirman's Persian Gulf Initiative. Soros's group gave $46,000, and the Samuel Rubin Foundation gave $5,000. An anonymous donor, and another donor whose identity he does not know, provided the balance, Tirman said. The Lancet II study cost about $100,000, according to Tirman, including about $45,000 for publicity and travel. That means that nearly half of the study's funding came from an outspoken billionaire who has repeatedly criticized the Iraq campaign and who spent $30 million trying to defeat Bush in 2004.
# Partisan considerations. Soros is not the only person associated with the Lancet studies who had one eye on the data and the other on the U.S. political calendar. In 2004, Roberts conceded that he opposed the Iraq invasion from the outset, and -- in a much more troubling admission -- said that he had e-mailed the first study to The Lancet on September 30, 2004, "under the condition that it come out before the election." Burnham admitted that he set the same condition for Lancet II. "We wanted to get the survey out before the election, if at all possible," he said.
"Les and Gil put themselves in position to be criticized on the basis of their views," Garfield concedes, before adding, "But you can have an opinion and still do good science." Perhaps, but the Lancet editor who agreed to rush their study into print, with an expedited peer-review process and without seeing the surveyors' original data, also makes no secret of his leftist politics. At a September 2006 rally in Manchester, England, Horton declared, "This axis of Anglo-American imperialism extends its influence through war and conflict, gathering power and wealth as it goes, so millions of people are left to die in poverty and disease." His speech can be viewed on YouTube.
# Mr. Roberts tries to go to Washington. Roberts, who opposed removing Saddam from power, is the most politically outspoken of the authors. He initiated the first Lancet study and repeatedly used its conclusions to criticize Bush. "I consider myself an advocate," Roberts told an interviewer in early 2007. "When you start working documenting events in war, the public health response -- the most important public health response -- is ending the war."
In 2006, he acted on this belief, seeking the Democratic nomination for New York's 24th Congressional District before dropping out in favor of the eventual winner, Democrat Michael Arcuri. Asked why he ran for office, Roberts told NJ: "It was a combination of Iraq and [Hurricane] Katrina that just put me over the top. I thought the country was going in the desperately wrong direction, particularly with regard to public health and science."
Politics At Work
Roberts was hardly the only American to lose confidence in Bush. The question is whether he and his team lost their objectivity as scientists as well. Unanimously, the authors insist that the answer is no.
Roberts concedes that the only certain way to collect information for a study of Iraqi war casualties would be through a full census, something he says is impossible in the midst of sectarian civil war. His study's method "has limitations," he told NJ. "It works less well when bombs are killing people in clusters -- and they are killing people in clusters in Iraq -- but it remains a fundamentally robust way of determining changes in mortality rates." Asked if he remains certain that Lafta's Iraqi teams truly collected the data they turned in, Roberts answered, "I'm just absolutely confident this data is not fabricated."
"Dr. Burnham and his colleagues are confident that the data presented in the 2004 and 2006 are accurate, and they fully stand by the conclusions of their research," according to a November 27 statement from the Bloomberg School of Public Health. "The findings of independent surveys of Iraqis conducted by the United Nations in March 2005, by the BBC in March 2007, and by the British polling firm ORB in September 2007 support the conclusions of the Hopkins mortality studies."
Critics say, however, that the other national reports cited in the Johns Hopkins statement, particularly the ORB poll, have methodological flaws and political overtones similar to those in the Lancet studies.
"Just stating, 'We have no biases of that type' isn't very convincing," says Oxford University's Johnson. "Using 'I am an expert' arguments sounds to me like 'Trust me, I am a doctor.' " Johnson and two of his colleagues have called on the scientific community to conduct an in-depth re-evaluation of both Lancet studies. "It's almost a crime to let it go unchallenged," Johnson said.
Even Garfield, a co-author of the first Lancet article, is backing away from his previous defense of his fellow authors. In December, Garfield told National Journal that he guesses that 250,000 Iraqis had died by late 2007. That total requires an underlying casualty rate only one-quarter of that offered by Lancet II.
The authors -- Lafta excepted -- have been willing to engage their critics in debate, returning journalists' calls and, for the most part, avoiding ad hominem arguments. Yet, sometimes their defenses raise new questions. Burnham says, for instance, that Lafta offered to take reporters to visit some of the neighborhoods used in the clusters, although he declined to say whether the reporters would be allowed to visit the surveyed households or to pick the clusters to see.
Roberts and his defenders emphasize that when their cluster method produced shockingly high mortality rates in the Congo, no one questioned them -- not seeming to understand that journalists looking at the Iraq study are now indeed wondering if the Congo results are valid.
Roberts, when asked if he timed the release of his Lancet studies to hurt the Republicans on Election Day, contends that his biggest concern was ensuring the safety of his researchers. "If this study was finished in September and not published until after the November elections -- and it was perceived that we were sitting on the results -- my Iraqi colleagues would have been killed," he told National Journal. Even if true, this assertion undermines his expressions of confidence in the integrity and skill of the Iraqi researchers. How can their data be trusted if their very lives depended on the results?
No matter whether a latent desire to feed the American public's opposition to the war might have shaped these studies, another audience was paying close attention: jihadists who used this research as a justification for killing Americans. Roberts already believed that jihadi attacks were, in part, driven by the international image of the United States. "The greatest threat to U.S. national security [is] the image that the United States is a violator of international laws and order and that there is no means other than violence to curb it," Roberts wrote in a July 2005 article for Tirman's center. When NJ asked Roberts about the risk that his estimate would incite more violence, his confidence seemed to waver for the only time during the interview. "This area of study is a minefield," he said. "The people you are talking about are the same kind of people who deny the Holocaust." Does it give him qualms that some of those people use his study to recruit suicide bombers? "It does," he replied after a pause. "My guess is that I've provided data that can be narrowly cited to incite hatred. On the other hand, I think it's worse to have our leaders downplaying the level of violence."
Burnham also paused when asked whether Iraqi factions manipulated him and his colleagues and then replied, "We're reasonably confident that we were not manipulated."
Professional Responsibilities
Officials at Iraq Body Count strongly opposed the Iraq war yet issued a detailed critique of the Lancet II study. Researchers wading into a field that is this fraught with danger have a responsibility not to be reckless with statistics, the group said. The numbers claimed by the Lancet study would, under the normal ratios of warfare, result in more than a million Iraqis wounded seriously enough to require medical treatment, according to this critique. Yet official sources in Iraq have not reported any such phenomenon. An Iraq Body Count analysis showed that the Lancet II numbers would have meant that 1,000 Iraqis were dying every day during the first half of 2006, "with less than a tenth of them being noticed by any public surveillance mechanisms." The February 2006 bombing of the Golden Mosque is widely credited with plunging Iraq into civil war, yet the Lancet II report posits the equivalent of five to 10 bombings of this magnitude in Iraq every day for three years.
"In the light of such extreme and improbable implications," the Iraq Body Count report stated, "a rational alternative conclusion to be considered is that the authors have drawn conclusions from unrepresentative data."
Against these criticisms, the authors maintain that they were using methods of study unfamiliar to human-rights groups and that the scientific community widely accepted the Lancet studies. "There have been 56 studies using this retrospective household survey method," Garfield said. "The estimation of crude mortality in a population does work.... It doesn't mean you can't do it wrong. It is the best method we have. The question is, 'Did they do it right?' "
When it comes to the question of peer review, the study's defenders sometimes seem to want it both ways. On the one hand, Roberts talks about the need "to step beyond peer review." Yet the authors insist that their study was peer-reviewed extensively (if rapidly, in order to be published before the election). The authors also maintain that one of the reasons they went to The Lancet with these studies is its quick turnaround time.
Surprisingly, not one of the peer reviewers seems to have thought to ask a basic question: Are the data in the two studies even true? The possibility of fakery, editor Horton told NJ, "did not come up in peer review." Medical journals can't afford to repeat every scientific study, he said, because "if for every paper we published we had to think, 'Is this fraud?' ... honestly, we would fold tomorrow."
In Belgium, Guha-Sapir's team is completing a paper outlining numerous mathematical and procedural errors in the Lancet II article, and its corrections will likely lower the estimate of dead Iraqis to 450,000, even without consideration of possible fraud during the surveying, a source said.
Perhaps medical journals, like respected news organizations, will learn that they have to factor the possibility of wartime fraud into their fact-checking. Horton knows the peacetime risks only too well: In a Lancet article in October 2005, exactly halfway between the two Iraq mortality studies, a Norwegian physician named Jon Sudbo wrote that a review of 454 patients showed that such common painkillers as ibuprofen and naproxen reduced smokers' risk of contracting oral cancer while increasing their risk for heart disease; it later turned out that Sudbo had faked his research.
Today, the journal's editor tacitly concedes discomfort with the Iraqi death estimates. "Anything [the authors] can do to strengthen the credibility of the Lancet paper," Horton told NJ, "would be very welcome." If clear evidence of misconduct is presented to The Lancet, "we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry."
http://news.nationaljournal.com/articles/databomb/index.htm
