Rapid response to: Civilian deaths from weapons used in the Syrian conflict
On November 4, 2015, the BMJ published our “Rapid Response” to Civilian deaths from weapons used in the Syrian conflict (BMJ 2015;351:h4736). The response was co-authored by Megan Price, Anita Gohdes, Jay Aronson (Carnegie Mellon University, Center for Human Rights Science), and Christopher McNaboe (Carter Center, Syria Conflict Mapping Project).
We have three concerns about this article. First, the article apportions responsibility for casualties to particular perpetrator organizations based on a single snapshot of territorial control that ignores the numerous (and well-documented) changes in this phenomenon over time. Second, combining Syrian Opposition Groups and Islamic State groups into one category (non-state armed groups; NSAG) obscures the fact that these groups are actively fighting each other. Finally, our work with Syrian human rights groups has taught us that their ability to document the conflict varies as a function of which armed groups are in control of each region. Therefore, the very high rate of missing data is likely to result in distorted patterns in the observed data. We will address these concerns in order in this response.
First, the authors present their findings stratified by geographic areas controlled by NSAG and the Syrian Government. This geographic stratification is based on a map of territorial control published by the BBC. The map shows areas controlled by different factions, and where high frequency conflict was reported between October and December 2014. Although they do not state this explicitly, this stratification choice implies an assessment of proportional responsibility. Such assessments will undoubtedly be important in a post-conflict transition and must be able to withstand scrutiny.
Over the course of the conflict, frontlines, battlefields and control of territory have changed rapidly across Syria. For example, analyses conducted by the Syria Conflict Mapping Project at the Carter Center have found that in the past year alone there have been close to 1,000 documented changes in territorial control among the areas they are monitoring. If they are to be accurate, quantitative claims about the geographic dynamics of violence require that analysts understand the detail of which groups controlled which areas, and when those lines of control changed. Occasionally, analysts may also have to recognize that territorial control at a particular time and place is too uncertain to attribute responsibility to a particular group. Instead of adjusting for territorial control over time, this article extrapolates backward three years from the static BBC map and forward six months from the same map. Similarly, temporal comparisons, as made in Figure 2 of the article, have not been adjusted to reflect temporal variation in territorial control. It is vital that any analyses based on territorial control take into account these rapid changes.
Our second concern pertains to the conflation of Islamic State and Syrian Opposition Groups. Since late 2013 these groups have been fighting each other. As a result, geographic regions that are treated in this analysis as being under homogenous territorial control are actually experiencing active armed confrontations between Islamic State and Syrian Opposition Groups. For example, the Washington Post describes community celebrations in February 2014 when the Free Syrian Army regained control over the town of Azaz after a “…five-month reign of terror” by factions of the Islamic State [1].
Lastly, as shared with us directly by Syrian human rights groups and described in “Stolen Futures” by the Oxford Research Group/Every Casualty [2], these groups’ ability to collect data varies dramatically depending on who has control over a territory. Data collection on both killed civilians and combatants is actively impeded by the Syrian government in areas controlled by their forces. In work previously published on estimating the dark figure in Syria [3] we provide evidence for higher levels of underreporting of fatalities in the immediate aftermath of territorial gains by the government.
The authors acknowledge these challenges in Appendix C: ‘Conflict deaths in government controlled areas may be more under-reported than those in opposition-controlled areas.’ ‘Access to bodies and access to the communications resources needed to report such casualties may be easier in urban areas compared to rural areas.’ ‘In this dataset the numbers of executions of civilians reported in government-controlled areas was extremely low…Underreporting of executions may be even higher in this area than for other deaths from weapons due to the highly controlled and secretive environment of executions…’ In other words, reporting rates for weapon types may be correlated with geography generally and territorial control specifically. Curiously, though, the authors present odds ratios in this paper without reflection about how they may be distorted by reporting issues. This is a serious methodological problem.
We agree that more attention should be drawn to the use of indiscriminate weapons and the possibility that already vulnerable populations may be disproportionately affected by such weapons. However, the analyses presented here rest on too many unexamined assumptions, uncontrolled biases, and inadequate data. Analyses such as the ones presented in this paper have the potential to influence the accountability efforts that will likely take place in the future. Without appropriate consideration of their potential limitations, they are more likely to mislead than to inform about important questions regarding culpability, reparation, reconciliation, and ultimately justice.
[1] https://www.washingtonpost.com/world/syria-extremists-retreat-from-key-t…[2] Salama, Hana, and Hamit Dardagan. “Stolen futures: the hidden toll of child casualties in Syria.” (2013).
[3] Price, Megan, Anita Gohdes, and Patrick Ball. “Documents of war: Understanding the Syrian conflict.” Significance 12.2 (2015): 14-19.
Reprinted with permission of the British Medical Journal.