Statisticians on the frontline

Counting refugees in conflict situations is a dangerous and imprecise task, sometimes clouded by politics

In a world of suffering, journalists play uncomfortable games with humanitarian disasters. Just how many people must die before a disaster makes the front page? How many must flee their homes and countries before it is a crisis of sufficient gravity to interest readers, viewers and listeners?

The answer is subjective, as each editor and audience is different. But the underlying principle is the same: tragedies can be measured and compared; suffering is quantifiable.

To do this, journalists need numbers: body counts, displaced person counts, refugee counts. So do international organisations such as the United Nations. They rely on numbers to figure out how much money and which resources to spend on a particular disaster. Humanitarian groups need the numbers to coordinate their responses and determine where to dispatch relief efforts first. Advocacy groups need the numbers to raise awareness of the disaster and funds to pay for their supplies and efforts.

These numbers are the foundation upon which the edifice of humanitarian intervention rests. They are the currency in the world of disaster response. Without them, organisations would be going in blind: unaware of the scale of the problem and therefore ill-equipped to deal with it appropriately.

The UN High Commissioner for Refugees (UNHCR) says 5.9m people worldwide were forced to leave their homes in the first six months of 2013, compared with 7.6m in 2012. In Africa, the UNHCR counted 3.7m refugees as of June 2013. More than 1,000 South Sudanese have fled their country every day into neighbouring Ethiopia, Kenya and Uganda since the outbreak of violence there in December 2013, reports Doctors Without Borders (MSF).

About 500,000 people have left northern Mali to become either Internally Displaced Persons (IDPs) or refugees in neighbouring countries since fighting began in January 2012, according to research by Refugees International, a US-based humanitarian organisation. Human Rights Watch claims that 55,000 Somali exiles are living in Nairobi, Kenya's capital.

Obviously, none of these numbers is exact. They are nice and round, rough calculations rather than precise counts. This makes sense, given the context in which the figures are generated. The people listed above are all fleeing or moving through war zones and natural disaster areas, places where the normal order has disintegrated so dreadfully that people are forced out of their homes. Record keeping is not a priority and often not possible.

This begs the following questions: where do these numbers come from? How Axelle Ronsse, an epidemiologist with MSF, is grappling with these questions in Bangui, the capital of the Central African Republic. Her job is to count people and bodies in the sprawling displaced persons' camp around the city's main airport. The numbers she gauges will determine MSF's response to the crisis. "We should look at where the problem is and the sectors where we have an epidemic or something like that," she told Africa in Fact. "It's also important to plan the number of structures we put in this camp and the numbers of staff we have to hire, and if we have to plan a vaccination or anything."

To estimate the number of people in the camps, Ms Ronsse uses three different techniques. First, she taps into the camp's leadership structure. The camp is divided into 88 sectors, each headed by a leader. Ms Ronsse liaises with each sector leader and asks them for an accurate count.

This is not straightforward and is subject to manipulation. Sector leaders are motivated to inflate their figures because food distribution is dependent on the number of people living in each sector.
 "When people see you counting them and asking them, they always think about food distribution and people try to increase their number," Ms Ronsse explained. "I understand. If it was me I would do the same in their situation."

To compensate, the MSF epidemiologist uses two other statistical methods based on random sampling of the camp's population. Both rely on a global positioning system (GPS) survey of the camp's area, which Ms Ronsse generates herself with a handheld GPS device.

With this information, Ms Ronsse asks a computer to identify 50 random points within the camp. At each of these points, she goes into the nearest house and counts the inhabitants. As Bangui's airport camp is an open, transient camp, she must also ask if anyone else will return at night. "Ideally I should do it at night but for security reasons I cannot." She then puts this data into special software, which uses carefully designed algorithms to extrapolate the sample into an overall estimation.

The third method is similar, but instead of 50 points the computer will specify 15 areas of 25 metres squared. Ms Ronsse and two of her team will cordon off each area with red and white construction tape, and survey all the houses that fall within. Again, the raw data is fed into the software, which comes up with an overall number.
 Taken together, these three methods should provide an accurate picture of the camp's IDPs. If Ms Ronsse had access to satellite images of the area, she would incorporate this method too. Satellite images used alone, however, can produce unreliable figures, she warned. "It's not enough by itself," she said. "I know that there are some empty houses and I cannot see what house is empty or not."

This explains the 10,000 fewer persons counted by MSF in the Bangui camp than the 100,000 identified in early January by the UNHCR, which has relied on satellite imagery, for the most part.
 As a rule, the UNHCR maintains the most comprehensive database of refugee statistics. It is the first port of call for anyone looking for reliable, big-picture information. It monitors and tracks refugees all over the world. It makes this information public through an online population database and yearbook. This task is just as daunting as it sounds.

"Data collections remains a complex process involving many actors- governments, NGOs, implementing and operating partners, and the United Nations country teams, among others," according to the 2012 yearbook, the most recent available. "This process is often a collaborative effort, in many cases requiring agreement by all parties involved. In the refugee context, data collection is typically coordinated by UNHCR and the government concerned."

The UNHCR has carefully refined this process over the 64 years of its existence. "Over the past decade, the methods for collecting refugee-related data have remained virtually unchanged, with the main techniques continuing to be censuses, registrations, and surveys or estimations," according to the UNHCR yearbook.

"Some countries rely on a single method while others use combinations to provide refugee statistics... registration has remained the main method used by UNHCR." By registration, the UNHCR is referring to the forms filled in by refugees or asylum seekers. This information is then fed into the proGres database, a specially-designed computer programme which allows UNHCR analysts to crunch through larger datasets than was possible previously.

To improve its accuracy, the agency also trains dozens of staff members every year in collection techniques and data management in the field, and has partnered with Statistics Norway, a Norwegian government unit, which provides the UNHCR with more skilled staff.

While the UNHCR is the leading source of refugee data, the Internal Displacement Monitoring Centre (IDMC), a Geneva-based advocacy group, has a UN mandate to track internally displaced people, an even more daunting task. "A few differences that stand out are that people that cross borders are generally accounted for by host country authorities either in the process of refugee status determination or as part of national or international assistance programmes," explained Alexandra Bilak, IDMC's head of policy and research, in an e-mail.

"Internal displacement is not a legal status, meaning that people need not be accounted for and registered (and this [is] normally not encouraged)," she wrote. "Also, people crossing borders tend to follow somewhat similar routes and crossing points, while people displaced in their own country may go to any place within their own country. Finally, at the end of crises, the return and repatriation of refugees are generally managed processes (either by governments or international agencies or both), which has an impact on keeping track of their numbers."

The IDMC uses satellite data, household surveys and extrapolation techniques to come up with its numbers.

The UNHCR and IDMC numbers are the most relied upon by international agencies because they are considered relatively rigorous. But not always. For example, governments provide more than a third of the UNHCR's refugee numbers. These can vary widely in quality, especially when governments bring a political agenda to bear on the collection of statistics.

"Perhaps the most important feature about internal displacement-especially if you factor in disaster-induced displacement-is that it happens worldwide and in countries with very different levels of political openness, data systems, and civil society and academic sectors," Ms Bilak said. "The accuracy and reliability of data on internal displacement varies accordingly."

As a result, major international humanitarian organisations do not always trust these numbers. Instead, they employ people like Brian Root, a quantitative analyst for Human Rights Watch, a New York-based pressure group. He is responsible for double and triple-checking all data before they are used in any kind of public communication.

Mr Root considers several items: "how something was measured, what were the definitions used, what social constructs were at play in deciding what/who/how to measure, what was the sampling methodology, what was the survey instrument, what assumptions are made in the statistical analysis, what biases are present, etc.," he wrote in an e-mail to Africa in Fact. "It's a long list. And therefore, we often don't feel comfortable citing numbers-especially those collected in active conflicts-because many, many numbers that are generated simply don't pass many of the methodological tests."

The Syrian report published in December 2013 by the American Bar Association (ABA) is an example of suspect methodology, according to Mr Root. The lawyers group went beyond merely counting displaced Syrians. They interviewed nearly 900 refugees. The ABA asked them why they left the country, what types and level of harm they suffered, and who was responsible. The international media widely publicised the report's headline conclusions.

"The issue with the ABA numbers is that they cannot be generalised to anyone besides the 800+ people that were interviewed," Mr Root wrote. "So when they put out a number that says that people claimed government responsibility for violations 19 times more than non-government forces, this number is only representative of the claims of 800 people and not a single person more. (This is not to say that the government probably is not responsible for the majority of attacks, because they likely are.) This is because of the difficulty in getting a true randomised sample in such an insane environment."

The very nature of working in a conflict zone means that it is misleading to use randomised, localised samples to explain national issues, Mr Root emphasised. "Even if [the ABA team] had gotten to everyone they had selected, the original sampling frame was people on the lists of aid agencies in these certain areas," he wrote. "So those people wouldn't be representative of all Syrian refugees-only those who had applied for aid from these specific agencies. It would not represent any refugees not on those original lists. For example, they will trend towards lower economic status, as those with money would probably not be receiving aid from these groups. Those who may be Alawite or other pro-government refugees have fled to the Syrian coast. Surely their responses on who was responsible would differ."

Other recent figures which Mr Root has had to treat with caution are the conflicting reports of casualty counts from South Sudan. The International Crisis Group, a Brussels-based think-tank, claimed in early January that the conflict between government and rebel forces had claimed nearly 10,000 lives. This was ten times the number cited just a few weeks earlier by Hilde Johnson, the UN special representative for South Sudan. Such a huge discrepancy could suggest that at least one of these figures is based more on guesswork than sound statistical methodology; or that new information suddenly became available.

Any quantification of mass refugee flows in Africa will have "some degree of uncertainty", concluded Mr Root.

The importance of good data in humanitarian work cannot be overestimated. "Reliable and accurate data are central to any effective policy-related decision-making on forced displacement," explained the UNHCR's yearbook. "In such situations, poor policy decisions, based on inaccurate or unreliable data, can have catastrophic humanitarian consequences."

Working to prevent these disasters are teams of fieldworkers, statisticians and analysts, who have developed rigorous procedures to produce sound data. Is the data perfect? Certainly not. But it is the best available. Humanitarian work would be rudderless without it.

Simon Allison is the Africa correspondent for the Daily Maverick, based in Johannesburg. He has previously reported from Egypt, Palestine and Somalia for the Asia Times and Agence France Presse. He holds a BA from Rhodes University and a master's degree from the School of Oriental and African Studies in London.

This article first appeared in the April edition of Africa in Fact, the journal of Good Governance Africa. The magazine can also be downloaded in PDF format here.

Click here to sign up for Good Governance Africa's free newsletter. GGA can also be followed on Twitter here.