The Afrobarometer is an award-winning international research project that collects data on Africans' attitudes and opinions about democracy and economics. Now conducted in 20 countries, and with an unprecedented four waves of data collected in 12 countries, no social science project has ever approached its scope on this continent. The project has been supported since 1999 by a wide range of multilateral and bilateral donors who use the data to understand the political atmosphere in Africa, and the results are increasingly seen and discussed at the highest levels of government across the continent. Afrobarometer data has supported articles and books in leading social science journals and publishing houses.
Thus, it was indeed surprising to come across an article last week written by the Democratic Alliance's Gareth Van Onselen ("Is Afrobarometer's Latest Poll Reliable?" - see here). Certainly, no research project is above criticism. However, one can't escape the conclusion that the motivation for the piece was not academic debate but the DA's pique at the results to one single question in the most recent Afrobarometer South Africa survey (on party support, which I will come to below) which then led him to download the summary of the questionnaire and top-line results from the Idasa website (Idasa is one of the core partners of the Afrobarometer) and embark on a "slash and burn" exercise, pulling out three or four questions to try and undermine the reliability of the entire project.
But because all of us should be prepared to learn from scholarly debate, however vituperative, let me address each of Van Onselen's points, one by one.
For starters, Van Onselen attacks the verbal, ordinal response scale we present to respondents answering a question about how often they were a victim of violent crime. The verbal rankings ("never," "once or twice," "several times"), Van Onselen suggests, are relative concepts and might mean different things to different people. He argues, rather, that we should have asked people to give us an actual quantitative count.
However, there are at least two different reasons why this is an appropriate way of posing the responses. First, in large-scale surveys like the Afrobarometer, it is incumbent on questionnaire designers to gather together sets of similar question items that can be answered with the same sets of responses. If he had bothered to look at the questionnaire more closely Van Onselen would have seen that this item was part of a larger battery of question items on human security, looking both at crime but also the frequency with which people experience shortages of basic necessities.
Second, it might seem on the face of it to make more sense to ask people for a more precise count of how many times they were attacked, or felt insecure, or went without enough to eat in the past year, But research shows that such responses provide a false sense of exactitude. Many people, especially those regularly subjected to violence and destitution, simply can't keep track to any meaningful degree of how many times they went hungry last year. Thus, it is safer - and more reliable - to ask for a more rough estimation: never, once or twice, several times, etc.... Indeed, we find that the aggregated responses present very reliable trends over time, with the overall percentages of those who are frequent victims of crime, for example, moving upward and downward in ways that generate important insights into national experiences of poverty and security and which provide invaluable alternatives to official data.