There is an important place for fact-checking non-profit groups in promoting the use of fact-based analysis and debate in the media and in public policy. But the group Africa Check in South Africa is getting it badly wrong. The problem with Africa Check is that their flawed methods and understanding of socio-economic data risks undermining fact-based debate and media reporting in South Africa - contrary to the very purpose behind their creation.
The trouble is that Africa Check is too quick to condemn socio-economic trends as 'wrong' or 'incorrect'. A recent example relates to a finding on protest data in South Africa. The data, collected by the police, showed that incidents of violent public order behaviour have increased by 96% since 2010. Using the argument that some actions may have been recorded incorrectly in the database, and that no consensus exists on what a protest action is, Africa Check concluded that the claim was ‘wrong’ (see here).
But this argument holds no water. Of course, there could be actions that were not recorded at all while others might have been recorded incorrectly. In fact there is little doubt but that this is the case. Just consider that tens of thousands of incidents of violent or other protest action have to be recorded, while many incidents will escape the notice of the police. Hence the data could never be perfectly complete.
That, however, is the nature of socio-economic data – it is never perfect and complete.
Take Africa Check’s thinking further and the problem with their methods becomes still more apparent. Government economists say that the South African economy grew by 1.4% in 2014. It is, of course, not possible to be certain that each and every transaction was recorded in that GDP data. In fact, we can be certain that many transactions were excluded. In practice it is impossible to ensure that all transactions are included.
Does that mean that the figure of 1.4% should be rejected as ‘incorrect’? Does it mean that comparisons of growth rates from one year to the next cannot be made? Of course not. As long as the data set suffers from no comprehensive errors then it is correct to say the economy grew at 1.4%. If the method of collecting the data has not changed significantly over time then it is also correct to use that data to calculate changes in the growth rate over time.