There is an important place for fact-checking non-profit groups in promoting the use of fact-based analysis and debate in the media and in public policy. But the group Africa Check in South Africa is getting it badly wrong. The problem with Africa Check is that their flawed methods and understanding of socio-economic data risks undermining fact-based debate and media reporting in South Africa - contrary to the very purpose behind their creation.

The trouble is that Africa Check is too quick to condemn socio-economic trends as 'wrong' or 'incorrect'. A recent example relates to a finding on protest data in South Africa. The data, collected by the police, showed that incidents of violent public order behaviour have increased by 96% since 2010. Using the argument that some actions may have been recorded incorrectly in the database, and that no consensus exists on what a protest action is, Africa Check concluded that the claim was ‘wrong’ (see here).

But this argument holds no water. Of course, there could be actions that were not recorded at all while others might have been recorded incorrectly. In fact there is little doubt but that this is the case. Just consider that tens of thousands of incidents of violent or other protest action have to be recorded, while many incidents will escape the notice of the police. Hence the data could never be perfectly complete.

That, however, is the nature of socio-economic data – it is never perfect and complete.

Take Africa Check’s thinking further and the problem with their methods becomes still more apparent. Government economists say that the South African economy grew by 1.4% in 2014. It is, of course, not possible to be certain that each and every transaction was recorded in that GDP data. In fact, we can be certain that many transactions were excluded. In practice it is impossible to ensure that all transactions are included.

Does that mean that the figure of 1.4% should be rejected as ‘incorrect’? Does it mean that comparisons of growth rates from one year to the next cannot be made? Of course not. As long as the data set suffers from no comprehensive errors then it is correct to say the economy grew at 1.4%. If the method of collecting the data has not changed significantly over time then it is also correct to use that data to calculate changes in the growth rate over time.

Take demographic data. StatsSA reports that there are 54 million people in the country. However, it is not possible to count them all, or even to account for all the migrants in the country. Is StatsSA therefore wrong – and should journalists stop reporting the size of the population?

Think where this would take us.

We would become a society that rejects fact-based analysis in its entirety – all because it is possible to argue here and there about the completeness of datasets and definitions. Quite the opposite should be true and analysts, politicians, and journalists must be encouraged to cite socio-economic data. Such data should in turn be embraced as the start of rich public policy debates and analyses not their death-knell.

We are currently engaged in an audit of Africa Check’s ‘reports’ and - while we don’t want to go into any detail on that exercise - there is a strong propensity to declare as ‘wrong’ or ‘incorrect’ fact-based claims that could easily be argued to be right.

This is dangerous as it will deter journalists, researchers, and politicians from citing data in their analyses. If there is definite evidence that an intentionally misleading claim has been made by a politician, for example, that is where a fact-checking service is valuable. It should, in such a case, explain why the claim is wrong and what the correct figure or indicator should read.

Short of such examples, ‘fact-checkers’ should be circumspect in their opinions and particularly careful about dismissive findings such as ‘wrong’ and ‘incorrect’ especially when applied to often very complex socio-economic data. If such flawed methods of analysis are not quickly exposed the risk is that they will harm the quality of debate in South Africa and thereby undermine our democracy.

Frans Cronje heads the IRR. Over the past eight decades, the IRR has produced tens of thousands of pages of social and economic data on South Africa in order to enhance the quality of public policy debate.