Equal Education supports the continuation of ANAs, but calls for major reforms

16 September 2015

When the Annual National Assessment was introduced in 2011, Equal Education welcomed this as an important step in monitoring the education system. We hoped the ANA’s could detect problems in teaching and learning at an early stage and, as the department noted then, could provide guidance for targeted interventions. We are still of the view that ANA’s should not be scrapped but redeveloped to give better results that can inform significant interventions in the education system.

However there are indeed very serious problems with the ANAs:

1. The ANAs cannot be compared across years, rendering them useless to monitoring educational progress.

Over the years we have realised that we cannot use the ANA’s to truly measure the state of education. The ANA’s, at present, cannot provide year on year comparisons, and cannot therefore be used diagnostically for targetted interventions. In an article in the Mail & Guardian in December 2014 educational researcher Nic Spaull wrote:

“The problem is that these tests are being used as evidence of “improvements” in education when the ANAs cannot show changes over time. There is absolutely no statistical or methodological foundation to make any comparison of ANA results over time or across grades. Any such comparison is inaccurate, misleading and irresponsible. The difficulty levels of these tests differ between years and across grades, yielding different scores that have nothing to do with improvements or deteriorations necessarily, but rather test difficulty and the content covered.

“On page 36 of [the Department's] 2014 report, it states: “Even though care is taken to develop appropriate ANA tests each year, the results may not be perfectly comparable across years as the difficulty level and composition of the tests may not be identical from year to year.”

“Every single reliable international and national assessment around the world uses [recognised] methods if they intend to compare results over time or grades, but not the ANAs. There are no common questions used across any of the ANAs, either grade to grade within one year of an ANA, or between the ANA cycles. Using the ANA results to talk about “improvements” or “deteriorations” has no methodological or statistical justification whatsoever.”

Between 2011 and 2012 there was a huge jump in the ANA results, this was meant to signify “improvement” and “progress”, but we dismissed this as improbable and asked whether the tests were set at the same difficulty level because the big jump did not make sense at all.

This means that we are unable to gauge what the real improvements or lack thereof have been by looking at the ANA results year in and year out.

2. External Moderation and Expert Input is not robust enough

The ANA’s were not externally moderated, but some attempts to address seem to have been made. But as they are administered solely by the DBE this can put into question the quality and authenticity of the results coming out. Furthermore, the DBE claims in their ANA reports that there is a reference panel of experts, but does not name who these people are, thereby rendering the process opaque.

3. Rapid Results Culture and the danger of high-stakes standardised testing

There is a culture of rapid results improvement fostered by the public, media and government, which is not helpful. This is most in evidence during the annual spectacle of releasing the matric results. These have become almost farcical because increasing numbers of students are being directed away from writing matric exams, and are being steered from Maths into Maths Literacy, in order to artificially inflate results. We need government to have a realistic discussion with the public based on three main themes: Firstly, there has been a significant improvement in educational access, throughput and outcomes since the demise of Apartheid. Government should set this out factually and fully. Secondly, educational improvement is gradual and expectations of rapid results actually hinder the ability to monitor, manage and intervene. Thirdly, government must set out a serious plan to close the educational inequality gap and improve quality – that currently does not exist in a serious form.

4. The ANAs can lead to teaching to the test

We agree with the unions that when there is more focus on testing and readiness for the ANAs, there is a decrease in the focus on the curriculum and usually we begin to see “gaming” because the teachers are under incredible amounts of pressure to performs and produce good test results. To support this claim, Campbells Law states that:

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

We also run the risk of collapsing the ANA’s if we continue to use them to call out underperforming schools and teachers. This claim is also supported by Goodhart’s Law which says:

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Turning the ANAs into a “high-stakes test” would create perverse incentives that undermine actual educational goals. The DBE needs to restate a very firm commitments that the ANAs are only ever going to be used for diagnostic purposes and not for so-called “high-stakes “ purposes which serve to reward or punish students or teachers.

We therefore make the following calls:

-We call on the task team that has been set up to study the best practices in the world on assessments to inform the reforms that will be applied to the ANA’s.

-We suggest that the ANAs be administered every 2 or 3 years like SACMEQ or PIRLS and TIMMS.

-We call on government to improve the working conditions of the teachers and learning conditions of pupils so that never again are teachers through their unions forced to threaten disruption of assessments in order to bargain for better working conditions and a better teaching and learning environment for their pupils. These struggles by teachers to improve their working conditions must be supported.

-We call on the unions to use this opportunity and their seat in the task team to propose better methods to administer the ANAs in order to make them work, and to recognise the importance of appropriate measurement, diagnosis, support and intervention.

Statement issued by Tshepo Motsepe, Equal Education General Secretary, 16 September 2015