Opening comments by Prof Sizwe Mabizela (Chair of Umalusi Council)
30 December 2011
ON THE OCCASION OF THE ANNOUNCEMENT OF APPROVAL FOR THE RELEASE OF THE FOLLOWING EXAMINATIONS
1. INDEPENDENT EXAMINATIONS BOARD (IEB) - NSC AND ABET
2. EKSAMENSRAAD VIR CHRISTELIKE ONDERWYS (ERCO) - NSC
3. DEPARTMENT OF BASIC EDUCATION (DBE) - NSC
Members of the media
Ladies and gentlemen,
1. Good afternoon, Absheni, molweni, dumelang, gotsong, goeie namiddag, sanibonani, ndaa, lotshani, thobela.
2. Welcome to this Media briefing in which we will make significant pronouncements on the outcome of Umalusi's quality assurance processes in respect of exit level examinations administered by the Independent Examinations Board (IEB), Eksamensraad vir Christelike Onderwys (ERCO) and the Department of Basic Education (DBE).
3. Today marks another significant milestone in the history of Umalusi.
4. You will recall that earlier this year, Umalusi took the unprecedented step of releasing the National Senior Certificate standardisation decisions. This action was necessitated by the heightened level of national interest in the work of Umalusi in quality assuring the results of the National Senior Certificate.
5. That public disclosure was important in ensuring that the general public had some insights into why Umalusi moderates learner performance and how that it does it.
6. Earlier this year, Umalusi Council took a decision that all standardisation decisions in respect of qualifications it quality assures will in future be made public.
7. This decision was not taken lightly, given that the international norm, standard and practice is to keep the raw mark moderation decisions confidential to protect the interests of candidates involved.
8. Before proceeding any further, allow me to explain Umalusi's mandate.
9. Umalusi is the Quality Council responsible for quality assurance in General and Further Education and Training. It is the accountable and certificating authority for assuring the quality of the following qualifications:
* The National Senior Certificate (NSC), which replaced the Senior Certificate; and
* General Education and Training Certificate (ABET level 4);
* the National Certificate Vocational NC(V); and
* the NATED N courses.
10. In addition to assuring the quality of the examinations administered by the Department of Basic Education and the Department of Higher Education and Training, Umalusi is also responsible for the quality assurance the examinations administered by two independent assessment bodies:
* The Independent Examinations Board (IEB); and
* The Eksamensraad vir Christelike Onderwys (ERCO).
11. Section 17 of the GENFETQA Act of 2001 (as amended in 2008) stipulates, inter alia, that:
(1) The Council must assure the quality of assessment at exit points;
(2) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has
(i) conducted the assessment free from any irregularity that may jeopardise the integrity of the assessment or its outcomes;
(ii) complied with the requirements prescribed by the Council for conducting assessments;
(iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and complied with every other condition determined by the Council.
(3) The Council must issue certificates to learners who have achieved qualifications or part qualifications.
12. From the above, it is clear that Umalusi is required to approve the release of results once it is satisfied that the examinations have been conducted in a fair, valid and credible manner.
13. In order to make this determination, there are various quality assurance processes that must be carried out. These are:
(1) Moderation and approval of question papers;
(2) Verification of the moderation of internal assessments;
(3) Monitoring of the conduct of examinations;
(4) Monitoring of the marking process;
(5) Verification of marking;
(6) Standardisation of the results.
14. As Umalusi, we would like to maintain the standard of the examination in all the subjects so that we would measure the real gains in quality of teaching and learning rather than false gains through examination papers of dubious quality.
15. Over the past few days, commencing on the 15th December 2011, Umalusi's Assessment Standards Committee has been hard at work standardising the examination results of the qualifications mentioned above.
16. Umalusi conducts not only the process of standardisation, which is required for national examinations, but it also, through rigorous procedures, assures the quality of the entire examination processes.
17. Gaining the approval of Umalusi Council for the release of the results is determined by the examinations' level of compliance with policies, directives and guidelines issued by both Umalusi and the Department of Basic Education.
18. Before such an approval is granted, Umalusi Council has to satisfy itself that no significant irregularities have occurred to undermine the integrity and the credibility of the examination processes.
The Standardisation process:
19. Let me briefly outline what the standardisation process is.
20. Any large-scale examination process invariably has many and varied sources of variability.
21. Most of these are unplanned, unintended, and undesirable. They vary from mistakes in a question paper to subtle issues of level of difficulty of a question paper and of possible multiple and valid interpretations of questions.
22. It is universally accepted that judging the level of difficulty of a question paper is a near impossible exercise. It is only after the paper has been written and marked that one is able to determine whether it was pitched at the correct level or not.
23. Standardisation of learner performance is an important quality assurance process used the world-over to mitigate the impact on learner performance caused by factors other than the learners' subject knowledge, abilities and aptitude.
24. There are therefore two main objectives for standardisation:
23.1. First, to ensure that a cohort of learners is not advantaged or disadvantaged by factors other than their knowledge of the subject, abilities and their aptitude;
23.2. Second, to achieve comparability and consistency of learner performance across the years.
25. In short, standardisation seeks to mitigate fluctuations in learner performance that are a result of factors within the examination processes themselves rather than the knowledge, aptitude or abilities of learners.
26. Umalusi's standardisation is done by the Assessment Standards Committee. This is a committee of Council which comprises men and women of impeccable credentials, personal integrity and credibility.
27. They are independent professionals who are not in the employ of Umalusi. They are appointed by Umalusi Council based on their extensive knowledge of, experience and expertise in statistical moderation, statistics, assessment, curriculum, and educational matters.
28. Some of them are affiliated with our universities and others serve on national and international bodies that deal with education and assessments.
29. Most of them have been doing this work for many years and have seen this process mature over time.
30. With your indulgence, I wish to publicly acknowledge and extend our sincere gratitude to the members of the Assessment Standards Committee. They are
* Prof Dan Kgwadi of North West University - Chairperson of the Committee
* Prof Paul Fatti (Professor emeritus, Wits University)
* Prof Moon Moodley (Professor emeritus, University of KwaZulu-Natal)
* Prof Sarah Howie, Professor at University of Pretoria;
* Prof John Aitchson, University of KwaZulu-Natal
* Prof Tim Dunne, University of Cape Town;
* Dr Nic Taylor, Research Fellow at JET;
* Mr Brian Shreuder, DoE, Western Cape;
* Dr Jeanne Gamble, University of Cape Town.
31. I also wish to express deep appreciation to Ms Fatima Dada, a member of Umalusi Council ExCo, who participated in this year's standardisation processes.
32. The standardisation decisions are informed by qualitative reports which include marking and moderation reports and the post examination analysis reports which provide comparative analyses of question papers over a period of three years.
33. Subjects are standardised individually, in a linear and non-iterative manner.
Principles of Standardisation
34. There are eight principles that guide the standardisation process. I will mention only two of these as they are particularly important.
(1) In general no adjustment should exceed 10% of the total mark of the paper or the historical average.
(2) In the case of the individual candidate, the adjustment effected should not exceed 50% of the raw mark obtained by the candidate. Put simply, this means that moderation of raw marks of individual candidates should not lead to the doubling or halving of a candidate.
35. It should be obvious from (1) and (2) above that one cannot have a fixed level of adjustment - an appropriate scaling operation must be employed in order to be consistent with these principles of standardisation.
36. In instances where raw examination marks are accepted, it is because
(1) First, the examination has been adjudged to be of appropriate level;
(2) Second, the learner performance is in line with the average historical learner performance profile in the subject;
(3) Third, the conduct and management of the examination in the subject were not compromised.
37. For the standardisation of the National Senior Certificate raw marks, we use the ogive (or cumulative frequency) curves to represent the cumulative frequency of learner marks. In constructing the ogive curve, we use the ‘less than or equal to' criterion.
38. Let us now turn to the results of the 2011 examinations:
1. Examinations administered by IEB
1.1 National Senior Certificate
* In total, 8 281 candidates sat for the National Senior Certificate examinations administered by the Independent Examinations Board.
* A total of 58 subjects were presented for standardisation.
* After moderation, raw marks were accepted for 45 the subjects. This figure represents 78% of the subjects.
* Moderation with minor upward shifts towards the average historical learner performance profile was effected in 5 subjects. This represents 9% of the subjects.
* Moderation with minor downward shifts towards the average historical learner performance profile, was made for 10 subjects. This represents 13% of the subjects.
* For detailed standardisation decisions, refer to slides.
Final declaration and release of results:
Having studied all the evidence at hand on the management and conduct of the examinations administered by the IEB, Umalusi is satisfied that nothing has compromised the integrity or credibility of the examinations process. We therefore conclude that the examinations were fair, valid and credible. We commend IEB for running a successful and credible examinations process. Accordingly, we hereby approve the release of the results of the National Senior Certificate Examinations administered by the Independent Examinations Board on a date to be determined by IEB.
1.2 General Education and Training Certificate - (ABET)
* In total, approximately 1 070 candidates sat for the GETC ABET examinations.
* Eight (8) learning areas were presented for standardisation.
* Following moderation, raw examination marks were accepted for all eight (8) learning areas.
* For detailed standardisation decisions, refer to the power-point slides.
Final declaration and release of results:
Having studied all the evidence at hand on the management and conduct of the examinations administered by the IEB, Umalusi is satisfied that nothing has compromised the integrity or credibility of the examination process. We therefore conclude that the examinations were fair, valid and credible. We commend IEB for running a successful and credible examinations process. Accordingly, we hereby approve the release of the results of the ABET Examinations administered by the Independent Examinations Board on a date to be determined by IEB.
2. Examinations administered by the Eksamensraad vir Christelike Onderwys
* In total, 1 273 candidates sat for the National Senior Certificate examinations administered by ERCO.
* A total of 21 subjects were presented for standardisation.
* After moderation, raw examination marks were accepted for 18 the subjects. This figure represents 86% of the subjects.
* Moderation with minor upward shifts towards the average historical learner performance profile was effected in 3 subjects. This represents 14% of the subjects.
* No downward adjustments were made.
* For detailed standardisation decisions refer to slides
Final Declaration and release of results:
Umalusi has the responsibility to safeguard the quality, standard, integrity and credibility of the National Senior Certificate. In doing so, we must ensure that learners are not unduly advantaged or disadvantaged by factors related to the conduct and management of the examination processes.
Umalusi has serious concerns about the Eksamensraad vir Christelike Onderwys' (ERCO's) ability and capacity to run a successful examination process. These concerns range from their ability to set question papers of appropriate standard to the actual management and conduct of the examinations process. Among other things, Umalusi has taken the decision not to accept the site-based assessment (SBA) marks of all ERCO candidates. To mitigate the adverse impact on learner performance, learner results will be computed without the SBA component.
In order to ensure a credible examination, Umalusi initiated a number of interventions in the period leading to and during ERCO's 2011 examinations. Umalusi also notes that ERCO used a substantial number of examination papers (39 papers in total) set by the Department of Basic Education and approved by Umalusi.
Notwithstanding the concerns raised above, Umalusi declares that the moderated marks are a fair reflection of learner performance in the examinations administered by ERCO. Accordingly, we hereby approve the release of the results of the National Senior Certificate Examinations administered by the Eksamensraad vir Christelike Onderwys on a date determined by ERCO.
3. Examinations administered by the Department of Basic Education
* In total, 620 266 (512 029 full time and 108 237 part time) candidates sat for the National Senior Certificate examinations administered by the Department of Basic Education.
* A total of 56 subjects were presented for standardisation.
* After moderation, raw marks were accepted for 45 the subjects. This figure represents 80% of the subjects.
* Moderation with minor upward shifts towards the average historical learner performance profile was effected in 3 subjects. This represents 5% of the subjects.
* Moderation with minor downward shifts towards the average historical learner performance profile, was made for 8 subjects. This represents 15% of the subjects.
* For detailed standardisation decisions, refer to slides.
Final Declaration and release of results:
Having studied all the evidence at hand on the management and conduct of the examinations administered by the Department of Basic Education, Umalusi is satisfied that nothing has compromised the integrity or credibility of the examinations process. We therefore conclude that the examinations were fair, valid and credible. We commend DBE for running a successful and credible examinations process. Accordingly, we hereby approve the release of the results of the National Senior Certificate Examinations administered by the Department of Basic Education.
Conclusion
39. Umalusi Council is pleased that the system is stabilising well and has reached a significant level of maturity as evidenced by the high percentage of subjects whose raw marks were accepted after the moderation process.
40. It is also worth noting that in the few instances where moderation shifts towards the average historical learner performance in a subject were approved, such adjustments were generally below the maximum of 10% provided for in the standardisation principles.
41. Umalusi has taken note of the intervention and improvement strategies implemented by both national and provincial education authorities and the positive impact of these on the overall assessment system as well as on teaching and learning.
42. On behalf of Umalusi Council, I wish to extend our hearty congratulations to all the learners who have done well in their examinations. Warm congratulations also to the teachers, parents, guardians and all those who have supported these learners in their studies.
43. To those who have not succeeded, there is another chance for you next year! Work hard and do well next year.
44. Wishing you all the best for 2012.
45. Thank you.
Prof Sizwe Mabizela
Chairperson of Council
Umalusi
Statement issued by Umalusi, January 3 2012
Click here to sign up to receive our free daily headline email newsletter
