Part 3: Assuring the consistency and quality of internal assessment

3.1
In this Part, we look at:

• external moderation, including the appeals process, moderator/teacher agreement rates, and training and support for moderators; and
• how NZQA reviews schools' administration of internal assessment for NCEA – including the role of the SRM, review of school achievement data, Managing National Assessment reviews, and working with schools to resolve issues.

Summary

3.2
Overall, NZQA has effective systems to support the consistency, quality, and effectiveness of how schools carry out internal assessment. NZQA monitors the quality of internal assessment in schools in many ways. These methods generate useful information. NZQA uses the information well to measure the consistency and appropriateness of internal assessment and to identify and address issues within schools.

3.3
In 2010, teachers and NZQA (through its moderators) agreed on 91% of assessments where students were awarded credits towards an NCEA qualification. Moderator and teacher agreement rates are improving at the level of the grade awarded (see paragraph 3.35). In 2010, the agreement rate was 84% (compared with 72% in 2008).

3.4
Recent changes to external moderation allow NZQA to target resources at the areas where the most support is needed – that is, in improving the assessment skills of teachers in schools where the quality of internal assessment is poorer. These changes show how NZQA has evaluated its practices and made ongoing improvements to better support internal assessment.

3.5
NZQA has a good framework to help schools maintain or improve their internal assessment capability. There are opportunities for NZQA and teachers to work with each other in ways that support ongoing improvements in internal assessment. These include feedback to teachers on their assessments of students' work and regular reviews of schools' quality systems. Schools use these reviews to request help from NZQA.

3.6
Overall, we consider that NZQA responds to issues within schools as they arise. NZQA deals with matters of poor performance professionally and collaboratively. NZQA can, and does, act if a school has not met its requirements for managing internal assessment.

External moderation

3.7
External moderation is one of the ways that NZQA monitors the consistency and quality of internal assessment. This involves NZQA moderators (teachers with subject expertise) checking the quality of teachers' assessments of students' work.

3.8
NZQA employs 34 full-time-equivalent moderators (National Assessment Moderators) and 235 part-time moderators. Most moderators are current or recent teachers and all are experts in particular subjects.

3.9
NZQA introduced the National Assessment Moderator positions in 2007/08. Before this, NZQA employed teachers on a part-time basis to do external moderation work (that is, reviewing the work of other teachers to ensure that it was fair and valid). Most people we spoke with during our fieldwork credited the introduction of full-time National Assessment Moderators with bringing greater consistency and quality to the moderation process.

3.10
Through external moderation, NZQA is able to:

• give feedback to teachers to help them assess students' work;
• identify aspects of assessment practice within schools, or particular subjects within schools, that could improve;
• provide assurance to schools and teachers that assessment practices are robust; and
• calculate moderator/teacher agreement rates (a measure of the extent to which moderators and teachers agree on whether samples of student work meet the applicable standards – see paragraphs 3.33-3.37).

3.11
Since the introduction of NCEA, the number of standards selected for external moderation has varied. Before 2008, about 3% of assessment tasks and teachers' assessments of student work were moderated. In response to public concerns about the credibility of internal assessment for NCEA, this was increased to 10% between 2008 and 2011. This equated to 154,000 samples of student work being externally moderated in 2011.

3.12
In September 2011, Cabinet approved a decrease in the amount of moderation required from 2012 to around 6.4%, or 100,000 samples of student work. NZQA's senior statistician told us that the smaller sample size can give a robust measure of consistency.

3.13
The reduced number of samples of student work to be moderated in 2012 reflects the greater moderator/teacher agreement rate in 2011 (87%), compared with the agreement rate in 2008 (69%) when the amount of moderation was increased.¹¹

3.14
Assessment tasks that teachers devise currently have a 97% agreement rate, and NZQA considers that moderating those tasks is not a good use of its time. Changes to external moderation from 2012 have been designed to allow moderators more time to spend on improving the assessment skills of teachers in schools with poor moderation history or practices.

3.15
We consider that that these changes allow for more efficient and better targeted use of NZQA's resources. These changes show that NZQA is evaluating its practices and delivering improvements that allow it to support ongoing improvement of its external moderation.

Moderation reports

3.16
NZQA selects the standards to be moderated at each school.¹² Schools then submit samples of students' work to NZQA, and the moderators check whether they agree with the teachers' assessments.

3.17
Moderators write a formal moderation report for each standard that is selected for moderation at each school. The reports indicate how many of the teachers' assessments were accurate for each standard, and provide advice about those that were not.

3.18
Each moderation report indicated whether the assessment tasks were suitable for assessing the standard or whether modifications to those tasks were required.

3.19
Our survey showed that 84% of teachers in our sample have had some of their assessment work moderated since January 2010.

3.20
Figure 4 shows that most teachers in our survey considered that moderation reports were:

• easy to understand;
• useful in providing feedback about assessment decisions;
• useful in providing feedback about assessment tasks; and
• written in a tone that was supportive and professional.

Figure 4
Teachers' feedback to our survey question: "Please indicate how much you agree with the following statements about the moderation reports you have received since January 2010"

Note: Percentages may not add to 100% because of rounding.

3.21
Our survey also invited teachers to comment on any moderation reports that they had received since January 2010. There were comments from 522 teachers. About one-quarter of the comments stated that moderation reports were useful and had improved in recent years. One teacher's comments reflected those of others who favoured these reports:

Very good. Informative and forward looking. The moderators know their work which is great. The feedback is very easy to interpret and assimilate into our future planning.

3.22
However, teachers responding to our survey also said that they would like to see improvements in moderation reports. These comments were similar to what we heard from teachers in the five schools we visited. The most common criticisms were that:

• moderator feedback was inconsistent from one year to the next; and
• the feedback pointed out problems but did not give advice on how to fix them.

3.23
As part of our audit, we reviewed 200 moderation reports. In general, we found moderation reports were of a good quality.¹³ We considered that most reports were written in a tone that was helpful for teachers. However, we also considered that some reports could have given more advice or direction where problems were identified.

Consistency in moderation reports

3.24
NZQA knows it must give consistent feedback to teachers in moderation reports and has taken steps to improve that consistency. These include:

• introducing guidelines to get better consistency between subjects;
• moderation team leaders reviewing a sample of moderation reports from different subjects;
• moderators of different subjects meeting and working together to ensure consistency; and
• moderators peer reviewing each other's moderation reports (called "check moderation").

3.25
Feedback that we received indicates that introducing full-time moderators in 2008 is also likely to have helped to increase the consistency and quality of moderation reports. National Assessment Moderators' reports and Clarifications of Standards (see paragraph 2.6) aim to give consistent advice to teachers.

3.26
Consistency in the feedback given in moderation reports is important because teachers use the feedback to improve their assessment tasks. In our view, NZQA needs to continue to improve its processes for increasing consistent feedback for individual standards. For teachers, receiving contradictory feedback from one year to the next is confusing and frustrating. It could lead to teachers losing confidence in the external moderation process.

Appealing a moderation report

3.27
If a teacher disagrees with aspects of a moderation report, they can either ask NZQA for clarification or appeal the decision.

Formal appeals

3.28
When a moderation report is appealed, a different NZQA moderator reviews the moderation report to establish whether the report was accurate or whether any changes are required. In 2011, there were 17 requests for clarification of moderation reports and 54 formal appeals of moderation reports. Thirteen of the appeals resulted in a change to the moderation report. If a teacher is dissatisfied with the outcome of an appeal, schools can ask for a review of an appeal. This did not happen in 2011.

3.29
In our survey responses, 83 respondents (6%) had appealed a moderation report since January 2010. Of these 83 respondents, 42% were satisfied with the outcome of the appeal and 25% were neither dissatisfied nor satisfied. The remaining 33% of teachers were dissatisfied with the outcome of their appeal.

3.30
The most common reason for dissatisfaction with the outcome of the appeal was that teachers disagreed with NZQA's assessment or interpretation of the standard. There is always likely to be some disagreement, given that interpreting standards and assessing students' work must involve some subjectivity.

3.31
Other survey respondents were dissatisfied because they considered that the appeals process took too long. NZQA told us that the usual time for an appeal is three weeks (although some appeals took longer to process in 2011 because of the effects of the Canterbury earthquakes).

3.32
We consider that the appeals process is useful in encouraging dialogue between NZQA and teachers and for allowing teachers the opportunity to seek clarification on NZQA's feedback. The process supports the ongoing improvement of internal assessment in schools. In our view, NZQA could consider whether it needs to improve the timeliness of the appeals process, or improve its communication with teachers about how long the appeals process takes.

Moderator/teacher agreement rates

3.33
Data from external moderation is used to calculate agreement rates – that is, the extent to which moderators and teachers agree on whether samples of student work meet the standards. In recent years, agreement rates have increased.

3.34
NZQA calculates two agreement rates:

• agreement at the level of credit – that is, whether the teacher and the moderator both assess that the student has met (or not achieved) the standard; and
• agreement at the level of the grade – that is, whether the teacher and the moderator agree on the specific grade (achieved, merit, or excellence) that the student was awarded for that standard.

3.35
In 2010, for all standards at all levels, moderators agreed with 91% of teachers' assessments at the level of credit. This was up from 83% in 2009, and was higher than in any previous year. Agreement rates for awarding at the level of credit are always higher than agreement rates at the level of the grade. This is because fewer assessments are involved when deciding whether to award credits. For many standards, there can be a fine distinction between awarding achievement and merit, or between merit and excellence.

3.36
In 2010, the overall moderator/teacher agreement rate for students' work at the level of the grade was 84%. This figure was 76% in 2009 and 72% in 2008.

3.37
Although there are some limitations when comparing agreement rates between years,¹⁴ it is possible to note steady improvement at both the level of credit and at the level of grade. This shows that teachers and schools, with the support of NZQA, are becoming more consistent in assessing students' work.

Training and support for moderators

3.38
Moderators are well-qualified teachers with recent teaching experience and expertise in a particular subject. Moderators receive training when they start work with NZQA. The moderators we interviewed considered that they received effective training and ongoing support from NZQA.

3.39
NZQA has a system for peer reviewing moderation reports within subjects. Some moderators we spoke to considered that the peer review process (called "check moderation") offered good support for their work and affirmed their decision-making for future moderation work.

Reviewing how schools administer internal assessment

3.40
NZQA works directly with schools to ensure that the school has effective internal quality assurance policies and procedures and resolves any issues as they arise.

School Relationship Managers

3.41
NZQA employs 11 School Relationship Managers (SRMs). Each has a portfolio of 25-50 schools. The role of an SRM is to liaise with schools about how the schools are administering NCEA. SRMs follow up on any issues identified in moderation reports and other analysis of each school's internal assessment performance. They also help schools to plan how to address any issues.

3.42
SRMs regularly review the quality of schools' internal assessment policies and procedures. The results of these reviews are "Managing National Assessment" reports, which are published on NZQA's website.

3.43
A school (usually through the Principal's Nominee) can readily contact its SRM with questions or issues about internal assessment. SRMs will usually visit each of their assigned schools at least once a year. NZQA surveyed Principals' Nominees in 2010 and received positive feedback about SRMs.

3.44
School staff in the five schools we visited reported positive relationships with their SRMs. We were told that SRMs work in partnership with schools to resolve issues and are responsive to schools' requests for information and advice.

3.45
We asked principals and Principals' Nominees at the five schools we visited to rate their interactions with NZQA using NCEA grades. All five schools rated the SRMs highly – merit or excellence. We do not often hear such positive feedback from stakeholders about the public entities we are auditing. We commend NZQA and its SRMs for the positive relationships they have with, and the improvements they support within, schools.

Reviewing school achievement data

3.46
NZQA compares each school's results from internal assessment with its results from external assessment for each standard in each subject. Any school's achievement rates for internal and external assessment will differ from the overall national achievement rates (for internal and external assessment, respectively). However, NZQA expects that each school will broadly reflect the national achievement rates. If a school's internal assessment results differ greatly from what is expected on the basis of its external results (and if the moderator/teacher agreement rates are well below the norm), NZQA will work with the school to improve its internal assessment policies and procedures.

3.47
SRMs monitor information from various sources about the schools that they work with. These sources include:

• the results of moderation reports for a school – the SRM will look for indications in the data of school departments that are performing poorly;
• outlier reports – statistical analysis of a school's data highlights where the results for internal assessment are outside the range expected, compared with the school's external results or compared with national internal assessment results (this may indicate a department is marking students too softly or too harshly); and
• "no not achieved reports" – data showing schools where no students have been awarded "not achieved".

3.48
NZQA uses data from these sources to identify potential issues for schools. When an SRM identifies a potential issue, they will investigate by talking to staff at the school, especially the Principal's Nominee, to find out what is happening, and work with the school to resolve the issues, if any.

Managing National Assessment reviews

3.49
A major component of NZQA's quality assurance of NCEA is an internal (school-based) moderation procedure. NZQA expects schools to have assessment policies and procedures to ensure that internal assessment is accurate, consistent, and appropriate (that is, the tasks are in line with the standards and allow students to demonstrate their ability). It expects schools to have effective and documented processes to ensure that they report reliable results from internal assessment to NZQA.

3.50
SRMs formally review each school's assessment procedures every one to four years. This is known as a Managing National Assessment (MNA) review. The result of this review is an MNA report, which is published on NZQA's website.

3.51
How often a school has an MNA review depends on the school's previous performance in complying with NZQA's requirements. Schools that NZQA identifies as struggling have an MNA review every year. Those that NZQA considers able to identify and deal with any issues well are reviewed every four years. Between MNA reviews, SRMs monitor their assigned schools' performance. They talk to teachers at their schools about potential issues and do not wait for the next MNA review to address it.

3.52
As part of an MNA review, SRMs check the school's assessment policies and procedures, including whether:

• the school is meeting its consent to assess;¹⁵
• the school has appropriate internal moderation processes and effectively manages external moderation processes; and
• teachers know and comply with school policies.

3.53
Part of these reviews involves checking that the school follows up on issues identified in moderation reports.

3.54
The five schools that we visited told us that the way that they self-assess to prepare for an MNA review helps them to identify where they must improve. NZQA chooses some areas of focus for the review, and the school may nominate departments for MNA review. Schools use the ability to nominate departments to seek NZQA's help for fixing problems in specific subject areas.

3.55
At the five schools we visited, those chairpersons of Boards of Trustees whom we spoke to valued the assurance that NZQA provided in the MNA reviews. The chairpersons and other school staff considered that the MNA review process was well run. School staff also considered that the SRM gave them enough information about the problems that had to be fixed and help with how to fix them.

Working with schools to resolve issues

3.56
As mentioned in paragraphs 3.47-3.48, SRMs often identify issues that arise within schools by reviewing schools' achievement data, moderation reports, and MNA reviews.

3.57
If a school has not complied with NZQA's requirements for managing internal assessment, SRMs can take a range of actions depending on the situation. These include:

• requiring the school to submit more material for moderation;
• stopping the school from submitting student results for the subject(s) in question; and
• since August 2011, withdrawing a school's consent to assess students' work.¹⁶

3.58
NZQA told us that, in recent years, it has rarely used its full non-compliance proceedings against a school. Sometimes NZQA has blocked a school from entering student results into NZQA systems until certain requirements of its consent to assess have been met. When NZQA has done this, it has found that the school works quickly to fix problems. School senior managers have often supported this action because it helped the school to address the issue. NZQA takes such extreme action only after it has made considerable effort to solve the problem.

3.59
The role of the Principal's Nominee is important for ensuring that the school is administering NCEA well. It is important that the Principal's Nominee has the right level of influence over the other teachers at the school and, when required, can encourage other teachers to comply with processes.

3.60
NZQA has no formal way to influence how the school appoints its Principal's Nominee. However, the SRM can discuss this appointment with the principal and say whether they consider that the Principal's Nominee is adequately carrying out the role. Figure 5 describes how NZQA worked with a school to resolve issues related to a Principal's Nominee who was performing poorly.

Figure 5
How the New Zealand Qualifications Authority worked in partnership with a school to address issues related to a Principal's Nominee who was performing poorly

3.61
Figure 6 describes the steps that NZQA can take to help a school to improve its internal assessment capability.

Figure 6
An example of the New Zealand Qualifications Authority dealing with an issue about internal assessment capability

3.62
Overall, we consider that NZQA is responsive to issues as they arise within schools. NZQA deals with issues professionally and collaboratively and has an effective framework to help schools maintain or improve their internal assessment capability.

11: There are agreement rates for the grades that students achieve and agreement rates for whether a student is awarded credits (see paragraphs 3.33-3.37). The percentages discussed here are for the combined grade and credit agreement rates.

12: NZQA has a process for selecting standards to moderate. It chooses different standards each year.

13: We looked at whether the moderation reports were, in our view, clear, complete, concise, and consistent.

14: One limitation, for example, arose in 2010, when there was a focus on moderating standards that teachers were having the most difficulty with when assessing students' work.

15: Having "consent to assess" (formerly referred to as accreditation) allows schools to assess against standards within the scope of the consent granted. See "Consent to assess" on NZQA's website (www.nzqa.govt.nz).

16: If either of the last two options were to occur, provision would be made for students' work to be assessed and quality assured by another school.