Part 9: Reporting student achievement in primary schools

9.1
Amendments to the Education Act 1989 (the Act) in 2001 require school boards of trustees (boards) to include in their annual report an analysis of any variance between the board's targets for what students would achieve (as set out annually in the school's charter) and what actually happened during the year.

9.2
The goal was that boards would prepare strategic plans, set targets, and report meaningfully on them – with a commitment to better outcomes for students informing all decision-making and resource allocation decisions at the school.

9.3
We assessed how well a sample of primary school boards reported on student achievement in the analysis of variance in their 2009 annual reports. We did so because we are interested in improving how public entities report on their performance and because the Ministry of Education (the Ministry) has not reviewed the effectiveness of the analysis of variance reports since the requirements were introduced.

9.4
Based on our work, we have produced a checklist for boards to use to assess and improve on the usefulness of their analysis of variance reports. We will send the checklist to schools in early 2011. The checklist is shown at the end of this Part.

9.5
Next year, we intend to build on this work. We will assess how well boards use information to underpin their strategic planning and self-review, and how coherently the school focuses its efforts on student achievement (including aligning resources, policies, and practices). We will also look at how well boards use and respond to assessment data.

Summary of our findings

9.6
Only 15% of the boards in our sample were largely meeting the requirements. These boards had related the annual targets to the board's strategic goals and better analysed what had or had not worked well this year, and on the basis of this information had planned what would be done next year.

9.7
We found that 72% of the boards in our sample were partly meeting the requirements for strategic planning and self-review. The analysis of variance reports prepared by these boards had clear annual targets and data measuring student achievement against the targets.

9.8
The remaining 13% of the boards in our sample had not met the requirements because the boards had not or only partly completed their analysis of variance reports.

9.9
There was wide disparity in the ease with which the boards communicated what they were doing in their analysis of variance reports. While some reports were very clear, others were quite difficult to understand. Boards need to think more carefully about their target audiences and tailor the information in the reports to their audiences.

About analysis of variance reports

9.10
According to the Ministry of Education (the Ministry), the analysis of variance is:

… a vital part of the board's annual report… [It] is a primary opportunity for the board to inform its community on how the school is performing in regard to student achievement against the targets set by the board in its strategic and annual plan. The presentation of the Analysis of Variance is a key indicator as to whether the board has student achievement plans in place, and is able to measure actual achievement against these targets.³⁹

9.11
The amendments to the Act in 2001 were aimed at making student outcomes a priority for boards. From 2003, boards have had to include three- to five-year strategic goals for student achievement in their school charters.⁴⁰ The school charters also have to contain annual targets for improved student achievement (or steps towards the strategic goal).

9.12
The Act requires each school to include in its annual report an analysis of any variance between the planned aims, objectives, directions, priorities, or targets (as set out in the school charter) and what the school has actually achieved during the year. The analysis of variance describes for the community how the school has addressed the board's priorities and how successful the school's approach has been. The analysis of variance should also provide the basis for plans for the coming year.

9.13
When the requirements for planning and self-review were introduced, it was expected that, once the process became embedded in board and management thinking, student achievement would inform all decision-making and resource allocation decisions within a school.

Content and format of the analysis of variance reports

9.14
The Ministry produced a series of circulars on "Planning for better student outcomes" during 2003. These circulars explained the new school planning and reporting process, the need for self-review and planning for improvement, and the reporting and accountability requirements. The circulars also provided examples of good planning and reporting practice.

9.15
There is no set format for the content of the analysis of variance reports.

9.16
We used the information in the circulars to extract what we consider to be a set of good practice principles. We use these principles to assess board performance in the detailed findings section of this Part (see paragraphs 9.28-9.50).

About our review

9.17
The requirements for strategic planning and self-review have been in place for seven years, and there has been no formal review of how well these arrangements are working. We noted in our June 2008 report⁴¹ on the Ministry's monitoring and support of boards that the Ministry was not systematically reviewing the analysis of variance reports to establish whether boards adequately understand and demonstrate their compliance with the National Education Guidelines.

9.18
When we provide the information in this Part to schools, it will be the first feedback boards will have received on the analysis of variance reports, so our intention is to:

• assess how well the reports are fulfilling their intended purpose;
• more clearly set out the principles of good practice for the content of the reports;
• provide constructive suggestions to boards on how they could improve their reports (illustrated with good practice that we have identified); and
• highlight matters that need care when boards are preparing the reports.

How we did our review

9.19
We selected a sample of 90 primary schools (both contributing and non-contributing⁴²) out of total of 1898 primary schools. The method that we used to select our sample was designed to cover all deciles⁴³ and sizes of schools. To do this, we obtained a list of all contributing and non-contributing primary schools from the Education Counts website.⁴⁴ The list included the number of children enrolled in each school and the school's decile. We then split the schools into three groups – large (those schools with 300 enrolled students or more), medium (those schools with 100 to 299 enrolled students), and small (those schools with fewer than 100 enrolled students). Within each group, we selected 30 schools, including schools from each decile and also from the different regions in New Zealand.

9.20
We assessed the quality of each school's analysis of variance report for the year ended 2009. We were particularly interested in whether a reader of the analysis of variance report (we took the view of a parent) could easily:

• identify the board's strategic goals and know from reading the report why the board had chosen these strategic goals;
• identify the annual targets and how these annual targets relate to the strategic goals;
• assess what outcomes had been achieved for the year and know what progress had been made;
• understand what had been done to achieve these results, whether the planned action had worked, and if not why not; and
• clearly understand what the board would be doing next year in response to its analysis of the results and be satisfied that the board was taking the appropriate action to achieve its strategic goals.

9.21
We did not read the schools' charter documents, because they are not published in the annual reports and most schools do not publish them on their websites. Therefore, the analysis of variance report has to stand alone and be able to be read and understood without the school charter.

Our overall findings

9.22
The quality of the analysis of variance reports prepared by boards in our sample varied widely. Some of the boards had clearly put a lot of thought and effort into preparing the analysis of variance report, while others appear to have treated it as more of a compliance exercise. We did not see any noticeable difference in the quality of the analysis of variance report between different sizes and deciles of schools.

9.23
We found that 15% of the boards in our sample had very good analysis of variance reports. In reading these reports, we considered that they had embraced the requirements for strategic planning and self-review. Their analysis of variance reports were focused on informing and engaging parents and other community members in what the school was trying to achieve for students and the progress the students were making. These reports had clear strategic goals (although the goals could be better linked to the charter). The board had set annual targets that were relevant to these goals. The reports detailed the action that the school planned to take to achieve the targets and analysed the student outcomes to establish whether the action was effective. The results from the current year were then used as a basis for the next year's planning.

9.24
We found that 72% of boards in our sample were partly meeting the requirements for strategic planning and self-review. The analysis of variance reports prepared by these boards had clear annual targets and data measuring student achievement against the targets. However, to fully meet the requirements, these boards needed to better relate the annual targets to their strategic goals and better analyse what did or did not work well this year and, on the basis of this information, plan what would be done next year.

9.25
We found that 13% of boards in our sample had not engaged with the requirements for strategic planning and self-review. For example, one board had not prepared an analysis of variance report for 2009 but had used the 2007 version and dated it 2009. In another four instances, the boards had prepared annual targets but had not reported what they had achieved against the targets. In two other instances, the boards' analysis of variance reports comprised pages and pages of student achievement data. However, the boards had not provided any targets or any analysis to help the reader interpret what the annual achievement data actually meant.

9.26
There was wide disparity in the ease with which the boards in our sample communicated what they were doing in their analysis of variance reports. Some reports were easier to read and understand than others.

9.27
The boards that we reviewed were using a wide variety of styles and formats for the analysis of variance reports. Some boards preferred a narrative report, while others used a table. We did not find that any particular style worked better than another. What was important was:

• that the board targeted the contents of the analysis of variance report to the parents and community members;
• whether the analysis of variance report contained meaningful headings to step the reader through the report (for example, one board had used the following headings: Why this target? What did we do? How did we resource it? What did we achieve? What do we believe made the difference? Where to next?);
• that any graphs and tables used were kept simple and included a brief explanation of what the data meant;
• that boards were frank about student achievement and had analysed why a programme had or had not worked; and
• that a plan for the next year had been established and the reader was satisfied from the data and analysis provided that this was the appropriate thing to do.

Identifying the board's strategic goals and why those strategic goals had been chosen

9.28
We consider that information about the board's strategic goals should be included in the analysis of variance report so that the report can be read without the reader having to locate a copy of the school's charter. During our review, we noted that most of the schools in our sample did not have a copy of their charter on their website.

9.29
We found that 58% of the boards in our sample included their strategic goals in their analysis of variance reports. However, only 15% of these boards (9% of the total sample) had established a link between the strategic goals in their analysis of variance reports and the strategic planning section in their charters.

9.30
Most often, the link to the strategic goals was made through an introductory section in the analysis of variance report. This section included a brief statement that the goals were contained in the school charter, why the goals had been chosen (for example, through an analysis of the available achievement data), and who had been consulted in deciding on the goals (for example, staff, educational providers, and parent groups).

9.31
One school included the annual planning section of its charter at the beginning of its analysis of variance report. Another school included the school's vision statement, its values, and its goals, with a target established for each goal. We consider that this is a good way to set the tone of the analysis of variance report, because it provides the reader with an understanding of the values that the school considers important.

9.32
We found that 58% of the boards in our sample had included goals. However, only 3% of those boards had set measurable goals. For example:

• Students who achieve successfully in literacy, to National Standards expectations.
• Raise student achievement in mathematics to National Standards.

9.33
For most of the measures chosen by boards, it was difficult to establish how the board would know when the goal had been achieved. Most of the strategic goals set by boards included measures to "increase", "improve", "develop", and "maximise" student achievement. However, because there was no target level, it was difficult to determine how boards would know when their goals had been achieved. We also noted a small sample of boards that had chosen strategic goals that would be more appropriate as part of the school's vision or mission statement. For example:

• To have high quality student achievement.
• To develop essential knowledge and skills to enable all students to progress and succeed.
• Quality learning comes with quality teaching.
• To have well developed literacy skills which will enable them to understand and become more aware of the world around them.
• Our children to develop strong skills and understanding in literacy and numeracy with students able to see the relevance to everyday life.

Identifying the annual targets and establishing how these annual targets relate to the strategic goals

9.34
We were pleased to find that about 87% of the boards in our sample had set annual targets and included the targets in their analysis of variance reports. In fact, one board had gone as far as choosing two targets:

• A challenging target – 100% of students will be working at their age-appropriate stage.
• A comfortable target – 80% of students will be working at their age-appropriate stage.

9.35
Of the 13% of boards in our sample that did not have targets, 44% had measured the student achievement level at the beginning of the year and then again at the end of the year and reported the difference. There was, however, no expectation about what the level of improvement would be. Thirty-three percent of these boards had reported student achievement data at the end of the year, and the remaining 23% of these boards had no targets or student achievement data.

9.36
Most of the boards in our sample chose student achievement targets to improve literacy (these included reading, comprehension, and writing) and to improve numeracy. However, about 26% of the boards in our sample chose targets in other curriculum areas – for example:

• fitness/swimming;
• self-management; and
• behaviour modification/emotional needs.

9.37
The most popular other targets were Information and Communication Technologies (ICT) and interacting with the environment.

9.38
Of boards that had set strategic goals (58% of the total sample), 67% had targets that measured their strategic goals.

9.39
Some schools had difficulty setting targets to measure their strategic goals because their goals were at too high a level or too wide. For example, one school had a strategic goal "To develop essential knowledge and skills to enable students to progress and succeed". The target set was "To significantly reduce, or eliminate, the groups of children who are well below their chronological reading age". In another example, the board had set a target "For all cohorts to make 10% increase in children at or above expectations in school-wide data gathering". However, they had only measured numeracy.

9.40
Of the 87% of boards that had set annual student achievement targets, we consider that 17% of them would have had difficulty in measuring whether the targets had been achieved because the targets were not quantified. Most of these boards had used targets that involved "increasing", "reducing", or "improving". For example, a board had set the target "we will improve student achievement in written language by the end of the year". In our view, this target could have been improved by identifying the level of expected improvement. For example, another board had set out the expected level of student improvement. The board's goal was "that at least 75% of our year 2, year 4 and year 6 children will be displaying accurate sentence structure at or above the expected national level by the end of 2009".

9.41
We noted examples where boards had set targets to "achieve at or above expectations", or measured targets as a percentage of "expectations". Because the board did not state what the level of expectations were, the reader was not able to tell how well the children were achieving.

9.42
We found that 14% of boards in our sample had used student achievement data from the previous two to three years in setting their annual targets, 40% of the boards had used data from the current year, and 21% used data from the 2008 year (this data was student achievement data from tests conducted in February or March of the current year or the end of the previous year). We were concerned to note that a quarter (25%) of boards in our sample had not used historical data when setting their targets.

9.43
Only 34% of the boards that had not used historical data were able to measure student achievement. This was because the target was a percentage achievement by the end of the year. For example, one board had a target "That 90% of year 3-6 children will be able to read and show understanding of material at or above their chronological age". This was able to be tested and measured at the end of the year. The remaining 66% either did not have an annual target or the achievement of the annual target was not able to be measured because the target was based on "increasing" or "improving" on current student achievement and no baseline data was provided.

9.44
Only one board in our sample had included longitudinal data (covering the past 10 years) in its analysis of variance report. Because the strategic goals are based on a three- to five-year period, the annual targets and measure of results should reflect movement towards the strategic goals.

Assessing what outcomes had been achieved for the year and knowing what progress had been made

9.45
In 62% of the boards in our sample, the outcome of what had been achieved against the targets was clear and easy to understand. For 13%, we were able to determine what the outcomes were after interpretation or analysis on the reader's part, but for 17% the outcomes could not be determined.

9.46
For the remaining 7% of the boards in our sample, the boards had not recorded any outcomes in their analysis of variance reports.⁴⁵

Understanding what had been done to achieve these results, whether the planned action had worked, and if not why not

9.47
Just over half of the boards in our sample included details of what action they had taken to achieve their targets in their analysis of variance reports. We also considered that we could work out what had been done, with some difficulty, in a further 6%. However, 38% of the boards in our sample did not have any details of what was done.

9.48
We were disappointed to find that only 30% of the boards in our sample had provided any analysis of what actions had worked or not worked and why. We consider that this is important because investigating the reasons for the differences between the actual student achievement outcomes and what was planned provides the board with a basis for deciding whether, and in what areas, professional programmes and resources need to be different in next year's plans. It is an important aspect of the school's ongoing cycle of self-review.

Clearly understanding what the board would be doing next year and being satisfied that appropriate action would be taken

9.49
One of the main purposes of the analysis of variance report is to provide the basis for plans for the coming year. We were disappointed to find that only 50% of boards had set out what was planned for the next year. Two of the boards had gone as far as setting out the board's plans and targets for the 2010 year.

9.50
Of the 50%, a half of the boards had provided enough analysis or explanation of the future plans to convince the reader that it was the logical course of action to take.

Checklist for boards of trustees – good practice for analysis of variance reports

39: In the Finance Education circular for the period 31 October 2008 to 31 October 2009.

40: The Education Act 1989 requires every school to have a charter. The charter communicates the school’s vision and direction, its goals for the long and short term, and its approach to meeting its legal responsibilities. It also outlines the school’s approach to meeting its legal responsibilities and to meeting national and local priorities for education.

41: Ministry of Education: Monitoring and supporting school boards of trustees (June 2008), www.oag.govt.nz.

42: Primary schools that cater for students from years 0 to 8 are called non-contributing schools or full primary schools. Schools that cater only for students to year 6 are called contributing schools.

43: A school’s decile indicates the extent to which it draws its students from low socio-economic communities. Decile 1 schools are the 10% of schools with the highest proportion of students from low socio-economic communities. Decile 10 schools are the 10% of schools with the lowest proportion of these students.

44: See www.educationcounts.govt.nz.

45: For rounding reasons, the percentages add up to 99 rather than 100.