Part 3: Information currently used and its limitations
3.1
In this Part, we discuss:
- what achievement and student information is currently used;
- gaps and issues with the quality of the information; and
- limitations with student management systems.
Summary
3.2
In some areas, the Ministry, agencies, and schools use information well. This includes achievement, participation, and engagement information.
3.3
However, there are currently significant gaps in the available information. This includes a lack of cost information, particularly for initiatives targeted at improving Māori student achievement. Also, information about Māori enjoying educational success as Māori is varied.
3.4
There are also limitations with the schools' student management systems and some issues with data quality that restrict how well information can be used.
Main information currently used
3.5
As we discussed in Part 2, the main sources of achievement information are National Standards and NCEA. The Ministry and NZQA administer this information, which is readily available and used throughout the education system.
3.6
Information about the number of students enrolled in schools is available. However, the Ministry and NZQA use different forms of enrolment information that involves different students. NZQA's data does not include students enrolled in schools not registered with NZQA.
3.7
The Ministry and NZQA also use different definitions to measure and report on NCEA achievement. The different reporting methods are used for different purposes. In our view, to support a "joined-up" education system working towards common goals, the different uses for the information needs to be made clearer.
3.8
Schools collect and use students' attendance information. Well-performing education systems have high levels of attendance and retention. Attendance and retention are strongly correlated to educational success.
3.9
The Ministry also collects information on suspensions and expulsions of students (which measures student engagement).
3.10
All this information is useful for the education sector to understand the achievement and performance of Māori students. This information is typically available in student management systems.
Soft information used in schools
3.11
Schools have a mixture of hard and soft information for informing their work on, and tracking their progress with, Māori students' achievement. The importance of soft information to provide a more holistic view of student success is nicely summarised by the statement from one of the school staff we spoke with, who said that "There's a story behind the data." The stories are the broader information that schools value and use in their day-to-day practice.
3.12
Some better-performing schools used softer information more effectively than some of the poorer-performing schools. Several schools were also looking for better capability in their student management systems to record this sort of information. This softer information provides important context about circumstances that could affect a student's learning.
3.13
The Ministry's guidance does not focus on how to effectively use soft information. Figure 16 sets out some examples of the types of information collected by the schools we visited. Schools do not necessarily formally record all of this information.
Figure 16
Examples of information that schools we visited use to track Māori educational success
| Soft information | Hard information |
|---|---|
Feedback (face-to-face, by telephone, email, or other written communication such as a survey) from:
Some information from health, welfare, and educational support entities, and the Police |
NCEA results National Standards Some information from health, welfare, educational support entities, and the Police Enrolment information Some entry-assessment information from contributing schools Ministry-produced infographics Student self-assessment, including interests, hobbies, and goals Student observational and whānau surveys |
Source: Our school visits.
Gaps and limitations in information
3.14
There are two significant information gaps. These are information about Māori enjoying educational success as Māori and costing information for initiatives and programmes focused on Māori students. There are also issues with the collection of ethnicity data by schools.
Varied information about Māori students enjoying educational success as Māori
3.15
The overall goal of Ka Hikitia is to enable Māori to enjoy and achieve educational success as Māori.
3.16
Even though Ka Hikitia highlights Māori enjoying and achieving educational success as Māori, none of its measures are specifically about identity and culture. This raises the questions of:
- how the achievement of the overall goal of Ka Hikitia will be measured; and
- how the agencies contributing to the progress of Ka Hikitia can assess their performance.
3.17
In practice, maintaining and enhancing identity and culture as Māori depend on the context of the individual student, whānau, hapū, and iwi. Achieving as Māori means different things to different people.
3.18
This presents a challenge to the education sector in terms of the information that could be available and collected at the aggregate level to determine a holistic view of Māori achievement.
3.19
During our visits to schools, we asked what Māori enjoying educational success as Māori meant. We received a wide range of responses. In short, the responses can be summarised as what is valued by Māori as identified in the context of each school, student, and whānau.
3.20
One principal said that Māori enjoying educational success as Māori " ... is what we identify". A teacher told us "It is what Māori people value." A parent on a board of trustees said that it is when " ... the Māori aspect is part of the fabric of the school".
3.21
Figure 17 shows examples of what the schools we visited identified as Māori enjoying educational success as Māori.
Figure 17
Examples of Māori enjoying educational success as Māori
| Overall on par, and sometimes above par, performance of Māori compared with non-Māori Encouraging of identity and cultural activities Having a successful kapa haka group Children singing waiata every day Speaking successfully at a pōwhiri Being part of a kapa haka group Confidence in giving a mihi on a marae Achievement rates and positive choices after leaving school Achievement of defined academic, cultural, employment, and community competencies |
Source: Our school visits.
3.22
The better-performing schools we visited collected and used cultural information. This is a rich source of information. The challenge for the education sector will be to better collect this information at the aggregate level to inform and improve its decision-making. To improve the quality of this information, the range and quantity of guidance from the Ministry and other agencies on how to support and measure Māori enjoying educational success as Māori needs to improve. The guidance currently does not clearly and consistently set out what is required.
3.23
Schools could also learn from each other and share how they collect and use cultural information. Improvements in these areas would greatly enhance knowledge of cultural success and reduce this significant information gap.
| Recommendation 1 |
| We recommend that the Ministry of Education work with schools to establish a framework for collecting cultural information (for example, a student's ties with their iwi) and other information (for example, a student's goals and aspirations) about Māori enjoying educational success as Māori. |
| Recommendation 2 |
| We recommend that the Ministry of Education help those schools that do not have enough understanding about what Māori enjoying educational success as Māori means, by providing better guidance and information that they can use to measure Māori enjoying educational success as Māori. |
| Recommendation 3 |
| We recommend that the Ministry of Education use currently available information to investigate the variation in Māori educational achievement of similar schools in similar circumstances and help the lower-performing schools to do better. |
Limited cost information available
3.24
The education sector uses some cost information. This includes information from school financial statements on staffing, the day-to-day running of the school, property, and student-funded places.
3.25
Other cost information is available for some programmes the Ministry funds. However, this is limited. Overall, it is difficult to identify the direct costs and value for money of specific interventions. There are few examples of cost evaluations of programmes aimed at supporting Māori students.
3.26
The Ministry could not accurately identify the funding for all programmes focused at Māori students. Appendix 1 shows the many programmes and initiatives that support student achievement, including Māori student achievement. Attributing the effect of specific programmes on individual Māori students is difficult. However, it is important to identify what value these programmes and initiatives have added.
3.27
Because it is using public money to fund these programmes and initiatives, the Ministry needs to work out how much they cost, whether they are effective, and whether they add any value overall and to Māori students in particular.
Issues with data quality
Collecting ethnicity data is variable
3.28
Schools collect ethnicity data when a student enrols, and the Ministry uses this data to identify who Māori students are.
3.29
We wanted to test whether schools collect this data consistently. We reviewed a random sample of 50 online school enrolment forms. We saw large variations in the design of enrolment forms that schools use to collect basic student data, which resulted in variations in the quality of data collected.
3.30
Schools generally followed the Ministry's guidance on collecting ethnicity data, but there are varying approaches to collecting such data. In our view, the Ministry's guidance could be improved to better reflect Statistics New Zealand's best-practice guidelines on collecting ethnicity data.
3.31
Although nearly all schools asked an ethnicity question, most did not say why they collect this data. Many schools have no instructions on how to answer the ethnicity question. Those that do have varying instructions about how to answer the question. Schools also use free text fields widely, which contributes to categorisation and coding errors with this data.
3.32
The methods of collecting ethnicity data in schools raise questions about the quality of other data collected.
3.33
Inconsistent collection processes by schools might have a significant effect on Communities of Learning. This is an initiative where schools are expected to work together to share expertise in teaching and learning, and support each other.
3.34
At the local and regional level, improving the consistency and standard of ethnicity data collection would give the education system a more accurate picture of the number and distribution of Māori students. It would also improve the accuracy of ethnicity data for all ethnic groups.
Data quality is validated by the Ministry, but controls are variable in schools
3.35
Where the Ministry and NZQA collect data from schools, there are controls to ensure that schools comply with data standards for inputting and transmitting the data. This includes reminders on the checks schools need to do to ensure that they enter the data correctly. The Ministry also has computer checks to ensure that the data it receives is complete and appropriately recorded.
3.36
Although this level of validation ensures that data is good enough for reporting at an education system level, it poses questions about ensuring that data quality is of a high and consistent level at schools. Different standards of data collection by schools is likely to have an immediate effect at a regional and local level. This is especially so where schools are asked to work together in Communities of Learning because their collaboration requires information sharing. This view is supported by the inconsistent collection of ethnicity data outlined in paragraphs 3.28 to 3.34, together with the potential variability in National Standards assessment information.
3.37
During our fieldwork, some schools raised questions about the reliability of National Standards information. In particular, the large number of potential assessment tools, the lack of moderation of results, and the high level of discretion teachers have, have raised doubts about the reliability of National Standards information.
3.38
One school we visited ignored some National Standards information because the school was of the view that it was too unreliable. We cannot say how widespread this practice is, but it does raise questions about the use of National Standards information if schools perceive it to be unreliable. High-quality information is needed to ensure that all policies, including National Standards, can be implemented appropriately.
Limitations with student management systems
3.39
Schools use a wide variety of student management systems. Schools use these systems to collect student achievement, enrolment, and attendance information for use nationally and in schools.
3.40
Student management systems should have a profound effect on the way schools are managed and how effective teaching is. The OECD reports:
In the best cases, student data from a range of assessment resources is held within the SMS, follows the student from class to class and is used for reporting to parents, families and whānau. Teachers use aggregated student data to adapt and plan their classroom programmes, tailoring instruction according to student need. School leaders use school-wide aggregated data to investigate the effectiveness of school programmes and student learning, set targets for achievement, make resourcing decisions and determine professional development priorities.11
3.41
During our visits, schools highlighted the problems of not having consistent student management systems. They told us of the difficulty in exchanging information between different student management systems and in transferring existing information to a new system when changing to a new provider.
3.42
The student management systems currently in use do not always enable schools to use the information effectively.
3.43
One teacher responsible for maintaining a school's student management system told us that "A lot of what the Ministry of Education want[s] is different to what we produce."
3.44
The variety of student management systems do not always support schools to share information or to collect cultural and pastoral information. Student management systems that are able to interact with others might help to avoid the manual transfer of information that schools said is common for providing information about incoming students. Also, student management systems that are capable of collecting softer information could be used within schools and for national purposes.
3.45
The Ministry is aware of these issues and has set up a special project to improve student management systems. The Ministry has received and published feedback from users about the two main problems.
3.46
The first problem is that there is no common understanding of, or approach to, what student information is captured and how to capture it. The second problem is that there is no common model for the development of future student management systems.
3.47
In our view, it is important that the Ministry continues to progress its project to address these issues.
11: Ministry of Education (2010), OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes – New Zealand Country Background Report 2010, Wellington, page 66.