Appendix 2: Our methodology for analysing the Ministry of Education's data
Request for data to the Ministry of Education
In September 2015, we decided we would carry out our own analysis of data held about schools by the Ministry of Education (the Ministry). We did this to provide assurance that data about school-level achievement of Māori students is accessible, able to be integrated with other (mainly financial) data, and able to be analysed to provide information about achievement.
We asked the Ministry for, and were provided with, aggregated achievement data and aggregated administrative data about schools from a range of different Ministry databases. The data was for 2014.
Importing and organising the data
Using secure transfer methods, we imported data into an Excel data model to three main tables. One table was about the characteristics of schools (for example, size, decile, location). Another table was about student achievement (at the school level, not individual students), and a final table contained aggregated school financial information. Other tables were added to the data model as the analysis progressed (such as aggregated information about school staff).
Data testing of results using Excel
We looked at the range and distribution of the data. We had earlier decided on an analysis plan after examining the achievement of Māori students within similar schools. We asked the Ministry to include aggregate data for Māori students by school for National Standards and NCEA results. We did not want to compare Māori and non-Māori students because our focus is on the performance of schools and the education system. Instead, we decided to look at whether Māori students succeeded consistently throughout schools. To support this reasoning, we identified an analytical way to compare schools so they were alike in important ways.
We chose decile, school type, and school size (roll) as a set of parameters to create groups of schools that were alike.
We grouped schools into three categories by decile: 1-3, 4-7, and 8-10. We chose these categories for consistency with our second Education for Māori audit report.
We combined school type (primary, secondary, composite, and special) and school size to produce 12 categories – small, medium, and large for each of the four school types.
Because of the "long-tailed" distribution of school size (many small schools, some with a school roll measured in single digits; many more schools with a few hundred students to a thousand or so; and a small number of very large schools, some with rolls in the thousands), we based the division of small, medium, and large for each school type on the cumulative percentage of the total school roll for each school type.
For example, we arranged primary schools in order from smallest to largest (according to school roll numbers as at May 2015, which were also made available to us) and then calculated the percentage and cumulative percentage that each school contributed to the total count of students. We then divided the cumulative percentage into thirds and gave the smallest third the label of "small primary", the middle third "medium primary", and the largest third "large primary". If we had divided the list of smallest to largest primary schools into three equal numbered groups, we would still have had quite a range of schools within each grouping, especially the schools with the largest number of students on their roll.
Figure 23 shows the number of schools in each category, by decile, that we used as the basis of our analysis. Some data is missing because some schools have not been assigned a decile rating.
Figure 23
Number of schools in each category of our analysis, by decile
| School type/size | Decile 1-3 | Decile 4-7 | Decile 8-10 | Missing data |
|---|---|---|---|---|
| Large secondary | 4 | 21 | 23 | 1 |
| Medium secondary | 14 | 44 | 27 | 2 |
| Small secondary | 68 | 87 | 34 | 41 |
| Large composite | 1 | 0 | 3 | 5 |
| Medium composite | 3 | 9 | 9 | 9 |
| Small composite | 64 | 26 | 9 | 28 |
| Large primary | 47 | 80 | 125 | 3 |
| Medium primary | 109 | 165 | 140 | 2 |
| Small primary | 406 | 519 | 336 | 29 |
| Large special | 2 | 4 | 1 | 0 |
| Medium special | 3 | 5 | 1 | 0 |
| Small special | 12 | 8 | 1 | 1 |
| Total | 733 | 968 | 709 | 121 |
Source: Our analysis of the Ministry of Education's data.
Figure 24 shows the distribution of the total school rolls and total Māori rolls for the 12 school size and type categories.
Figure 24
Student rolls by school category
| School type/size | Total rolls | % of total rolls | Total of Māori rolls | % of total Māori rolls |
|---|---|---|---|---|
| Large secondary | 90,874 | 12 | 13,149 | 7 |
| Medium secondary | 91,195 | 12 | 17,907 | 10 |
| Small secondary | 90,244 | 12 | 25,623 | 14 |
| Large composite | 17,620 | 2 | 2,474 | 1 |
| Medium composite | 17,455 | 2 | 2,580 | 1 |
| Small composite | 17,303 | 2 | 9,833 | 6 |
| Large primary | 149,841 | 20 | 26,185 | 15 |
| Medium primary | 144,845 | 19 | 35,055 | 20 |
| Small primary | 144,808 | 19 | 45,155 | 25 |
| Large special | 1,123 | 0.1 | 255 | 0.1 |
| Medium special | 974 | 0.1 | 242 | 0.1 |
| Small special | 976 | 0.1 | 295 | 0.2 |
| Total | 767,258 | 100 | 178,753 | 100 |
Source: Our analysis of the Ministry of Education's data.
Figure 24 shows that about 45% of Māori students attend a small school – whether that school is a secondary, composite, primary, or special school.
Our main intention was to test in a practical way whether the data could be used once it had been accessed. Our intention was not to research possible causes of, or contributors to, Māori student achievement. We also wanted to know whether the data we had asked for could be linked with financial and other data. We explored the practical considerations of making this linkage, and we consider that we have been able to generate new ideas and questions worth further investigation (see Part 2).
To start our exploration, we selected independent variables that we considered related to student achievement. We used the Chi-square function in Excel to examine relationships between the variables. We selected National Standards, NCEA, and "remained at school at the age of 17" as our dependent variables. We did not look at Ngā Whanaketanga Rumaki Māori results.
The dependent variables were the proportion of Māori students in a school who:
- were below, at, or above National Standards (an average rating of the three assessments made in reading, writing, and mathematics);
- were below, at, or above NCEA Level 2; or
- remained at school at the age of 17.
The first variable applied mainly to primary, intermediate, and some contributing schools. The last two applied mainly to secondary and some contributing schools.
The dependent variables and independent variables were usually organised as two by two, and sometimes as two by three, tables.
The Chi-square Test of Independence is a hypothesis test. We stated the null hypothesis and alternative hypotheses respectively as:
- H0: The data are consistent with a specified distribution – the expected frequencies.
- Ha: The data are not consistent with a specified distribution – the observed frequencies.
We set a high significance level of 0.001. This meant that we rejected the null hypothesis only when the likelihood of observing the relationship was equal to or smaller than 1 in 1000. Moreover, our intention was to explore and focus on performance throughout different dependent variables, so we looked for a consistency of the relationship and its direction between the independent variable and the dependent variables. We converted ratio variables, specifically the financial variables measured in dollar units, to equal categorical units by dividing the range of dollar values into thirds: small, medium, and large.
As we calculated the proportions of Māori students below or at and above the two main achievement categories, we noted a varied distribution of results throughout schools of the same decile group and school type and size. This invited further analysis, so we investigated individual decile levels. This investigation also showed a wide distribution of results between schools. This led us to make a qualitative selection of schools at either end of the distribution of achievement so that we could observe, in our school fieldwork, how they collect and use information.
School visit sample methodology
The first two dependent variables used in the data analysis (about National Standards and NCEA Level 2) formed the criteria to select schools for audit fieldwork. We categorised the proportions a school achieved in the 0-25%, 26-50%, 51-75%, or 76-100% range. To obtain a clearer picture, we included only schools with more than 30 Māori students in the calculation.
To maximise the chances of selecting schools that had the biggest differences, we labelled those schools categorised as 0-25% and 26-50% as "low" and those as 76-100% as "high". We then sorted the list of schools by location, decile, and school type to identify suitable candidate schools. We preferred schools in the upper North Island, to concentrate our fieldwork and reduce costs associated with travel and accommodation.
The sample schools were paired, but the pairing and reasons for it were kept to one member of the audit team. We did this so those visiting the schools could carry out their fieldwork without bias toward the selection criteria.
After we identified the pairs of schools, we sought advice from the Ministry and ERO about any reasons why we should not visit any of the schools in the sample – for example, because the school was under or about to be put under management intervention or an ERO review was scheduled at the same time. Two schools from an initial sample of 12 were identified as inappropriate to visit at the time, so the sample was reduced to 10 schools. The sample pairing was not exact, and we used some flexibility to find a match – for example, we matched a large primary school with a large intermediate school.
We contacted schools by telephone and email to ask them to take part in the audit. We asked to talk with people responsible for collecting, managing, and using information and those responsible for achievement, in particular Māori student achievement.
Including our pilot visits, we visited 13 schools between June and November 2015.