Part 3: Non-financial performance reporting by tertiary education institutions

Education sector: Results of the 2010/11 audits.

3.1
In this Part, we discuss our review of the performance information in the 2011–13 investment plans of most tertiary education institutions (TEIs).9 The performance information in these plans will be the basis of TEIs' reports of their service performance in their 2011, 2012, and 2013 annual reports. We note the legislative content requirements for the investment plans and discuss general requirements for performance reporting before outlining our preliminary observations.

3.2
For this review, our comments are limited to presentation and content matters rather than the overall appropriateness of the performance information. From next year, auditors will assess performance information within the context of their knowledge and understanding of the entity's strategies, operations, and priorities, and the performance management arrangements underpinning them.10

3.3
Overall, the presentation of performance or outcomes frameworks is a step forward for performance reporting by TEIs. We are aware of several initiatives under way to further develop performance information for the sector. We trust that our observations will be useful to TEIs as they make progress in this regard.

Background

3.4
The Education Act 1989 requires a TEI's investment plan to describe its proposed outcomes – what it will achieve through its education programmes and services – and the performance indicators it will use to measure whether those outcomes have been achieved. The annual report has to contain a statement that reports the TEI's performance in relation to those outcomes (a statement of service performance, or SSP).

3.5
Under generally accepted accounting practice (GAAP), an SSP reports primarily on output or service delivery performance – effectively, the services produced. The Tertiary Education Commission (the TEC) has clarified that:

Those parts of an institution's Plan that make up the forecast Statement of Service Performance will be expected to reflect the outcome framework agreed with TEC and be prepared in accordance with generally accepted accounting principles.11

3.6
The TEC has set out separate outcomes frameworks for universities and for institutes of technology and polytechnics (ITPs). Although aspects of the outcomes frameworks use slightly different terms from those used in GAAP, they capture the significant parts of a performance framework. The significant parts include inputs, services, the immediate effects of those services, and longer-term outcomes, together with associated performance measures.

3.7
TEIs' investment plans must include the following parts:

  • The Plan Context, which covers a variety of background information. For example, the TEI's response to government priorities in the Tertiary Education Strategy 2010–2015 (the Strategy), the needs of learners and industry and employers, and how the TEI will improve its educational performance.
  • The Summary of Activity, which sets out information on the services the TEI plans to deliver (commonly referred to as the "mix of provision") during the period of the plan.
  • The Performance Commitments, which sets out the proposed performance commitments and the associated indicators used in evaluating performance. TEIs are expected to use common "educational performance indicators" (EPIs).

3.8
From 2012, 5% of the Student Achievement Component funding received by TEIs will be based on their performance against mandatory EPIs. The EPIs, all related to student achievement, are: successful course completion; student retention; qualification completion; and student progression. Dimensions relevant to the Strategy, such as age and ethnicity, are presented for each indicator.

3.9
The TEC provided a template to TEIs for recording Performance Commitments. The template included the mandatory indicators, measures on participation of targeted priority groups,12 and measures relating to literacy and numeracy.

Performance frameworks

3.10
A non-financial performance framework, in both the forecast and historical (annual) reports, typically comprises information on the entity's outcomes, impacts, and related performance measures and targets, linked to outputs, performance measures, and targets. The reporting is about how well services are delivered and the effects of the services on the community.

3.11
In their investment plans, only a small number of TEIs presented performance frameworks that explicitly distinguished outcomes, impacts, and outputs and/or provided a diagram that identified these and indicated the links between them.

3.12
However, they discussed to varying degrees in the investment plans what they were trying to achieve and how they planned to do it. This made it possible to infer the formal performance elements (outcomes, impacts, and outputs), and the links between them, from the narrative (the performance story). For example:

  • planned courses are linked to the needs of local industry and employers - Waiariki Institute of Technology supported this discussion with a table that linked its planned programmes with historical and predicted employment trends; and
  • initiatives related to the quality of teaching are discussed within the context of influencing student achievement rates - Nelson Marlborough Institute of Technology stated:

    We aim to improve the educational performance of our learners by … improving teaching and learning. … For NMIT students, the most important indicator of quality is the standard of teaching and learning they receive. … The Teaching and Learning Observation programme is … designed to ensure maximum effectiveness of curriculum delivery by encouraging good practice within NMIT to be shared …

3.13
We encourage TEIs to consider presenting the SSP so that it clearly identifies the relevant aspects of performance (including costs of outputs), logically groups these aspects, and explicitly attaches measures to these aspects. Such a coherent, cohesive presentation of performance information helps the reader to understand the relevance of the performance information and is consistent with external performance reporting requirements (based on legislative requirements, GAAP, and the TEC's expectations). This information and these linkages would ideally be drawn from, and feed into, TEIs' internal strategic planning.

Outcomes and impacts – the difference tertiary education institutions make

3.14
The TEC-published outcomes frameworks relate high-level outcome measures (measures of whether outcomes are achieved) to economic development, social and community development, and environmental enhancement. The impacts – the more immediate differences a TEI will make from delivering its outputs or services – in TEIs' investment plans are different for universities and ITPs (see Figure 11).

Figure 11
Impacts, as described in the outcomes frameworks for tertiary education institutions

Universities Institutes of technology and polytechnics
People with critical thinking skills People progress to higher levels of learning
Educated and skilled workforce People progress to (self) employment
Value to society Value added to industry and society
More engaged communities
Enhanced environmental sustainability

3.15
In the TEIs' investment plans, the outcomes and impacts we found were often addressed (to varying extents) through the narrative rather than through formal outcome and/or impact statements.

3.16
In measuring their achievement of, or progress towards, the outcomes and impacts, we note that:

  • The TEC-published outcomes frameworks note that central government agencies will carry out research and analysis of system-level measures for TEIs' outcomes and impacts. An example of this is the Employment Outcomes of Tertiary Education project, which looks at the post-study earnings of recent participants in tertiary education and training. The feasibility study was a joint project by Statistics New Zealand, the Ministry of Education, the Department of Labour, the Inland Revenue Department, and the TEC.13
  • Consistent with this, it is likely to be appropriate that performance measures for individual TEIs focus on measuring lower-level impacts (such as the employment rate of graduates), but with commentary, linkages, and, where relevant, measurement of the expected contribution of these to the bigger picture.
  • In their investment plans, several TEIs presented a measure on the destination of graduates – such as the percentage of those who were in employment or further education within six months of graduation. At least one TEI commented on efforts to improve response rates to the survey. A number also presented a measure on the satisfaction of employers with graduates from their institute.
  • A new longitudinal study of university graduates has replaced the Graduates Destination Survey that questioned all New Zealand university graduates about their employment outcomes six months after graduation. The study aims to determine the ongoing effect of a tertiary education on university graduates.14 This may provide useful data for future performance reporting in TEIs' annual reports.
  • The mandatory EPIs on student achievement were the most commonly used measures in TEIs' investment plans. Student achievement measures best fit as impact measures given they are a consequence of service or output (that is, courses or programmes taught) delivery. In our view, the TEIs' performance stories should help the reader to understand how to interpret planned (and actual) performance against these measures.
  • At least some of the universities used impact measures for research. Measuring research outcomes seems to be a "work in progress". Some TEC guidance for universities discusses the links between the research produced by universities and economic, social, health, and environmental outcomes, and the importance of strong connections with firms to enable the use and commercialisation of research. The guidance says that the "TEC will discuss with universities how their research contributions to national goals can be best measured."15

Outputs – what tertiary education institutions produce or do

3.17
Based on the core roles of TEIs (see Figure 12), there is potential for fairly standard descriptions of TEIs' outputs in their (forecast) SSPs. Outputs are services delivered to external parties.

Figure 12
Core roles of universities and institutes of technology and polytechnics, as described in the Tertiary Education Strategy 2010-2015

Universities Institutes of technology and polytechnics
To carry out research that adds to the store of knowledge To deliver vocational education that provides skills for employment
To provide a wide range of research-led degree and postgraduate education that is of an international standard To carry out applied research that supports vocational learning and technology transfer
To act as sources of critical thinking and intellectual talent To assist progression to higher levels of learning or work through foundation education

3.18
In their investment plans, the forecast SSPs for ITPs typically consisted of the completed standard Performance Commitments template provided by the TEC. This template groups the measures under participation, educational performance, and financial performance. Some TEIs added other groups of measures into the template. There was no description of the output at the start of each group of measures.

3.19
Universities relied less on the standard template for presenting their performance measures but did not necessarily explicitly describe their "outputs". For example, about half of the universities set out their performance measures by objectives.

3.20
Although outputs did not tend to be explicitly identified as such, or described when presenting the performance measures, it was clear that they related primarily to teaching and (applied) research.

3.21
There are risks if outputs are not explicitly identified, including:

  • groups of internally focused measures, such as financial, asset, and resourcing measures, being incorrectly presented in the SSP;
  • a lack of explicit distinction between impact and outputs and their measures. This differentiation is important for internal performance management and for telling a complete and meaningful performance story, and for clearly depicting TEIs' accountability for their service performance (what they plan to deliver), as opposed to the effects of providing those services (over which TEIs have varying levels of influence), and for allowing a better view of the completeness of the output performance information; and
  • non-compliance with GAAP. As well as disclosing output performance, GAAP also requires that TEIs disclose the cost of each output. For example, if universities identify research as an output, they need to separately disclose how much they plan to spend on it.

Output performance measures

3.22
Investment plans serve two main purposes – they are a funding agreement between the TEIs and the TEC, on behalf of the Government, and they inform the annual report, which is the accountability document to Parliament and the public.

3.23
The TEC template of performance measures contains useful measures. However, in our view, a broader set of performance measures is more appropriate for accountability purposes. The output measures and targets need to provide a balanced and rounded performance story about service delivery by covering various dimensions of performance, such as quantity and quality of service delivery. Appropriate performance information is relevant, reliable, understandable, and comparable.

3.24
Nearly half the ITPs relied primarily on the measures in the TEC template. The other ITPs, and most of the universities, presented a more comprehensive set of performance measures and, in our view, provided a more meaningful and complete picture of their forecast service performance.

3.25
The investment plans, as reviewed, present a list of courses offered and the planned number of Equivalent Full-Time Students for each course. TEIs could consider what level of information on courses and/or faculties might be appropriate to include in the SSP in the annual report – for instance: Were the courses provided as planned and were there any significant changes?

3.26
It might be difficult for a lay reader to understand the different levels of courses (such as Levels 1 – 3), the different types of qualifications (such as certificates, diplomas, degrees), and the cross-over between schools, ITPs, and universities.

3.27
There are existing quality assurance mechanisms in place for course approval, accreditation, and external review. It would be worthwhile considering to what extent, if at all, aspects of these processes should be reflected in formal performance measures. Some TEIs do reflect them.

3.28
A common indirect measure of the quality of teaching/course provision is student satisfaction. A few TEIs presented measures from a benchmarked Australasian Survey of Student Engagement (AUSSE) and one university presented a measure from the Staff Student Engagement Survey.

3.29
As would be expected, the universities had a much stronger focus on research and more measures on research. In general, they appeared to present a reasonable set of measures, covering a range of performance dimensions.

3.30
Measures impart costs to institutions and should only be included if they are useful to the business decisions of the institution. The value of some measures might be questionable without associated analysis, interpretation, and indication of planned responses.

Performance story

3.31
Finally, it is important that a TEI's performance story, in both the investment plan and the annual report, gives a complete and meaningful picture of the entity and its performance, especially where similar measures are being presented. TEIs have the opportunity in their investment plans and annual reports to explain their own context, including their performance on the mandatory performance indicators of student achievement.

3.32
Figure 13 defines outcomes, impacts, outputs, and inputs.16 It presents extracts from some TEIs' investment plans and annual reports for the teaching/education output and shows the type of logical flow of performance information we would expect to see. (Note that these are not the only TEIs presenting this information – they are just used as examples.)

Figure 13
Performance framework definitions and examples from current investment plans

Outcomes – definition Example outcome Example outcome measures
Changes in the state of, condition of, effects on, or consequences for the community, society, economy, or environment resulting from the existence and operations of the entity. Through skills in leadership, in communication and in critical and creative thinking, our graduates contribute to a more educated and skilled workforce and society. (Victoria University of Wellington) Earnings gained by new graduates.

Standard of living.

(Unitec Institute of Technology)
Impacts – definition Example impact Example impact measures
The contributions made to an outcome by a specified set of outputs. Often referred to as "intermediate outcomes", they represent the relatively immediate or direct effect on stakeholders of the entity's outputs. Graduates who are advanced practitioners, highly sought after and who will make an impact. (Auckland University of Technology) Proportion of graduates gaining employment or going on to further study (annual graduate destination survey).

Relevant qualifications (annual employer satisfaction survey).

Work readiness of graduates (annual employer satisfaction survey).

(Wellington Institute of Technology)
Student progression rates.

Qualification completion rates.

Student retention rates.

Course completion rates.

(mandatory EPIs)
Outputs – definition Example output Example output measures
The goods or services that are produced by the entity for third parties. The major output provided by the Polytechnic is education and training to students. Full-time, short, and part-time, pre-vocational, vocational, and personal development courses are provided to Certificate and/or Diploma level. (Aoraki Polytechnic*)

Deliver high quality, relevant courses and qualifications and support students. (University of Canterbury)		 External Evaluation and Review reports "High Confidence" in both Educational Performance and Capability in Self Assessment. (Nelson Marlborough Institute of Technology)

Level and nature of students' contacts with teaching staff (benchmarked, percentage- based measure from the Australasian Survey of Student Engagement – AUSSE). (University of Canterbury)

Percentage of students who would recommend the TEI to others. (Auckland University of Technology)

Proportion of students evaluating lecturers as "excellent" or "good". (Lincoln University)

Student satisfaction with student support services and facilities. (Unitec Institute of Technology)

Student engagement – proportion of students who are withdrawn by the Polytechnic because of non-engagement. (The Open Polytechnic of New Zealand)

Academic staff with a doctoral qualification ratio (% of full-time equivalent). (Massey University).
Inputs – definition Example input
The resources used by the entity to produce its outputs Implement the annual published programme of staff development. (Victoria University of Wellington)

* This example is from the annual report, rather than the investment plan.

9: The investment plans are public documents (as required under section 159YO of the Education Act 1989) and are available from each TEI. Our review did not include the three wānanga.

10: For 2012 reporting, auditors will apply the Auditor-General's revised auditing standard, AG-4 The audit of service performance reports, to their audits of TEIs' performance information. The most significant change in the revised standard is that auditors will be required to attest to whether the statement of service performance (SSP) fairly reflects actual service performance for the year – previously, they attested to whether the SSP reported faithfully against the forecast SSP (which will still be the basis of the audit opinion on the 2011 annual reports).

11: New Zealand Gazette (3 June 2010), "Requirements and Exemptions for Investment Plans from 2011 onwards for Tertiary Education Organisations", Notice 4030.

12: Māori, Pasifika, those aged under 25, and international students.

13: Available on Statistics New Zealand's website (www.stats.govt.nz).

14: See "Graduate Longitudinal Study New Zealand" at www.glsnz.org.nz.

15: See "Strengthen research and economic outcomes" in the universities' section of the TEC's website (www.tec.govt.nz).

16: Based on GAAP and definitions in the Public Finance Act 1989 and the Crown Entities Act 2004.

page top