Part 3: Non-financial performance reporting in the tertiary sector
3.1
This Part builds on an article in our December 2011 report, Education sector: Results of the 2010/11 audits. The observations and examples presented in that article were based on our review of TEIs' investment plans. In this Part, we discuss aspects of performance reporting that could be improved in TEIs' 2012 annual reports, based on our review of a selection of TEIs' 2011 annual reports.
3.2
We confirm the essential elements of a good performance framework, identify attributes specific to annual reports, and give illustrative examples, where appropriate.
3.3
We have been working for some years now to lift the quality of non-financial performance reporting in the public sector. We published guidance about this in our June 2008 report to Parliament, The Auditor-General's observations on the quality of performance reporting. This and other relevant guidance is readily available on our website.
3.4
We have noted that entities that are doing well in their non-financial performance reporting have integrated their strategic planning and objectives with their reporting requirements and use both these processes to enhance their governance and wider performance.
3.5
Non-financial performance reporting should provide an easily understood picture of core activities during the reporting period and give a sense of progress and where improvements are being made. It should help with making choices and setting priorities.
3.6
One of the Auditor-General's auditing standards, AG-4 (revised), is about auditing performance information. Under AG-4 (revised), our auditors attest whether what is reported is a "fair reflection of the performance of the entity".
3.7
We have applied this standard to audits of local government entities since 30 June 2010 and to large government departments and Crown entities for the year ended 30 June 2011. We are continuing to progressively apply it to the rest of the public sector. We will audit TEIs in keeping with this standard beginning with the 2012 audits.
3.8
Our auditors have been working with TEIs to help ensure that their reporting for 2012 will be a fair reflection of their performance (for 2011, the audit report attested only to whether the SSP reported faithfully against the forecast SSP or investment plan).
Areas of focus
3.9
Services provided by TEIs are primarily teaching14 and research (for universities and some ITPs). Our opinion on whether the non-financial reporting fairly reflects TEIs' performance will be informed by whether the annual report for these entities clearly outlines the services the TEI is providing, how well it is providing them, and the effects of the services on the student community (impacts/outcomes).
3.10
We will also consider whether the annual report includes enough performance measures to give a balanced and rounded performance story about service delivery and the effect of those services. Ideally, the performance story will also be represented in a one-page performance framework that demonstrates a logical flow between the core services the TEI offers and the effects/outcomes it seeks as a result of offering those services.
3.11
Each TEI needs performance measures to capture what is most relevant to its performance – it should base its external performance reports on the information used to guide its internal performance management. This will help ensure that information and measures are highly relevant to the business decisions of the TEI.
3.12
Legislation and government policy implemented through the TEC sets TEIs' non-financial performance reporting requirements. The TEC's guidance15 for the 2013 investment plans notes that reporting on mandatory performance commitments, including EPIs and participation rates of targeted priority groups,16 should not be assumed to cover the full scope of a TEI's activities.
3.13
The EPIs all relate to student achievement17 and are therefore "impacts", in that student achievement is a consequence of the teaching or course provision output. Student achievement/EPIs and participation rates are useful as measures of the knowledge and skills a student has gained, but measures for teaching, research, or other outputs are also required if the full scope of the TEI's performance is to be assessed.
3.14
We expect the annual report of a TEI to include more measures that relate directly to service quality and the direct effects of services provided (that is, impact measures, including EPIs) than higher-level measures that indicate the broader outcomes intended from the services. We give examples in Figure 9.
3.15
Auditors will continue to work with individual TEIs on their plans for reporting performance, to consider what would most appropriately reflect the TEI's performance for the year. It is perfectly acceptable to report on more measures than those signalled in the investment plan, provided the TEI discloses that a measure is new and the relevance of the measure is clear.
Attributes and examples of appropriate performance reporting
3.16
Our review of a small selection of TEI annual reports suggests that some TEIs already use a reasonably broad range of performance measures for how well they deliver their services. They also monitor one or two higher-level indicators, such as graduate employment levels. We found examples of the performance elements in Figure 9 relatively easily. However, most TEIs' annual reports are likely to need some improvements to provide a complete service performance reporting framework.
3.17
In some instances, performance information is discussed in narrative information but not captured in performance measures – for example, the results of external evaluations and reviews. Although narrative information is useful to explain the importance of the measures, we encourage TEIs to set formal performance measures and targets for all important aspects of performance in their investment plans and then report against these in their annual reports. This would clearly convey the TEI's performance intentions and also help assure readers, at year's end, that the TEI's performance is fairly reflected in the annual report.
3.18
As well as setting out examples from the 2011 annual reports (or investment plans) in Figure 9, we have repeated most of the examples presented last year. Other TEIs will have similar examples that we could have used.
Figure 9
Attributes and examples of appropriate performance reporting
| Attribute | Examples |
|---|---|
| Performance framework presented A performance framework clearly identifies the relevant aspects of performance (outcomes, impacts, services), logically presents these, and explicitly attaches measures to these aspects. (We noted last year that outcomes, impacts, and services tended to not be explicitly presented and were sometimes confused with inputs.) |
Victoria University of Wellington's outcomes framework, set out on page 49 of its 2011 annual report (see Appendix 3). A further important step would be to explicitly identify the University's outputs/services in the framework. |
| Outcomes and/or impacts defined Outcomes are changes in the state, condition, effects on, or consequences for the community, society, economy, or environment resulting from the existence and operations of the entity. Impacts are the contributions made to an outcome by a specified set of outputs. They represent the relatively immediate or direct effect on stakeholders of the entity's outputs. |
Outcome: A more educated and skilled workforce and society. Impact: Victoria's graduates raise the skills and knowledge of the current and future workforce to meet labour market demand and social needs. (Victoria University's 2011 annual report, page 49). Impact: Graduates who are advanced practitioners, highly sought after and who will make an impact. (AUT, 2011-13 Investment Plan, page 9). |
| Outcomes/impacts measured TEIs' performance information should give some indication of the effects of services on the community (as well as how well the services are delivered). We acknowledge that there are sector initiatives, such as the longitudinal study of graduates' outcomes. We also recognise that research carried out by TEIs is often, by its nature, related to societal well-being and outcomes. The annual reports we reviewed gave useful snapshots of the type of research carried out. |
1. Proportion of graduates gaining employment or going on to further study (annual graduate destination survey) 2. Relevant qualifications (annual employer satisfaction survey) 3. Work readiness of graduates (annual employer satisfaction survey) (WelTec, 2011-13 Investment Plan, page 21). Lincoln University also reports on graduate employment outcomes (Lincoln University's 2011 annual report, page 29). Research
|
| Impact/proxy quality measures |
|
| Services identified Teaching is clearly a universal service for TEIs, as well as research for universities and for some ITPs. There may be other services that TEIs choose to identify and formally report on in their SSP. For a complete performance framework, these should include all services, not just those funded by the Crown. |
UCOL reports against the Performance Commitments from its investment plan (they are proxy impact measures), but also presents its services and associated performance measures under six output classes relating to different faculties. It gives a brief, explicit description under each output class. For example: Output 1: Agriculture, Horticulture and Science Objective: To provide to students training and education through the provision of programmes and courses in Agriculture, Horticulture and Science. Programmes at degree, diploma and certificate levels were offered during 2011 in science, animal science, exercise physiology, and general sciences. Secondary Tertiary Alignment Resource (STAR) programmes were also provided. (UCOL's 2011 annual report, page 18). UCOL also presents generic measures for each of these output classes. The measures are explained at the start of this section of its annual report. University of Otago identifies three services (outputs) – research and postgraduate teaching, teaching and learning, and community service. The next step would be to display these in a performance framework, explicitly linking these services to the six strategic objectives in the SSP, under which the performance measures are set out (University of Otago's 2011 annual report, page 32). |
| Service delivery measured Appropriate performance information will cover a range of dimensions of performance. Quality is an important aspect of performance – the examples focus on measures of quality. References to quality standards or criteria and/or approval processes should guide the reader to a summary or easy viewing of those standards, criteria, or approval processes. Survey measures should disclose the sample size and response rate. |
Lincoln University reports on the proportion of students evaluating lecturers as "excellent" or "good", and on a further five student satisfaction measures. It also indicates the frequency and number of evaluations in supporting narrative (Lincoln University's 2011 annual report, pages 28-29). University of Otago reports on a survey of its graduates' perceptions of their improvement in various attributes, such as independent judgement (University of Otago's 2011 annual report, page 36).* External Evaluation and Review reports "High Confidence" in both Educational performance and Capability in Self Assessment (NMIT, 2011-13 Investment Plan, page 106). Courses provided with the TEI's formal Academic Board's approval (UCOL's 2011 annual report, page 17). Improved student engagement – active learning, student/staff interactions, supportive learning environment (benchmarked, percentage-based measures from the Australasian Survey of Student Engagement – AUSSE; involvement suspended in 2011 due to the earthquakes) (University of Canterbury's 2011 annual report, page 20). Percentage of students who would recommend the TEI to others (AUT, 2011-13 Investment Plan, page 26). Student satisfaction with student support services and facilities (Unitec, 2011-13 Investment Plan, page 41). Student engagement – proportion of students who are withdrawn by the Polytechnic because of non-engagement (Open Polytechnic, 2011-13 Investment Plan, page 17). Research
|
| Supporting narrative and disclosures Performance measures are often just the starting point for the performance story. Useful supporting narrative will explain the relevance of the performance measures and what the results actually mean. |
The annual reports that we reviewed had useful supporting narrative, either in separate sections or accompanying the performance measures. For example:
Brief narrative on the number of courses and changes to courses can provide useful information to the reader. The University of Otago reports the number of programmes compared with the previous year, and identifies new programmes introduced (University of Otago's 2011 annual report, page 20). UCOL also explains the levels of courses – useful background for the external reader. (UCOL's 2011 annual report, page 13). |
| Costs of services disclosed Disclosing the costs of each output in the annual report gives an indication of relative expenditure in each area and may inform service delivery prioritisation decisions. |
The University of Otago identifies the costs of outputs for three outputs:
As noted earlier, the next step would be to link these to the performance measures under the strategic objectives in the SSP (University of Otago's 2011 annual report, pages 32-33). UCOL reports on the gross costs for each EFTS and the total cost for each of its six output classes (UCOL's 2011 annual report, page 18). |
| Comparative information Comparative information over time allows the reader to see trends in performance. |
Baseline data for the previous year is commonly presented by TEIs. |
| Variances explained Where a target was not met, the TEI should comment on this. |
With the number of EFTS generated by Pacific students being less than the target, the University of Waikato notes that the proportion is higher than the demographic for the region, and that it has prepared a Pacific Plan as part of its strategic planning framework. The formal infrastructure, with relevant responsibilities and accountability, will be established in 2012 (University of Waikato's 2011 annual report, page 93). |
* As with the EPIs, this could be seen as either an impact of the TEI's services or a quality measure of its services. Narrative would help the reader understand the TEI's perspective on what performance story these measures tell.
Measures and targets
3.19
Further attributes and examples of appropriate performance reporting are:
- All measures and targets identified in the forecast SSP in the investment plan must be clearly presented and reported on.
- All performance measures need to be reliable (that is, supported internally by detailed definitions and appropriate systems to collect the relevant data).
14: We use the term teaching broadly, to encompass concepts such as training, educating, or providing courses.
15: Available on the TEC's website, www.tec.govt.nz.
16: These measures are in the TEC's template of Performance Commitments.
17: Successful course completion, student retention, qualification completion, and student progression.