Here’s what auditors noticed during the 2012 audits

Continuing to improve how you report on your TEI’s service performance.

Introduction

The observations we make here are a summary, to guide your discussions about improvement. Please continue to liaise with your appointed auditor about improvements specific to your organisation.

It is important to note that we applied a revised standard to our audits of TEIs in 2012. Although we’re still working on applying the standard consistently, we trust you’ll find these observations useful.

Key messages for finance and strategy staff
Organisations that report their service performance well have integrated their strategic planning and objectives with their reporting requirements. They use both these processes to enhance their governance, prioritisation, decision-making, and overall performance.
Reporting on service performance should provide an easily understood picture of your core activities, give a sense of progress, and note where improvements are being made.
Based on what the auditors saw during the 2012 audits, tertiary education institutions are clearly working to improve their use of, and reporting on, service performance information.
Although you’re heading in the right direction, there remains considerable room for improvement.
Auditors will continue to work with you to help with those improvements.
You could work more with other TEIs, too – auditors noted that several TEIs want to work together more to prepare outcome measures that meet common requirements.

How we audited your performance reporting in 2012

When we applied our revised standard to TEI audits, we recognised the unique statutory requirements for TEIs while ensuring that TEIs can use our findings to improve the quality of their reporting. Essentially, we used a two-pronged approach to assess service performance reporting in the sector. This means that:

if there were deficiencies in reporting against investment plans, then we considered whether there was an effect on the audit opinion; and
if there were no deficiencies in reporting against the investment plan, but we identified parts of the annual report that did not fairly reflect the performance of the TEI, then we included these extra findings in our management letter to the TEI.

In our 2012 audits, we did not modify any of our audit opinions because of deficiencies in parent TEIs’ service performance information.

TEIs’ services are primarily teaching and research. Our assessment of whether the service reporting fairly reflected TEIs’ performance was informed by whether your annual report clearly outlined the services you provided, how well you provided those services, and the effects of providing those services on the student and wider community (impacts/outcomes).

Our assessment included considering whether the performance measures in your annual report were useful to understanding service delivery and the outcomes influenced by providing those services.

We looked for a logical flow or “performance story” showing the relationship between your TEI’s core services and the effects/outcomes you seek from offering those services.

We considered that each TEI needs performance measures to capture what is most relevant to its performance and that external performance reporting should be based on the information used to guide internal performance management. This would help ensure that externally reported information and measures were also highly relevant to the business decisions of the TEI.

We expected your annual report to include more measures that relate directly to service quality and the direct effects of the services provided (that is, impact measures, including educational performance indicators (EPIs)) as well as some higher-level measures that indicate the broader societal outcomes intended from the services.

Our overall assessment

Although improvements ranged in quality and scope, auditors noticed an encouraging and definite trend toward improvement. Good points included:

the gradual broadening of reporting to include all dimensions of the TEI’s services rather than just reporting against EPIs; and
management teams’ and governance bodies’ increased ownership of the performance information.

For the most part, TEIs had responded to the recommendations that auditors had made the year before.

Top recommendations
Continued development of baselines, systems, controls and better narrative to explain variances between years.
Improved definition, logic flow, and links between inputs, outputs, impacts, and outcomes.
Improved narrative explaining why measures are used and why they help to demonstrate results. (Also, including measures other than just those required by the investment plan process, to fairly reflect performance and the TEI’s broader contribution.)
Continued development of impact/outcome measures that enable, for example, the TEI to demonstrate the employment readiness of its graduates.
Increase the use of performance information by senior management and governance bodies. This might mean assessing, monitoring, and reporting measures more regularly. Such use would test whether the performance information is useful to decision-making in the TEI.
Link the financial and service performance reporting information. It would enable better trend analysis and measurement of, for example, cost-effectiveness.

Performance framework or story

Performance frameworks should provide a clear picture of what your TEI is trying to achieve, including:

outcomes at an appropriate level;
outputs (services) the TEI is accountable for;
the relationship (intervention logic) between outputs and outcomes sought; and
accountability documents (investment plans and annual reports) that clearly reflect the way the TEI manages performance.

An appropriate narrative or explanation is necessary to support performance information and measures. The narrative should explain variances between forecast and actual results, should include deficiencies and planned remedial actions, and should explain areas of success.

The most common recommendation was about the need to improve clarity of the links between (a) the components of the performance framework (inputs/outputs/outcomes and related measures), (b) the performance framework or story in the investment plan and that reported in the annual report, and (c) service and financial information.

Another common recommendation was about the comprehensiveness of the performance framework. Some TEIs reported on only what was in their investment plans and/or only on their EPIs.

EPIs are primarily impact measures. They do not capture all the important aspects of a TEI’s performance or the full range of TEIs’ effects on stakeholders.

Outcomes and impacts

Outcomes should:

clearly show what the TEI is seeking to achieve or contribute to;
include a change in state or condition in society or the community resulting from the TEI’s operations;
be supported by performance measures and targets for the full period of the investment plan; and
include historical and comparative performance information, where appropriate.

Most TEIs report well on impacts, but not on higher-level outcomes.

Several TEIs looked to others in the sector to prepare measures for higher-level outcomes. Others sourced indicators from other agencies, such as Statistics New Zealand.

Auditors noted poor links between organisational strategy documents and external reporting documents as well as poor links between the different components (such as outputs and outcomes) of performance frameworks. Some confusion might be created through poor definition or inconsistent use of terms – for example, “objectives” that were in fact inputs (such as finance) being positioned and reported against as outcomes.

Outputs

Output classes should:

be appropriate aggregations of outputs and provide a clear overview of those outputs/services;
be consistent within and between accountability documents; and
reconcile with financial information.

Outputs should include:

all the TEI’s significant services to third parties;
measures and targets covering a range of performance dimensions;
commentary to explain the setting of targets and/or any variance in performance achieved against targets;
historical and comparative information, to provide performance context to the extent possible;
SSP results that are verifiable and accurate; and
SSP results that are presented in a balanced and unbiased way.

TEIs that organised their outputs into output classes tended to aggregate their outputs around course and/or research provision types. Auditors recommended that inputs be clearly and separately identified and not confused with outputs – accurately identifying components makes it easier to assess how the cost of service delivery changes over time.

Output classes should only be used if they help to clarify output information for readers. Many TEIs continue to review their output reporting and how those outputs should most usefully be aggregated or separated.

Collecting cost information against output classes and/or outputs remains a “work in progress”. Some TEIs reported output costs, but most have not matched their financial reporting with their service reporting.

Without such data, it is very difficult to assess the cost of service delivery and cost-effectiveness and to use service information to make decisions at governance or senior management level. Auditors have noted that the data that might enable better matching of service data with financial data is often in TEIs’ financial systems or reported internally in other documents.

Auditors noted, too, that TEIs’ reporting would be improved by better narrative explaining the variance in performance against targets or measures between years.

Improved monitoring/use of measures throughout the year would demonstrate data quality and whether measures were useful to the governance and management of the organisation.

Measures and information: supporting systems and controls

Auditors noted matters about supporting systems and controls, and the investment plan and the annual report. They included:

whether supporting systems and controls were integrated into how the TEI manages and reports performance, and whether they could be relied on to produce forecast and actual performance information;
the clarity, readability, and consistency within and between documents;
whether information about the TEI’s activities, and forecast and actual performance, was consistent with our auditor’s understanding of the TEI;
information on the TEI’s capacity and resources to deliver its outputs and achieve the outcomes sought; and
whether there was enough information to allow the reader to assess cost-effectiveness.

Auditors noted that a few TEIs used service performance measures to report monthly or quarterly to their governing body or senior management teams. Others only collated and reported data to comply with external reporting requirements.

On the whole, the narrative about TEIs’ operating context was generally appropriate and annual reports were presented in a readable and logical manner. Auditors noted that consistency between documents could be improved, especially between measures in business and/or strategic plans and those in investment plans.

Calculating cost-effectiveness remains difficult for most TEIs because of a lack of cost information against outputs. When that information is collected, there are compatibility problems between financial and service data, and more broadly between the elements of performance frameworks.

Auditors often observed that a lack of co-ordination between strategy and finance personnel was a likely cause of poor links between various documents.