Part 2: Auditor-General's views on the quality of performance reports
2.1
The quality of performance reporting represents a significant weakness in the public sector's accountability to its stakeholders. Despite the sector having nearly 20 years' experience in preparing and using performance reports, performance reports are:
- not prepared as robustly as they should be to serve external readers' needs;
- not used as well as they might be by external readers as part of the accountability process; and
- not used as well as they might be by internal readers – managers and governors of public entities – to improve public service effectiveness.
2.2
In my view, many entities' performance reports:
- do not seem to set out coherent performance frameworks showing logical links from the medium-term1 outcomes information and organisational strategies to the annual output information (or, for those entities required to prepare an SCI, showing logical links from objectives to the performance measures and targets); and
- do not have well-specified, relevant, understandable, reliable, and comparable performance measures and targets. For local authorities, government departments, and Crown entities, this observation applies to both the medium-term outcomes-based information and the annual output-based SSP information.
2.3
For local authorities, government departments, and Crown entities, I am particularly concerned about:
- the weak links of the medium-term contextual and strategic information to the annual forecast SSP; and
- the identification and specification of the elements (primarily outcomes and outputs), measures, and targets for both output information and medium-term outcomes achievement.
2.4
For entities required to prepare an SCI, I am particularly concerned about wide variations in the clarity and coverage of linking performance measures and targets to entities' stated objectives.
2.5
For all entities, I am also concerned at the lack, in many instances, of robust best estimate-based targets combined with historical or benchmark information that gives context to those performance targets.
2.6
Those preparing performance reports also need to better set out the elements of the reports by:
- applying the definitions of the elements (in particular, of outcomes and outputs) – if the underlying elements are not properly identified and presented, the basis of the reporting is undermined; and
- considering how to meaningfully aggregate elements (in particular, outputs and output classes) with enough detail to communicate a comprehensive yet succinct and coherent account of the outputs they deliver.
2.7
Parliament and the public rely on these documents. If they report poor performance, entities should be held accountable for remedying it. If Parliament is unable to adequately assess the performance of entities because of the poor quality of their performance reporting, then I expect those entities, and their oversight agencies, to be held accountable.
My Office's work during the past two years
2.8
During the past two years, my staff have:
- reviewed most government department and Crown entity SOIs;
- audited the 2006-16 LTCCPs of local authorities, which included meeting a requirement to attest to the extent to which the forecast information and performance measures provide an appropriate framework to meaningfully assess the actual levels of service provision; and
- carried out a performance audit looking at the SCIs prepared by port and energy companies, State-owned enterprises, and Crown research institutes, as well as the SOIs prepared by council-controlled organisations and council-controlled trading organisations.
2.9
There is a summary of the results for each of these pieces of audit work in Parts 3 to 5. The full reports can be found on my website.2
2.10
This is not the first time that I have raised concerns about the quality and usefulness of the information prepared by various sectors.3 There needs to be a better understanding of why the public sector accountability framework4 does not promote quality public sector performance reporting – or at least appears to sanction inadequate reporting. To promote this understanding, I first outline the audiences for performance reporting, and the legislative requirements and generally accepted accounting practice applying to performance reporting. I discuss why, in my view, performance reporting is difficult, and look forward to consider how improvements might be achieved. I also set out my intentions for the work of my Office.
The audiences for performance reporting
2.11
Stakeholders of public entities are interested in outcome and output performance because those entities' provide goods and services for community or social benefit. The need for performance reporting in the public sector has an added dimension. In providing these goods and services, public entities use public resources (for example, taxes and rates) and sometimes have coercive powers to regulate the behaviour of others.
2.12
However, in my view, the questions of who performance reporting information is for and how it is intended to be used have not been well debated.
2.13
It is clear for state sector entities that Ministers and their monitoring agents use performance reporting information. However, as part of the governance system for state sector entities, performance information for Ministers could equally be provided through “special purpose reporting”.5
2.14
Parliament also needs performance reporting information, to hold Ministers and state sector entities to account. Parliament is required to make decisions about the funds to be voted through the annual Budget process, and to review the performance of the entities that have used those funds.
2.15
However, the needs of individual members of Parliament vary widely, depending on their individual views, their portfolio responsibilities, their political alignments, and the topical issues of the day.
2.16
The elected representatives of entities such as local authorities, which operate within statutory authority but independently from Parliament, also need information to hold the managers and staff of those entities to account. Local communities also need information about local authorities to allow them to participate in democratic decision-making about services and their level and cost, and to hold elected representatives and entities to account.
2.17
In a democratic society, openness and transparency are the foundations underpinning the use of taxing and regulatory powers. Therefore, the media, academics, political and financial analysts and commentators, and interested and concerned members of the public need this information too. My staff also need the information, to give my assurance to Parliament and the public about the performance of public entities.
2.18
All of these audiences – and more – have an interest, if not in any single agency's performance accountability, then in the quality and availability of this information collectively. Their reasons for wanting information will vary. However, generally, they use the information to bring issues to the attention of the wider community, perform research about the nature and state of society and how the services of public entities affect the community, and recommend investment and other decisions to those they advise.
2.19
Because there are so many users of, and uses for, performance information, there is unlikely to be a single or complete solution to the range of complex and varying information needs. Performance reporting is therefore premised on providing accurate information relevant to the needs of users to serve accountability purposes and to aid decision-making, with public accountability legislation requiring a common set of performance reporting information. As with all general purpose financial information, performance reporting might not be of equal relevance to all users, but it provides a starting point for more specific purposes.
2.20
This common set of information must be prepared subject to GAAP. Although the information is general purpose, it does not follow that any person should be able to use and understand it. There is an assumption that users of the information are already moderately informed and willing to make the effort to understand it. There is a range of means by which users can also access information for their special needs – for example, through the Official Information Act 1982 or through questions in the House of Representatives.
Legislative requirements and generally accepted accounting practice
2.21
Legislative requirements for the performance reporting of local authorities, government departments, and Crown entities are similar. The differences relate to the number of years required in the forecast information, process requirements, and the terminology used (although definitions are generally similar). Likewise, legislative requirements for council-controlled organisations, port and energy companies, State-owned enterprises, and Crown research institutes are similar. Appendix 2 sets out, for each type of entity, the legislative requirements for performance reporting, and the respective audit requirements.
2.22
The qualitative characteristics within the Framework for the Preparation and Presentation of Financial Statements6 applying to financial reports – relevance, understandability, reliability, and comparability – apply equally to performance reports in respect of selecting outcomes and outputs, selecting performance measures, determining targets, and any other related information disclosed in performance reports.
2.23
Overall, therefore, I expect public entities to have a performance reporting framework that reflects the statutory requirements and GAAP. Figure 1 sets out what I consider this would comprise for government departments, Crown entities, and local authorities.
Figure 1
What a performance reporting framework should include
The medium-term component should include information on the reporting entity's objectives, outcomes, impacts, and operating intentions, together with related performance measures and targets and other information required by legislation and GAAP:
Throughout these reports, there should be links between financial and non-financial information that, together, provide a coherent and reasonably complete picture of overall entity forecast and actual performance. |
Why performance reporting is difficult
2.24
In this section, I outline some of the issues that, in my view, contribute to the poor quality of performance reporting, and the reasons that the public accountability framework is not promoting improved reporting. Overall, I consider that those preparing performance reports have little incentive to prepare quality reports. Instead, they often:
- report what can be reported rather than what should be reported;
- are reluctant to report information that may reflect poorly on the entity and its governors; and
- do not adequately report the quality of output delivery and information to allow users to assess how outputs affect intended outcomes.
Lack of comprehensive standards
2.25
Compiling meaningful performance reports presents many more challenges than preparing conventional financial reports. For financial statements, their structure, composition, measurement, and disclosure are heavily prescribed by financial reporting standards.
2.26
There are no reporting standards in New Zealand for non-financial performance statements. Therefore, every entity needs to prepare and customise its own framework and the elements (primarily outcomes and outputs) within that framework to reflect its own nature.
2.27
Each reporting entity must judge which elements (outcomes and outputs) are most relevant and significant, and how they are best aggregated for the purpose of external reporting. These judgements, from my observations, appear to lie behind many of the quality concerns I have raised.
2.28
Financial reporting standards are generally silent on performance reporting issues. Accounting standard-setters helped address this by preparing Technical Practice Aid No. 9: Service Performance Reporting (TPA-9) in 2002. It contains a rich discussion that is still highly relevant. In my view, this discussion should be in a statement of greater authority than a technical practice aid.7 However, there has been no subsequent development of the 2002 TPA-9 initiative. Indeed, it has been some years since the question was raised of how standards should support service performance reporting (and reflect developments in practice) and help entities to make judgements about the presentation of information.
2.29
In my view, the decision to base New Zealand standards on International Financial Reporting Standards highlighted the different needs of the public sector and therefore the need for different treatment. Given that New Zealand equivalents to International Financial Reporting Standards (NZ IFRS) are written to serve the needs of large, profit-oriented entities in the private sector, it is unlikely that they will cater for the public sector's specific need for performance reporting standards.
2.30
The Public Finance Act 1989 provides for the Minister of Finance to make recommendations to the Governor-General for regulations prescribing non- financial reporting standards for the state sector. During the consideration of the 2004 amendments to the Public Finance Act, I suggested that the Financial Reporting Act 1993 be amended to give the Accounting Standards Review Board the mandate and resources to develop and approve non-financial reporting standards.
2.31
I consider that, over time and as experience develops, the Treasury, in consultation with interested groups, should facilitate and ensure the preparation of such reporting standards. It should also consider how such standards are applied and maintained.
No clear responsibility for professional leadership and oversight
2.32
Accountancy as a profession provides, among other things, leadership and oversight of preparation of financial information. Members of the profession who prepare financial information are required to apply the profession's technical standards and, in doing so, to meet certain standards of professional competence. However, there is no professional body that provides equivalent leadership for preparing performance information. Just as there are no technical standards governing the preparation of performance information, there is no purpose-built set of professional competency standards or professional support for preparers of such information.
2.33
Because of this, most of the leadership for the quality and presentation of performance reports in the public sector in the past 20 years has had to be assumed by the Treasury and the State Services Commission, other sector agencies with leadership roles in the public sector (such as the Society of Local Government Managers), and my Office. Although performance reporting requirements have been supported by guidance materials and processes, these have not helped to significantly improve the quality of reported information. Central government entities often tell my staff they are confused by what they see as conflicting messages about the quality, and improvement needs, of their performance reporting. This confusion appears to arise because of their multiple interactions with a range of interested parties (primarily central agencies and my Office, but also other central government monitoring departments).
2.34
Initiatives that central agencies, other sector agencies with leadership roles in the public sector, and my Office have taken to help improve performance reporting have often lacked co-ordination, sustained effort, and focus. For example, initiatives often:
- are not co-ordinated in terms of overall priorities with other parties who have roles in preparing and using performance reports, or through collaborative work with these parties;
- have not had ongoing support to sustain them; nor a constant set of messages about the elements of performance reporting and their application. Initiatives have tended to be associated with legislative or other change involving specific guidance products. For example, guidance material was produced to support the Managing for Outcomes/Results initiatives. However, the initiatives were regarded as “mainstreamed” after a short time without, in my view, enough work to ensure that entities had understood and responded appropriately. The 2004 changes to the Public Finance Act 1989 introduced requirements for measures around the cost-effectiveness of interventions (section 40D). Despite widespread confusion among entities about the intention of the provisions, there has been limited further development work by central agencies; and
- have not focused on helping entities to apply general guidance to their own circumstances and on how to make incremental improvements and adjustments over time.
2.35
Part of the difficulty in ensuring co-ordinated and sustained effort is that there are many parties with an interest in preparing and using performance reporting. In my view, there is a need to consider the question of leadership, co-ordination, and accountability within the public sector for ensuring the application of standards and expectations for performance reporting, and for improving its quality.
Relationships between outcomes and outputs are often not predictable or understood
2.36
Relationships between outcomes and outputs are complex, relationships and expectations change over time, and performance can be influenced by a range of circumstances and events. In the past decade, there has been extensive interest and exploration in the public sector on how to better collaborate and co-ordinate government entities to provide more effective outputs and achieve desired outcomes. Increasingly, such efforts require many-to-many, rather than one-to-one, links. Some of these many-to-many links are:
- several entities working towards their own outcomes, where the outcomes together will also achieve a complex overall outcome;
- multiple entities each contributing one part of a shared outcome;
- stakeholder interest in the collective performance of multiple entities, a sector, or multiple sectors; and
- world-wide considerations, such as international group or global commitments.
2.37
It is important that entities base outcomes on societal or community factors they can reasonably expect to influence, and outputs on services they control and deliver to third parties. Where there is confusion about the degree of influence or control exerted by an entity, there can be confusion about the nature of intended effects on outcomes or about the nature of the service provided and how its quality can be assessed. However, the relationship between outcomes and outputs is not always understood or predictable, and these relationships can change over time or in response to other events. Therefore, in reporting on actual results, entities need to evaluate and assess these relationships and the effect of services on desired outcomes over time. Reports by my Office have previously discussed evaluation in the public sector.8
2.38
It is natural that there be public debate and sometimes disagreement about what policy objectives or services should be pursued and what the results are indicating. In this context, performance reporting is about making actions and decisions understandable, and explaining subsequent events. Good quality reporting, both non-financial and financial, allows informed consideration in Parliament and the public sector, and across communities, about what is happening and how the reporting can be done better.
2.39
The difficulties in attributing the effects of outputs on outcomes appear to have been compounded in central government by expectations that entities will demonstrate how their efforts contribute to the very high level outcomes of the Government. Every public entity is part of wider public sector efforts and the actions of groups and individuals within society as a whole. For each entity to demonstrate its contribution to very high level outcomes often requires additional layers of outcome, intermediate outcome, and impact information, with reductions in the likely influence of the entity at the higher layers.
2.40
I do not think entity-based reporting can be expected to deal with the question of whether the collective efforts of public entities have a reasonable likelihood of influencing very high-level outcomes. There may therefore need to be consideration of sectoral or whole-of-government performance reporting. Some recent reports show an encouraging evolution toward sector-based reporting that seeks to set out and understand high-level outcomes and the public sector's influence on them. Examples are the Ministry for the Environment's Environment New Zealand 2007 (which builds on its 1997 report, The State of New Zealand's Environment), and the Ministry of Social Development's The Social Report, published annually since 2001. I would like to see this kind of reporting continue, widen, and improve over time.
Wider set of information needed for operational management
2.41
Creating and using performance information within an organisation is commonsense management practice. I am concerned when public entities tell my staff that external service performance information is not useful for the public entities' internal management.
2.42
In my view, performance reports should be a reflection of internal management – clearly articulating strategy, linking strategy to operational and other business plans, monitoring the delivery of operational and business plans, and evaluating strategy effects and results.
2.43
If external performance information is not prepared well, perhaps because of the issues discussed above, it may well be that the information is not useful for internal management. For example, if elements such as outcomes and outputs are not well identified or relevant, measures and targets of outcomes or outputs will not be well selected.
2.44
A range of other information is also needed for internal management, and this information might be equally, and sometimes more, important for governance and management decision-making than output and outcome information. Such information includes management of inputs (for example, finances, resources, and capability) and of risks.
2.45
I expect to see, within a public entity, a broad set of input, process, output, and outcome information being collected, and monitored at periods appropriate to the nature of the information. I also expect different sets of information to be relevant to different parts of an entity. External performance reporting information is highly aggregated information about an entity's wider environment, strategy, and planned service delivery. Like other highly aggregated information, it should be of greatest interest to senior managers and the governors of a public entity, allowing them to discharge their decision-making governance and accountability responsibilities.
2.46
However, I am concerned that senior managers and governors do not give external performance reports the attention they merit. The reports are not aligned to the information used to plan and monitor performance. I have found that the report preparation is often contracted out and is not connected with organisational strategy development and implementation, creating the risk that the reports become vehicles for public promotion rather than a basis for accountability and transparency.
2.47
Public entities should prepare internal information that meets day-today management needs and align this information to external reporting responsibilities. Accordingly, external reports should stem from information that is used to guide internal management of the entity.
2.48
In my view, if a public entity does not have a system for monitoring and aggregating the range of information used for internal day-to-day management and governance, governors and managers cannot be confident that they are fulfilling their responsibilities.
Weak incentives to improve
2.49
Performance reporting requirements for public entities have existed for nearly 20 years, but it is not clear how the accountability framework provides incentives for quality reporting or sanctions for poor reporting. I have found little use of both forecast and actual reported information to assess the performance of each entity and its future challenges and requirements. It raises the circular question of whether reports are of poor quality because they are not well used or whether they are not well used because they are of poor quality.
2.50
A state sector entity is expected to involve its responsible Minister in preparing its forecast performance report. Ministers are part of the governance of state sector entities. They therefore have ready access to a range of information about an entity and its performance that is much greater than that which should be presented in a general purpose report. They do not need to rely solely on the quality of external performance reporting to meet their own governance information needs. Nonetheless, they have a governance responsibility to ensure that the entity provides quality information for other external users.
2.51
Elected representatives also have a key role to play in creating demand for the use of performance reports. Those such as members of Parliament and local authority councillors need to consistently encourage general use of performance reports. In particular, elected representatives can insist on generally accepted (and expected) practice, such as requiring well-specified links between outputs and outcomes.
2.52
In audit work carried out by my staff over the past two years, similar issues have emerged across the public sector. However, discussions since then with local authorities suggest that they are finding LTCCPs to be a useful planning tool to assess achievements and decisions against. Independent review work commissioned by my Office has noted improvements in service performance information between the first LTCCPs prepared in 2003 and 2004 and those prepared and audited for the first time in 2006.
2.53
Under the Local Government Act 2002, the purpose and intended uses of the LTCCP are clearly articulated. Strategic planning is to be based on community engagement and is linked to organisational service delivery. Communication of a range of information (including financial and performance reports) is required in ways that take account of stakeholders' needs and preferences for receiving information.
2.54
In my view, there has not been enough debate about who information is for and how different audiences might obtain information relevant to their needs. Public entities need to explore the needs and interests of their different users, how these influence their use of performance reports, and how to provide reports to better meet different needs. For example, communities making choices about the costs and services provided by their local authority might require different information from that sought by the media to publicise information about the results of policy changes in government departments. Although public entities need to provide the minimum information required for accountability, they should also be responsible for adapting information to meet users' needs. By doing so, the entities will enable better use of both forecast and actual reported information to assess the performance of each entity and its future challenges and requirements.
Looking forward – improving the quality of performance reporting
2.55
Enduring improvements in performance information will require clear and consistent objectives, strong central co-ordination and direction, well-established good management practices, and an unwavering accountability focus on understanding performance results. In my view, the public sector accountability framework and those responsible for its operation have struggled to provide these conditions since the start of the performance reporting requirements.
2.56
It is also clear that, for performance information to be useful, it needs to be specific to, and owned by, the entity preparing it. Therefore, preparing extensive amounts of general high-level guidance and specific case study material is only part of the action required. I urge public entities, central agencies, and others with an interest in public sector management to pay attention to the quality and the substance of information that appears in both forecast and annual reports as well as to its presentation. In my view, improvement will require deeper, more sustained, and focused attention, and should include:
- the Treasury, in consultation with entities with monitoring responsibilities and others with an interest in public sector management, facilitating the preparation of public sector standards for reporting, including considering how such standards are applied and maintained;
- identifying clearer responsibilities among central agencies, entities with monitoring responsibilities, and others interested in public sector management on co-ordination, leadership, and accountability for preparing and improving performance reporting within the public sector;
- central agencies and those with monitoring responsibilities giving consistent messages about the elements of performance reporting and their application; considering whole-of-government and sector-level outcome information needs, and how these can be identified, co-ordinated, collected, and reported; and
- considering the identifiable users of external performance reports and their needs, so that public sector regimes can better accommodate both internal and external uses of reported information.
My intentions for the work of my Office
2.57
I intend to maintain my Office's focus on performance reporting and hope that, in tandem with the efforts of others such as central agencies and the Society of Local Government Managers, this will help bring about the improvement needed in public sector performance information.
2.58
I have recently decided that audit opinions should provide assurance that the audit has assessed whether the entity's performance framework will allow its actual output performance to be fairly presented. For the local government sector, this is already part of the assurance that the audit opinion on the LTCCP provides. My staff are revising my auditing standard AG-4 The Audit of Service Performance Reports, which I will issue later this year as a basis for consultation, and which is intended to take effect for future reporting periods.
2.59
My staff are also assessing the effects of the intended changes to standards and expectations of auditors on the hours and costs of audit work. Providing this greater and more useful assurance to Parliament, public entities, and the public will probably result in some increases in audit costs. I am not yet sure of the extent of such increases, and will discuss these with Parliament and others before I make any decisions. Nonetheless, I am committed to ensuring that the annual audits address my statutory mandate, to the fullest extent they reasonably can within efficiency constraints.
2.60
While there are some good examples of performance reports, these are, in my view, few and far between. My staff will more actively identify and highlight examples of good practice to help public entities improve their performance reporting. This will include looking at practices in other similar countries and the work of other Auditors-General.
2.61
At this stage, I have not focused specifically on audit work that should be carried out to report on the performance targets and other measures for entities such as energy companies and council-controlled organisations. These entities do not have requirements similar to those for government departments, most Crown entities, and local authorities to set out outcome and impact-based medium-term and annual output information. Instead, for these entities, performance measures are to be set to assess achievement of corporate objectives. I am required to report on the actual performance achievements against those measures. I will assess the appropriateness of performance reports for these entities after my staff complete the work outlined in paragraphs 2.57-2.60.
Conclusions
2.62
In my view, improving the quality of information about public entities' performance is critical, not just for demonstrating accountability, but also for achieving continuous improvement in public sector effectiveness. I note that this accountability through performance reporting is about making actions and decisions understandable, and explaining subsequent events.
2.63
Addressing the weaknesses identified in performance reporting requires greater clarity, persistence, and consistency throughout the public sector. As long as the weaknesses described in this report persist, parliamentarians and the public can have limited assurance that the performance information of public entities reflects the purpose, effects, and effectiveness of their endeavours.
1: Forecast information is required to cover different periods across the public sector. For government departments and Crown entities, medium-term information must cover a three-year period. For local authorities, the LTCCP must cover at least a ten-year period.
2: Central government: Results of the 2006/07 audits (2007), Part 5; Matters arising from the 2006-16 Long-Term Council Community Plans (2007); and Statements of corporate intent: Legislative compliance and performance reporting (2007).
3: See, for example, my reports on the results of the 2004/05 and 2005/06 audits for central government and my report Local Government: Results of the 2002/03 audits.
4: This is the systems and processes used to prepare non-financial performance information and to hold public entities and their governors to account.
5: Special purpose financial (and non-financial) reports are reports tailored to meet the specific information needs of users who are able to require, or contract for, their preparation and provision.
6: New Zealand Institute of Chartered Accountants, New Zealand Equivalent to the IASB Framework for the Preparation and Presentation of Financial Statements, June 2005.
7: Some of the definitions in TPA-9 are now reflected in legislation passed since 2002, and TPA-9 has recently been updated for consistency with NZ IFRS and legislative requirements. However, a technical practice aid has little authority compared with a financial reporting standard.
8: For example, First Report for 2000, parliamentary paper B.29[00a], Part 6 “Impact Evaluation – Its Purpose and Use”.
page top