Forecasting and reporting performance: The search for the Holy Grail?

More than 500 public entities are required to publicly report prospective and actual performance information. Most of the requirements have been in place for more than 15 years, their origins lying in the public sector reforms of the 1980s.

There is strong theoretical logic supporting the usefulness of publicly disclosed performance information, with disclosure of non-financial performance information, a growing phenomenon internationally. Indeed last year I attended an international gathering of Auditors-General at which it was clear that the demand for such information is growing.

Yet all of us here would have at some time questioned the relevance and usefulness of non-financial performance information. I think we perceive ourselves at a cross roads - needing to decide whether the aim is achievable and the cost and effort is justified by the benefits.

Over the last 18 months, my Office, in response to statutory changes over the period 2002 – 2004, has extensively reviewed the prospective information prepared across the public sector. Our conclusion? Despite more than 15 years of experience, there remain significant issues and limitations with the prospective information currently produced, impacting on the usefulness of subsequent reporting, and I remain concerned that such information:

• is not prepared and reported on as robustly as it ought to be to serve external user needs; and
• is not used as well as it might be by internal users - managers and governors of public entities - to improve public service effectiveness.

So today I want to take the opportunity to take stock and question what we think non-financial performance reporting is for.

In doing so, I want to outline what I see from looking across the non-financial performance reports produced by the public sector and the issues I think bedevil the use and usefulness of this information.

I want to go back to the fundamental elements and qualitative criteria that underlie the preparation of non-financial performance information.

Finally, I want to ask whether some of our expectations are misplaced or unachievable and if so, whether we are seeking a holy grail and what a future direction might involve.

So why do it and who needs it?

From more than 15 years experience, I think we can all agree that compiling a meaningful non-financial performance statement presents many more challenges than that of preparing conventional financial statements. For financial statements, their structure, composition, measurement, and disclosure are heavily prescribed by financial reporting standards. There are no such standards for non-financial performance statements, which need to be customised to reflect the nature of the reporting entity. Therefore judgements need to be made about which outcomes and outputs are most relevant and significant for the purpose of external reporting. It is these judgements that seem to create many of the issues and problems – both for preparers and users.

However preparing information about entities’ performance and the wider impact of this seems a bit inescapable one way or another. I’ve always found it hard to understand why the internal existence and use of such information could be anything beyond common sense management practice.

Likewise there are common sense reasons for the public availability of performance information. Stakeholders are interested in service performance because public sector entities’ exist to provide goods and services for the public’s benefit. However the vital point for the public sector that underscores the importance of well-prepared performance information lies in its use of "coercive powers" – the power to impose taxes or regulate the behaviour of others - powers provided to protect and serve the "public good". So a core purpose of public sector performance reporting lies in the need to demonstrate efficient and effective service in the "public good".

Who needs it? – is a question that is not so easy. Getting greater agreement about who information is for and how different audiences might access it or be provided with it according to their needs has not, in my view, been very well debated.

It’s obvious for state sector entities that Ministers and their monitoring agents are users. However, as part of the governance system for state sector entities, performance information for Ministers could equally be provided through “special purpose reporting”.

Information is also needed by Parliament to allow it to hold Ministers and state sector entities to account. So Parliament is a user required to make decisions about the funds to be voted through the annual Budget process and to review the performance of the entities that have used these funds.

However the need of any individual MP varies widely depending on their individual views, their portfolio responsibilities, their political alignments, and the topical issues of the day.

Of course we are fortunate to live in a democratic society, in which openness and transparency are the foundations underpinning the use of taxing and regulatory powers. So the media, academics, political and financial analysts and commentators, interested and concerned members of the public are users. I too am a user in that I give assurance to Parliament and the public about the performance of public entities to Parliament and the public.

It’s hard to imagine a functioning democracy in which all of these users – and more - don’t have a stake, if not in any single agency’s performance accountability, then in the quality and availability of this information collectively. Their uses and purposes vary but include bringing issues to the attention of the wider community, performing research about the nature and state of our society and how the services of public entities are impacting on us, and in recommending investment and other decisions to those they advise.

So could any single set of information meet all these individuals’ needs – and still have addressed Parliament or the public’s interests? Currently public accountability legislation tends to provides for one set of information or a ‘one size fits all’ approach. While a common set of information might not be of equal relevance to all users, it can provide a starting point from which to explore and question for their more specific purposes - just as for financial statements the bottom line is the starting point for analysis.

Equally one of the things that we might begin to explore is the different needs and interests of users, how this influences their uses of non-financial performance information, and therefore how it might be provided to better meet their different needs. For example, communities’ making choices about the costs and services provided by their Council might require different information from that of the media providing public information about the results of policy changes in government departments. An alternative approach might be to require strategic planning to be linked to organisational performance management and communicated in ways that take account of stakeholders and stakeholders’ needs and preferences for receiving information. This might provide another way of addressing our desire for service improvement, as well as signalling that information should be prepared based on an understanding of internal and external users needs.

In recognising that there is a range of users and uses of non-financial performance information, there is one use that I have seen expounded both recently and over time that I think we need to agree to reject. This is the idea that the purpose of accountability is blame-ability.

Idle curiosity here – I consulted the online Oxford Dictionary which was surprisingly silent on accountability, but did find ‘accountable’ which it defined as ‘required or expected to justify actions or decisions’; and as ‘understandable’. Likewise under ‘account’ I found ‘a description of an event or experience’.

We are all aware that relationships between outcomes and outputs are complex; that relationships and expectations change over time; and that performance can be influenced by a range of circumstances and events. So the idea of non-financial information being about making actions and decisions understandable, and explaining subsequent events is that which I think fits best. It is true that sometimes information and its consideration lead to blame, but this is a secondary effect rather than the purpose of information.

I hear people talk a lot about the need for joined up government for collaboration, and it seems to me that the key to joining up is to be able to talk openly and honestly about what we think is happening and how this can be improved – in Parliament and the public sector but also across communities. Good quality information – both non-financial and financial is the oil that allows this conversation.

Now perhaps, to quote Rodgers and Hammerstein, I am being ‘a cock-eyed optimist’. Perhaps for many of us who have a part to play in achieving public sector performance, our fear of blame is greater than our desire for gain? I am distressed to hear people questioning the usefulness of preparing non-financial performance information. Perhaps we need to look at ourselves and our behaviour and consider how we create the incentives for true openness and collaboration rather than simply walking away from the challenge of performance information.

Qualities of service performance information

As with all general purpose financial information, service performance information is premised on providing accurate information relevant to the needs of users to aid their decision-making.The qualitative characteristics within the Framework for the Preparation and Presentation of Financial Statements (relevance, understandability, reliability, and comparability) applying to financial reports apply equally to the elements of service performance reports - to the selection of outcomes and outputs, the selection of performance measures, the determination of targets, and any other related information disclosed in the service performance reports

Therefore overall I expect entities to have a performance reporting framework that reflects the statutory requirements, which in my view would comprise:

• The medium-term component, which should include information on the reporting entity’s objectives, outcomes, impacts, and operating intentions, together with related performance measures and standards and other information required by legislation and generally accepted accounting practice (GAAP):
  • clearly identified outcomes and supporting discussion on the entity’s role, functions, strategic priorities, challenges, and risks to provide the context for the entity’s role and functions;
  • main measures and standards for outcomes, objectives or impacts that are clearly specified cover a period of 3 years and provide baseline data that places measures and standards in a more meaningful context and allows progress to be tracked.
• The annual forecast SSP component, which should include information on the reporting entity’s intended outputs, together with related performance measures and standards and other information required by legislation and GAAP;
  • logically aggregated output classes/outputs with clearly specified outputs that are external impact focused;
  • clearly specified performance measures and standards that are relevant and balanced and provided baseline data for measures and standards
• A coherent structure and integrated contextual information that makes evident, through linking within and between the information in the two components:
  • the reasons for the entity’s outputs;
  • the focus of its reporting including the rationale for, and the relationships among, the elements, performance measures, and standards; and of course
• the annual historical SSP in the annual report which reports and explains actual performance against the standards in the forecast SSP.

My observations on the state of non-financial performance information

Over the last couple of years, my Office has:

• reviewed most government department and Crown entity Statements of Intent in depth;
• audited the 2006 – 16 Long Term Council Community Plans of local authorities; and
• conducted a performance audit looking at the statements of corporate intent (SCI) prepared by entities such as council-controlled organisations, state owned enterprises and crown research institutes.

Overall I’ve been disappointed that many entities’ service performance information:

• did not seem to set out coherent performance frameworks showing logical linkages from the medium-term outcomes information and organisational strategies to the annual output information, and
• did not have well-specified, relevant performance measures and standards for both the medium-term and SSP information.

Of particular concern were:

• the weak linkages of the medium-term contextual and strategic information to the annual forecast SSP. These linkages should clearly set out the rationale for the outputs and identify key dimensions of service performance for each output. Assessments of the relevance of performance measures and standards, and subsequent achievements against standards, can be made only in the context of the entity’s operating environment and strategic direction. Therefore a logical linkage between strategy and service delivery is vital not simply for external accountability but more importantly for management evaluation and future service planning.
• identification and specification of the elements, measures and standards in respect of both output information and medium-term achievement. In my view, just as for financial reporting standards, if the underlying elements of the SSP are not properly identified and treated, the basis of the reporting is undermined for external accountability and for the usefulness and relevance of information for management and business improvement purposes.
• the lack in many instances of robust best estimate-based standards combined with historical or benchmark information that gives context to the anticipated achievement.

Through our work we have identified a range of areas for improvement in service performance reports to better meet the expectations laid out in legislation and GAAP. These include a need to better:

• apply the definitions of the elements for service reporting (in particular of outputs);
• consider how to meaningfully aggregate elements to strike a reasonable balance in the extent of detail to communicate a reasonably complete yet succinct and coherent account;
• link the medium-term contextual and strategic information to the annual forecast SSP to clearly set out the rationale for the delivery of service and also identify key dimensions to service performance for each output;
• identify measures for both medium-term achievement (in terms of the influence or impact sought on the wider state or community) and for output delivery; and
• identify robust best estimate-based standards combined with historical or benchmark information that gives context to the set standards.

In my view, the quality of non-financial performance reporting is a significant weakness in the public sector’s accountability to its stakeholders.

I recognise that in 2007 the Treasury undertook a review of accountability documents and I’m supportive of any initiatives intended to improve the state of information and its usefulness. I confess to some anxiety, however, that undue focus on structural change could displace effort that might have been directed to improving the quality of information – and 15 years of not doing this well is quite enough.

There are wider improvement options that could be considered that are beyond my scope to discuss but in my – very ‘auditor’ – view of the world we need to focus on substance over form. By this I mean attention by both entities and central agencies to the quality of information that appears in both forecast and annual reports rather simply than to its presentation.

Enduring improvement in performance information will require clear and consistent policy objectives, strong central coordination and direction, well-established good management practices, and an unwavering accountability focus on understanding results. Despite legislation and other high level accounting and practice guidance, we have struggled to provide this over the last 15 years in the central government sector. One of the things we all recognise is that for performance information to be useful, it needs to be nuanced for and owned by the entity preparing it – so high level guidance will only take us part of the way and improvement is likely to require deeper and more sustained attention.

My Office's intended work on non-financial performance information

Over the upcoming year, I intend maintaining my focus on performance information and hope that in tandem with the efforts of central agencies this will help drive the improvement needed in public sector performance information. Indeed as a result of the legislative changes and with our emphasis on prospective service performance information over the last year, we have been reviewing and updating our own audit methodology and standards for service performance information to identify where we too can do better.

I started by noting that:

• many users would say that the information currently produced doesn’t meet their information needs or that it is not understandable for a lay-reader; and
• entities themselves do not use the information for management and business improvement purposes.

Should we be surprised and admit defeat to the challenge of public sector accountability?

Because there are so many users of, and uses for, performance information, there is unlikely to be a silver bullet solution to their complex and varied information needs and I’m not sure why we would delude ourselves with such a simple suggestion. This is why our public sector accountability arrangements don’t rely simply on performance information but include a range of rights and means to obtain information for the range of users.

My own view is that the very diversity of expectations and uses of performance information is part of what bedevils it – and so long as we continue to expect it to answer every question, non-financial performance information will remain a quest for the Holy Grail. I recall still that when the legislative requirements for SSPs were first introduced, some individuals in central agencies thought there would be no need for other work such as the performance audits my Office undertakes because all the information would be in the SSP. We are a long way now from thinking that kind of information can appear in one statement – but no further advanced in having a better answer to service performance improvement and accountability. Despite the obvious inherent tensions and flaws, neither the ‘status quo’ nor the ‘abandon’ options appear tenable. So we need to knuckle down with this, in my view, not fully implemented part of the reforms, and genuinely give non-financial information the consideration it deserves, in the spirit it deserves – be we preparers, reviewers, or users.

Many people don’t understand a set of financial statements, but that doesn’t discredit the value of the information or its purpose. Likewise, while we should endeavour to ensure service performance information is well expressed in commonly used language, we should also expect the reader to come to the information moderately informed and willing to make the effort to understand it.

The question of the use that entities make of their service performance information is a little more perturbing. By way of parallel – what confidence would we have in an organisation that said it did not use its financial information for management and business improvement purposes?

The preparation of service performance information should be the reflection of good management practice involving clear articulation of strategy, linking of strategy to operational and other business plans, monitoring of the delivery of operational and business planning and evaluation of strategy impacts and results.

Non-financial performance reports are essential documents in ensuring government departments and Crown entities are held accountable to Parliament. Parliament and the public rely on these documents. If those documents report poor performance, democracy requires departments to be held accountable for ensuring they remedy performance issues. If Parliament is unable to adequately assess entity performance because of the poor quality of performance reporting, then we would expect those entities and their oversight agencies, responsible for the quality of reporting to be held accountable for their inadequate reporting.

I note again that here I speak not of blame but of making actions and decisions understandable, and explaining subsequent events. I speak not just of demonstrating accountability but of achieving continuous improvement in public sector effectiveness.

While there are some good examples, these are in my view too few and far between. The only thing I can generally say about the state of performance information currently is that there is a level of public information about government departments and Crown entities and their services.

Addressing the weaknesses in performance information requires greater clarity, persistence and consistency at a government and public sector-wide level. As long as the weaknesses I’ve described persist, parliamentarians and the public can have limited assurance that the performance information of public entities reflects the purpose and impact and effectiveness of their endeavours.

page top