Part 3: Changes to our reporting to entities, Ministers, and select committees
3.101
During the past 13 years, we have reported to government departments, Crown entities, and State-owned enterprises (referred to collectively as "entities" in this Part) our assessment ratings of aspects of financial and service performance management (which we have referred to as the "five management aspects"). We have also reported our assessment ratings to responsible Ministers and select committees.
3.102
This is the last year that we will be reporting ratings under the current five management aspects framework (see Part 2 for the results of the 2005/06 government department audits).
3.103
In 2006/07, we will begin reporting under a new assessment framework, which is designed to be simpler and, in our view, clearer and easier to understand. Our new reporting will address the same subject matter as the previous framework – the areas of the management control environment, information systems, and controls necessary to produce the audited financial statements, including Statements of Service Performance (SSPs).
3.104
Our shorthand term for the new assessment framework is "ESCO", which emphasises that the assessment is of the environment, systems, and controls underlying the financial statements.
Why we are changing our assessment framework
3.105
We have reviewed our terminology to help avoid the implication that our assessment covers overall management performance. The assessment is a byproduct of the financial statements audit (including SSPs), and we consider that our new terminology reflects this more accurately.
3.106
We have also reviewed and refreshed our assessment framework to provide greater transparency about how grades under the new framework are assigned. We have made substantive changes to our assessment and reporting model, which we expect will improve the understandability and usefulness of our reporting to Ministers, select committees, and the entities that we audit.
3.107
During the last couple of years, entities have expressed some concerns that the basis for their five management aspects ratings was not clear. They were uncertain whether auditor expectations were the same for all entities, and about how any particular concern identified by the auditor might affect their rating for any specific aspect. This means they did not always understand why they received their rating, and were not certain what action might be required to improve it.
3.108
We have also been concerned that users' perceptions of the five management aspects assessment sometimes differed from that intended. Our changes are aimed at improving the clarity and transparency of our assessment so that users can better understand what the commentary and grades in the new framework are based on, and what they do and do not provide information about.
3.109
In addition, we have sought to ensure that our revised ESCO framework aligns with our obligations under international auditing standards that are likely to come into effect in the next couple of years.
The new assessment framework
3.110
The most substantive change is to the basis on which grades will be assigned. Grading will now be based purely on any deficiencies observed by auditors and their corresponding recommendations for improvement. We expect that the reasons for the auditor's choice of grade will be clearer than under the current framework, and the commentary will support continual improvement by entities.
We will report audit conclusions on three "aspects"
3.111
The new framework will cover the same areas in the simpler form of three aspects:1
- management control environment;
- financial information systems and controls; and
- service performance information and associated systems and controls.
3.112
The management control environment will now cover the organisational context for the production of the financial statements and, if applicable, the SSP.2 It includes attitudes, policies, and practices relating to strategic planning; ethics; governance and management styles; organisational structure and the assignment of authority and responsibility; human resources; risk management; and key entity-level control policies, procedures, information systems, and communication.
3.113
The two other aspects represent the information systems and controls that underlie the two main areas to which the audit opinion relates – the financial statements and the SSPs (if applicable).
Our commentary will emphasise areas for improvement
3.114
Under each of the three aspects, auditors will identify any significant deficiencies observed during the annual audit. For each deficiency identified, they will recommend improvement. The recommendations for improvement, individually and collectively, will determine which grade the auditor assigns for the aspect. In addition to focusing on areas for improvement, auditors will use their discretion to provide brief general comments to place their discussion of deficiencies in a proper context.
We will use a four-point grading scale
3.115
The previous five management aspects framework had a five-point scale.3 We have reduced the points on the scale by one for ESCO, with our deficiency- and recommendations-based scale offering a simpler and clearer explanation of why entities receive the grades they get.
Figure 3.1
Our four-point grading scale in the new assessment framework
| Grade | Explanation of grade |
|---|---|
| Very good | We recommend that no improvements are necessary. |
| Good | We recommend that improvements would be beneficial and that the entity addresses these. |
| Needs improvement | We recommend that improvements are necessary and should be addressed at the earliest reasonable opportunity. |
| Poor | We recommend that major improvements are required, to which the entity should give urgent attention. |
3.116
We expect that the majority of entities will fall within the two middle grades, which offer distinctly different signals about the importance and urgency of the improvements recommended.
We will provide better explanatory notes
3.117
To enhance users' understanding of what is covered in the ESCO assessment and how the grades are derived, the notes in our reporting to entities, Ministers, and select committees will explain each aspect, as well as the assumptions underlying the auditors' assessments.
3.118
The most important point to note is that the auditor's conclusions on deficiencies (that is, the gap between "how things are" and "how they should be"), and the associated recommendations for improvement, are based on the auditor's assessment of how far short things are from "best practice". The auditor's notion of "best practice" is based on their professional expertise and judgement, taking into account what is deemed appropriate for each entity, given its size, nature, and complexity.
3.119
Another important assumption relates to the cost-benefit of introducing improvements. Auditors will recommend improvements only when they consider (in their professional judgement) that the benefit of the improvement will justify its cost.
3.120
Our explanatory notes will also explain why grades may fluctuate from year to year. This could happen even if nothing has changed within the entity. Some of the factors that may cause the "goalposts to move" include changes in the operating environment, standards, best practice expectations, or auditor emphasis. This suggests that the long-term trend in grade movement will be a more useful indicator of progress than the shorter term, year-to-year grade changes.
A greater emphasis on the appropriateness of service performance information
3.121
To coincide with the introduction of the new ESCO assessment framework, we will be placing a greater emphasis on the appropriateness of the service performance information reported in SSPs. This is necessary because, unlike financial statements, SSPs do not have prescribed standards that govern their content, measurement, disclosure, and presentation requirements. We therefore need to review Statements of Intent (SOI) to determine the context for SSP performance measures, as well as entities' processes and rationale for selecting them, to form our conclusions on their appropriateness. Our review of SOIs is discussed further in Part 8.
3.122
Our shift in emphasis takes into account the recent changes in the Public Finance Act 1989 and the new Crown Entities Act 2004, reflecting the Managing for Outcomes and Managing for Results initiatives. Last year, we reported to Parliament that the relationship between the outcomes a department seeks to achieve and the outputs it delivers in order to contribute to those outcomes (Parts A and B of the SOI) has generally not been well enough developed, and considerable improvement is required.
3.123
We expect that departments should now have adapted more to the new requirements for SOIs and their outcome-to-output links. Service performance reporting, in particular the "appropriateness" of the service performance measures used, will be an area of emphasis in our 2006/07 annual audits. This emphasis will be reflected in the assessments we provide in our ESCO reports to entities, Ministers, and select committees.
When the new assessment framework will take effect
3.124
We will begin reporting under the new assessment framework from 2006/07, issuing grades for the first two aspects (management control environment, and financial information systems and controls). We will not assign grades for the service performance aspect in 2006/07, but we will provide comments on where improvements can be made.
3.125
We expect that shortcomings identified in our reviews of service performance reporting will, in future, affect entities' grades more significantly than they have to date, for two reasons – our move to a deficiency-and-recommendations-based grading system, and our auditors giving greater emphasis in their reports to the appropriateness of performance measures. Our transitional approach will allow entities time to adjust to this change of emphasis. We will grade the service performance aspect for the first time in the 2007/08 ESCO assessment.4
1: The current five management aspects are (1) Financial Control Systems, (2) Financial Management Information Systems, (3) Financial Management Control Environment, (4) Service Performance Information Systems, and (5) Service Performance Management Control Environment.
2: The ESCO report will encompass the Statements of Service Performance (SSPs) of government departments and Crown entities (other than Crown Research Institutes). Crown Research Institutes and State-owned enterprises are not required by legislation to produce SSPs; accordingly, we will not cover information systems and controls relating to service performance for these entities.
3: (1) Excellent, (2) Good, (3) Satisfactory, (4) Just adequate, and (5) Not adequate.
4: This will cover the 2008/09 SOI and the 2007/08 SSP.
page top