Part 2: Assessing entities' environment, systems, and controls

2.1
In this Part, we report on our 2008/09 assessments of the environment, systems, and controls of government departments, Crown entities (excluding school boards of trustees and tertiary education institutions), and State-owned enterprises (SOEs).

2.2
We set out our findings on the management control environment and financial information systems and controls for the past three years. We also separately set out our findings for the first year of grading service performance information and associated systems and controls.

Background

2.3
As part of the annual financial audit, our auditors examine, assess, and grade central government entities' environment, systems, and controls for managing and reporting financial and service performance information.

2.4
This examination is part of the auditor's work in forming an opinion on the financial and service performance statements. Deficiencies identified by auditors are highlighted as areas for improvement. The grades assigned directly reflect the recommendations for improvement as at the end of the financial year. The framework and commentary is intended to support continual improvement by public entities.

2.5
We report our assessments to the entity, the responsible Ministers, and the relevant select committees. In the 2006/07 financial year, we introduced a new assessment framework to improve the transparency, usefulness, and understandability of our reporting.

The areas we examine

2.6
We assess and report on three areas:

• management control environment;
• financial information systems and controls; and
• service performance information and associated systems and controls, where applicable.

2.7
The management control environment is the foundation of the control environment, and the areas that our audit may consider include:

• strategic planning, policies and practices;
• ethics;
• governance;
• organisational structure and assignment of authority and responsibility;
• human resources;
• risk management;
• overall legislative compliance arrangements;
• key entity-level control policies, procedures, information systems, and communication;
• management style; and
• emphasis on effectiveness and efficiency.

2.8
Financial information systems and controls are the systems and controls (including application-level computer controls) over financial performance and financial reporting.

2.9
Service performance information and associated systems and controls refers to the quality of the service performance measures selected for reporting against, and the systems and controls (including application-level computer controls) over service performance reporting.

2.10
This, 2008/09, is the third year that we have graded the management control environment and financial information systems and controls. It is the first year that we have graded service performance information and associated systems and controls.

2.11
We allowed for a transitional period before we graded service performance information and associated systems and controls by not grading them in 2006/07 or 2007/08. However, our auditors did assess them in both these years and provided comments to entities about improvements they could make.

2.12
The Auditor-General has a statutory requirement to attest to the statement of service performance included in the annual reports of government departments and Crown entities (excluding school boards of trustees and tertiary education institutions). Grading in 2008/09 was part of our phasing in of improvements to the way we audit service performance information to meet this requirement. Part 5 sets out more information about our intentions for auditing service performance information.

Our grading system

2.13
Auditors base the grades that they assign in their assessment on deficiencies observed through the audit, and on the associated recommendations for improvement. Auditors' conclusions on deficiencies (that is, the gap between "actual practice" and "how practice should be"), and the associated recommendations for improvement, are based on their assessment of how far the entity's practice falls short of good practice. Good practice is based on auditors' professional expertise and judgement, taking into account what is deemed appropriate for each entity, given its size, nature, and complexity. Auditors' professional judgement is informed by many factors, including national and international standards, knowledge of good practice, and standards and expectations for the public sector. The auditor makes recommendations for improvements only when, in their judgement, the benefits of the improvement would justify the costs.

2.14
Our grading scale is shown in Figure 1.

Figure 1:
Grading scale for assessment of environment, systems, and controls

Grade	Explanation of grade
Very good	No improvements are necessary.
Good	Improvements would be beneficial and the entity should address these.
Needs improvement	Improvements are necessary and we recommend that the entity should address these at the earliest reasonable opportunity.
Poor	Major improvements are required and we recommend that the entity should urgently address these.

Interpretation of results

2.15
Our auditors' approach and the standards they apply reflect the unique circumstances of each entity in each financial year. Entities vary greatly in size and organisational structure, and sometimes undergo restructuring or other organisational changes. Grades for a particular entity may fluctuate from year to year. Some of the factors that may cause fluctuations include changes in the operating environment, standards, good practice expectations, and auditor emphasis. For these reasons, we advise caution when comparing grades between years and between different entities.

2.16
How an entity responds to the auditor's recommendations for improvement, as they arise, is more important than the grade change from year to year. A downward shift in grade, for example, may not indicate deterioration – it may be that the entity has not kept pace with good practice expectations for similar entities between one year and the next. Consequently, the long-term trend in grade movement is a more useful indication of progress than year-to-year grade changes.

Results for the management control environment and financial information systems and controls

2.17
Results for 2008/09 are pleasing, with 137 of the 150 entities¹ (91%) receiving either a very good (no improvements needed) or good (only beneficial improvements needed) grade for the management control environment. Likewise, 138 entities received either a very good or good grade for the financial information systems and controls.

2.18
Figure 2 shows a summary of the 2008/09 grades, by type of entity, for the management control environment and for financial information systems and controls. There were no poor grades for either aspect in 2008/09.

Figure 2:
Summary of grades for 2008/09

		Grades received for MCE			Grades received for FISC
Type of entity*	Number of entities	VG	G	NI	VG	G	NI
Government departments	39	15	21	3	12	24	3
District health boards	21	1	15	5	0	15	6
Crown research institutes	8	6	2	0	1	7	0
Other Crown entities	65	39	22	4	34	29	2
State-owned enterprises	17	12	4	1	6	10	1
Total	150	73	64	13	53	85	12

MCE – Management control environment. FISC – Financial information systems and controls.

Ratings used are: VG – Very good, G – Good, NI – Needs improvement.

Government departments exclude Offices of Parliament, the Government Communications Security Bureau, and the Security Intelligence Service. School boards of trustees and tertiary education institutions are not included in the above analysis for other Crown entities. Crown research institutes reduced from nine entities in 2007/08 to eight entities in 2008/09. Air New Zealand Limited has been included as if it were a State-owned enterprise.

2.19
Improvements would be beneficial to the management control environment of 43% of all central government entities and to the financial information systems and controls of 57% of all entities.

2.20
Thirteen entities received needs improvement grades for the management control environment, and 12 received needs improvement grades for financial information systems and controls. District health boards (DHBs) were over-represented in each aspect, with five and six DHBs respectively.

2.21
It is disappointing that five entities (one government department, one SOE, and three DHBs) received needs improvements grades for both their management control environment and their financial information systems and controls.

2.22
We examined the detailed recommendations for improvement in the 9% of all entities that received needs improvement grades in 2008/09 for both the management control environment and financial information systems and controls. We did not find any particular recurring themes or issues. However, we will continue to carry out such analysis to see if there are any particular common themes or issues on which we can provide further guidance to both entities and auditors.

2.23
We will also examine the recommendations for improvement in entities receiving good grades to see whether there is any further assistance or support we can provide auditors or entities, as appropriate.

2.24
We have published entity names and grades in this report only for the DHB sector (see Part 6). Although this information is available to the public through other means, we consider that there would be benefit in making this information more accessible to the public. We intend to look into this further.

Trends from 2006/07 to 2008/09

2.25
Figures 3 and 4 below show the trends in grades for all 150 entities for the management control environment and financial information systems and controls during the past three years.

2.26
The overall trend is pleasing for both the management control environment and financial information systems and controls, because it shows that entities have generally improved their grades. This reflects that entities have satisfactorily made the changes that we recommended to them.

Figure 3:
Management control environment – grades for 2006/07 to 2008/09, as percentages

Figure 4:
Financial information systems and controls – grades for 2006/07 to 2008/09, as percentages

2.27
The results show an increase in very good grades for the management control environment, from 38% of all entities in 2006/07 to 48% in 2008/09. Good grades have decreased from 51% in 2006/07 to 43% in 2008/09, showing that entities have generally improved from good to very good.

2.28
Results for the financial information systems and controls of all entities show an increase in very good grades, from 21% of all entities in 2006/07 to 35% of all entities in 2008/09. Good grades decreased from 68% in 2006/07 to 57% in 2008/09, also showing that entities have generally improved from good to very good for this aspect.

2.29
It is pleasing that no entities have received a poor grading for either aspect since the introduction of this framework in 2006/07. However, there has been only a slight decrease in the proportion of needs improvement grades, from 11% in 2006/07 to 9% in 2008/09 for the management control environment, and to 8% in 2008/09 for financial information systems and controls.

2.30
As noted above (paragraph 2.15), grades may fluctuate from year to year depending on a range of factors. We are interested in how entities respond to auditors' recommendations for improvement as well as the trends in grade movements.

2.31
Most entities responded either in full or in part to recommendations from the 2007/08 audit for both beneficial and necessary improvements.

2.32
Deficiencies in the management control environment were not resolved at all for six entities (three government departments, two DHBs, and one other Crown entity). Deficiencies in financial information systems and controls were also not resolved at all for three entities (two DHBs and one other Crown entity).

2.33
It is always concerning when entities do not take appropriate action to address the matters raised by our auditors and do not achieve the improvements recommended. Our auditors continue to work with the entities concerned to ensure that the recommendations are implemented.

Figure 5:
Management control environment – grades for 2008/09, by type of entity, as percentages

Figure 6:
Financial information systems and controls – grades for 2008/09, by type of entity, as percentages

Sector analysis

2.34
Figures 5 and 6 show the proportion of grades within each of the central government sectors for 2008/09. Results for each sector are discussed below.

Government departments

2.35
Figures 7 and 8 show steady improvement overall in both the management control environment and financial information systems and control grades for government departments since 2006/07.

2.36
During the last three years, the proportion of government departments receiving either a very good (no improvements) or good grade (only beneficial improvements) increased from 79% to 92% for the management control environment and from 84% to 92% for financial information systems and controls.

Figure 7:
Management control environment – grades for government departments from 2006/07 to 2008/09, as percentages

Figure 8:
Financial information systems and controls – grades for government departments from 2006/07 to 2008/09, as percentages

2.37
For both aspects, the proportion of government departments receiving very good grades has increased year after year, while the proportion has decreased for needs improvement grades.

2.38
In 2008/09, three government departments (8%) were graded as needs improvement for the management control environment, and three were graded as needs improvement for financial information systems and controls.

2.39
Six government departments fully resolved deficiencies that we had identified and improved their management control environment after the 2007/08 audit, helping them to achieve very good grades in 2008/09.

2.40
However, three government departments that were graded as needs improvement for their management control environment in 2008/09 were unchanged from 2007/08. Four of the five government departments graded as needs improvement in 2007/08 for financial information systems and controls have resolved deficiencies, resulting in good grades in 2008/09.

2.41
One government department has been graded as needs improvement for both the management control environment and financial information systems and controls in all three years of grading, which is a serious concern.

2.42
We expect entities to take appropriate action to address the matters raised by our auditors and to make the improvements that we recommend. In the case referred to above, some deficiencies identified in previous audits have been resolved, but improvements are still needed in both aspects. We continue to report our findings to the relevant Minister and select committee.

Other Crown entities

2.43
The other Crown entities group is the largest of the central government sectors, with 65 entities. It includes Crown agents (excluding DHBs), autonomous Crown entities, and independent Crown entities.

2.44
In the three years of grading, at least 94% of other Crown entities received a grade of either very good (no improvements necessary) or good (only beneficial improvements needed) for their management control environment or financial information systems and controls.

2.45
Figures 9 and 10 show the trend since 2006/07 for an increase in very good grades for both the management control environment (from 53% to 60%) and for financial information systems and associated controls (from 32% to 52%). Good grades have decreased during this period, showing that entities have addressed our recommendations for improvement.

Figure 9:
Management control environment – grades for other Crown entities from 2006/07 to 2008/09, as percentages

Figure 10:
Financial information systems and controls - grades for other Crown entities from 2006/07 to 2008/09, as percentages

2.46
The results show a small increase in the number of entities needing to improve their management control environment, from three entities in 2007/08 to four in 2008/09. Two of these entities only partially resolved the deficiencies identified in the 2007/08 audit, and again received needs improvement grades in 2008/09.

2.47
Two entities received needs improvement grades for their financial information systems and controls in both 2008/09 and in 2007/08. Deficiencies identified in the 2007/08 audit were only partially resolved in one case and not at all in the other.

2.48
Five entities addressed our recommendations, after the 2007/08 audit, for making beneficial improvements to their management control environment, helping them to achieve very good grades. Similarly, seven entities improved to very good grades for their financial information systems and controls.

2.49
As noted above (paragraph 2.15), grades can be lower from one year to another because of a range of factors, such as organisational change or auditor emphasis. Six entities received lower grades in 2008/09 than in 2007/08 for their management control environment, and four entities received lower grades for their financial information systems and controls.

Crown research institutes

2.50
Figures 11 and 12 show that, in 2008/09, all Crown research institutes (CRIs) continued to be assessed as either very good or good for both the management control environment and financial information systems and controls. This is consistent with results from the previous two years, and shows that CRIs have generally sound environments, systems, and controls. We are pleased to see this continued strong performance.

Figure 11: Management control environment – grades for Crown research institutes from 2006/07 to 2008/09, as percentages

Figure 12:
Financial information systems and controls – grades for Crown research institutes from 2006/07 to 2008/09, as percentages

2.51
Six of the eight CRIs had their management control environments assessed as very good in 2008/09. One CRI's financial information systems and controls were assessed as very good.

2.52
Looking at how CRIs have responded to the auditor's 2007/08 recommendations for beneficial improvements to financial information systems and controls, one CRI fully resolved the deficiencies identified by auditors and improved its grading from good in 2007/08 to very good in 2008/09. Five CRIs partially resolved their deficiencies and received unchanged good grades for financial information systems and controls in 2008/09.

2.53
As Figure 12 shows, seven CRIs (88% of the sector) could move to a very good grade by implementing the auditor's recommendations identified in 2008/09 for their financial information systems and controls.

District health boards

2.54
Figures 13 and 14 show that DHBs continue to be over-represented in the lower grades, with necessary improvements identified for both the management control environment (24% of DHBs in 2008/09, compared to 19% in 2007/08²) and financial information systems and controls (29% of DHBs in 2008/09, compared to 33% in 2007/08).

Figure 13:
Management control environment – grades for district health boards from 2006/07 to 2008/09, as percentages

Figure 14:
Financial information systems and controls – grades for district health boards from 2006/07 to 2008/09, as percentages

2.55
We discuss the results of DHB audits for 2008/09 in more detail in Part 6.

State-owned enterprises

2.56
Overall, the results in 2008/09 show that the SOEs have generally sound management control environments and financial information systems and controls, with 94% receiving either a very good or a good grade for these aspects. Twelve of the seventeen SOEs received very good grades for their management control environments, and six received very good grades for their financial information systems and controls.

2.57
Figures 15 shows that SOEs have generally sound management control environments, with between 66% and 76% of SOEs having received very good grades in each of the years since 2006/07.

Figure 15:
Management control environment – grades for State-owned enterprises from 2006/07 to 2008/09, as percentages

2.58
Figure 16 shows an increase in very good grades for financial information systems and associated controls, from 17% in 2006/07 to 35% in 2008/09. Good grades have decreased during this period, showing that entities have followed the auditor's recommendations.

Figure 16:
Financial information systems and controls – grades for State-owned enterprises from 2006/07 to 2008/09, as percentages

2.59
As Figure 16 shows, 10 entities (59% of the sector) could move to a very good grade by implementing the auditor's recommendations in 2008/09 for their financial information systems and controls.

2.60
In 2007/08, 12 SOEs received good grades for their financial information systems and controls. Four of these entities fully resolved, and eight entities partially resolved, deficiencies identified by auditors in 2007/08. This resulted in two of the entities receiving very good grades in 2008/09 for this aspect.

2.61
One entity received a needs improvement grade for both the management control environment and financial information systems and controls in 2008/09. This entity also received a needs improvement grade for its management control environment in 2007/08 and needs improvement grades for both aspects in 2006/07.

Results for service performance information and associated systems and controls

2.62
As we noted in paragraph 2.12, the Auditor-General has a statutory requirement to attest to the statement of service performance included in the annual reports of government departments and Crown entities (excluding school boards of trustees and tertiary education institutions). We graded service performance information and associated systems and controls for 125 entities (detailed below). There is no statutory requirement relating to CRIs and SOEs. We discuss this further in Part 5.

2.63
The auditor's expectations for the quality of the service performance information are derived from:

• legislation (Public Finance Act 1989 and Crown Entities Act 2004);
• accounting standards and guidance;
• Technical Practice Aid 9: Service Performance Reporting; and
• guidance and instructions from the Treasury and State Services Commission.

2.64
These expectations are reflected in the Auditor-General's Auditing Standard 4 (Revised) – The Audit of Service Performance Reports.

2.65
Our primary objective in examining this aspect was to assess the quality of the forecast performance reports and supporting systems and controls. Auditors reviewed the Statement of Service Performance included in the 2008/09 annual report, the 2009-2012 Statement of Intent, 2009/10 Forecast Statement of Service Performance, and the associated information contained in the Information Supporting the Estimates.

2.66
In reviewing and grading forecast reports, our auditors considered the relevance, reliability, understandability, and comparability of information in presenting a clear and cohesive description of performance. Performance elements (outcomes, impacts, and outputs) and their associated measures should be presented in a clear and informative context.

2.67
We recognise that the absence of reporting standards for non-financial performance reports requires more judgement, both for those preparing reports and for auditors in issuing opinions on reports.³ As well as our normal quality assurance processes, we supported consistency in grading among our appointed auditors by carrying out team-based peer reviews of proposed grades. This resulted in team-based peer reviews of more than 60% of the service performance grades.

2.68
Auditors reviewed and commented on (but did not grade) performance reports in the 2006/07 and 2007/08 audits. Results of these reviews showed that the overall quality of performance reporting by central government agencies was poor.

2.69
Results of the first year of grading service performance information and associated systems and controls are still disappointing and show that there continues to be room for improvement. In our view, improving the quality of performance reporting is critical, not only for demonstrating accountability but also for improving public sector effectiveness.

2.70
In recent years, our Office has placed much emphasis on improving auditing standards, methodology, and reporting by appointed auditors. We are also working with the Treasury on a wider framework for service performance management and accountability in the public sector.

2.71
In 2008/09, auditors assessed and graded 39 government departments, 21 DHBs, and 65 other Crown entities for service performance information and associated systems and controls. Figure 17 shows a summary of grades for departments and other Crown entities. DHBs were all graded as poor/needs improvement, but are excluded from our analysis in this section and discussed separately in Part 6.

Figure 17:
Summary of grades for 2008/09

		Grades received for service performance information and associated systems and controls
	Number of entities	G	NI	P
Government departments	39	9	26	4
Other Crown entities	65	22	38	5
Total	104	31	64	9

Ratings used are: VG – Very good, G – Good, NI – Needs improvement.

2.72
No entities were rated as very good, but 31 entities were graded as good (where we recommended beneficial improvements).

2.73
We assessed 64 entities as needing to improve their service performance reports and associated systems and controls. Nine entities were assessed as poor, requiring major improvements because of significant deficiencies.

2.74
Of the 64 entities needing to improve, the quality of their service performance information and associated systems and controls varied widely. Some entities' service performance information would have required improvements in only a few aspects to achieve a good grade. Other entities' service performance information needed to improve in a range of aspects or had more significant deficiencies, which meant they were closer to a poor grade.

2.75
Our assessments were not significantly affected by the type or size of entity.⁴ We did find that, among government departments, policy departments with significant non-departmental activities were more likely to have been assessed as poor compared with departments having other types of functions.⁵ We have been discussing this issue with the Treasury, which is currently leading work in developing service performance reporting guidance for policy advice.

Recurring themes and conclusions

2.76
Our overall conclusions on the service performance information are similar to our findings from the assessments of service performance reporting from the 2007/08 audits.⁶ We found that:

• Many entities were not clear about what the entity was trying to achieve and what its services were.
• There was often confusion about how an entity would know that it was achieving what it hoped to through delivering quality services.
• Important information about the quality of services and their impact was often omitted. There appeared to be an over-reliance on measures that assessed service quantity and very few measures that assessed service quality.
• There was a lack of contextual information to help the reader understand the targets set and the relative level of performance that an entity was seeking to achieve.
• There were also instances where the information in the Statement of Intent was inconsistent with that in the Information Supporting the Estimates.

2.77
Improvements were recommended to the 31% of entities who received good grades. Commonly, the information was easy to comprehend, and provided a clear picture of what an entity was trying to achieve and how it could be achieved. Overall, the information provided a basis for assessing performance. Although the service performance information was not necessarily complete:

• The information did set out all of the performance framework components (outcomes, impacts, operating intentions, outputs, and performance measures and targets).
• There was a logical flow that linked the outcomes and impacts to outputs. Often the linkages were strengthened through use of tables, pictures, or diagrams.
• There was an adequate range of relevant output performance measures and targets that reflected a range of performance dimensions (such as quantity, quality, and timeliness).

2.78
We expect entities to take appropriate action to address the matters raised by our auditors and to achieve the recommended improvement to service performance reporting and associated systems and controls. We recognise that this is a challenging area, but believe that it is integral to improving the effectiveness and efficiency of the public sector – both in actual performance and demonstrating it through better accountability. Part 5 sets out our intentions and work programme for improving service performance information and reporting.

---

1: Grades for three entities had not been finalised at the time of publication. Results for previous years may also differ to previously published results for those years if the grades were finalised after publication. Figures have been rounded down in some cases to ensure that totals add correctly.

2: Results for district health boards were incorrectly reported last year as 24% needing improvement in 2007/08.

3: See, for example, our publications The Auditor-General's observations on the quality of performance reporting (June 2008) and Central government: Results of the 2007/08 audits (May 2009).

4: Entity size was assessed primarily based on audit effort (hours) as a proxy for a range of factors such as size, complexity, and risk.

5: Government department functions are stated in the Treasury report: Public Sector Financial Management Capability, August 2008.

6: Controller and Auditor-General (May 2009), Central government: Results of the 2007/08 audits.

page top