Part 1: Our performance and effect

Assessing the performance of the Office of the Auditor-General against International Standards.

1.1
This report summarises our findings from assessing ourselves against the Supreme Audit Institution Performance Measurement Framework (the Framework). The assessment provides an integrated picture of the performance of the Auditor-General of New Zealand (the Office). A quality assurance statement from the International Organisation of Supreme Audit Institutions Development Initiative (see Appendix) indicates that our work and the report have been completed to the standard expected by the Framework’s developers.

1.2
The assessment covers the audit functions of the Office, its organisational management, and the effect of governance on performance. This report draws on the detailed assessment and gives an integrated evaluation of the Office’s strengths and weaknesses.

1.3
The detailed report available on our website includes the full findings of our assessment of the Office using the Framework.2

1.4
Our assessment of the performance of the Office is presented in this Part in five sections:

Our operating environment

1.5
New Zealand has a mature public sector accountability regime, which is comprehensively set out in various pieces of legislation. An important part of the regime is our own legislation, the Public Audit Act 2001.

1.6
The Public Audit Act:

  • establishes the independence of the Auditor-General from Parliament and the entities we audit;
  • ensures that the Office has operational autonomy; and
  • provides the Auditor-General with a broad mandate and access to information, so that Parliament and the public can effectively and efficiently hold the public sector to account.

1.7
International Standards recommend that the role of the Auditor-General be written into the state’s legal framework at the highest level, ideally within its constitution. New Zealand does not have a written constitution. Despite the comprehensive nature of the Public Audit Act, it is possible that the Auditor-General’s independence and mandate could be altered or repealed by a simple majority vote of Parliament. However, within New Zealand’s constitutional context, the Public Audit Act is considered to provide a strong framework and is the envy of Auditors-General around the world.

1.8
In New Zealand, legislation sets out the reporting requirements for public sector entities, particularly the content of reports and their time frames. We are fortunate in New Zealand that enough resources are invested into preparing and auditing accountability documents.

Our audit work

1.9
The International Standards for Supreme Audit Institutions (ISSAIs) identify three types of audit: financial, performance, and compliance audits.3 The Public Audit Act makes the Auditor-General the auditor of every public entity in New Zealand. It also enables the Office to perform financial audits, performance audits, and other services, as well as carry out inquiries on request or on the Auditor-General’s own initiative.

1.10
To assess the Office’s compliance with the ISSAIs, we applied the results from our quality assurance programme, which we had assessed under the Framework as robust. The programme covers all appointed auditors during a three-year cycle and assesses the quality of audits in the same areas covered by the Framework’s criteria.

1.11
Before we started this assessment, we considered that we did not do separate compliance audits. However, when we looked at the definition of a compliance audit more closely, and at several assignments our financial auditors carry out in conjunction with financial audits, we realised that we do carry out a small number of compliance audits.

1.12
These audits represent a small proportion of the total audit work of the Office. However, because the Office has not recognised these as a separate type of audit, they have not been subject to the same systematic approaches that govern the Office’s other audit work. We discuss the quality of our compliance audit work in more detail in paragraphs 1.36-1.39.

Strengths

Financial and performance audit standards

1.13
The audit work of the Auditor-General can generally be assessed positively against the Framework. We identified that the Office is particularly strong in financial and performance audit standards and its approach to quality.

1.14
All appointed auditors carrying out financial audits on behalf of the Auditor- General are required to apply the New Zealand equivalents of International Standards on Auditing and, where applicable, the additional and complementary standards of the Auditor-General. These cover all of the fundamental principles set out in the ISSAIs, setting a solid platform for all public sector audits.

1.15
Audit New Zealand is a business unit of the Office. It carries out about 50%4 of the financial audits on behalf of the Auditor-General. Audit New Zealand has an Audit Manual and a Quality Control Manual that provide guidance on how to meet the standards.

1.16
For performance audits, we have a Performance Audit Manual that is aligned to the ISSAIs. This is supported by policies and procedures that set out how to implement requirements to produce high-quality performance audits. The Performance Audit Manual also sets out the requirements for review throughout the performance audit process. Except for missing the recommended step of identifying the design approach5 in the planning of each audit, the Performance Audit Manual complies with all assessment criteria.

Outsourcing audits

1.17
A robust and complete set of financial audit standards is particularly important because we outsource a significant proportion of financial audits.

1.18
We have strong policies and procedures for managing the outsourcing of audits. We meet all of the ISSAI assessment criteria. We have a comprehensive regime for selecting appointed auditors6 that ensures that they have the competence and capability to carry out financial audit work to the Auditor-General’s standards. We also have a comprehensive system of cyclical quality assurance reviews of appointed auditors to confirm they are meeting the required quality standards.

1.19
Audits are primarily allocated to appointed auditors, although a small number of tenders occur. The allocation approach is based on six principles, one of which is that Audit New Zealand is maintained as a “strong and viable” audit service provider.

1.20
Our quality assurance programme is robust. Overall, the quality assurance programme results show that the standard of financial audit performed on behalf of the Auditor-General is high, with few exceptions.

1.21
The quality assurance programme covers all appointed auditors during a three-year cycle and assesses the quality of their audits. The Office has established the Auditor-General’s Statement on Quality Control (AG PES 3), which sets out the review procedures that all appointed auditors must adhere to. This includes procedures for clearing certain non-standard audit opinions through the Office.

Professional development and training

1.22
Audit New Zealand’s leadership framework outlines all of the professional competencies and soft skills that audit staff at every level need to carry out financial audits to a high standard. It also provides the direction for all of Audit New Zealand’s professional development programmes.

1.23
Audit New Zealand’s Performance Planning and Review policy provides the basis for monitoring staff performance against the competencies in its leadership framework. The leadership framework addresses all of the professional competencies and skills identified by the ISSAIs.

1.24
The Office also provides support to all Audit Service Providers to supplement their in-house training programmes by issuing annual sector briefs with information about sector-specific issues and risks. Together, these result in an approach to audit team management that meets all the criteria of the ISSAIs.

1.25
Likewise, we have established an approach to provide performance auditors with the necessary professional competence and skills to carry out performance audits that meet all the criteria of the ISSAIs.

Issues and opportunities for improvement

Quality of small audits

1.26
The Office’s quality assurance programme assigns appointed auditors a quality grade (on a scale of “excellent”, “very good”, “good”, “satisfactory”, and “re-review”) based on the results of review of a sample of their work every three years. In 2015/16, 97% of appointed auditors were graded at least “satisfactory” (up from 95% in 2014/15). The 3% who were graded below satisfactory were exclusively school auditors working in small auditing firms.

1.27
The Auditor-General’s mandate covers about 3700 public entities that are required to report separately, of which about 2500 are schools.

1.28
Our quality assurance programme identified shortfalls in the standard of some of the school audit work. This is an ongoing issue for the Office, which we are monitoring closely. The volume of these audits, the complexity of the schools’ reporting and accountability regime (as well as the increasing complexity and extent of auditing standards), and the low audit fees revenue available have contributed to larger auditing practices progressively leaving this sector. As a result, it has become challenging to identify appointed auditors competent enough to conduct school audits to a consistently high quality.

1.29
It is possible that other small audits carried out by smaller auditing practices may have similar issues. However, we have not identified a substantive trend in any sector other than schools.

Performance audit planning and reporting

1.30
The performance audit process largely meets ISSAI criteria, although we identified inconsistent evidence of compliance with several of the ISSAI criteria for audit planning and reporting. This inconsistency is mainly because of our broad and flexible approach to performance auditing, which means that, in some cases, our reporting provides an overview and observations of a subject matter area. Therefore, several of the ISSAI criteria, such as consideration of materiality and an assessment of the risk of fraud, are not relevant to our audit approach.

1.31
We need to give more consideration to whether we need to explicitly document matters such as the relevance of materiality in the context of the audit topic and the approach to the risk of fraud during the planning and reporting of performance audits with this focus. Alternatively, we might alter the Performance Audit Manual to draw these matters to the attention of the auditor so that they can evaluate the relevance of the ISSAI criteria and document in the audit records the professional judgement they applied.

Audit recommendations follow-up

1.32
The ISSAIs clearly state the importance of completing audits in a timely way and to a high quality. The ISSAIs also emphasise that the audited entity should act on the audit findings and, particularly, the recommendations for improvement. The ISSAIs recommend that there is a system for following up the auditor’s observations and recommendations.

1.33
New Zealand Auditing Standards and The Auditor-General’s Auditing Standards do not specifically require auditors to report on whether entities have addressed recommendations made after financial audits. As a result, our quality assurance processes do not formally monitor this, and it is reported publicly only at an entity level.

1.34
Nevertheless, any follow-up of responses to audit recommendations occurs at an individual audit level. However, we could monitor this part of the audit process more cohesively. There are benefits to having a system to track entities’ responses to audit recommendations, including being in a stronger position to assess trends, able to more effectively monitor how recommendations are implemented,and able to report the audit’s effect on the improvement of the public sector to Parliament and the public.

1.35
In contrast, there are sound processes in place to monitor the response of entities to recommendations made because of performance audits. This allows us to more effectively identify the effects of performance audits.

Compliance audits

1.36
As noted in paragraph 1.11, we had not formally identified compliance audits as a separate product of the Office before completing this assessment. In practice, financial audit teams carry out several compliance audits in conjunction with the annual financial audit. These audits assess compliance with various regulatory and external reporting obligations. This includes the audit of debenture trust deeds, energy regulation disclosures, and tertiary education sector performancebased research funds.

1.37
As a result, although applicable standards within New Zealand Auditing Standards and also The Auditor-General’s Auditing Standards cover non-financial audit work and guidance is provided through applicable annual sector briefs, there is no overarching systematic consideration of these compliance audits.

1.38
In comparison with our approach to financial audits, this compliance work is not consistently addressed in training programmes for audit staff.7 There are inconsistent standards for collecting evidence to support compliance audit conclusions, and there are variable systems in place for appointed auditors to report on the completion of these audits.

1.39
Because of these weaknesses, it is not possible to assess whether the audits are completed within required time frames and the Auditor-General’s annual report does not include any reporting on this work. We are considering a proposal to adapt our data capture systems to enable appointed auditors to submit data about these audits.

Independent review of our practice

1.40
The Framework recommends that, in addition to the quality assurance regime, an audit practice be periodically, externally, and independently assessed. Our financial and performance audit functions have not been subject to a full practice review in recent years.

1.41
Although we were assessed to a high standard against the Framework, it may be timely to consider the value of seeking a full independent practice review.However, we note that the Australian National Audit Office reviews a sample of performance edits biennially, and the Chartered Accountants Australia and New Zealand and the Financial Markets Authority have reviews in progress to assess our financial audit practice. These, along with other reviews of aspects of our audit practice, may be enough.

Our internal culture

1.42
We scored at the highest possible level in the Framework for leadership and internal communications, the operation of human resource management, and staff training.

1.43
Communications from leadership are firmly grounded in the vision and values of the Office, which are clearly communicated in corporate documents, including The Auditor-General’s strategy 2013-17, annual plans, and annual reports. This is a significant strength. The Auditor-General’s Professional and Ethical Standards also set the tone and expectations from which all actions are assessed.

1.44
We have a staff of about 370 who are spread over seven locations and span a wide range of professional expertise and experience. Communications with such a diverse staff is challenging, and there are always opportunities for improvement. However, based on the results of the assessment, the main components for effective leadership and internal communications are in place. The results of the 2016 staff engagement survey largely support this finding, as do previous surveys.8

1.45
Managing human resources is critical to delivering high-quality audits. We have a well-resourced and capable human resources team, which meets all the criteria identified as important to deliver the services we need.

1.46
Sound recruitment processes are in place and are supported by appropriate remuneration, promotion, and staff welfare policies and processes. It would be best practice for all of these policies and processes to be guided by an overall human resources strategy.

1.47
Although we do not currently have a human resources strategy, one is in development, along with a future-focused workforce development plan. Once these are in place, they will provide a critical link between the staff capability goals in the Office’s Strategic Plan and the current professional development and training framework.

1.48
The training and professional development approach supporting financial and performance auditors is highly developed and, as a result, is rated highly by the Framework (see paragraph 1.23). However, the training and professional development for non-audit professional staff and non-professional staff is not managed as effectively. Training for these employees is largely staff-driven and would benefit from being more structured, including having better processes to evaluate how effective the training is.

Our relationship with Parliament

1.49
Based on the Public Audit Act, we are required to report the results of our audits to the House of Representatives at least once a year. We do this through annual reporting on central and local government audits, supplemented by cyclical and issues-focused reporting. We also provide an extensive programme of support with briefings to Parliamentary select committees. We also support the induction programme for new members of Parliament.

1.50
We continue to develop and improve these processes to increase the value and effectiveness of our interactions with Parliament, particularly through our interactions with select committees. The feedback from the 2016 independent survey of external stakeholders included feedback from select committee chairs stating they are “uniformly positive about the role of the Office in helping Parliament hold public sector agencies to account”.9

Our corporate functions

1.51
One of the four strategic objectives of the Office is to lead by example as a model organisation. We recently reported in our 2015/16 Annual Report that we are making good progress against this objective, but “there is still work to do”.10

Strengths

Planning processes

1.52
Organisational planning processes are an area of ongoing focus for us, but this aspect of the strategic planning cycle is rated highly against the Framework criteria. Work programme planning is a highly inclusive, multi-year process using cross-office workshops and involving a large number of appointed auditors and other staff.

1.53
There is a well-established timeline for this process that includes time for consultation with external stakeholders. The process is evaluated and refined annually. The leadership team actively monitors the implementation of the annual work programme.

Ethics and integrity systems and quality control

1.54
The Auditor-General has professional ethical standards that all appointed auditors and other staff must follow. These standards are based on the External Reporting Board ethical standards, which in turn are based on standards established by the International Federation of Accountants. There is also a comprehensive employee independence declaration system to mitigate risks, support ethical behaviour, and address any breach in ethical values. These policies and processes address all the criteria of the ISSAIs and are recognised as best practice.

1.55
We operate a best practice framework of standards to manage quality control and to ensure independence. We play a role in promoting quality and good practice to public entities through our publications. Our quality assurance function over financial audits is highly developed and operates effectively by following a scheduled cycle of reviews.

1.56
The score against this component of the Framework would have been higher if the quality assurance systems were applied annually to other Office products as consistently as they are applied to financial audit work.

Transparent reporting

1.57
We are committed to transparently reporting our performance. Consistent with the reporting requirements that other public sector entities must meet, we report on our financial and operational performance through our annual report. The annual report is routinely completed, tabled in Parliament, and released on our website within three months of balance date.

Issues and opportunities for improvement

Strategic and annual plan content

1.58
We followed a robust process in 2012 to develop our current strategy, incorporating a comprehensive needs assessment focused on the changing face of the public sector environment and our opportunity to positively influence the performance of public entities. The development process included a needs assessment, an environmental scan, and internal and stakeholder engagement processes.

1.59
Although The Auditor-General’s strategy 2013-17 has proved an important foundation document, it has some shortcomings. No implementation matrix or prioritisation was included in the plan, nor is this provided in any other corporate documents. It is important to establish clear priorities related to strategic goals and identify any interdependencies or risks. There is currently a lack of links between the strategy and the annual plan, with one focused at a very high level on outcome and impacts and the other being product-focused.

1.60
To more effectively evaluate whether our strategy goals have been achieved, future revisions of the strategy should include a more explicit prioritisation and cascading implementation framework and a risk assessment tool. Evaluating the effectiveness of the Office’s work should also include a mechanism to capture input from a citizen perspective.

Internal control systems

1.61
We have well-established internal control processes and procedures. However, testing how those controls operate through internal audit reviews has not been as systematic, regular, or comprehensive as it could have been. We have established systems to evaluate and monitor risk, but these are not fully developed or formalised. Risk mitigation systems are fundamental to a robust internal control system. This weakness has significantly affected the score for this element of the assessment.

Support services

1.62
Overall, we have robust financial and operational practices. We do not own significant assets. Our assets are primarily information technology equipment, which are governed and managed under the Information Systems Strategic Plan. This is monitored regularly.

1.63
We lease office premises, and lease renewals are used as the driver to re-evaluate size and location needs. Our approach to deciding recent lease renewals has been comprehensive. However, having an accommodation strategy in place would be best practice and would ensure that leasing decisions are consistent with broader organisational goals.

1.64
It would also be best practice for administrative support functions to be reviewed on a five-yearly cycle. However, we have not carried out such reviews consistently or regularly, but rather in response to issues.

2: See our website: www.oag.govt.nz/2016/sai-pmf/detailed-assessment

3: The Public Audit Act mandate to carry out inquiries does not readily fit within any of the three audit types. As a result, we decided that our inquiries work was outside the scope of this assessment.

4: Audit New Zealand carries out about 50% of financial audits based on audit fees and 23% by number of audits. Excluding the 2500 school audits from the total audit portfolio, Audit New Zealand carries out 64% of audits based on audit fees and 60% of the total number of audits.

5: Three design approaches are set out in the ISSAIs. These are a result-, problem-, or system-oriented design. The New Zealand approach to performance audits is more flexible, enabling performance audits to take any of these approaches, use a blend of approaches, or simply provide an overview of a topic area. This is not considered a significant weakness in the New Zealand Performance Audit Manual and does not affect the quality of the resulting audit reports.

6: The Office uses the following terminology to refer to an auditor carrying out outsourced financial audits. Auditing firms that carry out audits on behalf of the Auditor-General are referred to as “Audit Service Providers”, and the individual auditor contracted to carry out a specific audit on behalf of the Auditor-General is referred to as an “appointed auditor”.

7: Knowledge of how to do this often technically complex audit work is passed on through informal on-the-job training mechanisms that maintain the pool of knowledgeable staff who carry out these audits.

8: For further details, refer to SAI 6 – Dimension 2 and SAI 23 – Dimension 4 of the detailed SAI PMF report.

9: For the full quote from the survey response, refer to SAI 25 – Dimension 2 of the detailed SAI PMF report.

10: Controller and Auditor-General (2016), Annual Report 2015/16, page 19.