Part 2: Methodology

Collecting evidence

2.1
We completed the assessment by reviewing documentation and interviewing staff at all functions of the Office. Wherever possible, interviews were confirmed with documentary evidence. We were allowed access to all requested documentation and interviews during our work.

2.2
We did not directly contact audit service providers, other than Audit New Zealand. We were able to get relevant evidence (for example, the quality control of outsourced audits) from the Office.

2.3
We received evidence on the performance of financial, performance, and compliance audits from the results of the Office’s Quality Assurance (QA) reviews. This was supported by the additional reviews we carried out.

2.4
Annex 2 provides a comprehensive list of the sources used to support the assessment.

Selecting samples of audits for assessment

2.5
The SAI PMF methodology requires a sample of financial, performance, and compliance audits be tested against specified criteria in order to determine the dimension scores for implementing audits (SAI-10 ii, SAI-13 ii, and SAI-16 ii).

2.6
For financial audits, we mainly relied on the Office’s QA programme, which is considered robust and reliable. It covers all appointed auditors from Audit New Zealand and other audit service providers over a three-year cycle, and assesses the quality of audits using the same criteria as the SAI PMF. Our sample was therefore limited to one file: the most recent financial statements of government (FSG) audit for the year ended 30 June 2015. This is the Office’s highest profile annual audit, comprising the Government's consolidated financial performance and financial position.

2.7
When using the QA work, we assessed as “not met” any criteria relating to topics mentioned as common findings in the QA team’s annual report to the Leadership Team for the 2014/15 year.

2.8
We did not rely on the Office’s QA programme for performance audits, because there have been updates to the Performance Audit Manual since the most recent QA review. Instead, we reviewed two recent performance audits carried out by the performance audit group. They were selected at random from performance audits completed during 2015/16. Follow-up audits were excluded from the selection. One audit was assessed by a member of the assessment team; the other by the Office’s QA Director. We assessed as “not met” any PMF criteria that one or both audits failed to meet.

2.9
Section 17 of the Public Audit Act 2001 gives the Auditor-General the authority to perform other reasonable and appropriate audit services in addition to financial and performance audits. This provides a mandate for the completion of compliance audits. There is also an additional mandate within section 18 of the Public Audit Act 2001 allowing the Auditor-General to inquire into a public entity’s use of its resources.

2.10
In New Zealand, there are elements of a compliance audit² in annual financial audits, performance audits, inquiries, and other assurance work covered in section 17 of the Public Audit Act 2001. For the purposes of this assessment, the framework we are using is limited to three types of audit and assurance work. These are:

• Performance Based Research Funds (PBRF) – the Tertiary Education Commission requires this annual return to be audited where a certain level of performance-based research funding is provided.
• Debenture Trust Deed reporting certificates – where there is a requirement in the public sector entity’s Trust Deed to perform an annual assurance engagement.
• Energy regulation audits – there is a requirement in section 53ZD of the Commerce Act 1986 for the audit of the information disclosure determination and price-quality path reporting when this is periodically required by electricity distribution businesses.

2.11
For compliance audits, we heavily relied on the Office’s QA programme. As part of its 2015/16 work, the QA team assessed a sample of debenture trust deed reviews and energy regulations audits. PBRF audits were not covered by the QA work, so we supported the QA team’s findings by looking at a sample of one PBRF audit.

2.12
Audit samples that fulfil the criteria are assessed as being met.

2.13
It is important to note that these compliance audits are a very small part of the Office’s audit work.

2.14
Inquiries, controller function, and work of a compliance nature (such as legislative compliance, sensitive expenditure, compliance with Cabinet circulars), where no separate engagement is established or audit report issued, have been left out of this assessment.

Scoring

2.15
The complete SAI PMF (v3.2) includes 26 performance indicators. Of these, one indicator is applicable only to SAIs using the Court of Audit model. Three indicators are optional and can be tailored to the specific needs of the SAI. We excluded from our assessment four indicators (SAI-18 to SAI-21). The remaining 22 indicators were included as part of the assessment.

2.16
Of the 22 indicators included in our assessment, only five dimensions were not scored. The reasons for not scoring those dimension are as follows:

Quality control

2.17
One of the project leaders on our assessment team is a certified SAI PMF assessor and trainer who attended a training programme run by the IDI in April 2014. The rest of the assessment team were trained on how to use the assessment tool before the review.

2.18
We made assessments based on sufficient and appropriate evidence. All work was subject to review by another, more senior, assessment team member.

2.19
During the assessment, we asked an IDI advisor any questions we had about how to interpret the methodology and criteria in a New Zealand context.

2.20
Details of all assessments and supporting evidence were recorded in the Office’s document management system. The assessment against each criteria records who carried out and reviewed the assessment.

2.21
The Deputy Auditor-General sponsored the assessment project. He was kept up-to-date with the review’s progress and preliminary findings. In some cases, his input gave us  access to additional evidence or allowed us to better understand the context. However, he was not directly involved in scoring or the assessment.

2.22
Deputy Director General of INTOSAI’s Donor Secretariat, in IDI, provided a quality assurance review of the assessment report. This is detailed in the Quality Assurance Statement, which is attached as an appendix to the report.

2: As defined by ISSAI 400 – Fundamental Principles of Compliance Auditing.