Part 2: The scope of our audit

2.1
While NZQA can delegate¹ its quality assurance functions, it is still responsible for ensuring that the delegation is being operated in an effective way. The NZQA Board, which delegated the functions, needs to be kept informed about the operation of the delegation.

2.2
In this Part, we assess NZQA against our expectations of how it:

• monitored the operation of the delegation of quality assurance functions;
• reviewed ITP Quality’s academic audit reports;
• informed the NZQA Board about the operation of the academic auditing function; and
• ensured that there was a relationship with ITP Quality on quality assurance matters.

Monitoring the delegations to Institutes of Technology and Polytechnics Quality

2.3
We expected that, having delegated its quality assurance functions (course approvals, accreditation, and audits of academic programmes) to ITP Quality, NZQA would check on a regular basis to ensure that the functions were being carried out in accordance with the scope of the delegation and to agreed standards.

2.4
We found that NZQA audits the operation of its delegated quality assurance functions every two to three years against a set of standards, including standards for course approvals, accreditation, and academic audits.

2.5
A panel convened by NZQA conducted an audit in 2004 as part of this regular cycle. At the time of our review, the 2004 NZQA audit was the most recent audit of ITP Quality and the first audit against a new set of standards drawn up by NZQA for auditing quality assurance bodies.

2.6
The NZQA audit examined ITP Quality’s processes and procedures for the delegated quality assurance functions and found that these complied with the relevant standards set by NZQA. The audit panel also commended ITP Quality for the quality of its academic audit reports.

2.7
The audit found that ITP Quality did not comply with one standard for the delegation. ITP Quality had created a quality assurance subcommittee that met monthly to approve courses and accredit providers. The audit found that, in terms of the delegation from NZQA, such approvals could be granted only by the ITP Quality Board. ITP Quality took action to ensure that it complied with the standard.

2.8
NZQA planned a further audit of ITP Quality for 2006, but deferred it in September 2006 because of NZQA’s other audit commitments.

2.9
As part of the audit standards for quality assurance bodies, ITP Quality submits an annual report to NZQA that details the activities of ITP Quality during the calendar year. This provides a useful summary of the work carried out by ITP Quality under the delegation from NZQA.

Review of academic audit reports

Institutes of Technology and Polytechnics Quality

2.10
We expected that NZQA would systematically review the academic audit reports received from ITP Quality and determine whether there were issues arising from the reports that required further consideration. Audit reports are important documents. If the course approval and accreditation functions have been carried out effectively, this will be demonstrated in the academic audits carried out in polytechnics.

2.11
A group within NZQA, the Board Services and Audit group, was responsible for the relationship with ITP Quality and for monitoring the delegation of quality assurance functions to ITP Quality. This group was disestablished in mid-2006 after a restructuring within NZQA. Its functions have been incorporated into the Quality Assurance Division, which has a broader mandate than the Board Services and Audit group. However, almost all of the ITP Quality academic audit reports that we reviewed were submitted to NZQA in the period before the restructuring.

2.12
We found that the Board Services and Audit group did not systematically review these reports.²

2.13
In our view, systematic reviews are important. In paragraphs 2.14-2.18, we illustrate the benefits of systematic review by referring to the issues that arose during the auditing of one polytechnic. We emphasise that this example is intended to illustrate the need for NZQA to review the audit reports and is not intended to be representative of the ITP Quality’s academic audit reports that we looked at.

2.14
Since 2000, ITP Quality had completed four academic audits at the polytechnic. The audits revealed a pattern of non-compliance with ITP Quality’s academic audit standards. The polytechnic took action to comply, and ITP Quality believed the situation was improving.

2.15
However, our review of the 2006 academic audit report prepared by ITP Quality auditors showed that, despite the auditors concluding that the polytechnic complied with all academic audit standards, their comments in the audit report indicated that they had serious reservations about some aspects of the academic quality systems in place at the polytechnic. Indeed, the audit report cited an internal polytechnic report that identified serious shortcomings, including a fundamental non-compliance with legislation. However, the audit team believed that significant improvements were under way at the polytechnic, and stated that: “It would have been unnecessarily limiting to have identified corrective actions with narrow foci and short time frames.” Instead, the audit team agreed with the chief executive and senior managers at the polytechnic that the work to improve academic quality systems “must be completed and shown to be effective”.

2.16
The Board of ITP Quality debated this report and sought further evidence from the audit team before deciding to award the polytechnic quality assured status for a period of four years and to do a limited-scope audit of the polytechnic after two years.

2.17
As the NZQA Board is the body with the overall responsibility for the quality assurance arrangements in polytechnics, we expected NZQA’s management to have informed the Board of NZQA about the audit report and the actions taken by ITP Quality. The NZQA Board may have agreed with the position adopted by ITP Quality, or the Board may have decided that it wished to comment on this situation. In any event, the NZQA Board should have been given the opportunity to consider this matter.

2.18
We could find no record of any discussions within NZQA about that report.

Approvals, Accreditation and Audit group

2.19
Unitec New Zealand is the only polytechnic that is not audited by ITP Quality. Although it was set up as a polytechnic, Unitec withdrew from ITP New Zealand in 2000. The 12 academic audit standards developed by ITP Quality apply only to polytechnics that are members of ITP New Zealand.

2.20
NZQA had to decide which academic standards would be applied to Unitec and which body would carry out the audit work. It had a meeting with Unitec in November 2002 to discuss this issue. NZQA had developed “Quality Assurance Standard One”, which applies only to PTEs, Government Training Organisations, and wānanga. It was agreed that NZQA would audit Unitec against those elements of Quality Assurance Standard One that could be applied to a polytechnic.

2.21
NZQA’s AAA group carried out an academic audit of Unitec in September 2003. At the time of our audit in late 2006, the AAA group was about to start a further academic audit of Unitec.³

Informing the New Zealand Qualifications Authority Board

Institutes of Technology and Polytechnics Quality

2.22
We expected that NZQA’s management would inform the Board of NZQA of academic audit issues within the polytechnic sector, and that the Board would receive a summary of all the areas for corrective action identified in the academic audits and of the main issues arising from the audits.

2.23
We reviewed papers presented to the NZQA Board and its subcommittees on quality assurance matters relating to polytechnics from mid-2004 until early 2006. The Board and its subcommittees received a large number of papers on quality assurance in the polytechnic sector. For example, a Board meeting in early 2006 showed that the Board considered the:

• review of audit standards for quality assurance bodies with delegations from NZQA;
• implementation of special focus audits;
• effect of proposed tertiary reforms on quality assurance projects under way;
• changes to a coursework option in a master’s degree offered at one polytechnic; and
• changes to a polytechnic degree that involved offshore work.

2.24
In our review of all the papers submitted to the Board during the two-year period, there was no analysis by the Board Services and Audit group (which received ITP Quality’s audit reports) of matters that could keep the NZQA Board up to date with the auditing carried out by ITP Quality. In particular, there was no analysis of the outcomes from the auditing work or of significant findings from the audits.

2.25
As the NZQA Board is ultimately responsible for the quality assurance of polytechnics, we expected that such information would have been provided to the Board.

Approvals, Accreditation and Audit group

2.26
We expected NZQA’s management to have informed the Board of NZQA of the audit of Unitec carried out in 2003.

2.27
The results of this audit were not reported formally to the NZQA Board. However, NZQA intends to report formally to its Board on the results of the audit completed in late 2006.

Maintaining the relationship with Institutes of Technology and Polytechnics Quality

2.28
In terms of ensuring that NZQA has an effective relationship with ITP Quality, we expected that there would be regular meetings between NZQA and ITP Quality to discuss issues arising from auditing polytechnics and other quality assurance issues.

2.29
Our expectation is similar to a requirement in NZQA’s document Audit Standards for Quality Assurance Bodies with Delegations from NZQA, dated February 2006. These standards require that there be quarterly meetings to discuss complaints or issues.

2.30
Staff of both NZQA and ITP Quality work together in a number of different forums. For example, there is a regular meeting between the staff of NZQA and staff of the quality assurance bodies, including ITP Quality, about the entire quality assurance process. ITP Quality staff and auditing staff from NZQA are currently shared across the audits of polytechnic and PTEs to help develop a standardised and consistent approach to the quality assurance process. In addition, discussions have been held on regular inter-agency workshops on quality assurance best practice.

2.31
A representative from NZQA attends meetings of the Board of ITP Quality to advise on matters such as NZQA policy directions.

2.32
However, there are no regular, formal meetings between NZQA and ITP Quality to discuss issues arising from the academic auditing of polytechnics. There was a meeting in May 2006 that covered a wide range of significant issues of interest to ITP Quality and NZQA. In our view, the large number of issues discussed at this meeting suggests a need for regular formal meetings between NZQA and ITP Quality.

1: NZQA is able to do this under section 260(2) of the Act.

2: The Board of NZQA has since decided that a summary of all polytechnic academic audit reports will be presented to the Board.

3: An academic audit report on Unitec was produced in February 2007. The report was reviewed within NZQA and will be reported to the May 2007 meeting of the NZQA Board.