Part 3: Auditing polytechnics

3.1
In this Part, we assess how NZQA obtains assurance that there has been effective auditing of the polytechnic sector. We assess the actions taken by ITP Quality, the AAA group, and the rest of NZQA against our expectations that:

• academic audits are conducted on a regular basis;
• audit reports present enough evidence to provide assurance that the academic audit standards have been complied with;
• a risk-based approach is used for selecting academic programmes for audit;
• there is an ability to commission special audits where a possible risk to quality has been identified.

Academic audits by Institutes of Technology and Polytechnics Quality

The frequency and follow-up of Institutes of Technology and Polytechnics Quality academic audits

3.2
We expected that ITP Quality would conduct academic audits on a regular basis and that, if it found that audit standards had not been complied with, it would ensure that the polytechnic took action to correct any faults.

3.3
We found that ITP Quality has adopted a systematic approach to academic audits. ITP Quality normally carries out academic audits that cover all 12 of its academic audit standards every four years. An audit panel of three to four experienced auditors will spend up to a week auditing each polytechnic. If the polytechnic is found to have met all the standards, it is awarded quality assured status for a period of up to four years.

3.4
If standards have not been complied with, the polytechnic is required to take action to ensure that it complies, and a member of the audit team may revisit the polytechnic to ensure that it has remedied areas of non-compliance.

3.5
We found that, where a polytechnic was required to take action to comply, both the polytechnic concerned and ITP Quality were assiduous in ensuring that the action was taken.

3.6
ITP Quality also conducts mid-term reviews of polytechnics that have been awarded quality assured status. This provides ITP Quality with an assurance that polytechnics are using effective internal audit and review processes to ensure that they are maintaining their academic standards.

Presentation of evidence

3.7
We expected that the audit reports prepared by ITP Quality would present enough evidence to explain why standards had, or had not, been complied with. Audit reports are important documents, as they provide information for stakeholders about the quality of the teaching and learning at polytechnics.

3.8
The audit reports that we reviewed were detailed and clearly presented, and contained constructive recommendations for quality improvements identified by the auditors. The reports also had a good practice section that identified areas where the polytechnic had demonstrated particular effectiveness in assuring quality.

3.9
Where the auditors had found that practices did not comply with the relevant academic audit standard, evidence was presented to support such conclusions.

3.10
However, in many cases where standards were assessed as being complied with, there was only a limited amount of evidence presented in the audit reports to support these conclusions. In our sampling of standards assessed as being complied with, we found that fewer than half (44%) had enough evidence to support a finding that the polytechnic had complied with the standard.

3.11
For example, the academic audit standard for the delivery of polytechnic academic programmes requires polytechnics to define and implement effective teaching and learning practices. In one polytechnic, this standard was assessed as being complied with, even though the auditors noted that the policies for implementing good teaching practice had not been applied consistently throughout the polytechnic. Several other faults were observed, but there was no evidence presented to explain why, despite the failings identified, the auditors concluded that the standard had been complied with.

3.12
The academic audit reporting approach adopted by ITP Quality can be characterised as an “exceptions-based” approach. That is, evidence is described in detail to support findings where standards have not been complied with or where there are recommendations for improvement, but there is often only a brief description of the evidence that supports findings of compliance.

3.13
This approach to auditing, where the balance of the evidence presented refers to problems, can raise questions as to why a polytechnic was assessed as complying with the standards.

3.14
ITP Quality’s approach can be contrasted with that of NZQA’s requirements for its audits of quality assurance bodies. In its revised set of standards issued in February 2006, NZQA requires that audit conclusions demonstrate that the “nature and balance of evidence provides assurance that the standard has been met”.

3.15
In our view, a similar requirement for ITP Quality’s audit reports would provide further assurance to NZQA on the reasons ITP Quality concludes that academic quality standards have been complied with.

3.16
ITP Quality could use its guidance statements (written to show polytechnics how they might comply with academic audit standards) in academic audit reports to show how a standard has been met. (Refer to Appendix 1 for an example of a guidance statement.)

Risk-based approach

3.17
We expected that some programmes selected for audit in a polytechnic would be selected on a risk-based approach. That is, programmes would be ranked on the basis of any identified concerns about their quality or on their potential for risk (for example, newly established courses, courses offered overseas, and courses with a high proportion of new staff), and the highest ranked programmes would be selected for audit.

3.18
The ITP Quality audit guidelines describe the procedure ITP Quality auditors follow for selecting programmes. This procedure provides for the lead auditor to visit the polytechnic six weeks before the audit. At this visit, the lead auditor consults with the polytechnic on the programmes to be selected for audit.

3.19
The ITP Quality audit guidelines list 16 criteria for selecting programmes for audit. These include selecting programmes where there may be specific concerns about their quality. Other criteria listed in the guidelines that have an element of a risk-based approach include whether the programme is delivered off-site, the involvement of full- and part-time students, and the time since the programme was first approved.

3.20
The audit reports state the programmes that have been audited, but do not indicate why these particular programmes were selected. Programmes may have been selected on a risk basis, but it is not possible to tell this from the audit report.

3.21
Analysis of course completion data, discussed in paragraphs 3.34 and 3.35, is another method ITP Quality could use when deciding which courses to select for audit.

Adult and community education

3.22
NZQA now requires ITP Quality to review the quality of adult and community education (ACE) programmes,¹ and ITP Quality has included these programmes in its audits since the beginning of 2006.

3.23
We expected that a proportion of the ACE programmes selected for audit would be on the basis of identified actual or potential quality concerns.

3.24
Two of the audit reports for ACE programmes that we reviewed contained the statement that: “In the absence of any specific comment it may be assumed that the Polytechnic is deemed to be compliant.” However, it is not possible to tell from such a finding whether the ACE programmes that were audited were selected on the basis of any identified quality concerns. It is also not possible to determine why the auditors considered that these programmes complied with the standards.

Academic audits by the Approvals, Accreditation and Audit group

3.25
As discussed in paragraphs 2.19-2.21, the AAA group audited Unitec in 2003. We reviewed the 2003 audit report to confirm that this auditing arrangement provided assurance that Unitec complied with appropriate quality standards.

3.26
The auditors selected 12 academic programmes for audit. The reasons for selecting these programmes were stated in the audit report. It was clear from the reasons provided that some programmes were selected on the basis of potential risk. For example, two programmes were selected on the basis that they had only recently been approved. Another programme was selected on the basis that it was offered overseas.

3.27
We reviewed the audit report to confirm whether enough evidence was presented to provide a convincing case as to why the relevant parts of the standard were assessed as being met (or, if not met, that there was evidence to explain why).

3.28
The audit report was carefully argued and presented enough evidence to justify the conclusions. For example, when concluding that Unitec complied with the standard for recruitment and development of staff, the audit report referred to the clear and effective policies and procedures for all personnel issues. The auditors referred to the staff files that had been examined and explained how the audit team interviewed staff and sought their views. The audit team was able to conclude that Unitec recruits, manages, and develops staff in compliance with the standard, and provided convincing evidence as to how these conclusions were reached.

3.29
Four areas of non-compliance were identified, none of which was considered to represent a major risk to the quality of the education provided at Unitec.

3.30
Unitec and NZQA agreed a plan to rectify the areas of non-compliance. Unitec was assessed as complying with the specified quality standards.

3.31
In summary, programmes for audit by the AAA group were selected on a risk-based approach and the audit report presented clear evidence in support of findings.

Special audits

3.32
We expected that it would be possible to initiate special audits of polytechnics outside the existing audit arrangements in situations where quality issues and concerns may arise. ITP Quality did initiate special audits where there was seen to be a need for such work. However, historically there has been no additional funding for such work.

3.33
In March 2006, the NZQA Board approved arrangements for additional audits, known as special focus audits. Additional government funding was provided for such audits, which could be initiated if, for example, the NZQA Board decided to investigate quality concerns with groups of courses across different polytechnics. So far, there have been no special focus audits in the polytechnic sector.

3.34
One way for the NZQA Board to initiate and investigate academic quality issues would be by analysing course completion data, which provides information on the proportion of students who start and who complete (that is, pass) a course.

3.35
In our view, analysing course completion data could assist NZQA in identifying potential quality risks.

1: ACE is intended to allow adults to engage in a range of education pursuits, generally focused on personal development and skill enhancement. Most ACE programmes do not lead to a qualification.