Part 5: The examination system

Inquiry into the Plumbers, Gasfitters, and Drainlayers Board.

5.1
The Board's registration and craftsman examinations have a high failure rate, and have been a source of concern for some time. Previous reports on the Board have been prompted by concerns with the examination system.

5.2
We heard a range of criticisms about the questions asked in the examinations and about the way they are prepared and marked. Therefore, a significant part of our work concentrated on reviewing the Board's processes for ensuring that the examinations are fair, relevant, and accurate.

5.3
In this Part, we discuss our findings on the examination system, including:

  • the legal requirements to hold examinations;
  • the Board's process for setting questions and exam papers;
  • the quality of the examination questions; and
  • the practical link with the teaching curriculum used in training institutions and the National Certificate qualification.

Summary of our findings

5.4
Overall, we found that:

  • the system used by the Board to prepare examination questions was not as reliable or robust as the Board believed;
  • there were unanswerable questions in some examination papers and mistakes in some questions that would make them unnecessarily difficult to answer; and
  • the current prescriptions for examinations do not match the listed competencies in the New Zealand Gazette notices about registration.

Registration examinations

5.5
The 1976 Act required that candidates for registration as plumbers or gasfitters had to sit a registration examination. Candidates for registration as craftsman plumbers or gasfitters were also required to sit two examinations – an examination on technical matters and another on business matters (the craftsman common exam). The Plumbers, Gasfitters, and Drainlayers Regulations 1977 set out the type of examination, the topics to be examined, and eligibility to sit the examinations.

5.6
The regulations stated that, for a candidate who had completed an apprenticeship in plumbing only, the registration examination was to consist of two written examinations, each of not less than three hours' duration. A practical test of workmanship was also required. The examination requirements were the same for someone who had completed an apprenticeship in gasfitting only.

5.7
If a candidate had completed an apprenticeship in both plumbing and gasfitting, the requirements in the regulations were for three written papers, each of not less that three hours' duration, and for a practical test of workmanship.

5.8
For a candidate who wanted to become qualified as a craftsman plumber or gasfitter, the regulations required two written examinations, each of three hours' duration. The craftsman common examination contained material that all craftsmen were required to be examined on, including about running a business and health and safety requirements.

5.9
The regulations also set out, in general terms, the topics to be covered by each examination at both registration and craftsman level. If a person was sitting the craftsman examinations in both trades, the regulations stated that they did not need to be examined on the common material twice.

The Board's system for setting questions

5.10
The Board had in place what appears to be a comprehensive process for checking the accuracy and fairness of questions. NZQA also checked the questions. To understand how examination questions might nonetheless contain errors, we reviewed the processes the Board followed for checking examination questions. The processes were such that it is possible for incorrect questions to be included in the examinations.

5.11
The questions for the registration and craftsman examinations were set at "moderation meetings". The Board examination manual stated that the purpose of the moderation meeting was to consider new questions and answers for inclusion in the Board examination database. The Board organised the meetings, which were held twice a year. Representatives of all training providers and the examiners appointed by the Board to set and mark the examinations attended the meetings. Representatives from the Board also attended these meetings.

5.12
The examiners provided most of the questions and answers. All questions and answers that had been moderated and agreed to were then compiled into a "Question and Answer Examination Database".

Process for setting examination papers

5.13
Examiners appointed by the Board selected questions from the examination database and prepared a draft examination paper. An NZQA-appointed examination expert reviewed the draft examination paper. The examination expert met with the examiners at a meeting described by the Board as the "linguistics meeting". This meeting took three to four days and reviewed the examination questions and answers to ensure that the questions were clearly written and that matters such as formatting were correct.

5.14
As the title of these meetings implies, the purpose of the meeting was to check the clarity of the questions. The meeting did not address the technical accuracy of the questions. The assumption was that this would have been addressed in the moderation meetings. When we spoke with the NZQA reviewer, he was clear that he did not have technical expertise and was not reviewing the questions from that perspective.

5.15
After the linguistics meeting, the draft examination papers were reviewed at a meeting with industry representatives. The examination manual said that the purpose of this meeting was to check that the questions were at a level that candidates could answer. No record was kept of the matters discussed at these meetings.

5.16
The Board told us that these industry meetings provided the opportunity for industry representatives to provide feedback on the extent to which the examination questions reflected current industry practices. Our understanding of these meetings was that there was no review of individual questions and answers to check their accuracy. Changes made to the questions would negate the work of the moderation committee. The examination papers were then reviewed at a meeting of NZQA and the examiners, and any final changes made.

What we found in 2008/09

5.17
The 1977 regulations were very out of date. For example, they still referred to the New Zealand Trades Certification Board as the examining authority. That Board was abolished in 1990, when the NZQA system was established.

5.18
NZQA had a different role, and at first there was some uncertainty and debate about which agency was now responsible for running the examinations and what NZQA's role was. At a practical level, this issue was resolved when the Board entered into a service level agreement with NZQA. The Board now effectively runs the examination, with support from NZQA, even though this is not what the 1976 Act provided.

5.19
We also note that the basic shape of the assessment system set out in the regulations – of extensive written examinations – may now be somewhat dated. Several people we spoke with questioned whether there might be a better way of assessing the relevant knowledge and skills than lengthy written examinations.

The Board system for setting questions

5.20
The examination process seemed, on paper, to be soundly based and well organised. A crucial part of the process was the moderation meeting where new questions are agreed to. We assumed that a meeting where all questions were considered in detail would guard against any faulty questions. Board staff told us that those attending the meeting had the power to reject any questions that they did not agree with.

5.21
No minutes of these meetings were kept, so it was not possible to confirm the procedures that were followed or whether there were objections to questions and how such objections were resolved. However, we interviewed some of the representatives of the training providers who attended these meetings. They told us that questions could be included in the database even if those attending the meeting did not agree that they should be included. They also told us that the moderation meetings were required to consider a large number of questions in a short period of time, and it did not appear that there was enough time to consider the accuracy of the questions included.

5.22
The Board told us that training providers were expected to teach the topics covered by the National Certificate. It stated that questions in the examination database were drawn from the National Certificate syllabus. If a training provider objected to a question because the topic was not taught, and it was established that the topic was in the National Certificate syllabus, their objection would not constitute a valid objection.

5.23
We were told that disagreeing with examination questions and the moderation process could be difficult. On one occasion, a representative of a training provider said that he had expressed concerns to the Board about issues arising from a moderation meeting. He received a telephone call from the Board telling him to stop raising issues at the moderation meetings. The Board also wrote to his superiors at his work place and he was not allowed to write directly to the Board on these matters. He understood from the Board actions that it would not be in the interests of his career to raise objections at the moderation meeting or to write to the Board expressing his concerns.

The quality of the examination questions

5.24
The most serious allegation made to us was that some of the examination questions contained incorrect data or incomplete information, making it difficult to correctly answer the question. We were told that the alleged faults with the questions had been drawn to the attention of the Board on several occasions but had been ignored.

5.25
We decided to have some of these questions independently reviewed to confirm whether the questions could be answered. Our reviewer was a qualified engineer, who had no involvement in the Board examinations or in any other Board activity. He was asked to review 13 questions in the registration examinations and the craftsman registration examinations for 2007/08. He consulted with two other members of his engineering team.

5.26
The reviewer and his colleagues found mistakes in 10 of the 13 questions. The mistakes were such that it would have been difficult for candidates to correctly answer the question. In our view, some of the mistakes were so serious that it would be unreasonable to expect candidates to be able to correctly answer these questions. The mistakes included:

  • A question in a craftsman common examination asked candidates to calculate the discharge power from two pipes of different sizes. There is no such term as "discharge power" in fluid dynamics. Asking candidates to answer a question about a term that does not exist means that the question would be very difficult to answer.
  • A question in a craftsman plumbing examination asked a series of questions about pump performance. The vertical height between the pump and the water supply was not provided. Without knowing the vertical height, the question cannot be answered.

5.27
There were mistakes in several other questions that would make them unnecessarily difficult to answer:

  • A question in the craftsman gasfitting examination asked candidates to calculate the gas input rate for a gas heater. There were several mistakes in this question. The specific heat capacity of water had been incorrectly stated. The heating value of gas had been incorrectly written. The formula given in the examination to calculate the input rate was incomplete.
  • A craftsman gasfitting examination asked candidates questions about installing a gas supply for a motel. Candidates had to make a number of calculations, including calculating the size of gas pipe work from the first stage regulator to the second stage regulator. There were several errors in this question. The data provided to answer the questions stated "pipe size m", when it should be "pipe size mm". The unit rating for the water heater was incorrectly stated. The gas pressure from the outlet was not specified. Another part to this question asked about the sizing of LPG pipe work. Information in the form of a chart was provided to enable candidates to answer the question. However, not enough information was provided to use the chart accurately.
  • Another question said that 1kg of LPG will deliver 250m3 of gas, when in fact 1kg of LPG will deliver 0.5m3 of gas. The review team stated that this was "out by a factor of 500".

5.28
We gave the Board a copy of the report from our reviewers. The Board did not agree that in all cases the mistakes meant that it was difficult for people to correctly answer the questions. In its view, there were two questions that could not be answered (the two questions described in paragraph 5.26), and there were technical shortcomings in four other questions.

5.29
Although we have different views on the extent of the problems with examination questions, we and the Board agree that some questions that have been used are fundamentally flawed, and that others have shortcomings.

5.30
These problems take on more significance when put in the context of an examination that comprises 11 to 16 questions, with a 60% pass mark. Having some faulty questions could materially reduce the candidate's chances of passing the examination.

5.31
As already noted, the examinations have a high failure rate. Figure 1 sets out recent pass rates for the different registration and craftsman (now certifying) plumbing, gasfitting, and drainlaying examinations.

Figure 1
Percentage of applicants who passed examinations, 2004-2008


 2004 pass
rate
%		 2005 pass
rate
%		 2006 pass
rate
%		 2007 pass
rate*
%		 2008 pass
rate*
%
Plumbing registration 31 12 37 32 36
Gasfitting registration 65 58 49 72 36
Craftsman common 32 58 37 23 34
Craftsman plumbing 31 38 63 35 20
Craftsman gasfitting 49 62 47 41 30
Drainlaying registration N/A N/A 75 55 51

* Combines the June and November examination cycles.
Note: N/A means not applicable.
Source: Plumbers, Gasfitters and Drainlayers Board (2009) Annual Report Year ended 31 March 2009, Wellington.

Use of unrealistic scenarios in questions

5.32
Another issue raised with us by representatives from training providers was that some of the examination questions describe situations that are unrealistic. For example, a question in a craftsman plumbing examination described the design of a hot water system to be installed in a sports club to feed a number of showers. It then asked candidates to calculate the plumbing requirements for the system. One part of this question asked candidates to calculate the volume of water that had to be heated and stored. The calculations showed that the hot water tank would need to hold 2500 litres of water. We were told by a tutor that most plumbers would not have encountered a hot water tank of this size and would assume that their calculations were wrong and would not attempt the rest of the questions.

5.33
Another plumbing tutor, with extensive experience in the plumbing industry, was more explicit in his criticisms of this question. He told us that a 2500 litre hot water tank does not exist and that the scenario described in this question makes no sense. In the situation described in the question, where the hot water is to be used only twice a day, for 30 minutes, plumbers would install instantaneous water heaters, which would be a cheaper and more efficient system.

5.34
The issue of presenting unrealistic scenarios in questions had been raised with the Board but rejected. The Board took the view that the point of such questions was for candidates to be able to demonstrate that they can work through the calculations and that they should ignore the scenario presented in the question.

Recent changes

Changes brought in by the 2006 Act

5.35
Under the 2006 Act, the Board can prescribe for each class of registration the minimum standards for registration, which can include requiring a person to have passed an examination to be registered.

5.36
In the Registration and Licensing Notice published in the New Zealand Gazette, the Board set out that the minimum standards for registration include:

  • certifying plumbers, gasfitters, and drainlayers are required to pass the Board's three-hour certifying exam for that trade as well as the Board's three-hour common exam; and
  • licensed plumbers, gasfitters, and drainlayers are required to pass the Board's three-hour licensing exam.

5.37
The notice also sets out that, to pass the Board's examinations, candidates must demonstrate a series of competencies that are listed in the notice for each class of registration.

5.38
The Board has published prescriptions for its examinations. These prescriptions are largely the same as the prescriptions issued for the examinations that the Board held under the Plumbers, Gasfitters, and Drainlayers Regulations 1977.

5.39
We note that, under the 2006 Act, the Board does not have to hold examinations itself, or require people to pass examinations to be registered.

What the Board has been doing to address the problems

5.40
The Board takes the concerns with the examinations seriously and has resolved to address them. In a newsletter dated March 2009, the Chairperson of the Board explained that more than 100 candidates who had passed the November examinations had been interviewed. Candidates who had failed were also interviewed. In addition, the Board called together the examiners, representatives from training providers, NZQA, the ITO, and the Master Plumbers Association to further discuss the reasons for the low pass rates. Some of the problems identified included that:

  • there is no standard text book for students to study from and for tutors to teach from; and
  • the teaching material varies throughout New Zealand. The Board believes this makes it difficult for students who shift around the country and who then find that other training providers teach different topics.

5.41
The Board commissioned a review of the examinations. At its meeting in November 2009, it adopted the reviewer's recommendations, including:

  • working with the ITO to move towards three staged and shorter qualifying examinations embedded within the National Certificate;
  • development with the ITO of new unit standards to meet the Board requirements;
  • fostering alignment between training and examinations by formally involving the ITO in the examination-setting process; and
  • supporting the development of baseline reference documents.

5.42
The Board is also considering a number of changes to the examination system, including:

  • allowing open book examinations;
  • providing model answers to questions; and
  • ensuring that all training providers teach to a standard text and that questions set for the examination reflect the topics taught to students.

5.43
The introduction of the 2006 Act has given the Board an opportunity to review the current examination system and introduce a system that is more modern and fit for purpose. The changes the Board is considering should, if implemented, go a long way towards removing unnecessary impediments and improving the pass rates.

Issues that still need attention

5.44
Until the Board implements a new examination system, the current system will continue. As discussed above, we have significant concerns about the robustness of the current examination system – in particular, about the quality of the questions.

5.45
We encourage the Board to consider what steps it needs to take to ensure that the current examination system operates fairly while it remains in place. We are pleased to record that the Board told us that it has now "quarantined" the questions in its database. It will prepare new written examinations using a new process involving external training providers.

5.46
We also note that the new prescriptions for the examinations do not match the listed competencies in the registration and licensing New Zealand Gazette notice issued by the Board. The prescriptions meet the requirements of the Plumbers, Gasfitters, and Drainlayers Regulations 1977, which are no longer relevant. The Board will need to ensure that any questions it includes in its examinations meet the new requirements rather than those of the 1977 regulations.

Recommendation 7
We recommend that the Plumbers, Gasfitters, and Drainlayers Board, in preparing questions for any future examinations, ensure that the questions are appropriate for assessment under the Plumbers, Gasfitters, and Drainlayers Act 2006, are able to be answered, are free of mistakes, and do not contain unrealistic scenarios.

Recommendation 8
We recommend that the Plumbers, Gasfitters, and Drainlayers Board review its processes for preparing and moderating questions, and for setting examination papers.
page top