4: The Ministry’s process for dealing with correspondence

How the Ministry of Education managed the 2008 national school bus transport tender process.
  1. We examined the extent to which the Ministry responded promptly and effectively when concerns were expressed about aspects of the 2008 bus tender process.

Correspondence about the 2008 bus tender process

  1. Most of the correspondence the Ministry received about elements of the 2008 bus tender process related to qualification ratings and the outcomes of the pricing submissions phase.
  2. Qualification ratings were announced in the middle of June 2008. Twenty-nine of the 165 bus operators who participated in this phase sought reviews of their qualification ratings after the Ministry announced the results. All of the review requests were dealt with by the end of August 2008.
  3. Of the 165 bus operators, 151 were invited through to the pricing submissions phase of the 2008 bus tender process, with 146 bus operators electing to submit pricing bids. Many concerns were raised about the outcome of the 2008 bus tender process after the results of the pricing submissions phase were announced in late September 2008.23 Concerns continued to be raised after the terms of reference for our inquiry were announced in February 2009.24

What we did

  1. We reviewed the Ministry’s process for dealing with 13 of the 29 requests for reviews of qualification ratings. The requests were received during June and July 2008.
  2. Of the large amount of correspondence the Ministry received about the outcomes of the 2008 bus tender process, we selected another sample of 12 correspondents. Some but not all of the correspondents were bus operators who had participated in the 2008 bus tender process. Some of the correspondents had written to the Ministry several times between October 2008 and February 2009. We considered all of their correspondence. We reviewed the Ministry’s process for dealing with each of these 12 correspondents, as well as the Ministry’s replies to the correspondents in our sample.
  3. We interviewed a range of Ministry staff about the Ministry’s process for dealing with correspondence about the 2008 bus tender process. We also reviewed a significant amount of documentation generated by correspondents and by the Ministry.
  4. We did not look into the individual matters each correspondent raised with the Ministry about the 2008 bus tender process and its outcomes. This was not the purpose of our inquiry. We examined the Ministry’s process for addressing each of the matters that correspondents raised.

General process for dealing with correspondence

Recording correspondence about the 2008 bus tender process

  1. The Ministry did not establish a central register of the correspondence it received about the 2008 bus tender process. To help us with this part of our inquiry, the Ministry retrospectively assembled a central register of correspondence. In total, 129 correspondents were recorded as having communicated with the Ministry in some form about the 2008 bus tender process.
  2. The Ministry told us that it did not create a central register at the time the tender was under way because correspondence could arrive as Ministerial letters, Official Information Act requests, or general correspondence by letter or email. We consider that a central register (with cross-references to other systems as appropriate) should have been established. A central register would have ensured that the Ministry accurately monitored the scale and theme of correspondence as it came in.

Replying to correspondents about the 2008 bus tender process

  1. Staff involved in evaluating the tenders were also involved in replying to correspondence about the 2008 bus tender process. It was sensible to draw on these staff members’ knowledge of the 2008 bus tender process.
  2. However, this presented the Ministry with a risk that responses were biased, although we saw no evidence of this in the sample of correspondence that we reviewed. The Ministry appeared to have managed this risk by making the signatories of the correspondence different from the drafter of the reply. From time to time, the drafter also sought advice from other Ministry staff to construct the Ministry’s reply.
  3. Senior Ministry staff were involved in making important decisions when the volume and nature of concerns escalated after the outcomes of the 2008 bus tender process were announced.

Dealing with requests for reviews

  1. After the qualification ratings were published on 11 June 2008 for each of the 165 bus operators, 28 bus operators asked the Ministry to review their qualification ratings. The Ministry issued the review results for 28 bus operators on 26 June 2008. Five further review requests were received after 26 June 2008. Four of these bus operators had previously submitted review requests, and one was requesting a review for the first time.
  2. Of the 29 review requests (the original 28 and the one new review request), 13 qualification ratings stayed the same and 16 ratings changed (15 increased and one decreased).25

The Ministry’s effectiveness when dealing with requests for reviews

  1. We selected a sample of eight of the 28 bus operators who requested reviews of their qualification ratings before 26 June 2008. The Ministry used a consistent process to review these qualification ratings. The TEC made a recommendation to the Ministry and the Ministry replied to each of the eight bus operators with the outcome of their review request.
  2. Five of the eight qualification ratings increased through this review process. The changes were largely because the vehicle age scores had been initially calculated on incomplete information. As previously noted, new vehicles were inadvertently excluded from the initial assessment of vehicle age scores. Once the prevalence of this omission was identified, the Ministry reviewed every qualification submission that included new vehicles to ensure that the vehicle age score was accurate, regardless of whether the bus operator had requested a review. This was appropriate action for the Ministry to take.
  3. We acknowledge that the Ministry took appropriate action to rectify matters. This does not, however, lessen our dissatisfaction about the review requests leading to so many changes in qualification ratings.

The Ministry’s timeliness when dealing with requests for reviews

  1. For the eight bus operators in our sample, the Ministry considered their review requests in a timely fashion and communicated its review decision on 26 June 2008. This was within 11 days of having received the request, and before the pricing submissions closed on 11 July 2008. We consider that this was sufficiently prompt to enable those bus operators to participate in the next phase of the 2008 bus tender process.

Further review requests

  1. The Ministry received five qualification rating review requests after 26 June. Four of these bus operators had previously queried their qualification rating with the Ministry. We examined the Ministry’s process for dealing with these five review requests.26
  2. The Ministry responded efficiently to three of these review requests (within 10 days). The Ministry took longer to complete its reviews for the other two bus operators – up to three weeks. It is not clear why these reviews took longer. We consider that the Ministry should have completed the reviews faster, because the period for submitting pricing bids closed while the Ministry was considering these two review requests.
  3. These reviews resulted in changes to four of the five qualification ratings. Three changes were made because of the inaccurate scoring of new vehicles for the vehicle age criteria, and three changes were made to increase driver development scores. We are concerned that further changes to qualification ratings were made at this stage – this was the third review of qualification ratings for four of these five bus operators. It suggests a lack of robustness in the Ministry’s quality assurance processes.

The Ministry’s quality assurance processes

  1. As well as reviewing the vehicle age scores for all qualification submissions that included new vehicles, the Ministry told us that it “eyeballed” the other evaluation scores (such as driver development and business sustainability) after it received a number of review requests. Ministry staff told us that the “eyeball test” assessed whether the scores awarded for each criteria met their expectations for each bus operator, and whether the scoring was consistent between bus operators.
  2. The Ministry was not able to provide us with any documentation to verify that this “eyeball test” took place. We are unclear how the “eyeball test” could have been done objectively and effectively, given the minimal supporting information provided for each evaluation criteria score to the TEC and the Ministry. We found an instance where one bus operator had no change made to their driver development score in June when they initially requested a review of their qualification rating (this is also when the “eyeball test” was said to have been done), but had their driver development score increased when their qualification rating was reviewed again in late July.
  3. The “eyeball test” did not identify the inconsistencies in how the service agents had applied some of the evaluation criteria.

Summary

  1. The Ministry was generally efficient in how it dealt with requests to review qualification ratings for bus operators. The number of requests caused the Ministry to review the evaluation scores that made up the qualification ratings in mid-June, through the Ministry’s “eyeball test”.
  2. This “eyeball test” was neither documented nor effective enough to identify the inconsistent application of some evaluation criteria by the service agents. Changes to evaluation scores were also made after the “eyeball test” had occurred. In our view, the Ministry’s approach to gaining assurance about the outcomes of the qualification submissions phase before proceeding with the pricing submissions phase was not robust enough.
  3. It appeared that the persistence of some bus operators led to increases in their qualification ratings, rather than any quality assurance process that the Ministry initiated after concerns were expressed about the accuracy of some qualification ratings. The lack of a robust quality assurance process at this point of the 2008 bus tender process compounded the problems we noted earlier about the inconsistent application of some of the RFP criteria.
  4. The Ministry made changes to qualification ratings where the information supported such a change being made. We did not find any evidence that bus operators had been incorrectly advanced to the pricing submissions phase.

Dealing with correspondence about the outcomes

  1. Of the 129 correspondents to the Ministry about the outcomes of the 2008 bus tender process, we drew a sample of 12 correspondents. There were 28 separate pieces of correspondence between the Ministry and those 12 correspondents.27

The Ministry’s effectiveness in dealing with correspondence about the outcomes

  1. The documentation we reviewed suggested that the Ministry considered each piece of correspondence carefully and, for the most part, replied in a consistent and customised way.28
  2. In two cases, the tone of the Ministry’s reply to correspondents could have been seen as antagonistic in already fraught circumstances. Overall, however, the Ministry’s replies to the correspondents in our sample were both courteous and neutral in tone.
  3. In January 2009, the Ministry started using standard text about our inquiry when replying to any matters raised. Staff were acting under direction from the Minister of Education’s office. We consider that it would have been more effective for the Ministry to note our inquiry work as well as address the concerns raised in the correspondence, given the Auditor-General’s mandate and role and the scope of our terms of reference (once announced in February 2009).
  4. In our sample of 12, three operators made repeated complaints to the Ministry. This suggests that these correspondents were not satisfied with the Ministry’s response. However, we acknowledge that there were some environmental and specific constraints on the Ministry’s ability to respond more fully to each of these complaints. For example, one of these correspondents was requesting information that the Ministry could not provide because of privacy and commercial sensitivity considerations.

The Ministry’s timeliness in dealing with correspondence about the outcomes

  1. Generally, the Ministry’s responses to our sample of correspondents were timely.29 We assessed timeliness using the Ministry’s own performance measures for certain types of correspondence, for example, 20 days for a non-urgent Official Information Act request. In a small number of cases (four), the Ministry did not meet these performance measure time frames.
  2. The Ministry took a little more than two months to respond to one correspondent. In our view, this was too long and unacceptable, regardless of the amount of correspondence received about the 2008 bus tender process.

Lessons learned

  1. In any subsequent school bus tender processes, the Ministry should:
    • establish a central register of correspondence linking, as appropriate, to the Ministry’s existing correspondence tracking systems, so that it can accurately and effectively monitor the scale and themes of correspondence;
    • establish a complaints process with a single point of contact, so that complaints can be centrally managed in a timely way; and
    • document the analysis it carries out to be satisfied about the veracity of evaluation criteria scores, qualification ratings, and any other decisions when queries are raised.

Conclusions

  1. Based on our sample, the Ministry reacted promptly when concerns began to be expressed about aspects of the tender process, during both the qualification submissions and pricing submissions phases. We found only a handful of cases where the Ministry could have been more timely.
  2. The effects of the inconsistent and sometimes inaccurate application of the RFP rules became evident after the qualification ratings were announced in mid-June. The Ministry’s process for dealing with requests that it review qualification ratings lacked the robustness we expected. For some bus operators, their tenacity rather than the Ministry’s quality assurance processes led to an increase in their qualification ratings. This must be improved for any subsequent bus tenders.
  3. After the outcomes of the tender process became known in September 2008, and up until a standard set of wording formed the Ministry’s response to any and all correspondence about the 2008 bus tender process, we are confident that the Ministry responded to correspondents in a thoughtful and courteous manner.
  4. Some correspondents were clearly dissatisfied with the Ministry’s response. However, in our view, some correspondents were likely to be satisfied only if the Ministry stopped the entire 2008 bus tender process or changed its outcomes.
  5. It is inevitable that some businesses will be unsuccessful in an open tender process. The fact that many of the businesses involved in the 2008 bus tender process were small, rurally located, and that the tender was about transporting school children in small tight-knit communities seemed to amplify the dissatisfaction that was expressed about the outcomes of the 2008 bus tender process.
  6. We consider that the use of a central register of correspondence would have helped the Ministry’s senior management to more effectively monitor the scale and theme of complaints, and to consider what (if any) action it needed to take to manage its risks.
  7. With the benefit of hindsight, we consider that the scale and nature of complaints received should have caused the Ministry to more effectively review its processes and decisions when the results of both phases were announced and before the Ministry entered into contracts with preferred suppliers.30 This could have identified some of the issues highlighted in this inquiry report.

23: The list of successful bus operators was published on the Ministry’s website: www.minedu.govt.nz/NZEducation/EducationPolicies/Schools/SchoolOperations/SchoolTransport/SchoolTransportHomePage/SuccessfulTenderers2008NationalTenderSchoolBusTransport.aspx.

24: We limited our inquiries into the Ministry’s handling of correspondence about the outcomes of the 2008 bus tender process to correspondence received between October 2008 and February 2009.

25: Eleven of these 16 rating changes related to the omission of the vehicle age scores for new vehicles.

26: While four of the five bus operators requesting these reviews had previously requested reviews, they did not form part of our sample of eight review requests dealt with by 26 June 2008. In total, we examined 13 separate bus operators’ requests for qualification rating reviews.

27: Some correspondents wrote to the Ministry on multiple occasions. We reviewed each piece of correspondence for each correspondent in our sample.

28: There were some notable exceptions with Official Information Act requests. A lack of consistency in the nature of these responses is appropriate.

29: There were two instances in our sample where the Ministry was unable to provide us with evidence that a final response to correspondents had been sent. A draft was available for us to view, but it was not clear that the letter had been finalised and sent.

30: The Ministry deferred its plan to carry out a post-tender review when we were asked to inquire into the 2008 bus tender process.

page top