3: How the Ministry applied the RFP rules

52. We examined the extent to which the RFP rules were applied correctly and consistently by Ministry staff, contractors, and the TEC during the 2008 bus tender process.

Phases of the 2008 bus tender process

53. For this aspect of our inquiry, we considered the 2008 bus tender process in its two distinct phases:
    • the qualification submissions phase, which began when qualification submissions were due on 30 April 2008; and
    • the pricing submissions phase, which began when the period for lodging pricing submissions closed on 11 July 2008.
54. The Ministry’s two service agents used the criteria¹¹ set out in the RFP to evaluate each of the qualification submissions and arrive at a provisional rating (out of 95).¹² The service agents’ provisional ratings were considered by the TEC and either confirmed or amended before being submitted to the Ministry’s Senior Manager – Resourcing for approval. Bus operators who scored more than zero for each of the vehicle safety, business sustainability, customer service history, and driver development criteria and whose total provisional rating was more than 30 for vehicle safety, business sustainability, customer service history, driver development, and vehicle age were invited through to the pricing submissions phase.
55. During the qualification submissions phase, Land Transport New Zealand (LTNZ)¹³ verified some of the information bus operators submitted about vehicle age and vehicle safety.¹⁴
56. The pricing submissions phase involved each bus operator submitting bids for any routes, route groupings, or clusters. The pricing submissions were analysed and considered by the TEC. The TEC made recommendations for the Ministry to approve. Although price was an important factor in decision-making, it was not necessarily the only factor.
57. A filtering process occurred so that all bid types (routes, route groupings, and clusters) were considered before a preferred bus operator for each route was identified. For each route, the lowest bid was identified along with any bids within 10% of that lowest bid. These were considered acceptable bids. For some routes, there was only one acceptable bid (usually because no other bids were within 10% of that lowest bid price). The bus operator who submitted the sole acceptable bid was considered the preferred supplier. Where two or more acceptable bids were the same price, the bus operator with the highest qualification rating was considered the preferred supplier.
58. Once all the preferred suppliers were approved and the Ministry published that information, it began its next phase of work to ensure that all bus operators were ready to deliver bus services at the beginning of the 2009 school year.

What we did

59. Of the 165 bus operators who participated in the qualification submissions phase, we selected a sample of 25 bus operators. We evaluated their scores using the criteria in the RFP. Essentially, we repeated the qualification submissions phase tasks carried out by each service agent. In doing so, we considered the consistency with which each service agent applied the evaluation criteria to the submissions in our sample, and whether the service agents were consistent with each other in how they applied the evaluation criteria.
60. The Ministry used LTNZ to verify the vehicle age and safety information in the qualification submissions. LTNZ did not verify the completeness of the information that was submitted about vehicle age and safety. We checked this information for each of the 25 qualification submissions in our sample.
61. Two of our sample of 25 did not participate in the pricing submissions phase.
62. The remaining sample of 23 submitted prices for 34 route groupings. Fourteen of those route groupings were part of 10 cluster pricing bids. We evaluated all 34 route groupings and 10 cluster pricing bids against the rules in the RFP for the pricing submissions phase. In doing so, we were examining the extent to which the RFP rules were applied correctly and consistently.
63. During this part of our inquiry, we interviewed Ministry staff, contractors, the service agents, TEC members, and staff from LTNZ.

Our findings about the qualification submissions phase

How each service agent applied the RFP evaluation criteria

64. In our sample, each service agent had followed the same approach for each qualification submission they evaluated. We found one minor inconsistency in how the service agents awarded some points for the driver development criterion. While this is not ideal, it did not have a material effect on the provisional qualification rating awarded to each bus operator in our sample.

Consistency between the service agents

65. For three criteria, the service agents had applied the evaluation criteria in different ways. The criteria were:
    • Business sustainability: one service agent used an historic LTNZ review to arrive at scores, while the other service agent used the bus operator’s business plans and other prospective information.
    • Customer service history: because of the subjectivity of this criterion, the service agents were inconsistent in how they arrived at scores based on customer feedback about bus operators.
    • Driver development: one service agent awarded scores based on the number of drivers qualified as at 30 April 2008, while the other service agent awarded scores based on the expected number of qualified drivers in 12 months’ time.
66. In scoring the business sustainability criterion using an historic LTNZ review, the service agent effectively doubled the weighting accorded to vehicle safety. This additional emphasis on vehicle safety was consistent with the safety principle underpinning the school transport policy. However, it was not clear exactly how the service agent considered business sustainability. The inconsistent application of the business sustainability criterion between the service agents should have been avoided.
67. The retrospective approach to scoring business sustainability also suggested to us that not all of the information included in the qualification submissions was used to inform that service agent’s provisional ratings. The service agent used only some of the submitted information to form his judgments.
68. The service agents scored the customer service history criterion differently. While they both reviewed feedback from schools and other stakeholders to inform their score, they arrived at their score in different ways. Further, some of these scores changed when the TEC moderated the scoring based on its knowledge of the bus operators. The TEC also assigned customer service history scores when the qualification submission did not include any customer references from schools. The TEC often awarded maximum scores (15 points), using its knowledge of the sector and individual bus operators. In most cases, there was insufficient documentation in the Ministry files or in the TEC minutes to explain the scores the TEC awarded.
69. The Ministry told us that it expected service agents’ scores for driver development to be based on the number of qualified drivers the bus operator had when qualification submissions closed in April 2008. As well as the service agents scoring this criterion on a different basis, we noted that the TEC made some changes to the driver development scores in our sample. We do not know what information these changes were based on because the Ministry was unable to provide us with enough documentation to support the changes that were made.
70. With the geographic split of the service agents’ portfolios, and their inconsistent approaches to scoring three of the six criteria, we decided to examine whether these differences could have changed the outcomes of the 2008 bus tender process. There was potential for the different approaches to scoring to affect the outcomes for lower North Island and South Island routes only if the bus operators that competed across service agents’ geographic boundaries were competing for these routes. This did occur, and the results of our analysis are discussed in paragraph 85.
71. The service agents worked largely in isolation from each other. There was little opportunity for the service agents, either independently of the Ministry or through the Ministry’s facilitation, to work together to share their approaches to evaluating the qualification submissions. In our view, the service agents working more closely together could have reduced the level of inconsistency between their scoring of the business sustainability, customer service history, and driver development criteria.

TEC’s quality assurance role

72. The TEC acted as a quality assurance mechanism for the provisional ratings that the service agents awarded during the qualification submissions phase. However, the TEC did not identify the different approaches that the service agents had taken to three of the criteria. In our view, this was an essential part of the TEC’s function.
73. It was not clear in the TEC’s documentation why the TEC had adjusted some of the scores that the service agents had awarded for some criteria. Although the Ministry had not intended for the qualification submissions to be provided to the TEC to help it perform its moderation role, little rationale for each service agent’s scores against each criterion was provided in the information they supplied to the TEC. Given that the TEC did not review the qualification submissions, we consider that the Ministry should have required the service agents to provide more detail to the TEC to support their evaluation scores.
74. The Ministry appointed an independent advisor to the TEC. The independent advisor’s role involved him participating in the TEC’s decision-making. We consider that the independent advisor could have strengthened the quality assurance role of the TEC if he had not been involved in decision-making. In our view, the Ministry should revisit the purpose for using an independent advisor in any future bus tender processes.

LTNZ’s verification role

75. After the outcomes of the 2008 bus tender process were announced, some people expressed concern that a number of vehicle types other than buses were listed by bus operators during the qualification submissions phase. In our sample, submissions from four bus operators cited vehicles that appeared to be inappropriate for school transport use (that is, they were not buses) but were included in LTNZ’s initial assessment. For one bus operator, a vehicle’s age was incorrectly calculated. When these errors were identified by the bus operators, LTNZ reassessed the information. This did not result in changes to the vehicle age and safety scores for these five bus operators in our sample.
76. There was no evidence to suggest that incorrect assessments by LTNZ were widespread during the qualification submissions phase.¹⁵ Some bus operators identified, when the results of the qualification submissions phase were announced, that their vehicle age and vehicle safety scores had been based on incomplete information. Some were also belatedly identified through quality assurance mechanisms. Most of the omissions related to the exclusion of new buses.¹⁶ The Ministry’s written advice did not clearly instruct LTNZ to include this information in its initial verification exercise. In each case in our sample, this was later corrected before the pricing submission phase.

Qualification submissions phase – lessons to be learned

77. The Ministry needs to closely examine the areas for improvement that we have identified in the qualification submissions phase of the 2008 bus tender process, and make sure that improvements are made for any subsequent school bus tender processes. Specifically, the Ministry should:
    • review the RFP guidance for each of the evaluation criteria to ensure that the requirements are clear and that the full range of circumstances are provided for;
    • review the information the RFP requires from bus operators, so that it seeks and assesses only information that is essential to the evaluation process;
    • clearly specify its expectations of, and requirements of, all contractors and consultants involved in evaluating tenders. This includes providing guidance on how contractors and consultants should interpret the evaluation criteria, and the minimum level of evidence or information required to award certain scores for each of the evaluation criteria;
    • facilitate contractors working together under Ministry guidance to minimise the risk of them inconsistently applying criteria during their evaluation of tenders;
    • review the quality assurance arrangements it has in place to ensure that it more effectively identifies any inaccuracies and inconsistencies in applying the evaluation criteria;
    • enhance the level of supporting information supplied to the TEC about the evaluation scores awarded against each of the criteria, so it can better moderate the provisional ratings and justify any changes that it makes to provisional ratings; and
    • revisit the purpose of appointing an independent advisor to the TEC, to strengthen the quality assurance arrangements.

Our findings about the pricing submissions phase

78. For our sample, we evaluated all 34 route groupings and 10 cluster pricing bids against the rules in the RFP for the pricing submissions phase. We looked at the extent to which the RFP rules were correctly and consistently applied.
79. Price was an important determining factor in the 2008 bus tender process. For 30% of our sample of route groupings, the price tendered was 10% lower than any competing bids and therefore the only determinant in awarding that route grouping. Because of the two-phase design of the 2008 bus tender process, there was no suggestion that safety, an underpinning principle of the school transport policy, was compromised by this emphasis on price.

Evaluating the route groupings

80. Feasible bids¹⁷ were identified for each route in each route grouping. Acceptable bids were then identified: the feasible bid that had the lowest price for each kilometre, along with any other feasible bid that was within 10% of that lowest bid (see Rule 130 in the RFP for more information about this process). When there were a number of acceptable bids, the bus operator with the highest qualification rating became the preferred supplier that the TEC recommended to the Ministry. If there was only one acceptable bid for a route grouping, then the TEC recommended that bus operator to the Ministry.¹⁸
81. When evaluating route grouping bids, the Ministry changed the system between national tender rounds for awarding, and thereby the influence of, incumbency scores. For route groupings, the number of incumbency points awarded between zero and five depended on the number of routes held by the incumbent bus operator, and the number of routes they were bidding for in a route grouping.¹⁹ It disadvantaged bus operators who had bid for a smaller number of routes.
82. We considered whether there was evidence supporting the concerns people had raised about market monopolies and unsustainable prices. In our sample, in 10 of the 34 route groupings only one bus operator advanced to the acceptable bid stage because there were no other feasible bids within 10% of that bus operator’s price. These bus operators automatically became the preferred supplier for these 10 route groupings. The final filter – using qualification ratings – did not need to be used to determine the preferred supplier.
83. Essentially, in 10 instances in our sample, price determined which bus operators won the route groupings, after they had met all of the minimum service quality and safety standard requirements in the qualification submissions phase. Six of the 10 sole acceptable bids were from one of the bus operators that competed across service agents’ geographic boundaries in the 2008 bus tender process.²⁰
84. To manage the risks of unsustainable prices, the Ministry prepared an expected price range for each of the school bus routes. If the preferred supplier’s price was outside this price range, then the Ministry sought further information from the preferred supplier to be satisfied about the sustainability of the price submitted. In our sample, we found evidence that this occurred once.
85. As we noted in paragraph 70, we assessed the extent to which the errors we identified in the qualification submissions phase for our sample of 25 affected the results of the pricing submissions phase. We found no evidence that the outcome would have been any different. Our sample included a comparison of the qualification ratings given by the two service agents.

Evaluating the cluster bids

86. The TEC changed the evaluation method it used during the pricing submissions phase to allow qualification ratings to be considered. This change meant that the cluster bid evaluation method matched that used for route groupings. However, it also constituted a change to the RFP rules. This change was not communicated to bus operators. In our view, the Ministry should have communicated through GETS this change in its method for evaluating cluster bids.
87. All of the qualification evaluation scores²¹ except incumbency carried through to the pricing submissions phase for evaluating cluster bids. The Ministry told us that it was too complex a task to include incumbency scores in this analysis. If this was the case, the Ministry should have anticipated the complexity of including incumbency scores in the evaluation of cluster bids, and specifically noted in the RFP that the scores would be excluded from cluster bid evaluations.
88. For six of the 10 cluster bids, the qualification rating determined the preferred supplier. For two of the remaining four cluster bids, the qualification rating of each of the bus operators was the same so price became the final determinant. For the other two cluster bids, price also determined the eventual recommendation.
89. In total, for eight of the 10 cluster bids in our sample, the decision about the preferred supplier was affected by the bus operator’s overall qualification ratings. While we did not re-perform the Ministry’s evaluation of all cluster bids, we calculated that including the incumbency scores in the decision-making for these eight cluster bids could have influenced the outcome in two cases.
90. In our sample, cluster pricing did not have a substantial effect on which bus operator won the routes. However, cluster pricing did enable the Ministry to achieve a significantly lower price for a number of routes. For the 14 route groupings included in our sample of 10 cluster bids:
    • 5 route groupings: the bus operator and the price remained the same for each route grouping;
    • 6 route groupings: the bus operator remained the same but the Ministry achieved a lower price from that bus operator for those route groupings within the cluster bid;
    • 1 route grouping: the bus operator remained the same, but the price of that route grouping increased;
    • 1 route grouping: the bus operator changed, but the price of that route grouping decreased; and
    • 1 route grouping: the bus operator changed and the price of that route grouping increased.

Other observations

91. For our sample, we assessed whether the Ministry followed the other RFP rules that we had not explicitly examined when we repeated the evaluation of the qualification and pricing submissions phases. Some of the rules were not followed.
92. Although we do not consider that the rule breaches were integral to the outcomes of the 2008 bus tender process, we consider that the Ministry should address these areas for any subsequent school bus tender processes.²² Specifically, the Ministry should:
    • date-stamp all pricing submissions when they arrive at the Ministry (Rule 9);
    • ensure that no pricing submissions are opened until after the submission deadline (Rule 9) – we found evidence that seven of our sample were opened before 11 July;
    • return any unsigned statutory declarations to bus operators for signing (Rule 44 and Template 13);
    • consider what is the most effective and efficient method for communicating any changes to the RFP and the tender process results, and adhere to that method (Rules 10 and 30). This would not preclude the Ministry using additional communication methods;
    • ensure that the rules about considering late submissions are strictly adhered to (Rule 25) – we found evidence that the Ministry considered two late qualification submissions; and
    • ensure that it has a robust system for satisfying itself that bus operators have enough buses for the number of routes they submit bids for. In the 2008 bus tender process, all of the bus operators had enough buses for the number of routes they won. However, we did observe an error in one of the Ministry’s calculations that the Ministry was unable to explain. The error inflated the number of buses the bus operator had in its fleet by 45. Even with this error, the bus operator still had enough buses to operate the number of routes it won.

Conclusions

93. There was some inconsistency in how the service agents carried out their own scoring, with more pronounced inconsistencies between the service agents’ scoring. With any in-depth inquiry, we expect to find inconsistencies and identify areas for improvement – no process is perfect. On their own, the inconsistencies we found were not fundamental. However, in combination, there was an unacceptable number of them. The Ministry should address the lessons learned from this inquiry in any subsequent bus tender processes.
94. There was potential for the inconsistent application of criteria between service agents to distort the outcome. This could occur only when the qualification ratings awarded by the different service agents were compared with each other, and when qualification ratings influenced the decision about the preferred supplier (that is, where national bus operators were competing for South Island or lower North Island bus routes). Based on our sample, we found no evidence that the outcome of the 2008 bus tender process would have been any different. Our sample included school bus routes where the qualification ratings awarded by the two different service agents were compared.
95. If the Ministry chooses to use contractors or consultants in its bus tender processes, it needs to more tightly specify the requirements for these contractors and consultants, and monitor their performance against those requirements. This, along with more effective quality assurance arrangements, will help the Ministry to better manage its exposure to any risks in how criteria are applied.
96. The Ministry’s quality assurance arrangements did not operate as effectively as they could have. We consider that they need to be enhanced for any subsequent bus tender processes. While a quality assurance system may not identify all errors, a number of the errors and identifiable inconsistencies that occurred in the 2008 bus tender process were, in our view, easily avoidable.
97. When evaluating the pricing bids, most of the RFP rules were applied correctly and consistently. The main exception related to the exclusion of incumbency scores for cluster bids.
98. It is not clear to us whether the Ministry chose its method for recognising incumbency in the 2008 bus tender process knowing that it was not easily able to be used in the evaluation of cluster bids. The Ministry is entitled to determine how it recognises incumbency. However, we consider that the extent of feedback during consultation should have alerted the Ministry to the benefit of clearly setting out how incumbency would be treated when cluster bids were evaluated.
99. Overall, price was an important determining factor in the 2008 bus tender process. The use of route grouping and cluster pricing was influential in this regard. The Ministry used a two-phase process to ensure that minimum service quality and safety standards were met before price could be a decision-making factor. It also considered the risks to long-term competition in how it approached the pricing submissions phase.
100. The influence of price in the 2008 school bus tender process does not cause us concern, given the Ministry’s stated value-for-money objectives. There is no suggestion that safety was compromised as a result of the emphasis on price.

11: The evaluation criteria were: vehicle age (25 points), vehicle safety (20 points), business sustainability (20 points), customer service history (15 points), driver development (15 points), and incumbency (5 points). Further information on each of the criteria was included in the RFP.

12: Incumbency scores were not awarded until the pricing submissions phase. At that point, qualification ratings were a mark out of 100.

13: The functions of LTNZ and Transit New Zealand were brought together as the New Zealand Transport Agency from 1 August 2008.

14: LTNZ was not required to assess the completeness of the information that bus operators submitted about their vehicles.

15: LTNZ provided an assessment of vehicle age and vehicle safety to service agents, who translated this information into vehicle age and vehicle safety scores.

16: This omission affected 10 of the 28 bus operators who had asked the Ministry to review their qualification ratings.

17: Feasible bids included fleet pricing bids submitted by bus operators for the route grouping and any composite bids assembled by the Ministry (where the Ministry combined the bids for individual routes that made up a route grouping to make them a route grouping bid).

18: The Ministry told us that if the price for each kilometre was too high when there was only one acceptable bid (about 75% above the current incumbent’s rate), then the route or route grouping was recommended for re-tendering.

19: For example, if an incumbent bus operator bid on one route in a 12-route grouping, that bus operator would be awarded only 1 point because it was less than 20% of the routes that made up that route grouping.

20: Combined, the bus operators that competed across service agents’ geographic boundaries were the preferred suppliers for 37% of all the school bus routes included in the 2008 bus tender process.

21: For vehicle age, vehicle safety, business sustainability, driver development, and customer service history.

22: For ease of reference, we have included the relevant rule number from the RFP.