Part 3: Maintaining relationships and monitoring contracts
3.1
In this Part, we set out our findings about how NZTA:
- manages relationships with its consultants and contractors;
- monitors their performance;
- extends contracts; and
- responds to issues that arise.
3.2
In this Part, we make three recommendations.
Managing relationships with consultants and contractors
NZTA has clear accountabilities and responsibilities for managing relationships and monitoring contracts, and had generally maintained good working relationships with its consultants and contractors in the areas we visited.
3.3
NZTA's Senior Asset Managers from the HNO group are based in NZTA's regional offices throughout the country. Senior Asset Managers are responsible for managing the relationship between NZTA and its consultants and contractors within each area and for procuring, managing, and monitoring contracts. As a result of staff shortages in some of the areas we visited, NZTA had engaged additional consultants to provide advice and support to manage the Auckland Harbour Bridge and structures more generally at a national level.
3.4
NZTA's network management consultants are responsible for preparing, evaluating, and administering physical works contracts for maintenance and renewal work, as well as their management, surveillance, and verification. These responsibilities are outlined in NZTA's contract documents with its network management consultants and in standard professional services specifications.
3.5
NZTA's regional bridge consultants are responsible for identifying any routine or structural maintenance required as a result of their inspections. They are required to forward a schedule of routine maintenance identified from inspections to the network management consultant responsible for programming the work to be carried out. For structural maintenance projects, the regional bridge consultant is responsible for preparing an offer of service; for investigation, design, and/or contract preparation; and for managing the project. If projects are required to be tendered, the regional bridge consultant is responsible for preparing the appropriate contract documents.
3.6
In the areas we visited, NZTA's network management consultants, regional bridge consultants, and physical works contractors all considered they had a good working relationship with NZTA's Senior Asset Managers and other regional NZTA office staff. Although issues arose in some of the areas we visited (more detail is outlined in case studies 3 and 5), consultants and contractors generally commented that NZTA's staff were supportive and co-operative, and communicated with them in an open and transparent manner. In the areas we visited, NZTA staff were customer-focused (in line with one of NZTA's strategic priorities) and worked very closely with their consultants and contractors to ensure that they worked with the perspective of road users and other important stakeholders in mind.
3.7
In all areas we visited, the primary means of communication between NZTA and its consultants and contractors were regular formal monthly meetings. Detailed progress reports from consultants and contractors were circulated before the meeting, and tabled and discussed at the meeting. There was also a range of more frequent informal communication between NZTA's Senior Asset Managers and consultants and contractors through email and telephone conversations, which often occurred daily. Weekly catch-ups between NZTA's Senior Asset Managers and its consultants and contractors were considered valuable as a way of "front-footing" issues.
Monitoring the performance of consultants and contractors
NZTA regularly monitors the performance of its consultants and contractors through monthly reporting processes, performance evaluation reviews, and from time to time contract management and "lessons learnt" reviews. NZTA needs to be more consistent in carrying out its performance evaluations and carry out formal reviews toward the end of its contracts.
3.8
NZTA receives and discusses regular reports from its consultants and contractors. Monthly reports detail the past month's progress (including financial progress), highlighting the key milestones, summarising the current status of the contract, and comparing it with the overall maintenance and renewal programme. NZTA also regularly receives and reviews financial, public relations, accident (when relevant), and construction reports from its consultants and contractors.
3.9
The Auckland Motorway Alliance had its own specific key results areas, which were regularly monitored and measured. More detail about how performance and innovation is measured and reported in the Auckland Motorway Alliance is outlined in case study 2.
| Case study 2: Auckland Motorway Alliance – Measuring performance and innovation In October 2008, the Auckland Motorway Alliance (AMA) took over responsibility for the maintenance of the Auckland Motorway network and State Highway 22 for almost 10 years. The AMA was set up under an alliance agreement and is made up of NZTA and alliance partners Fulton Hogan Limited, Opus International Consultants Limited, Beca Infrastructure Limited, Resolve Group Limited, and Armitage Systems Limited. The alliance model was chosen for this part of the state highway network for its potential to deliver a high level of service and value for money, and give all the alliance participants incentives to innovate. The AMA has five specific key result areas (KRAs) for the purposes of measuring the AMA achievements. The KRAs relate to the AMA's five overall objectives. The AMA's objectives are:
The AMA annual report for the year ended 30 June 2010 outlined its performance in achieving value for money against its objectives for the year. It also reviewed the progress made during the second year of the AMA. The annual report emphasises that delivering enhanced value for money for the AMA is about five key components:
The AMA achieved a lower than anticipated overall KRA score of 57.99% in its first full year of measurement. This was from a baseline of 50% and against a commitment to achieve 65%. The overall score was adversely affected by a high number of fatalities and serious injuries from motor vehicle accidents and low travel time reliability on the network, mainly as a result of new capital projects starting that the AMA was not responsible for. In November 2010, a clear strategy to improve the AMA's performance against its KRAs was agreed. The strategy involved "champions" within AMA staff being assigned to prepare an improvement plan for each KRA measure. The strategy sets the overall KRA target score to be achieved in June 2011 as 67.7%. In the March 2011 quarter, the AMA achieved 62.9%. The AMA has a clear performance framework, is generally performing well across the wide range of its performance measures, and is demonstrating innovation in its work systems and practices. However, improvement is required to lift its performance against its KRAs. This case study highlights some aspects for NZTA to consider when measuring performance and promoting innovation – in particular, the need for:
|
* Three target outturn cost (TOC) periods have been defined for the AMA contract as follows: TOC 1 – the first 3.75 years, TOC 2 – 3 years and TOC 3 – 3 years.
3.10
NZTA formally monitors the performance of its consultants and contractors by carrying out regular performance evaluations through its Performance Assessment by Coordinated Evaluation (PACE) system. The PACE system is a database that records all the performance evaluations of NZTA's consultants and contractors on a central register. NZTA's Senior Asset Managers or its consultants (measuring the performance of contractors) generally carry out performance evaluations against four key criteria relating to management, production, health and safety, and administration. For each criterion, grades are given that add up to an overall evaluation score.
3.11
For those carrying out performance evaluations, there is guidance outlining when the evaluations are required to be done and what is involved. However, except for recent workshops outlining the improvements made to the PACE system, there has been limited training for those carrying out performance evaluations. Also, although NZTA head office staff have from time to time published and circulated PACE evaluation scores across areas, there is also no formal or regular review or evaluation of scores at a national level.
3.12
Although they generally considered it to be fair, most consultants and contractors we spoke to considered NZTA's performance evaluation scoring to be subjective and the weighting of criteria confusing in some areas. Some considered that the performance evaluation scores reflected the relationship the consultant and contractor has with NZTA. Others commented that the scoring was inconsistent throughout the country, with some NZTA areas considered more generous than others.
3.13
We examined a range of NZTA's performance evaluations of its consultants and contractors from the areas we visited. The overall performance evaluation scores of the consultants and contractors generally met or exceeded their contractual requirements.3 The average overall evaluation scores ranged from 61% to 86%, with the average scores from Northland, Auckland Harbour Bridge, and the Auckland Motorway Alliance generally being higher than those from Wellington and Southland.
3.14
We found that the evaluations varied in the depth of explanation or comment about the scores. Some evaluations provided explanation about why scores were given (particularly if there had been a change in score since the last evaluation), while others provided very little or no explanation. Also, for the Southland, Wellington, Auckland Harbour Bridge, and Northland areas, performance evaluations were not always carried out as frequently as required.
3.15
NZTA was aware of some of these issues. In February 2010, it set up an internal working group to review and improve the PACE system. Issues with the PACE system identified by the group included the inconsistency of assessments, the need for better use of PACE outputs, limited understanding of some aspects of the system by staff, pressures exerted by suppliers to improve or negotiate better assessments, and meeting the required frequency of evaluations. The group introduced a number of changes, including clarifying the objectives, frequency, and overall ratings of performance evaluations and updating the guidance available on performance evaluation.
3.16
From time to time, NZTA also carries out contract management and "lessons learnt" reviews. Contract management reviews evaluate the compliance and effectiveness of management systems against contractual requirements and plans for maintenance work. "Lessons learnt" reviews evaluate the effectiveness of contract and project management measures and key outcomes to identify positive outcomes, common issues, and trends. NZTA regional offices nominate projects and programmes to be reviewed as part of the contract management and "lessons learnt" review processes, which are conducted by an independent consultant working alongside relevant NZTA staff.
3.17
There were five contract management reviews (three focused on professional services contracts, including two in the Wellington area, and two on physical works contracts) and one "lessons learnt" review conducted in the areas we visited from 2006 to 2010. Although some contract management reviews found some detailed non-compliance and made improvement recommendations to address these, in general the reviews highlighted that the contracts were being appropriately managed and administered.
3.18
However, one review of the professional services contract from the Wellington area identified issues with the consultant's delivery, asset management, and information, and some relationship and communication problems within the wider team. More detail about the issues that arose in the performance evaluations and contract management review and how they were addressed is outlined in case study 3.
| Case study 3: Wellington Network Management Area – Monitoring performance Since 2007, NZTA has operated a hybrid model for the maintenance and renewal of the Wellington network management area. Previously, there had been a traditional model in place. Under the current hybrid contract, MWH New Zealand Limited (MWH) provides professional services, and Fulton Hogan Limited provides physical works services. Both are contracted by NZTA until 2012. From the beginning of the contract, a number of issues arose from a lack of recognition that the hybrid model represented a new way of working that involved new requirements, different performance expectations, and more collaboration among all parties. Some of the main issues that arose were:
NZTA extensively reviewed MWH's deliverable and reporting requirements, and actively monitors MWH against these. Also, a more conscious split in the management of the contract was put into place so the senior management team focuses on high-level issues and the technical management team focuses on operational issues. Regular formal and informal meetings with MWH and Fulton Hogan were also put in place. NZTA also realigned the responsibilities of some staff within its office and created a single point of contact for both MWH and Fulton Hogan. NZTA also implemented clearer processes to better track and record finances generally. MWH focused on improving the quality and timeliness of its reporting. It also realigned the responsibilities of some staff within its organisation and brought in staff capacity and expertise from other areas within its organisation to help facilitate performance improvements. It spent time and effort resolving outstanding issues, particularly around the accuracy and integrity of RAMM data. Fulton Hogan focused on improving the accuracy and integrity of RAMM data and realigned the responsibilities of some staff within its organisation. All parties recognise that considerable improvements have been made and acknowledge that performance is only now where it should have been at the beginning of the contract. This case study highlights some aspects for NZTA to consider when entering into new delivery models – in particular, the need for:
|
3.19
The one "lessons learnt" review between 2006 and 2010 for the areas we visited focused on network management in Southland. It highlighted some clear practical lessons about prioritising work, progressing projects, the importance of information management, and the role of the consultant relevant for areas across the network that experience lower traffic volumes.
3.20
A contract was extended twice in the Southland area. For all maintenance contracts with extension periods, NZTA is required to carry out contract extension performance evaluations before the extension date. More detail about the contract extensions is outlined in case study 4.
| Case study 4: Southland Network Management Area – Extending contracts NZTA operates a traditional model for the maintenance and renewal of the Southland network management area. Under the current model, Opus International Consultants Limited (Opus) provides professional services and a range of network contractors in Southland provide physical works services. These contractors include Downer EDI, which manages the maintenance and renewal of the Milford Road. Opus is contracted by NZTA until 2015. Opus was the network management consultant for the previous traditional professional services contract from 2005 to 2010. That contract was for five years (3 years + 1 year + 1 year), allowing for two extension periods, which were approved by NZTA in September 2008 and September 2009. NZTA's approval of the extension periods was based on the consultant successfully completing the first three years of the contract, ongoing good performance, and performance evaluation scores averaging 61% for the first extension and 63% for the second extension. Although it was a straightforward process of extending the previous professional services contract, NZTA was required to carry out a contract extension performance evaluation before awarding the contract extensions. This did not occur before the extensions were granted in 2008 or 2009. Also, NZTA did not do the required quarterly evaluations for the contract. Although the consultant was performing and justifiably had their contract extended twice, this case study highlights some aspects for NZTA to consider when extending a contract – in particular, the need for:
|
3.21
Although it was clear that the consultant was performing well and justifiably had their contract extended twice in the Southland case study, it is important that performance evaluations are carried out before contracts are extended to demonstrate the rationale for extending the contract.
3.22
We recognise that NZTA has recently improved the performance evaluation process and that there will always be some level of subjectivity to evaluations that will create a range of scores. But as NZTA's formal monitoring process, performance evaluations need to be carried out consistently and as frequently as required throughout the whole contract period to accurately track and monitor the performance of NZTA's consultants and contractors. Greater consistency could be achieved through a range of measures, including more frequent and targeted training for those carrying out the performance evaluations, and more frequent monitoring and review of the performance evaluation requirements and scores at a national level.
3.23
NZTA has noted that, in most instances, there is a high-level review of the performance of previous contracts as part of preparing procurement strategies. However, we consider that there would be value in a detailed review toward the end of each contract to assess how well the delivery model has delivered quality and value for money, and to identify any wider lessons that could be drawn from NZTA's management and monitoring of the contract and applied to the management of other contracts.
| Recommendation 3 |
|---|
| We recommend that the New Zealand Transport Agency introduce measures to ensure that all requirements for the completion of contract performance evaluations are carried out in a consistent and timely way. |
| Recommendation 4 |
|---|
| We recommend that the New Zealand Transport Agency carry out a detailed review toward the end of each contract to assess how well the delivery model has delivered in terms of quality and value for money, and to identify any wider lessons that could be drawn from the management and monitoring of the contract and applied to the management of other contracts. |
Responding to issues
There were performance issues with NZTA's consultants and contractors in two areas we visited. Although the issues were appropriately resolved in both areas, NZTA needs to be more responsive and timely in the way it resolves performance issues with its consultants and contractors.
3.24
Performance issues can be identified through a range of ways, including monthly reporting processes, performance evaluation reviews, and contract management and "lessons learnt" reviews.
3.25
For below average or unacceptable performance identified through performance evaluation and contract management review processes, NZTA's consultants and contractors are generally required to prepare and agree with NZTA improvement plans and corrective actions.
3.26
There were performance issues with NZTA's consultants and contractors in the Wellington (see case study 3) and Northland areas that we visited as part of the audit. More detail about the issues in Northland and how they were addressed is outlined in case study 5.
| Case Study 5: Northland Network Management Area – Responding to issues NZTA currently operates a traditional model for the maintenance and renewal of the Northland network management area. Under the current model, Opus International Consultants Limited (Opus) provides professional services and there are two different physical works contractors in the northern and southern parts of the area. Opus and the physical works contractors are contracted by NZTA until 2014. Before 2011, the Northland network management area operated under a 10-year Performance Specified Maintenance Contract (PSMC) model. A number of issues arose from the beginning of the PSMC contract, including:
Initial work to prepare a procurement plan for a new contract in 2010 also identified lessons learnt from the tender evaluation method, tender prices, co-operative relationship and NZTA contract structure, asset deterioration, and additional works associated with the PSMC. Also, there were other matters NZTA considered when preparing the new contract for Northland. These included the reduction in size and sustainability of the bordering Auckland North area (as a result of the establishment of the Auckland Motorway Alliance), the wish to align contract terms with financial year dates, the ability to introduce a longer-term strategy in 2014, and ongoing discussions with local authorities to explore the benefits of combining the management of state highways and local roads. NZTA considered a three-year traditional model the best approach to address the issues with the previous contract, combining the Northland and Auckland North areas and giving time to prepare for a sustainable long-term strategy for the northern area in the future. Components of the traditional model were put in place by NZTA to address some of the key issues with the previous contract. These included making area-wide treatments part of the design and build requirements, establishing management boards between NZTA and its consultants and contractors for two distinct areas within the network and the network as a whole, providing for an independent auditor to provide increased quality assurance, and prioritising data collection to better understand the condition of the network. Although there was an acknowledgment that lessons had been learnt from the previous contract, this case study highlights some things for NZTA to consider when entering into new delivery models – in particular, the need for:
|
3.27
Although the performance issues were complex and a range of important wider issues were resolved effectively and lessons learnt by NZTA and its consultants and contractors in the Northland and Wellington case studies, in both cases it took almost half the contract period (in the Wellington area two years and in the Northland area five years) for appropriate actions to be put in place to address the issues identified.
3.28
We recognise that maintaining good working relationships with its consultants and contractors requires NZTA at times to make difficult practical judgements about how best to motivate and encourage performance improvement. We also acknowledge that there were some complex contractual, performance, and relationship management issues that arose in the Northland and Wellington case studies.
3.29
However, in our view, in these instances it took too long for NZTA to address some of the key issues that arose, particularly issues around communication, engagement, and team working. NZTA needs to introduce, and adhere to, procedures to be more consistent, responsive, and timely in the way it resolves performance issues with its consultants and contractors.
| Recommendation 5 |
|---|
| We recommend that the New Zealand Transport Agency introduce, and adhere to, procedures to more consistently respond in a timely way to resolve performance issues with its consultants and contractors. |
3: PACE performance evaluation scores between 60% and 70% mean that contractual requirements are fully met, and scores between 71% and 85% mean that requirements are exceeded.