Part 4: How well are the programme management arrangements working?
4.1
In this Part, we assess how well the programme management arrangements are working. We look at:
- how the Programme Control Group works − in terms of composition, independence of the people involved, understanding of roles and responsibilities, and management of risks;
- how the AMETI Advisory Group works;
- how information is used to manage the programme; and
- whether reporting of progress towards benefits realised is adequate.
4.2
We expected that the Programme Control Group would consist of people chosen to bring the right balance of skills and experience to assessing the programme's delivery risks. In practice, the Programme Control Group has stronger skills and experience in technical aspects of the programme than in communication and stakeholder engagement, which are significant risks for the programme. This imbalance in skills and experience has implications for the breadth of leadership that the Programme Control Group can provide.
4.3
Auckland Transport needs to ensure that the chairperson of the Programme Control Group is not involved in day-to-day management of the programme. It also needs to ensure that members of the group clearly understand their roles and responsibilities, have the collective capability to exercise those, and rely less on informal reporting of information to the group. Formal reporting must be enough to support programme monitoring and management, and risk management within the programme.
How the Programme Control Group works
Composition of the Programme Control Group
4.4
When we began our audit in February 2015, there had been some changes to the Programme Control Group membership from that set out in the July 2014 terms of reference and that we discussed in Part 2.
4.5
The changed composition of the Programme Control Group brings together an experienced group of Auckland Transport staff with mainly technical skills. Most members are senior people with authority, and group meetings are well attended. The group has a member with commercial contracting expertise and another who can advise on the strategic fit of AMETI with other projects in Auckland.
4.6
We consider that the Programme Control Group has made a positive contribution on the technical aspects of the programme, such as design, procurement, and budget management.
4.7
However, our case studies showed that the Programme Control Group did not always manage stakeholder relationships well. We understand that Project Directors have the main responsibility for communication about a project stage and that a communications person is attached to the AMETI programme management team. However, the programme has many stakeholders, and the two biggest risks to Stage 2a involved stakeholders.
4.8
In this context, we expected the Programme Control Group to have a member who brought a specialist communications and stakeholder engagement perspective. Staff told us that this was the case in Stage 1 and did not know why it changed for Stage 2a. We observed that the Group Manager had taken on some of this responsibility by, for example, becoming involved in the discussions with Ngāti Paoa about Mokoia Pā.
4.9
Auckland Transport needs to ensure that, collectively, the members of the Programme Control Group have the full range of skills and experience to lead the programme's technical direction and stakeholder relationships and are able to hold staff to account for how they manage all the programme's aspects.
Independence of the Programme Control Group
4.10
We noted in paragraph 2.43 that it is good practice for the chairperson of the Programme Control Group to be independent of programme delivery, to bring an impartial perspective. This also introduces more assurance and ensures that people are not overseeing their own work.
4.11
The previous chairperson of the Programme Control Group was more independent from programme delivery than the current chairperson. This is because the previous chairperson held a higher management role than the current chairperson (second tier rather than third tier) and did not have line management responsibility for programme delivery. Our comments on these arrangements refer to the structural arrangements and not the individual people in those roles.
Figure 6
Programme Control Group membership, February 2015
| Group Manager, Investment and Development (3rd tier) – Programme sponsor and chairperson |
| Group Manager, Construction (3rd tier) |
| Manager, Property and Planning (4th tier) |
| Group Manager, Road Design and Development (3rd tier) |
| Group Manager, Public Transport (3rd tier) |
| Project Director, Key Strategic Initiatives |
Source: Auckland Transport project initiation document for AMETI.
4.12
When the composition of the Programme Control Group was changed, a third-tier manager became the chairperson of the Programme Control Group (see Figure 6). This Group Manager also sponsors the programme, manages the division responsible for delivery, and is the line manager of AMETI's Programme Director.
4.13
This Group Manager also chairs the AMETI Advisory Group and would chair any main stakeholder forum. In our view, this places too much responsibility in the hands of one person. It has reduced the assurance that the Board of Auckland Transport should place on the Programme Control Group for independent guidance and advice.
4.14
Auckland Transport is considering asking the Capital Review Committee to "sign off" all appointments to the Programme Control Group for major capital projects from now on. This should help to ensure that a formal assessment of skills, experience, and independence informs the composition of programme control groups.
| Recommendation 6 We recommend that Auckland Transport ensure that Programme Control Group members have the full range of skills and experience needed and that the Group's chairperson is not involved in managing the programme. |
Understanding the Programme Control Group's roles and responsibilities
4.15
The purpose of the Programme Control Group was not well understood. There were inconsistencies between the Group's stated purpose and practice.
4.16
Changing from individual project control groups to an overall Programme Control Group is likely to have contributed to this confusion. In our view, that change was sensible because it meant that governance and programme management were more in keeping with AMETI's purpose.
4.17
In paragraphs 2.37 to 2.41, we set out the purpose and responsibilities of the Programme Control Group as they are described in October 2013 documents and the July 2014 terms of reference.
4.18
In summary, these terms of reference state that the purpose of the Programme Control Group is to be responsible for assurance and decision-making for the programme; to control the scope, cost, and programme of projects within the authority that the Board of Auckland Transport delegates; and to be the focus of project management.
4.19
In the main AMETI programme document, dated July 2014, the stated role is to "provide guidance, and endorsement of recommendations". The group meets monthly and deals with "contract variations, project changes, departures from standard processes or policies, and commercial proposals".
4.20
However, at the April 2014 meeting of the Programme Control Group, the chairperson (Group Manager, Investment and Development) stated that the Group's purpose was to provide guidance, rather than manage the programme. In future, the focus would be on what was coming up, the main risks and problems, getting consent for plans, and accelerating the programme. The group would no longer routinely scrutinise budgetary information.
4.21
The Programme Control Group endorses papers for submission to the Board of Auckland Transport, although any or all of the Programme Sponsor, the Chief Infrastructure Officer, and the Chief Executive can amend papers after the Programme Control Group meeting.
4.22
The Programme Control Group's role was unclear to us. When we asked the Programme Control Group members directly, most considered that their role was to bringing a wider perspective to the programme, together with critical peer challenge and support. However, some stated that the role was to "make decisions" or to work on strategy.
4.23
Many staff − including members of the Programme Control Group and the programme management team − were confused about how and where some decisions were made. We considered that the main cause of the confusion was that the Group Manager was directing the Programme Control Group as chairperson and also making decisions as a line manager and delegated financial authority holder.
| Recommendation 7 We recommend that Auckland Transport address inconsistencies in the Programme Control Group's roles and responsibilities and ensure that all staff working in or with the Group are aware of any revisions to documents and practice. |
How the Programme Control Group manages risks to the project
4.24
If managers do not refer problems to the Programme Control Group promptly, they limit the group's effectiveness. AMETI has no clear process on what should be referred upward. When a matter is reported to the programme management team, it decides whether the matter needs to be referred upward. Auckland Transport told us that the lack of strict rules or criteria for referral is because it is difficult to be consistent throughout all the projects.
4.25
In our view, the confusion about the purpose of the Programme Control Group increases the risk of problems not being referred upward promptly.
4.26
Further, the Programme Control Group has no formal process for referring matters to the Board of Auckland Transport. The Board expects that it will be told of project risks when necessary, but no process gives effect to that expectation.
4.27
We consider that Auckland Transport should prepare guidance on matters of significance that should be referred to the Programme Control Group and from the Programme Control Group to the Board. Without such guidance, the Board cannot be certain that its advice and direction is always sought when it is needed.
| Recommendation 8 We recommend that Auckland Transport prepare guidance on significant matters that should be referred upwards – for example, to programme control groups, senior managers, or the Board of Auckland Transport. |
4.28
The Programme Control Group needs to provide guidance and advice on managing project risks more effectively. It should evaluate risks and look at what mitigations are in place, to get assurance that the programme management team and project teams are dealing with day-to-day risks. The group should also ensure that residual risks are referred to the Executive Leadership team or to the Board of Auckland Transport, as appropriate.
4.29
In practice, we saw evidence that some risks were scrutinised, but these were mostly technical matters. The Programme Control Group provided no direction on the stakeholder relationship problems that our case studies raise. Some members of the group knew about the risks informally because they had heard discussions outside the group's meetings.
4.30
Also, members of the Programme Control Group said that their role was to give advice on risks and not necessarily to follow through to see whether the project teams had taken action or that the Group Manager had referred a problem upward.
How the Advisory Group works
4.31
The purpose of the AMETI Advisory Group is to provide advice to Auckland Transport. In contrast, the Key Stakeholder Forum is meant to share programme and project information with the main stakeholders.
4.32
Members of the AMETI Advisory Group told us that they understand their roles and responsibilities − to provide advice and to bring a wider Auckland perspective. Group members felt that the Group's involvement was now "about right", as long as their meetings were held at the right time. During Stage 1, some AMETI Advisory Group members had served on the Programme Control Group and found that they received too much detailed information.
4.33
Since Stage 2a began, the AMETI Advisory Group met in July, September, and November 2014, and then not until April 2015. This is at odds with its terms of reference, which provide that meetings will take place every quarter and more often if necessary. Less frequent meetings mean members are less well appraised and less involved than intended. This might also limit the Group's effectiveness in fulfilling its objectives of assuring that strategy is in line throughout Auckland.
4.34
We tracked the two case studies we discussed in Part 3 through the Group's agendas and minutes, and spoke to the members of the AMETI Advisory Group individually. The AMETI Advisory Group knew little about the potential changes in Pakuranga and did not know about the relationship problems with Ngāti Paoa arising from Stage 2a at Mokoia Pā.
4.35
Although we did not expect the AMETI Advisory Group members to know the details of programme delivery, we expected them to be aware of changes to risks within the programme, because this could have wider implications.
4.36
At the time of our audit, we concluded that Auckland Transport needed to more effectively use the AMETI Advisory Group as a forum for sharing information about plans. This would mean that the Group is fully effective in fulfilling its objectives of ensuring that strategy is in line throughout Auckland.
How information for programme management is used
4.37
The Programme Control Group received good and improving information. Information was being made available with more time for the Group to consider it. In our view, the Group needs to share the information it receives and its decisions more formally. It relies too much on sharing information informally.
Reporting to the Programme Control Group
4.38
Members of the Programme Control Group, and managers and governors generally, need relevant, good, and timely information to monitor performance and risks so that they can fulfil their responsibilities. Relevant information that is targeted and fit for the purpose and the user supports delivery of, and accountability for, the programme outcomes. Good information is accurate, complete, and reliable. Timely information is up to date, available when needed, and can be acted on.
Relevance and quality of information
4.39
Project directors report mainly with a monthly project highlight report, along with monthly programme dashboard reports and more specific reporting as and when required. Auckland Transport's AMETI Planning Strategy helps managers to get the information they need to manage risk and performance. Programme governors also get other reports, including those commissioned from experts.
4.40
The format and quality of the project highlight report and monthly dashboard reports are good and improving. Individual projects or work streams report in more detail.
4.41
We expected that the project highlight reports would contain information about contractors' performance. They did not. This information would be useful when deciding about future contracts and should alert the Programme Control Group to any risks. We understand that Auckland Transport has started to roll out a method that will allow it to capture performance information about all its contracted suppliers.
4.42
We also expected to see more information about the health of stakeholder relationships. Although relationships do not have an easily quantifiable monetary value, they can significantly affect the deliverability and costs of a project, particularly where the stakeholder might object to Auckland Transport's plans. Again, this information was absent from the project highlight and monthly dashboard reports.
| Recommendation 9 We recommend that Auckland Transport, when reporting on programme performance, include information about how contractors perform and the health of stakeholder relationships, so that these matters can be better managed and governed. |
Timeliness of information
4.43
Although Programme Control Group meetings were well attended, Group members sometimes lacked time to prepare for meetings. The main cause was receiving information late, but Group members would also like the production of information and meeting cycles to be more in line. Auckland Transport has told us that this has improved since our visit, with members receiving papers at least 48 hours before a meeting.
4.44
In our view, this is something Auckland Transport should keep under review, to see whether the 48-hour threshold is supporting effective programme management. In our experience, at least five working days is more usual. More regular timetables allow managers to set aside preparation time, and meetings can be cancelled if there is no business to discuss.
| Recommendation 10 We recommend that Auckland Transport review the logistics of providing information and meeting cycles for the Programme Control Group to provide information earlier and get information and meeting times more in line. |
Sharing information
4.45
Some members of the Programme Control Group told us that, if their only source of information was what they received in the meeting packs, they would struggle to understand what was going on.
4.46
Many people, including members of the Executive Leadership Team, told us that a lot of information is passed on informally. One senior person told us that much informal conversation, such as discussions with individual management team members and informal catch-ups, takes place outside Programme Control Group meetings.
4.47
He felt that the combination of these activities and the information provided directly to the Programme Control Group is enough. Others found the haphazard nature of communication to be a problem – great if you were in the right place at the right time, but not so good otherwise.
4.48
Members of the management team told us that the outcomes of discussions at the Programme Control Group were sometimes ambiguous and that reading the minutes did not help. In those instances, they sought out someone who had attended the meeting to elaborate. This indicates that the flow of communication from the Programme Control Group needs to be more formalised. For example, the project team could be briefed after each Programme Control Group meeting.
4.49
Information to the Board of Auckland Transport is often not presented in a board paper. Instead, information is provided through Programme Control Group slides and presentations. Although minutes of decisions are recorded, working in this way carries potentially serious accountability risks. This is because it is difficult to see:
- what advice was given;
- how direction was given;
- how conflicts were managed; and
- how decisions were arrived at.
4.50
AMETI is a long programme, and before it ends many different members will have served on the Board of Auckland Transport. It will be important for the Board, at any time, to know how its predecessors got to a decision. In our 2012 report, Inquiry into the Mangawhai community wastewater scheme,11 we said:
Good records are the foundation for any accountability process. Records need to be able to explain what happened and why, and can also protect an organisation by providing evidence to rebut unfounded allegations of improper action.
| Recommendation 11 We recommend that Auckland Transport use more formal channels of communication to and from the Programme Control Group and to the Board of Auckland Transport. |
Reporting benefits realised
4.51
The Programme Control Group receives some information on the expected benefits of AMETI, but Auckland Transport does not yet have a systematic approach to defining and measuring AMETI's benefits.
4.52
The programme documents link the outputs and outcomes to the programme's anticipated benefits, but, in April 2015, Auckland Transport had not yet set quantifiable measures for most of the benefits.
4.53
We looked at a year's worth of papers that went to the Programme Control Group. Some of those papers contained calculations about the projected benefits that would arise when the project team delivered a completed "asset", such as a bus station or changes to road intersections.
4.54
These benefits were expressed in four ways:
- the strategic fit with the Auckland Plan and the Integrated Transport plan;
- those that could be measured when the asset was delivered, such as increased passenger boarding numbers or reduced journey times;
- those that would arise only when several or all of the project elements were delivered, such as further increases in passenger boarding numbers at a train station when it was connected to a rapid transit bus way; and
- theoretical benefit-to-cost ratios for the programme and individual parts of it.12
4.55
Auckland Transport told us it relies on NZTA's funding and review processes to define benefits in a way that can be measured in future. NZTA requires Auckland Transport to provide a rigorous assessment of effectiveness, efficiency, and strategic fit before NZTA approves funding. We acknowledge that this appears to be a very robust process. However, it should be a secondary source, not a primary source, of assurance for Auckland Transport.
4.56
Auckland Transport also told us that some of AMETI's benefits depended on completing other projects, such as City Rail Link. If Auckland Transport is unable to distinguish the effect of AMETI from those of other initiatives, it is difficult to work out whether the programme represents value for money.
4.57
Auckland Transport told us that it is now preparing a formal process and timetable for Board reporting to its two funding partners on major capital projects.
| Recommendation 12 We recommend that Auckland Transport clearly define the project's and programme's expected benefits and report regularly to funders on the benefits realised. |
11: Available on our website, www.oag.govt.nz.
12: Benefit-to-cost ratios model the anticipated benefits in economic terms, and express this as a ratio of the cost of doing the project. Benefit-to-cost ratios of greater than 1.0:1.0 indicate that a project will return benefits greater than the cost of the doing the project.