Part 6: Benefits delivered by Gateway reviews

6.1
In this Part, we explore whether there is evidence that Gateway reviews have delivered value to projects in New Zealand.

Our expectations

6.2
We expected to see:

• evidence that Gateway reviews have delivered value for individual projects;
• an assessment of the performance of Gateway reviews; and
• evidence that Gateway reviews have delivered broader benefits to capital asset management systems.

Summary of our findings

6.3
Gateway reviews have delivered benefits to individual projects. Sometimes Gateway reviews help projects by providing a sounding board for the project sponsor. In other instances, Gateway reviews have suggested specific actions, which have directly affected a project's success.

6.4
The effectiveness of Gateway reviews in New Zealand has not been measured. Analysis in the United Kingdom found that Gateway reviews delivered cost savings to projects. This analysis was complex and has not been reproduced here. However, in our view, Gateway reviews are likely to have resulted in cost savings for New Zealand projects.

6.5
One way that Gateway delivers value is through the recommendations made by review teams. These recommendations are of mixed quality. Most, but not all, are implemented. Poor-quality recommendations substantially reduce the value that Gateway reviews can add to a project.

6.6
Gateway reviews can benefit more than just the individual projects that are reviewed. The skill level of people taking part in reviews are improved by giving them exposure to different projects in different agencies and from working with different reviewers.

6.7
There is scope to improve the way lessons from Gateway reviews are shared. Currently, the Treasury categorises Gateway review recommendations by themes and publishes the results. Not everyone we spoke to knew about this. The Treasury has started to collect information about examples of good practice observed during reviews, but is not yet collating and analysing this information. Sharing lessons more effectively could benefit all projects.

6.8
Confidentiality is an important part of a Gateway review, and it does bring benefits. However, it also conflicts with transparency and accountability, which are important values in New Zealand's public sector.

Benefits for individual projects

Gateway reviews deliver value to individual projects

6.9
We found that Gateway reviews can deliver value to individual projects. Project sponsors gave examples where these reviews had made a difference to their projects. Feedback collected by the Gateway unit is also positive, with 100% of project sponsors³ agreeing, or agreeing strongly, that the Gateway review "was beneficial and will impact positively on the outcome of the project".

6.10
In many instances, the main benefit to the project sponsor came from using the review team as a sounding board to work through issues. Projects can also benefit more directly from Gateway reviews.

6.11
For one project, the agency had been struggling to get approval from its different stakeholders and was unable to progress. The Gateway review helped the project sponsor to decide what actions were needed to get stakeholders to agree. As a result, a way for the project to move forward was agreed.

6.12
In another instance, the agency was not confident that its business case for the project would be approved. The review team provided advice and guidance to help the agency to rewrite the business case so that it presented a more compelling proposal. Cabinet approved the business case and the project avoided delays and costly rewrites.

The Treasury has not measured the performance of Gateway reviews in New Zealand

6.13
The Treasury has not made a formal assessment of how Gateway reviews have performed in New Zealand. Without clear objectives (as we discussed in Part 3), it is impossible to say whether Gateway reviews have achieved what they were intended to achieve. However, it would still be possible to assess the impact that Gateway reviews have had.

6.14
A cost benefit analysis of Gateway in the United Kingdom estimated that 2-4% of project costs were avoided as a result of Gateway reviews. It is reasonable to assume there would be savings in New Zealand, but there has been no analysis to show what level of savings has been achieved here. Gateway reviews follow the same process here, but projects are much smaller. The analysis in the United Kingdom was skewed towards projects with a budget in excess of £1 billion.

6.15
The analysis in the United Kingdom was very complex and would be difficult to replicate. It would be particularly difficult to isolate the effect of Gateway reviews from other interventions in the same period. For example, Better Business Cases were introduced in 2010 and have been found to have significantly improved the quality of business cases. We note that one of the reasons for introducing Better Business Cases was the high number of recommendations about problems with business cases in early Gateway reviews.

6.16
Another way to consider whether Gateway reviews have been successful would be to find out whether the reviews had stopped any projects from failing. This would also be difficult, because we would need evidence of whether, and how many, projects were failing before Gateway reviews were introduced. Secondly, we would need to attribute a project's success to the Gateway reviews. Even if we found that a Gateway review had picked up a critical problem, we cannot assume that another process would not have done the same. However, what we do know is that no projects that have included Gateway reviews have failed. Some projects have been cancelled or re-scoped.

For most reviews, recommendations are an effective means of adding value to the project

6.17
Some of the recommendations in Gateway review reports are better than others. We expected to see variation in the number and type of recommendations, depending on the nature of projects. We looked at the recommendations in 44 review reports and found that those reports had between 5 and 21 recommendations, which varied in quality.

6.18
In our view, a high-quality recommendation is:

• specific about the action required, with sufficient detail;
• practical and achievable by the project sponsor;
• focused on the key issues; and
• ranked by priority.

6.19
Features of lower-quality recommendations that we found included:

• recommendations requiring action by people outside the project sponsor and project team;
• impractical recommendations;
• unclearly worded recommendations;
• recommendations that did not get to the bottom of issues; and
• obvious recommendations or recommendations about matters that were already being addressed.

6.20
Recommendations are one of the main ways that a Gateway review can influence a project. Poor-quality recommendations therefore reduce the value that a Gateway review can bring.

6.21
We also looked at whether the recommendations were implemented. Most Gateway reports include a table showing progress on recommendations from the previous review, where there was one.

6.22
We found that most recommendations were implemented but several were not. Sometimes this was because the recommendation was no longer applicable, but it does call into question the relevance of some recommendations. Although recommendations are only advisory, we would expect project sponsors to implement most of them.

Broader benefits

Taking part in Gateway reviews can improve skill levels for everyone

6.23
When the Government introduced Gateway reviews, one of the intended benefits was that by taking part in reviews, reviewers would improve their own skill levels through the experience of looking at projects outside their own agency. They would also benefit from the input of the other reviewers on the team. Those reviewers would then be able to bring back what they had learned and apply it to projects in their own agency.

6.24
We saw some evidence of this. For example, one chief executive told us that even after being part of only one review team, he was able to bring back lessons and apply them to projects in his own agency.

6.25
The impact of this is limited however. Although over 40 agencies have had employees take part in review teams, only about 20% of reviewers come from the New Zealand public service. It is often difficult for senior public servants to be part of a review team. Review team members have to dedicate an entire week for each review and, for most people, there is a limit to how often this is feasible.

There is scope to improve the communication and implementation of lessons learned from Gateway reviews

6.26
The Treasury already shares some of the lessons learned from Gateway reviews. There is room for improvement by making sure that the right people are informed and that lessons are applied to other projects.

6.27
Reports on the lessons learned categorise Gateway review recommendations by themes and summarise the main findings under each theme. These reports also include examples of recommendations from Gateway reports. The examples do not include any contextual information that could identify the projects they come from.

6.28
So far there have been three reports on lessons learned – one for every 50 Gateway reviews. The reports show how the distribution of recommendation themes has changed over time. Figure 7 shows the distribution of recommendations for each report.

Figure 7
Percentage of recommendations by category in the 2011, 2013, and 2015 reports on lessons learned

Source: Adapted from the Treasury (2015), New Zealand Gateway Reviews: Lessons Learned Report 2015, Wellington.

6.29
The Treasury could do more to make sure that these lessons are shared proactively. The reports are distributed to government agencies and are also available on the Treasury's website. Some people told us they find the reports useful, while others had not seen them.

6.30
There is also scope to share the examples of good practice observed by Gateway reviewers. The reports on lessons learned look only at recommendations on aspects of the projects that are not being done as well as they could be. We expect that there are also many examples of good practice, which could benefit other projects. Since 2015, Gateway reports have included a small section where reviewers can give these examples. The Gateway unit has not started to use this information yet.

6.31
Gateway reviews provide a unique opportunity to delve into a wide range of projects in the public sector. Sharing lessons from these projects effectively should help people to apply them to other projects.

The confidentiality of Gateway reviews has benefits but limits the reviews' potential

6.32
Confidentiality is a fundamental principle of Gateway reviews. Most project sponsors share the report within their agencies and sometimes more widely, but they can control what they share and when. This is important, because Gateway reviews are intended to help the project sponsor, not to assess their performance and hold them accountable.

6.33
Each interview during a review is also confidential. This helps people feel free to speak openly about the project. Without confidentiality, many Gateway reviews would not be able to find out what the real issues are.

6.34
However, transparency and accountability are important values in our public sector. Confidential Gateway reviews are not in keeping with these values.

6.35
Central agencies with a monitoring role may not be able to carry out this role effectively if they are not informed about important findings from a Gateway review. This risk has been mitigated by the escalation process described in Part 2, which makes sure that the relevant people are informed when a project is at risk of failing.

3: This is with an 89% total response rate since 2008.