Appendix 1: Detailed findings https://oag.parliament.nz/2018/auckland-pir/appendix-1 https://oag.parliament.nz/@@site-logo/logo.png

Appendix 1: Detailed findings

Assessment question	Contact Centre post-implementation review	Libraries post-implementation review	Overall conclusions
Have the PIRs been appropriately planned (clear scope and objectives)?	Yes, there was a detailed scope and a six-stage process was prepared, which built on previous internal reviews carried out by the Council. Although there was some debate on what was part of the scope, it became clear that the scope should focus on the achievement of benefits described in each project’s business case. Resource planning could have been improved.	Yes, there was a detailed scope and six-stage process prepared, which built on previous internal reviews carried out by the Council. The scope for the PIR was clear. Resource planning could have been improved.	Overall the planning was good, with a clear approach that built on past experience. Early planning meetings communicated the process well and ensured that there were no changes in scope. The reviews focused on the achievement of benefits described in the business cases. The lack of detailed resource planning did have an effect on the PIRs and the work required to deliver a high-quality and timely report.
Were the PIRs carried out at the right time?	Although there was some initial debate that the review was carried out too soon to assess whether the full objectives in the project’s business case had been achieved, it was considered that 12 months after implementation was a good time. The review was on the first stage of implementation. The findings from the PIR could then usefully be used to inform the next stages of the project.	The Libraries Fit for the Future PIR was done eight months after the project was implemented. This was considered enough time to establish whether the main objectives described in the business case were being achieved or on track to being achieved. Although this might have been considered early to do the PIR, an indication of progress towards business case targets was achievable.	The timing of the first two PIRs was requested by the Mayor’s office (who also commissioned them) and linked achievement of the major benefits of these projects to the Chief Executive’s performance. Enough time had passed after implementation of the projects to assess whether objectives were being, or were on the way to being, delivered.
Were appropriate timelines set for the PIRs and review milestones achieved?	Resourcing issues meant that there were some delays in meeting timelines and milestones. However, the delivery of the report on time and to appropriate quality to the Mayor’s office was achieved.	This PIR began after the Contact Centre Consolidation PIR. As a result, it benefited from lessons learned in setting up the first review. Resourcing issues still affected timelines and the PIR was delivered a little later than anticipated to the Mayor’s office but it did not constitute a significant delay.	Overall, reports were delivered to quality and on time, with some delays on milestones during the projects.
Were sufficient resources allocated to undertake the PIRs within agreed timelines?	The Council decided to build capacity and capability of its staff to carry out current and future PIRs. Although external consultants can provide an independent perspective, internal staff can add more value because they understand the Council and context for the reviews. It takes time to develop internal staff capacity and capability. For both PIRs, capacity was stretched because of new people starting the work, lack of contextual experience of the council service (although they were experienced in undertaking reviews), other demands and priorities drawing resources away from the reviews and staff turnover. Taking time to develop report writing skills also delayed the PIRs. A more focused approach to evidence collection and interviews would have made better use of resources. Resourcing the PIRs was the most significant issue affecting the process.	Although there were the same issues as for resourcing of the Contact Centre PIR, some lessons were learned and there was a more focused approach to evidence gathering, which reduced the burden on resourcing.	Overall, although the PIRs were completed broadly to time, resources were an issue and this put pressure on the team’s ability to complete the work. Capability will need to be developed to ensure that there is better resourcing for future PIRs.
Were the PIRs carried out by staff who had the requisite independence, experience, and expertise?	See above. Knowledge and skills need to be developed. Contextual knowledge of the Contact Centre review was lacking and staff were inexperienced. One staff member, who was leading the PIR team, joined the Council only a few weeks before the PIR started.	The PIR team had a little more experience in carrying out reviews because the PIR team leader was experienced in both working for the Council and undertaking reviews.	Overall, capability needs to be developed further to ensure that PIRs can be efficient and effective. The Council is aware of this and is intending to build this capacity.
Were roles and responsibilities for the PIR teams clear and understood?	Yes, the planned and structured approach that was followed ensured that roles and responsibilities were clear and well communicated. Management and sponsors of the PIRs worked well with appropriate input. One matter for improvement concerned the advisory groups, which could have provided invaluable context and understanding to help guide and challenge the team’s findings if somebody from the council service being reviewed had been included.	Yes, the planned and structured approach that was followed ensured that roles and responsibilities were clear and well communicated. Management and sponsors of the PIRs worked well with appropriate input. Although understanding the context was less of an issue, there was still an opportunity to consider using someone from the libraries council service on the advisory group.	Roles and responsibilities were clearly understood and worked well in managing the process, adding advice and challenge, and keeping senior managers aware of progress and what the findings of the PIRs were.
Have effective and appropriate quality assurance processes been included?	The management, governance, and advisory group structure provided for an effective challenge and quality assurance on the findings. There was effective management oversight of the report drafting and transparency and natural justice were achieved through consultation. The Contact Centre Management wanted their own submission in the report to provide context and an update on progress being made on benefits realisation.	Strong quality assurance. Consultation resulted in agreement on the findings.	Overall a strong quality assurance process ensured fair and balanced reporting.
Were the PIRs independent and did they report without “fear or favour”?	Yes, the review team were independent of the project being reviewed and were situated in the corporate function of the Council, which allowed them to maintain independence. The Mayor’s office feedback on the report considered it to be a fair and balanced view of benefits achieved. The report contained a balance of where targets had been achieved or not. A lesson would be to make the summaries at the beginning of each report stronger and to better reflect the detail contained within. This would enable a quick assessment of achievement of identified benefits.	Yes, the review team were independent of the project being reviewed and were situated in the corporate function of the Council, which allowed them to maintain independence.	Overall, the teams remained independent and reported in a balanced way.
Was there a clear and effective review process?	Yes, the process was effective, logical, and built on previous review processes.	Yes, the process was effective, logical, and built on previous review processes. The libraries review went a little more smoothly, highlighting that lessons were learned from the PIR for the Contact Centre Consolidation project.	There was a strong and effective PIR process that followed good practice.
Was effective collection and analysis of all relevant review data carried out?	Although evidence was collected and used to inform a fair and balanced report, the collection of evidence could have been more focused. The interviews could have been more focused to increase the effectiveness and efficiency of the PIR process. Too many interviews were conducted. Interview notetaking was not consistently done and was not matched up with the line of inquiry of the review. Developing the initial findings would have been easier if interview notes were collected in a more focused way. The collection of data was challenging, particularly because the Contact Centre itself was not effectively measuring performance and achievement of business case objectives. The advisory group and management oversight ensured that there was an effective analysis process to develop findings.	This PIR was more focused in its selection of interviewees, highlighting that lessons had been learned from the first PIR. However, focus groups tended to be dominated by concerns over how the change management process was implemented. Although there were some good insights, most of what was discussed was not in the scope of the project. Notetaking and matching these notes to the business case objectives would have saved time and focused the evidence collection.	Overall, all relevant information was gathered. However, time and effort were wasted on too many interviews, a lack of focus on what information to gather, and not marshalling the evidence at an earlier stage under the business case objectives.
Was reporting of the PIR findings appropriate and were the main messages communicated well?	Yes, the reports were balanced and received positive feedback from the Mayor’s office. The summaries of each report could have been stronger and better developed. It is not clear how wider lessons from the reports on realistic target setting in business cases and monitoring performance against those targets will be used to make improvements throughout the Council. It is clear that service improvements and implementing report recommendations will be monitored by the management teams. There is a role for the corporate strategy and investment teams to ensure that wider lessons are put in place throughout the Council.	Same for the libraries review.	Overall, review findings were appropriate and communicated well.
Are lessons learned from the projects and review process being used to make improvements?			Yes – using our review. Lessons and experience from the first two PIRs are being implemented by the teams. An important lesson for the Council is to develop internal capability and experience and carry out future reviews more efficiently.