Part 3: Delivering projects in the online services programme
3.1
In this Part, we discuss:
- choosing and approving the right projects;
- elements of good practice in delivering projects;
- improving reporting to governance groups; and
- effectively moving the completed projects to the responsible business units.
Summary of our findings
3.2
We did not see any evidence of a formal prioritisation framework for choosing and approving projects in the online services programme. Without such a framework, the Council risks prioritising projects that do not provide the greatest benefit in achieving its strategic outcomes, including making it easier for people to access and use the Council's services. The Council has identified that prioritising projects is an area for improvement.
3.3
We identified elements of good practice for delivering projects in the online services programme. The Council completed the Identity Management and Smart Forms projects largely on time and within budget.
3.4
The progress of individual projects against timelines and project costs, risks, and dependencies were reported to the steering committee and are reported to the subgroup. However, some of the reports were inaccurate, and they were presented as "a snapshot in time", without any trend information. Poor reporting means that the governance groups might not get the information they need to make good decisions in a timely manner, which could affect their ability to maintain effective oversight.
3.5
Council staff in the responsible business units and those working on the projects considered that projects have been moved effectively to the responsible business units. Aucklanders can now use a single log-in to access all of the Council's online services and can go online to register their dog or apply for the licences they need to run their businesses.
Choosing and approving the right projects
Improving prioritisation of projects
3.6
The process for choosing and approving projects in the online services programme consists of prioritising proposed projects, preparing business cases, and approving business cases.
3.7
Proposed projects are prioritised:
- within the online services programme;
- by the Customer-friendly services subgroup; and
- by the Investment Group.
3.8
Each of these has a slightly different focus. However, we did not see any evidence of a formal prioritisation framework. Without a framework, the Council risks making poor decisions about how it prioritises projects. A prioritisation framework would help ensure a common understanding between the three levels of the prioritisation process. It would also provide a consolidated view of factors that influence the prioritisation of projects at the different levels.
3.9
The business case for the online services programme recommended setting up a performance management framework with standard criteria to help prioritise projects. However, when the governance arrangements changed and the Investment Group was formed, the Council did not put this framework in place.
3.10
The Council has identified prioritisation as an area for improvement. Council staff have described the prioritisation process for 2017/18 as more rigorous than for the previous year. The Council told us that, for 2017/18, the Investment Group asked for more information to support requests for funding. All subgroups had to complete a more rigorous prioritisation process to meet the Investment Group's criteria, including testing the ability to deliver the projects based on the available resources.
3.11
The Council is also doing further work, including:
- developing a formal prioritisation framework; and
- planning to use information about the progress of projects in achieving the expected benefits to guide future prioritisation.
3.12
We encourage the Council to continue its work in improving prioritisation of projects. Limited funding makes it even more important that the Council has an effective prioritisation process to help it choose the projects that will provide the greatest benefit in achieving its strategic outcomes, including making it easier for people to access and use the Council's services.
Preparing business cases for projects
3.13
The way that the online services programme develops business cases follows elements of good practice. In our view, these will help to ensure that projects meet the needs of customers and the responsible business unit.
3.14
Each project in the online services programme needs to have its business case approved by the Investment Group before the project can begin. Each business case provides information on matters including:
- the rationale for the project;
- scope and deliverables;
- expected costs, timelines, and benefits; and
- how the project supports the Council's strategic outcomes.
3.15
Lessons from earlier projects were used to develop the Identity Management and Smart Forms projects. As part of the Identity Management pilot, the team talked with experts in identity management to understand what was needed and how to achieve it. The project manager for the Smart Forms project looked at what could be learnt from other projects that had used the chosen software.
3.16
The Council developed principles for the Smart Forms and Identity Management projects so it was clear what the projects needed to provide. The Council also checked whether the chosen technology would meet the requirements of each project. The Identity Management project did this through a pilot, while the Smart Forms project completed a proof of concept.
3.17
The online services programme used a co-design methodology, which meant that customers and business unit staff were involved in the design process. For example, as part of the Smart Forms project, researchers talked to café managers about how compliance activities and licensing could be more user-friendly. Council staff told us that this was a change from the Council's usual process.
3.18
Information from business units was gathered in several ways, including through workshops with staff and someone from the project sitting with the business unit to learn about what they do. Council staff told us that project teams within the online services programme formed a good understanding of the business before preparing the business case.
3.19
The project team then identified what needed to be delivered and what potential benefits might result. For example, the Smart Forms project team identified that improving the applications and putting them online could reduce the time Council staff spent on each application by an average of 14 minutes. In consultation with staff, the project team then decided how they would deliver the project, including what technology they would use, and calculated the project cost and time frames.
3.20
The Investment Group expects project teams to consult with the relevant parts of the Council before submitting business cases. This helps to ensure that the right people have considered and commented on the business case. Staff told us this was an improvement on their previous experiences of Council projects.
A more rigorous approval process for business cases
3.21
The process for approving business cases appears to be working well. The Council has a multi-layered approval process for projects supporting the organisational strategy and for information technology projects (see Figure 6). Business cases are reviewed by the head of the online services programme and then by the subgroup before being submitted to the Investment Group.
Figure 6
Sequence for reviewing and approving projects
Source: Office of the Auditor-General, using information from Auckland Council.
3.22
Members of the subgroup told us they had declined to endorse business cases that were not yet good enough to go to the Investment Group.
3.23
The Strategic Portfolio Office and Corporate Strategy teams jointly review business cases submitted to the Investment Group. They provide the Investment Group with an assessment of each business case, any concerns, and a recommendation about whether the Investment Group should approve it. These assessments also score business cases against the criteria of desirability, feasibility, and viability.
3.24
At the start of each Investment Group meeting, members are briefed on the overall programme of projects under the organisational strategy and the effect of approving funding for the submitted business cases. The Investment Group then considers each business case and questions the project manager, business owner, and head of the online services programme. For example, for the Identity Management project, members expressed concerns about how benefits from the project would be monitored and asked for targets based on use.
3.25
In the minutes of Investment Group meetings, we saw examples of the Investment Group deferring business cases and allocating resources to help project teams revise their business cases before re-submitting them. Members of the Investment Group told us that they also sometimes decline business cases. In one instance, this was because the business case was not clear about where ongoing operating expenditure would come from.
3.26
Council staff described the current approval process as "maturing". However, they also said that, despite this, it is more rigorous, provides better visibility of projects throughout the Council, and is generally an improvement on previous approval processes. This should help ensure that approved projects are co-ordinated and contribute towards achieving the Council's strategic outcomes, including making its services more customer-friendly.
3.27
We suggest that the Council explore opportunities to improve the process for developing and approving business cases to avoid some of the issues identified at the Investment Group stage.
Elements of good practice in delivering projects
3.28
The online services programme has elements of good practice for project delivery, as identified in our 2012 report Realising benefits from six public sector technology projects. This good practice helped the Council to deliver projects that were largely on time, that had justifiable increases in costs, and that meet the needs of the Council and its customers.
3.29
The first part of the Identity Management project was delayed (see Figure 7), but this had a limited effect when the overall system was made available to the public. The Smart Forms project was delivered earlier than planned.
Figure 7
Summary of the planned and actual dates for making online services available to the public
| Project | Planned date (in the business case) | Actual date |
|---|---|---|
| Smart Forms | 28 December 2016 | 12 December 2016 |
| Identity Management (phase 1) | 17 October 2016 | 14 November 2016 |
| Identity Management (phase 2) | 30 November 2016 | 12 December 2016 |
Source: Office of the Auditor-General, using information from Auckland Council.
3.30
The Council delivered both projects largely to budget, with justified increases in cost. The Council delivered the phases of the Identity Management project under budget (see Figure 8). The Smart Forms project needed an increase in budget that went through the appropriate approval process.
Figure 8
Summary of the planned and actual cost for the projects, as at May 2017
| Project | Planned | Actual |
|---|---|---|
| Smart Forms | $500,000 (Revised to $595,000) | $595,000 (100% of revised budget) |
| Identity Management (phase 1) | $257,400 | $249,678 (97% of budget) |
| Identity Management (phase 2) | $152,000 | $109,440 (72% of budget) |
Source: Office of the Auditor-General, using information from Auckland Council.
3.31
We identified elements of good practice in how the online services programme delivers projects. These elements include:
- using a business-led, flexible, and "agile" approach;
- working effectively with the right people, including those who will use the online service; and
- using the right technology.
3.32
The online services programme uses a "minimum viable product" approach to deliver projects.2 An example of this approach was the Council's decision to have a "soft launch" of the Smart Forms project, with limited publicity to minimise the initial number of users. This meant that the Council could test the resilience and readiness of the system and get feedback from staff and customers without affecting many people if there were significant problems.
3.33
The projects in the online services programme consider the needs of the Council and its customers by involving the business unit throughout the project and using a co-design methodology during the development of business cases. Relevant business units were involved in developing and delivering a project by business unit staff working with project teams to identify benefits, having some business unit staff on the project team, and business unit staff providing advice as subject matter experts.
3.34
When preparing the Smart Forms and Identity Management projects' business cases, both project teams took steps to ensure that the technology would be fit for purpose. Both projects used external software suppliers, selected through appropriate procurement processes. The Identity Management project team provided information to their suppliers on the design of webpages.
3.35
We heard positive feedback from Council staff about the online services programme. Staff from the business units said that they had a good relationship with project teams and that projects delivered what they wanted. Other interviewees said that Council staff used to try to avoid transformation. Now, the rest of the Council wants to work with the online services programme because of what the programme has delivered.
Managing project risks and issues
3.36
The Council appropriately managed risks and problems with the projects. Risks were identified within the projects' business cases, and both projects had regularly updated risk registers. Risks were also identified at specific points within the projects, such as when procuring services or during testing.
3.37
The head of the online services programme has promoted a culture among the project managers of being open about any problems or challenges. All staff working on projects in the online services programme gather for quick daily meetings. This is a chance for participants to provide updates on their projects, discuss any issues, and informally share what they have learned.
3.38
We heard several examples of governance groups intervening to manage problems in projects. We also observed governance groups intervening in projects to minimise delays or ensure that any increase in cost was justified.
3.39
An escalation process for change requests supports intervention by governance groups. The effect of the requested changes on cost, scope, quality, and timelines decides who needs to approve it. For example, the subgroup needs to approve increases in the cost of a project of 5% to 20%. An appropriate authority approved the change requests for the projects we looked at, and status reports to the governance groups included information on the change requests.
3.40
However, the escalation process applies only to individual increases. The reporting we saw did not record the cumulative effects of the changes on a project's cost, timelines, or scope. This creates a risk that a series of small increases with a significant cumulative effect may not be visible or fully appreciated when small, individual changes are approved. We suggest that the Council consider ways to provide greater visibility of the cumulative effect of approved changes to a project.
3.41
The Council appropriately managed issues that arose within the projects. The cost of the Smart Forms project increased by $95,000 (about 19%) because a planning oversight meant that the business case did not include the cost of setting up the Smart Forms platform. The budget increase went through a change request process, and the subgroup approved it after confirming that they had made the right decision about which technology to use. We were told that the Council had learned from this incident and now does more due diligence of costs when preparing business cases.
3.42
One of the identified issues for the Identity Management project was using RealMe3 as a way for users to log in to access the Council's services. This was because the Council needed to integrate RealMe into the new platforms and to migrate existing RealMe users on to these. The project had two main problems with integrating RealMe. These were:
- getting enough information from the Department of Internal Affairs about the requirements for using RealMe; and
- complying with RealMe requirements for error messages.
3.43
Working with an external contractor and the Department of Internal Affairs, the project team overcame these problems, with some delays to the project. The project team told the relevant people about these problems and used the change request process to get decisions made to resolve them.
Reporting to governance groups about projects needs to improve
3.44
The Council needs to improve the accuracy of reporting and find ways to track the performance of a project over time. Poor reporting risks the governance groups not getting the information they need to make good decisions in a timely manner and can affect their ability to effectively oversee projects.
3.45
Under the online services programme's previous governance arrangements, both the steering committee and the subgroup were responsible for monitoring the delivery of projects. Projects in the online services programme provided dashboard-style reporting to the steering committee, the subgroup, or both. Reporting covers risks, issues, timelines, cost, and dependencies.
3.46
We asked members of the governance groups about the usefulness of the reporting. Most members said that the reporting does its job of raising issues with projects. However, a senior manager also told us that one of the limitations of the current reports was that they did not provide any trend data on the project's progress.
3.47
We found numerous inaccuracies in reports from several projects, including the Identity Management project. These inaccuracies included a project being 80% complete in one month but then 50% complete in the next, and incorrect budget figures. When we asked about these inaccuracies, we were told that they were mistakes. Having trend data would have helped reduce inaccuracy, because some inaccuracies became clear only when we tracked reporting over time.
3.48
Reporting without this wider context or trend data can provide only "a snapshot in time". For example, a project team would report that they had spent 45% of their budget but not provide any information on whether this is what they expected to have spent at that stage.
3.49
All projects approved by the Investment Group now need to use the same reporting tool as the rest of the Council, including projects that are part of the online services programme. The Council expects that this will help mitigate the issues we found with the accuracy of the reporting.
| Recommendation 2 We recommend that Auckland Council provide all governance groups with reports that are accurate, enable progress to be tracked over time, and enable the people who govern projects to maintain effective oversight of risks and issues. |
Effectively moving projects to the responsible business units
3.50
Once a project in the online services programme is completed, it is transferred to the responsible business unit in the Council. This handover needs to be thorough and effective so that staff are confident in delivering services in a new way and the full benefits are achieved for Aucklanders and the Council.
3.51
We found that the transition of the projects, once launched, to "business as usual" was generally effective, with no significant issues.
3.52
In preparation for moving the projects to the responsible business units, both project teams did testing, prepared handover documentation, trained staff, and provided troubleshooting guides. The Council's reviews of the projects identified aspects that went well during this stage, including:
- There was a smooth transition to the new Identity Management systems.
- Staff could see the results of all the testing activities.
- Quick daily meetings about defects kept the project teams' focus on problems and priorities.
3.53
In the project teams' reviews, they identified things that they could have done better. Both project teams identified the need for a stable system for project testing and staff training. Time constraints also meant that different types of testing activities were running in parallel.
3.54
Both projects had issues before the online services were made available to the public. The Identity Management project's main issue was that it needed to train staff at the contact centre at the same time as the staff had training scheduled for another project. Instead of delaying the "go live" date, the Council trained a different group of staff to provide support until the contact centre staff could take over.
3.55
The launch of the Smart Forms project was delayed because staff had not seen how the forms would work in practice. They identified some problems that needed to be fixed before they were comfortable with people using the forms. We were also told that staff were used to an environment in which change is uncommon and slow, in contrast to an "agile" environment where continuous improvements are made after a project has been delivered.
3.56
Neither project experienced significant problems after online services were made available to the public. The few problems that did occur were quickly resolved. Council staff from the business units, as well as the project teams, considered the transition to be effective.
2: A minimum viable product has just enough features to satisfy early customers and to provide feedback for future improvements.
3: RealMe is a collaboration between New Zealand Post and the Department of Internal Affairs that provides a secure way to verify a person's identity online.