Part 4: Assessing the benefits of Infrastructure as a Service
4.1
In this Part, we:
- estimate the benefits of Infrastructure as a Service from the measures that the GCIO uses;
- discuss the deeper understanding of Infrastructure as a Service's benefits we got from our work; and
- discuss the need for the GCIO to better measure the benefits of all shared ICT services.
Summary of our findings
4.2
The GCIO collects information about three main measures. These measures assess the consolidation of demand and the benefits resulting from economies of scale.
4.3
The GCIO could improve the methods it uses to calculate benefits for two of its measures – "organisation uptake" and "business savings" – to reduce the risk of understating or overstating benefits. Nevertheless, demand has been consolidated and monetary benefits have been achieved.
4.4
People we talked to from ICT providers, the GCIO, and organisations using Infrastructure as a Service consider that Infrastructure as a Service is worthwhile and meeting expectations. Organisations were generally satisfied with their provider's performance in delivering the agreed services and levels of service under their contract.
4.5
In 2016, the GCIO had a private firm independently review the GCIO's assessment of business savings for all its shared ICT services, including Infrastructure as a Service. The review found that there have been measureable financial benefits, which have increased over time and would reach the Government's target of $100 million annual savings in July 2017. The target was exceeded by March 2017.
4.6
The reviewer recommended that the GCIO work with organisations to identify measures that better represent the range of benefits from shared ICT services and that could be consistently applied to all organisations that subscribe to these services. The GCIO plans to use the reviewer's report to help prepare a measureable framework for all shared ICT services.
Estimated benefits from the GCIO's measures
4.7
When Infrastructure as a Service was introduced, the GCIO was to collect information that accurately assessed progress towards achieving each of the expected benefits from Infrastructure as a Service and its use by organisations.
4.8
However, it was not practical for the GCIO or the organisations using Infrastructure as a Service to routinely and cost-effectively collect information about all the expected benefits.
4.9
Instead, the GCIO uses three main measures to assess and review Infrastructure as a Service's benefits:
- uptake – the number of organisations using Infrastructure as a Service, which is a proxy for consolidation;
- price reductions for services – services that are available through Infrastructure as a Service; and
- business savings – estimated as part of the GCIO's work to estimate sustainable annual savings for all shared ICT services.
4.10
These measures help assess the consolidation of demand for ICT infrastructure and the benefits resulting from economies of scale. For example, it is easier to reach the consumption levels needed to trigger price reductions for services when more organisations use Infrastructure as a Service.
4.11
The GCIO reports on some or all of the measures:
- to the relevant governance committee in the Department of Internal Affairs;
- in the Department's quarterly reports to the Minister of Internal Affairs; and
- to Cabinet at agreed intervals.17
What the measures show
4.12
The GCIO's measures show that:
- depending on the method used, between 76 and 100 organisations are using Infrastructure as a Service for some ICT infrastructure;
- the price reduction targets that the GCIO negotiated with ICT providers have been met and prices have reduced; and
- business savings for all shared ICT services have increased over time and exceeded the Government's target of $100 million annually by March 2017.
Limitations to the measures
4.13
The measures do not show the other benefits from using Infrastructure as a Service.
4.14
The "uptake" and "business savings" measures are less reliable than the "price reductions for services" measure:
- The GCIO does not calculate uptake in a consistent way, which is why the number of organisations that the GCIO reports using Infrastructure as a Service can vary.
- There are also some limitations with the way the GCIO calculates business savings, which we discuss below.
4.15
In 2016, the GCIO had a private firm independently review progress towards the Government's savings target of $100 million annually from 2017/18 for all shared ICT services, including Infrastructure as a Service. The reviewer took a conservative approach and largely agreed with the GCIO's estimates. The reviewer found that savings would reach the target by July 2017. The target was exceeded by March 2017.
4.16
To confirm the GCIO's estimates for Infrastructure as a Service, the reviewer assessed the procurement savings and cumulative price savings for services over time. The reviewer had some reservations about the savings estimates for Infrastructure as a Service because of the GCIO's static market pricing method and the limited data set used.
4.17
The static market pricing method calculates savings achieved based on the market rates when Infrastructure as a Service was introduced. Since then, prices have reduced by arrangement and from competition. The static market pricing method could overstate savings. The reviewer commented that, in the future, the GCIO might consider using a flexible pricing method based on current market demands to calibrate the results of its static marketing method.
4.18
Savings estimates were based on two organisations' business cases for adopting Infrastructure as a Service. The GCIO said that each organisations' circumstances are different. This means that such a small sample is unlikely to represent the savings achieved by all organisations using Infrastructure as a Service.
Getting a deeper understanding of the benefits of Infrastructure as a Service
4.19
To get a deeper understanding of the benefits of Infrastructure as a Service and whether there are any real or perceived barriers to increasing organisations' uptake and use of it, we:
- asked people about Infrastructure as a Service's benefits;
- looked at how ICT providers' performance is monitored; and
- looked at information about the costs to manage Infrastructure as a Service.
What people said about Infrastructure as a Service
Staff in the organisations using Infrastructure as a Service
4.20
Staff in the organisations using Infrastructure as a Service said that they consider the service part of their normal operations. Their focus is now on working out which of the other shared ICT services they should use next.
4.21
This suggests that Infrastructure as a Service has been successful in helping organisations change from a traditional model of managing ICT services themselves to an "as a service" model.
4.22
People said that, through Infrastructure as a Service:
- they have increased cost transparency;
- they can respond quickly to changing business needs;
- prices for the services have reduced;
- procurement is faster and efficient, which means that staff can do other work;
- ICT infrastructure risks have reduced, because ICT providers are able to apply government security requirements more consistently and rigorously; and
- the potential for interruptions to their business is reduced.18
4.23
People said that they were using Infrastructure as a Service effectively as a utility but that they were yet to move to the next stage of using it to encourage innovation and help advance their organisation's strategic goals.
People from ICT providers
4.24
The GCIO designed Infrastructure as a Service to encourage ICT providers to improve the range of ancillary services available and increase the effectiveness and efficiency of services. The ICT providers we talked to consider that they have more constructive and closer working relationships with organisations because of Infrastructure as a Service's design.
Staff from the GCIO
4.25
The GCIO considers that Infrastructure as a Service is successful because it changed the operating model of organisations from a traditional ICT infrastructure model to an "as a service" model for at least some of their operations. This means that demand has been consolidated to some extent.
4.26
The GCIO points to the number of organisations that chose to use Infrastructure as a Service as a sign that it is worthwhile. The GCIO considers that Infrastructure as a Service has also led to organisations' interest in using public cloud services, which the GCIO supports.
4.27
The GCIO said that, for organisations with old ICT infrastructure, poor maintenance or security practices, inadequate backups or staffing, and poor service levels, using Infrastructure as a Service delivered immediate improvements. It might also increase these organisations' total cost of ownership because they had underinvested before.
4.28
The extent to which service levels improved or individual organisations' total cost of ownership increased or decreased is not clear because the GCIO has not collected the before-and-after data.
Staff from organisations not using Infrastructure as a Service
4.29
The GCIO's reporting on uptake does not distinguish between organisations required to use Infrastructure as a Service and organisations choosing to use Infrastructure as a Service. The GCIO could not give us data on any barriers to organisations using Infrastructure as a Service it had identified.
4.30
Therefore, we asked some staff from organisations that are required to use Infrastructure as a Service why they are not using it. Their comments suggested that they understand the expected benefits of Infrastructure as a Service but did not accept that it would produce enough benefits for them. Their main reasons for not using Infrastructure as a Service were that:
- benefits would be few;
- Infrastructure as a Service does not align with their ICT strategy; and
- there are barriers to entry, such as:
- financial constraints;
- service complexity;
- current infrastructure; and
- the effort needed to implement Infrastructure as a Service.
4.31
Between them, the organisations said that they did not see Infrastructure as a Service:
- improving their security profile;
- providing technological innovation that they could take advantage of without extra investment;
- providing much benefit in terms of ability to scale up and down on demand so that they do not need to allow for future demand; and
- providing much benefit in:
- making costs transparent so that they could know what they are paying for and could manage their budget better; and
- shifting the workload to allow them to focus on their core business without worrying about ICT provisioning and management.
4.32
The organisations were satisfied that the GCIO had appropriately informed them about Infrastructure as a Service at the right time in their decision-making. We asked the organisations to say what extra support the GCIO could offer to make it easier for them to use Infrastructure as a Service in the future. They said that:
- no additional support was required;
- Infrastructure as a Service is not relevant or fit for their purposes; and
- Infrastructure as a Service was not affordable.
4.33
We shared the organisations' comments with the GCIO. The GCIO said that the comments aligned with its own understanding of the reasons organisations do not use Infrastructure as a Service. The GCIO also said that some organisations want to use public cloud services and do not want to do this through Infrastructure as a Service.
4.34
One of the benefits expected from Infrastructure as a Service is that organisations would understand their total cost of ownership of ICT infrastructure, even if they decide not to use it.
4.35
The GCIO said that some organisations' cost modelling for using Infrastructure as a Service is flawed because the organisations did not assess total cost of ownership. For example, the GCIO said that some organisations exclude employee costs from their analysis, which underestimates the financial benefits that Infrastructure as a Service could deliver.
4.36
It is not clear what action the GCIO is taking to ensure that all organisations required to use Infrastructure as a Service will use it or get an exemption. At the time of our audit, the GCIO had not issued any exemptions.
Providers' performance
4.37
Providers send organisations' monthly service performance and delivery reports that summarise, for example, contacts between them during the previous month, information about resolved and outstanding issues, and reports on any unplanned events. The reports will also show whether organisations are paying for what they are not using.
4.38
The monthly reports follow a template developed by the GCIO, which ensures that each organisation gets a consistent set of data regardless of which provider they use.
4.39
The people we talked to were generally satisfied with their provider's performance in delivering the agreed services and levels of service under their contract. The small sample of monthly reports we looked at confirmed what people said.
Management costs for Infrastructure as a Service
4.40
We asked the GCIO for information about its costs for managing Infrastructure as a Service from 2010/11 to 2016/17 compared to the revenue it gets from organisations' monthly fees.
4.41
The fees organisations pay to the GCIO now cover the GCIO's costs of managing Infrastructure as a Service. This improvement in the GCIO's overall financial position is because more services are being used through Infrastructure as a Service, confirming that demand is consolidating.
4.42
The GCIO reviews the fee every year. Until 31 December 2017, the GCIO set the monthly fee at 2.5% of the organisations' monthly payment to ICT providers. Because the costs of establishing Infrastructure as a Service have been recovered, the monthly fee was reduced to 1.5% of the organisations' monthly payment to ICT providers from 1 January 2018.
4.43
Figure 1 shows that the revenue that the GCIO received from organisations exceeded costs for the first time in 2013/14. The GCIO's overall financial position for Infrastructure as a Service was positive for the first time in 2016/17.
Figure 1
The GCIO's overall financial position for managing Infrastructure as a Service, revenue, and costs, from 2010/11 to 2016/17
Source: The Government Chief Information Office.
Note: There was a lag between designing Infrastructure as a Service and organisations starting to use services because it can take months for organisations to transition from an in-house model to Infrastructure as a Service. This means that organisations did not start to pay monthly fees until 2012/13.
Measuring the benefits of all shared ICT services better
4.44
The independent reviewer that the GCIO used in 2016 recommended that the GCIO work with organisations to identify measures for the benefits of shared ICT services, which could be consistently applied to all organisations that subscribe to the services.
4.45
The GCIO plans to use the reviewer's report to help prepare a measureable framework for all shared ICT services. The GCIO has not yet set a timetable for this work, and we consider that it should.
4.46
It is likely that, as well as demonstrating fuller accountability for the benefits of each shared ICT service and the funds organisations pay the GCIO to manage them, generating a deeper understanding of benefits would provide the GCIO with evidence that it could use to support greater uptake and use.
4.47
We encourage the GCIO to take this opportunity to ensure that the information it collects about all shared ICT services provides as thorough an understanding of their benefits as practical.
| Recommendation |
|---|
We recommend that the Government Chief Information Office work with public sector organisations to:
|
17: Some information is available on the GCIO's website at ict.govt.nz.
18: For example, organisations did not experience network outage or information loss after the 14 November 2016 earthquake, and ICT was quickly operational for organisations that needed to shift out of their buildings.