Summary

We were unable to complete our planned performance audit to identify and report changes to costs, time frames, and essential user requirements in recent major defence acquisition projects managed by the Ministry of Defence and the New Zealand Defence Force (the defence agencies).

We had difficulty getting all of the necessary information, particularly from the Ministry’s information systems and had to change our audit approach. We briefed the Foreign Affairs, Defence and Trade Committee (the Committee) about this on 20 March 2008.

Although we were unable to complete our audit, we were able to compile a high-level summary of how the costs and time frames have changed for each of the 10 acquisition projects we looked at.

For most of the 10 projects, estimated costs and time frames had increased, in some cases significantly, between when Cabinet gave approval for acquisition to commence and when Cabinet gave approval for the contract to be signed. Time frames increased further for some projects after the contract had been signed, but the defence agencies reported no further cost increases.

In our view, this information is not enough for the defence agencies to demonstrate how well they are managing the projects or for Parliament or other stakeholders to reach a view on this. There is scope for improvement to the quality, transparency, and usefulness of the reports the defence agencies provide about the progress of defence acquisition projects. We will work with the defence agencies during the next two financial years to make the changes needed.

Background

Projects to acquire new defence capabilities (for example, upgraded electronic systems for existing military planes, and new naval vessels or army vehicles) can take several years to complete. These projects involve defining a capability need, preparing a business case, getting approvals, following a procurement process, waiting for the capability to be produced and delivered, and introducing the new capability into service.

A significant amount of funding is provided to the defence agencies to acquire new defence capabilities. In 2006, a long-term plan listed $2.2 billion (excluding GST) of approved defence acquisition projects. The new capabilities were expected to be delivered between 2006 and 2013.

Because of the costs, time frames, and risks involved, the defence agencies must be able to report transparently to their Minister and to Parliament on the progress of these projects. We expect that the defence agencies would be able to clearly report how and why estimated costs change from those submitted to Cabinet, and how and why time frames and essential user requirements change as projects progress.

Our audit

We started a performance audit to identify and report changes to costs, time frames, and essential user requirements for 10 of the largest and highest-priority defence acquisition projects. The selected projects, and their forecast acquisition costs (excluding GST), were the Medium Utility Helicopter, $771 million; Multi-Role Vessel and Patrol Vessels (Project Protector), $500 million; P-3 Systems Upgrade, $373 million; C-130 Life Extension, $234 million; Boeing 757 Acquisition and Modification, $220 million; Training/Light Utility Helicopter, $110 million; Light Operational Vehicle, $93 million; Medium Range Anti-Armour Weapon, $24 million; Improvised Explosive Device Disposal, $22 million; and Very Low Level Air Defence Alerting and Cueing System, $14 million.

The acquisition process for these projects is set out in the New Zealand Defence Capability Management Framework (CMF). The CMF describes five stages (including four approval points): Ministerial Note, Approval to Initiate, Approval to Commence, Approval to Negotiate, and Approval to Commit.

Based on the CMF, we set out to compare estimated costs, time frames, and essential user requirements between the Approval to Commence point and the Approval to Commit point, and identify the causes of changes between these points. We also set out to compare the latest forecasts of costs, time frames, and essential user requirements with those approved at the Approval to Commit point, and identify the causes of changes.

We chose the Approval to Commence and Approval to Commit points for our audit because they are two main approval points where Cabinet is presented with estimates of the costs and time frames to acquire defence capabilities. The CMF describes the Approval to Commence point as the “Main Gate”, and states that accurate cost estimations are critical at that point. Cabinet notes the estimated costs and gives approval for the acquisition phase of the process to commence. Sometimes those cost estimates have been used to make public announcements on the details of the project. The Approval to Commit point is when the defence agencies seek final Cabinet approval to commit funds for the project before the contract is signed.

Our ability to complete the audit was hindered by the difficulties we encountered getting the necessary information, particularly from the Ministry’s information systems. We had to change our audit approach, and briefed the Foreign Affairs, Defence and Trade Committee (the Committee) about this on 20 March 2008.

In keeping with what we told the Committee we would do, this interim report shows, at a high level, the cost and time frame changes we were able to determine for the 10 defence acquisition projects that we looked at. This interim report also summarises the difficulties we encountered with the Ministry’s information systems and describes how we will work with the defence agencies during the next two financial years to encourage them to make the changes needed to improve the transparency and usefulness of their reporting about defence acquisition projects.

Our findings

This interim report does not make any conclusion about how well the defence agencies are managing specific defence acquisition projects, or on the causes or justifications for the changes in costs and time frames that we report. We do not have enough information or the right information to form a judgement on that larger question. The purpose of the audit was to improve the quality of reporting by the defence agencies, not to assess the quality of the decisions made in managing the projects.

Although we were unable to complete our audit, we were able to compile a high-level summary of how the costs and time frames have changed for each of the 10 acquisition projects we looked at. We do not report on essential user requirements, because of the difficulty we had getting that information.

Overall findings

The defence agencies must be able to report better and more complete information to demonstrate how well they are managing defence acquisition projects. In our view, the defence agencies must improve the transparency of reporting so that readers of the reports can have confidence that the defence agencies are appropriately managing the acquisition process for the projects.

For most of the 10 projects we looked at, estimated costs and time frames between the Approval to Commence and Approval to Commit points had increased, in some cases significantly. Between the Approval to Commit point and the latest forecasts (in December 2007), no further cost increases were forecast or had occurred for most of the projects, but most had experienced further delays. The delays were smaller than those between the Approval to Commence and Approval to Commit points.

We were unable to find complete information on the individual variations that made up the changes, or the reasons for them. We consider that it is important for the defence agencies to be able to report such information to Ministers, Parliament, and other stakeholders. We recognise that, when the defence agencies’ reporting systems were introduced, they were not designed to monitor the progress of projects in the way that we set out to. Nevertheless, we expected the sort of information we were looking for to have been more readily available and easier to extract than we found.

Changes between the Approval to Commence point and Approval to Commit point

We expected the tender and negotiation process to cause some changes to estimates between the Approval to Commence and Approval to Commit points. Given that the CMF states that accurate cost estimations are critical at the Approval to Commence point, we expected that these changes would not be significant.

Changes were sometimes significant, and cost estimates at the Approval to Commence point were not as robust as the CMF indicates they should be. The defence agencies advised us that cost estimates at this point are, at best, an “intelligent guess” of what new capabilities will cost. This is because the agencies cannot request costed proposals from the market until Cabinet has given Approval to Commence. We note that the CMF indicates that, to provide information that will support the development of robust cost estimates, the defence agencies can consult with suppliers before the Approval to Commence point.

In our view, the defence agencies should start reporting progress from the Approval to Commence point. The defence agencies disagreed with us because, in their view, it is too early in the acquisition process and cost estimates at this point are not robust enough to report against.

Although the defence agencies’ guidance states that cost estimates should be robust when they are submitted to Cabinet at this point, in practice they are not. The defence agencies do not indicate how they have estimated costs or the degree of uncertainty inherent in them. In our view, the defence agencies need to do this as part of their future reporting. They also need to address the mismatch between their practice and the CMF.

We understand that the Ministry will soon be making major changes to the CMF to incorporate the requirements of the Treasury’s Capital Asset Management Review, and the Ministry’s own examination of the level of accuracy that can be expected in cost forecasts at various stages of the acquisition process.

Changes from the Approval to Commit point to the defence agencies’ forecasts at the end of 2007

We expected there to be few changes to costs and time frames after the Approval to Commit point. The defence agencies advised us that the forecast costs and time frames at the Approval to Commit point are much more accurate than the estimates at the Approval to Commence point, because the Ministry has been able to get costed proposals from potential suppliers and is about to award a firm fixed-price contract.

At the aggregate level of total project cost, no changes had been reported for most projects after the Approval to Commit. However, there are underlying variations and movements in and between budget lines that are not reported. We were unable to find complete information on these variations and movements, or the reasons for them. In our view, it is important for the defence agencies to monitor and understand the reasons for such changes, and to be able to report on them so that any trade-offs being made are transparent.

Plan to improve progress reporting on defence acquisition projects

There is scope for improvement to the quality, transparency, and usefulness of the reports the defence agencies provide about the progress of defence acquisition projects. We will work with the defence agencies during the next two financial years to encourage them to make the changes needed.

Our future work with the defence agencies will be in two stages. We will reconsider our approach after each stage.

In stage 1, we will look at how well the defence agencies’ reporting of the progress of defence acquisition projects meets the information requirements of internal and external stakeholders, and recommend improvements where necessary.

In stage 2, we will consider, with the defence agencies and other stakeholders, how we can ensure that any recommended improvements are made. Our objective is to achieve a more effective and useful reporting framework that meets the needs of the defence agencies and external stakeholders.

We intend to use our annual plans to keep Parliament and other interested parties informed about the progress we and the defence agencies make against this plan.