Part 3: Problems with information about the selected defence acquisition projects

3.1
In this Part, we discuss the problems that prevented us from providing a more detailed and precise analysis of the cost and time frame performance of the selected defence acquisition projects. We recognise that, when the defence agencies’ reporting systems were introduced, they were not designed to monitor the progress of projects in the way that we expected. However, we still expected that the information we were looking for would have been more readily available and easier to extract.

3.2
As we highlighted in Part 2, there were changes to the costs and time frames between the Approval to Commence and Approval to Commit points for each of the 10 selected defence acquisition projects. Such changes are to be expected because the cost and delivery time frames become more precise during the acquisition process.

3.3
Changes can be caused by a range of factors, including:

• improved knowledge of available equipment or systems;
• contract negotiations;
• production difficulties;
• greater awareness of logistic support requirements and costs;
• personnel and training needs;
• specification changes; and
• foreign exchange fluctuations.

3.4
It is important for the defence agencies to maintain information so that they can monitor and report changes, and are able to explain the reasons for changes to Ministers, Parliament, and other stakeholders. This is the essence of accountability, and will provide confidence to readers of the reports that the defence agencies are appropriately managing projects through the acquisition process.

3.5
In compiling the information for each project, we had difficulty:

• getting information on the reasons for changes to estimated costs, time frames, and essential user requirements;
• accurately identifying the approval points for some projects; and
• extracting and interpreting historical information.

3.6
These difficulties have implications for the transparency of the acquisition process. There is also a risk that the outcome of an acquisition is a capability that does not meet the essential user requirements that were defined by the NZDF at the start of the project.

Difficulty getting information about the reasons for changes

3.7
The Ministry’s financial reporting system is designed to monitor expenditure against the budget that Cabinet approved for each defence acquisition project. From this system, we were able to identify high-level differences between the current costs for project main line items (for example, the prime contract and ancillary contracts) and the budget approved by Cabinet. However, this system does not identify the reasons for changes between expenditure categories. Some of this information is available from the responsible project manager’s working files. However, some of these files are held offshore (in cases where the project manager is based overseas) and do not monitor changes between expenditure categories consistently or comprehensively. We encountered similar problems when we tried to ascertain changes in time frames and essential user requirements.

3.8
As an example, we tried to ascertain the changes in cost for the Medium Range Anti-Armour Weapon project. Between Cabinet’s Approval to Commence in December 2002 and the Approval to Commit point in December 2003, the estimated cost of the project increased from $21.5 million to $23.9 million (an increase of $2.4 million). As at June 2007, the forecast cost to complete the project had reduced to $21.9 million (a decrease of $2 million). By December 2007, the forecast had again risen to $23.9 million. We were unable to identify the reasons for all of the cost movements using the Ministry’s project files and financial reports.

3.9
The $2.4 million increase in estimated cost between the Approval to Commence and Approval to Commit points included savings from favourable exchange rates (a decrease of $3.7 million) and an increase in the cost of various project line items (an increase of $6.1 million). Of the $6.1 million, we were able to find reasons for an increase of $0.5 million,2 leaving $5.6 million that the Ministry’s project manager could not explain in any further detail. The project manager assumed that the increase was the result of the supplier providing more accurate costs at the Approval to Commit point.

3.10
As at June 2007, the project was forecast to be $2 million under budget. We tried to identify the reasons for this. There were savings because of favourable exchange rates (a decrease of $2 million), and reductions in the prime contract, ancillary contracts, and project management costs (decreases of $0.5 million, $1.7 million, and $1 million respectively). Those savings were partly offset by an increase ($3.2 million) in the budgeted contingency funding committed or forecast to be spent, leaving the project $2 million under budget. We were unable to find detailed reasons for the savings under ancillary contracts and project management, because the Ministry’s finance and acquisition project management systems did not monitor costs at the level of detail that we required.

Difficulty accurately identifying approval points for some projects

3.11
We have used the Approval to Commence point as the baseline approval point for each project. This is one of the main milestones set out in the CMF. Not all of the projects included in our audit were approved under the CMF. Some pre-dated it, and for those projects it was sometimes difficult to identify the historical approvals that corresponded to the CMF’s milestones, because the approvals were not always clearly documented.

3.12
For example, the Very Low Level Air Defence Alerting and Cueing System (VACS) was originally a component of the Very Low Level Air Defence (VLLAD) project to procure shortrange anti-aircraft missiles for the Army. The project was first approved in June 1994. In August 1996, the project was split into two parts: one to procure the missiles and launchers, and the other to procure a radar alerting and cueing system to identify hostile aircraft and assign missiles to target them (the VACS). The delivery of missiles and launchers was completed in April 1998. However, no suitable VACS was available. The Ministry told us that the VACS part of the project was cancelled, and a new project to procure a VACS started in 2002.

3.13
The Ministry’s documentation did not clearly show the cancellation of the VACS part of the original VLLAD project and the start of the new project, including the dates when each happened. Therefore, it was not possible to formally identify the Approval to Commence point for the revived VACS project. For our analysis, we have used the cost and time frame given in the 2002 LTDP. Assessing the timeliness of the VACS project against the original June 1994 approval would show the overall project performance in a poor light, and one that does not reflect the more recent performance of the project.

Difficulty extracting and interpreting historical information

3.14
Information about some of the defence acquisition projects is dated. For example, the VACS and the Medium Range Anti-Armour Weapon projects began in the 1990s. Extracting the necessary information from the extensive project files proved very time-consuming. The information often required interpretation, based on an in-depth knowledge of the project, to extract the relevant details for our audit. We did not have that in-depth knowledge and, in some cases, project managers with the relevant historical knowledge had left the Ministry.

3.15
We expected the Ministry to have information in a format that we could readily access and understand. Because the information was not readily accessible, the NZDF, as the end user, may also not have adequate access to information about whether the acquisition process will deliver a capability that meets its requirements.

The Ministry’s reporting of the progress of acquisition projects

3.16
To compile the information in Part 2, we examined the Ministry’s internal financial reporting of the progress of projects, which was also reported in the Ministry’s annual report. In our view, both could be more complete and clearer in conveying the progress of projects. Our concern is that the Ministry could report more information on project progress, not that the reported figures are inaccurate.

3.17
The format of the Ministry’s internal reporting and its annual report is very similar. Both present highly aggregated information and are arranged by “total approved cost” and “total forecast to complete”:

• The total approved cost reflects the cost defined at the Approval to Commit point, factoring in any foreign exchange changes (which are costs outside the Ministry’s control).
• The total forecast to complete is made up of expenditure to date, future commitments and forecasts, and Goods and Services Tax.

3.18
However, a significant difference between the reports is that the annual report does not show the “total cost variance” that is included in the internal report. The total cost variance is the difference between the approved cost and forecast to complete. Figure 4 shows the internal report and annual report extract produced for the C-130 Life Extension project.

Figure 4
Illustration of the Ministry’s internal report and annual report information for the C-130 Life Extension project

3.19
For the 10 selected projects, we reviewed information from the Ministry’s internal financial status reports (as at the end of June 2007) to see what total cost variance was reported between the Approval to Commit point and the cost forecast at the end of June to complete the project. We note how small the variance was for each project, with each project reported to be under budget.

3.20
Figure 5 shows the reported cost variations and the percentages of the approved costs they represent. Two projects are not included – the Training/Light Utility Helicopter had not reached the Approval to Commit point as at the end of June 2007, and the Improvised Explosive Device Disposal project is managed by the NZDF.

Figure 5
Reported cost variations and their percentage of project approval, as at June 2007

Source: The Ministry of Defence’s Financial Status Reports.

3.21
The Ministry’s practice of entering into firm fixed-price contracts, where the risk is borne by the contractor, provides a reason for the variations being so small. However, without access to more complete information, we cannot confirm whether the defence agencies have made capability and/or time trade-offs to remain within the approved budget. In our view, the Ministry’s reports do not provide some important and useful information. For example, they do not show:

• what cost changes there have been compared with budgeted costs to make up the overall variance, and the reasons for the changes;
• how much of the project contingency has been used;
• how much of future expenditure is uncommitted; and
• how foreign exchange fluctuations have affected project costs set at the Approval to Commit point.

Use of project contingency and savings against budget

3.22
The project contingency for the Medium Range Anti-Armour Weapon was used to retain savings from other budget lines (see paragraph 3.10). It is sensible to retain savings from budget lines to use for any unexpected costs. However, the practice of placing those saved funds in the contingency reduces the transparency of project changes, especially savings.

3.23
In our view, information about how much of the contingency has been used, and what savings on budgeted costs have been made to date, could usefully be separately identified in the Ministry’s internal and external reports. Instead, contingency expenditure and budget savings are included within the aggregated “future commitments and forecasts” figure in the Ministry’s reports. Because the use of the contingency fund and any savings on budget lines are not reported separately, the Ministry’s reports do not transparently show that projects may be underspent, as happened with the Medium Range Anti-Armour Weapon project. Equally, any overspending that has been managed by the project manager re-assigning savings from elsewhere or by favourable foreign exchange fluctuations is not transparent.

Uncommitted expenditure

3.24
The Ministry’s reports on the projects do not distinguish between future expenditure that is committed to be spent under contracts that are in place, and future expenditure that is forecast as being required to complete the projects but has not yet been committed to under contract. Separately identifying these two types of expenditure – and identifying early where forecast expenditure may not be required – could be useful to the NZDF in deciding how to re-allocate any uncommitted funds.

Reporting foreign exchange fluctuations

3.25
Foreign exchange fluctuations are a standard feature of projects that span several years and currencies. The cost of the fluctuations is forecast, included as part of the total cost of the project, and submitted to Cabinet when the Ministry seeks its Approval to Commit. However, accounting for subsequent foreign exchange fluctuations in the Ministry’s reporting is complicated.

3.26
When there is a favourable change in the foreign exchange rate, the saving is deducted from the project’s total approved cost. When the change is unfavourable, the loss is added to the project’s total approved cost. Therefore, the total approved cost of a project increases or decreases according to foreign exchange fluctuations. In our view, a project’s total approved cost should be reported as a constant.

3.27
Figure 6 shows how foreign exchange fluctuations affect the total approved cost of the selected defence acquisition projects reported by the Ministry in its 2005, 2006, and 2007 annual reports. The Figure does not include three projects from our selection:

• The Medium Utility Helicopter project is not included because its total cost had not been defined until the period covered by the 2007 annual report.
• The Training/Light Utility Helicopter is not included because contract negotiations had not been completed at the time of the 2007 annual report.
• The Improvised Explosive Device Disposal project is not included because it is managed by the NZDF.

Figure 6
Effect of foreign exchange fluctuations on the total approved cost (including GST) for the selected defence acquisition projects

3.28
Figure 6 shows how the total approved cost of the selected projects changed because of foreign exchange fluctuations. When compared with the total cost of the projects, the changes are not significant. However, the changes for individual projects can be significant. For example, the total approved cost for the Boeing 757 Acquisition and Modification project changed by about $13 million between the 2005 and 2006 annual reports.

3.29
We are satisfied that the Ministry is managing its exposure to foreign currency fluctuations appropriately. However, we consider that the Ministry can improve its reporting on the effect these fluctuations have on a project’s cost. Because the Ministry’s submissions to Cabinet at the Approval to Commit point include the forecast cost of foreign exchange fluctuations, the subsequent effect of actual changes should be shown separately as a variation to the cost that Cabinet approved.

3.30
Our concern, which is shared by the Treasury, is that the Ministry’s current practice is to report the actual change in a “floating” total approved cost. This approach reduces the transparency of how foreign exchange fluctuations affect the cost of individual projects, and the projects overall. In our view, the Ministry’s reporting should show the fluctuations as part of the overall cost variance. We were told that the Ministry is reconsidering its approach to reporting project costs. This is one area that the Ministry could usefully review.

2 This $0.5m was made up of price increases for some project items, and savings in other areas of the project. Reasons for the changes included:

• price increases for weapons systems, missiles, and simulation and training equipment;
• reduced costs for depot level support, transportation, training, and testing;
• purchase of additional spare parts and equipment; and
• an increased budget for the Ministry’s project administration and infrastructure requirements.