Part 2: Methodology

The data we used

15
Appropriations are the basis on which Parliament authorises executive government to incur expenses and capital expenditure. Every year, the Treasury publishes appropriation data on its website. This includes the budgeted expenditure for the next financial year, estimated actual expenditure for the current financial year, and actual expenditure against appropriations for the previous four financial years.⁶

16
We merged three appropriation data sets to form a 10-year view of actual expenditure by appropriations from 2009 to 2018. The data set includes departmental and non-departmental expenditure (including capital expenditure).⁷ The data set we created contains almost 10,000 rows of data related to expenditure against individual appropriations for this period. We refer to this 10-year data set as “the appropriation data” in the rest of this document.

17
Each row of the appropriation data contains 17 items of information. The items we considered the most relevant for our analysis were:

• Department: name of the administering department of the appropriation.
• Vote: name of the Vote in which the relevant appropriations are reported. 
• Appropriation name: name of the appropriation as reported in the Vote. This information, together with the “category name” and “current scope” information, describe what the appropriation is about.
• Category name: the category of appropriated expenditure within a broader multi-category appropriation.
• Current scope: a description of what types of expenditure (that is, expenditure on which activities) are permitted within the appropriation or category.
• Portfolio: ministerial portfolio responsible for specified appropriations, which also allows us to identify the Minister responsible for each appropriation.
• Amount $000: the value of appropriation.

18
One item of information in the appropriation data is “Amount type”. There are several amount types, including actual, estimated actual, and main estimate. Our analysis uses only the actual amount, which represents actual expenditure by appropriation.

19
We excluded some items in the original data sets because they were not relevant to our analysis. Those items included: 

• the “M Number” item, which gives the identification number of the responsible Minister; and
• the “App ID” item, which is used to uniquely identify each appropriation.

Our approach

20
To answer the questions we set out in paragraph 3, we: 

• created the appropriation data set; 
• identified significant events and key words relating to natural hazards and/or disasters. This included obtaining input from the Department of Internal Affairs and the New Zealand Institute of Economic Research;⁸
• identified appropriations potentially related to natural hazards, risk reduction, or response expenditure by searching the appropriation data and using those key words;
• reviewed a sample of the results of each search to refine our criteria to minimise the risks of false positive results (that is, where a key word was present but the appropriation was unlikely to be relevant) and to pick up additional key words that were likely to result in identification of relevant expenditure;
• categorised these appropriations by the level of confidence we had in the relevance of the appropriation, the hazard type, the expenditure type, and the relevant event;
• validated our categorisation by manually checking samples of the data. More information about the quality assurance steps that we carried out is in paragraphs 41-46; and 
• analysed the resulting data set (we refer to this data set enriched with the above categorisation as “the enriched appropriation data” in the rest of this document), summarised our key findings, and recorded the limitations of our analysis.

Key word search

21
We developed a list of key words to identify appropriations relevant to natural hazards. The list of key words we developed was based on information about previous emergencies⁹ from the Civil Defence website, input from the Department of Internal Affairs and the New Zealand Institute of Economic Research, and our own research.

22
The key words we used for each category initially were: 

• Types of natural hazards: earthquake, tsunami, tornado, cyclone, flood, volcano, eruption, landslide, landslip, fire, snow, wind, rain, weather, climate or climatic, drought, and erosion.
• Natural hazard events and where they occurred: Christchurch, Canterbury, Hurunui, Kaikōura, Gita, Fehi, Nelson, Auckland, Taranaki, Tasman, Pam, and Lusi.
• Response, recovery, or risk reduction expenditure: response, enhancement, protection, demolition, rebuild, strengthening, awareness, severe, adverse, readiness, ex gratia, emergency, civil defence, disaster, hazard, and restoration.

23
We searched the appropriation data for key words within the “appropriation name”, “category name”, “vote”, “portfolio”, and “current scope” items. 

24
We used key word searches to pick up whole words and partial words. After each search, we further refined our search criteria both to minimise false positives and to pick up additional key words we thought would identify relevant expenditure. In the end, we carried out more than 200 searches.

Enriching the appropriation data

25
We enriched the appropriation data to facilitate our analysis (see example in Figure 1). We decided to categorise the data into: 

• our level of confidence in the appropriations relevance to natural hazards; 
• type of expenditure; 
• type of hazard; and 
• the specific hazard event.

Figure 1
How we enriched the appropriation data

Two tables show how one appropriation was enriched with four additional columns after categorising.

Before

After

Source: Categorised from appropriation data published on the Treasury’s website.

26
The categories are explained further below.

Confidence levels

27
We categorised appropriations as high, medium, or low, depending on the level of confidence we had in their relevance (Figure 2).

Figure 2
Criteria for confidence levels and the key words that returned results

The table contains the criteria we used to assign the confidence levels and the key words that we found in the appropriation data.

* Civil Defence National Emergency Management Agency. Resources: Previous emergencies: Declared States of Emergency at www.civildefence.govt.nz.

** For example, the appropriation Unwind of Discount Rate Used in the Present Value Calculation of Payment Under Crown Deed of Support with Southern Response Earthquake Services Ltd is limited to the expense incurred in unwinding the discount rate used in the present value calculation of the liability as the liability nears settlement.

Note: There are two entries for medium because there were two different criteria that met medium.

28
The appropriation data was categorised from high to low. After an appropriation had been assigned a category, it was removed from the data search. This ensured that appropriations would not be assigned to more than one category. We also manually reviewed a sample of data from each category to check that the categorisation had been correctly assigned. 

Type of expenditure

29
We have categorised the appropriations into three types of expenditure: “Risk reduction”, “Response and recovery”, and “Other civil defence” (Figure 3). If an appropriation did not fit under one of these three types of expenditure, we categorised it as “Unclear”.

Figure 3
Criteria for each category and the key words that returned results

The table below outlines the criteria used for each category and the key words found in the appropriation data.

Type of hazard

30
We further analysed the appropriation data to understand the different types of natural hazards that were referenced. We initially searched for the following natural hazard key words: earthquake, tsunami, tornado, cyclone, flood, volcano, eruption, landslide, landslip, fire, snow, wind, rain, weather, climate or climatic, drought, and erosion. We reviewed a sample of our search results and refined our search criteria to also include the keyword "environmental".

Figure 4
Types of natural hazard found in the appropriation data

The table lists the hazard types we found in the appropriation data. If an appropriation did not contain a specific hazard type, we categorised it as “Unknown”.

* For more about tsunamis, see Reference – Tsunamis on the National Geographic website: www.nationalgeographic.com.

Specific hazard event

31
We also categorised appropriations according to the specific natural hazard event (including the year when the event occurred) where this was possible (Figure 4). Where it was not possible to categorise an appropriation under a specific hazard event, it was categorised as “Unknown”.

32
We included expenditure associated with international hazard events (Figure 5) because we thought it reasonable to assume that the financial assistance New Zealand provides to other countries would be reciprocated when major events occur here.

33
We expected to see a number of other hazard events specified in the appropriation data – for example, Taranaki and Auckland storms in 2019 and ex-tropical Cyclone Gita in 2018 – but we did not find them.

Figure 5
Criteria for events and the key words that returned results

The table outlines the criteria used for each category, and the events and key words found in the appropriation data.

6: An example from 2019/20 is available on the Treasury website. The Treasury, B19-expenditure-data at www.treasury.govt.nz.

7: Departmental expenditure is directly incurred by departments. Non-departmental expenditure is administered by departments but incurred on behalf of the Crown.

8: The Department of Internal Affairs is also working with the New Zealand Institute of Economic Research to develop a better understanding of government department expenditure on response and risk reduction of natural hazards. We have consulted with both entities on our approach to this analysis. 

9: Resources: Previous emergencies at www.civildefence.govt.nz.