Part 2: Methodology

Analysing government expenditure related to natural hazards.

The data we used

15
Appropriations are the basis on which Parliament authorises executive government to incur expenses and capital expenditure. Every year, the Treasury publishes appropriation data on its website. This includes the budgeted expenditure for the next financial year, estimated actual expenditure for the current financial year, and actual expenditure against appropriations for the previous four financial years.6

16
We merged three appropriation data sets to form a 10-year view of actual expenditure by appropriations from 2009 to 2018. The data set includes departmental and non-departmental expenditure (including capital expenditure).7 The data set we created contains almost 10,000 rows of data related to expenditure against individual appropriations for this period. We refer to this 10-year data set as “the appropriation data” in the rest of this document.

17
Each row of the appropriation data contains 17 items of information. The items we considered the most relevant for our analysis were:

  • Department: name of the administering department of the appropriation.
  • Vote: name of the Vote in which the relevant appropriations are reported. 
  • Appropriation name: name of the appropriation as reported in the Vote. This information, together with the “category name” and “current scope” information, describe what the appropriation is about.
  • Category name: the category of appropriated expenditure within a broader multi-category appropriation.
  • Current scope: a description of what types of expenditure (that is, expenditure on which activities) are permitted within the appropriation or category.
  • Portfolio: ministerial portfolio responsible for specified appropriations, which also allows us to identify the Minister responsible for each appropriation.
  • Amount $000: the value of appropriation.

18
One item of information in the appropriation data is “Amount type”. There are several amount types, including actual, estimated actual, and main estimate. Our analysis uses only the actual amount, which represents actual expenditure by appropriation.

19
We excluded some items in the original data sets because they were not relevant to our analysis. Those items included: 

  • the “M Number” item, which gives the identification number of the responsible Minister; and
  • the “App ID” item, which is used to uniquely identify each appropriation.

Our approach

20
To answer the questions we set out in paragraph 3, we: 

  • created the appropriation data set; 
  • identified significant events and key words relating to natural hazards and/or disasters. This included obtaining input from the Department of Internal Affairs and the New Zealand Institute of Economic Research;8
  • identified appropriations potentially related to natural hazards, risk reduction, or response expenditure by searching the appropriation data and using those key words;
  • reviewed a sample of the results of each search to refine our criteria to minimise the risks of false positive results (that is, where a key word was present but the appropriation was unlikely to be relevant) and to pick up additional key words that were likely to result in identification of relevant expenditure;
  • categorised these appropriations by the level of confidence we had in the relevance of the appropriation, the hazard type, the expenditure type, and the relevant event;
  • validated our categorisation by manually checking samples of the data. More information about the quality assurance steps that we carried out is in paragraphs 41-46; and 
  • analysed the resulting data set (we refer to this data set enriched with the above categorisation as “the enriched appropriation data” in the rest of this document), summarised our key findings, and recorded the limitations of our analysis.

Key word search

21
We developed a list of key words to identify appropriations relevant to natural hazards. The list of key words we developed was based on information about previous emergencies9 from the Civil Defence website, input from the Department of Internal Affairs and the New Zealand Institute of Economic Research, and our own research.

22
The key words we used for each category initially were: 

  • Types of natural hazards: earthquake, tsunami, tornado, cyclone, flood, volcano, eruption, landslide, landslip, fire, snow, wind, rain, weather, climate or climatic, drought, and erosion.
  • Natural hazard events and where they occurred: Christchurch, Canterbury, Hurunui, Kaikōura, Gita, Fehi, Nelson, Auckland, Taranaki, Tasman, Pam, and Lusi.
  • Response, recovery, or risk reduction expenditure: response, enhancement, protection, demolition, rebuild, strengthening, awareness, severe, adverse, readiness, ex gratia, emergency, civil defence, disaster, hazard, and restoration.

23
We searched the appropriation data for key words within the “appropriation name”, “category name”, “vote”, “portfolio”, and “current scope” items. 

24
We used key word searches to pick up whole words and partial words. After each search, we further refined our search criteria both to minimise false positives and to pick up additional key words we thought would identify relevant expenditure. In the end, we carried out more than 200 searches.

Enriching the appropriation data

25
We enriched the appropriation data to facilitate our analysis (see example in Figure 1). We decided to categorise the data into: 

  • our level of confidence in the appropriations relevance to natural hazards; 
  • type of expenditure; 
  • type of hazard; and 
  • the specific hazard event.

Figure 1
How we enriched the appropriation data

Two tables show how one appropriation was enriched with four additional columns after categorising.

Before

Appropriation Year Amount
$million
Greater Christchurch Anchor Projects 2018 17.474

After

Appropriation Year Amount $million Confidence level Expenditure type Hazard type Event
Greater Christchurch Anchor Projects 2018 17.474 Medium Response and recovery Earthquake 2010/11 Canterbury earthquakes

Source: Categorised from appropriation data published on the Treasury’s website.

26
The categories are explained further below.

Confidence levels

27
We categorised appropriations as high, medium, or low, depending on the level of confidence we had in their relevance (Figure 2).

Figure 2
Criteria for confidence levels and the key words that returned results

The table contains the criteria we used to assign the confidence levels and the key words that we found in the appropriation data.

Category Criteria Key words that returned results that were then categorised Notes
High
We have a high degree of confidence that an appropriation was related to natural hazards response or risk reduction expenditure.
Specific mention of natural hazard or disaster event from the Civil Defence website of previous emergencies. 2009 Victoria bushfire

2011 Australian flood

2011 Canterbury earthquake

Canterbury rebuild

Canterbury recovery

Christchurch earthquake

2011 Japan earthquake/tsunami

2016 Kaikōura earthquake

2017 Edgecumbe flood
Medium
We have a moderate degree of confidence that an appropriation was related to natural hazards response or risk reduction expenditure.
Contains key words associated with natural hazards or disasters, and includes a response or risk reduction key word. Earthquake, flood, severe, weather, erosion, costs, arising, repair, recovery, strength, control, protect, relief, warning
Medium
We have a moderate degree of confidence that an appropriation was related to natural hazards response or risk reduction expenditure.
Special terms indirectly related to natural hazards response and reduction. Civil Defence, Minister for Greater Christchurch Regeneration All civil defence emergencies declared before 2020 have been related to natural disasters.*

Christchurch regeneration is related to the 2010/11 Canterbury earthquakes.
Low
We think the appropriation might be related to natural hazard response or risk reduction expenditure, but when we manually reviewed the content it was not clear that the appropriation was about natural hazards.**
Either contains key words associated with natural hazards or disasters,
or
has key words associated with response and/or risk reduction expenditure type.
Earthquake, climate or climatic (clima), hazard, post crisis, emergency services, disaster

Protection, environmental awareness, rebuild, adverse event, emergency readiness, restoration Christchurch, reinstatement Christchurch

* Civil Defence National Emergency Management Agency. Resources: Previous emergencies: Declared States of Emergency at www.civildefence.govt.nz.

** For example, the appropriation Unwind of Discount Rate Used in the Present Value Calculation of Payment Under Crown Deed of Support with Southern Response Earthquake Services Ltd is limited to the expense incurred in unwinding the discount rate used in the present value calculation of the liability as the liability nears settlement.

Note: There are two entries for medium because there were two different criteria that met medium.

28
The appropriation data was categorised from high to low. After an appropriation had been assigned a category, it was removed from the data search. This ensured that appropriations would not be assigned to more than one category. We also manually reviewed a sample of data from each category to check that the categorisation had been correctly assigned. 

Type of expenditure

29
We have categorised the appropriations into three types of expenditure: “Risk reduction”, “Response and recovery”, and “Other civil defence” (Figure 3). If an appropriation did not fit under one of these three types of expenditure, we categorised it as “Unclear”.

Figure 3
Criteria for each category and the key words that returned results

The table below outlines the criteria used for each category and the key words found in the appropriation data.

Category Criteria Key words that returned results that were then categorised Notes
Response and recovery
The wording of the appropriation was related to expenditure on a response to or recovery to an event that had happened.
Contains key words related to response and recovery, or a hazard event. Costs arising, repair, recovery, relief, regeneration, rebuild, restoration Christchurch.
Risk reduction
The wording of the appropriation indicates that the appropriation was related to reducing the risks associated with natural hazards.
Contains key words related to risk reduction. Strength, control, protect, warning, environmental awareness, emergency readiness.
Other civil defence
The wording of the appropriation indicates that the appropriation was related to civil defence expenditure, but we were not able to clearly identify a relationship to either risk reduction or response.
Contains the specific term civil defence. Civil defence. The type of appropriations that we identified tended to be related to ongoing expenses associated with civil defence – for example, policy advice on matters relating to civil defence and emergency management and drafting ministerial correspondence and questions.

Type of hazard

30
We further analysed the appropriation data to understand the different types of natural hazards that were referenced. We initially searched for the following natural hazard key words: earthquake, tsunami, tornado, cyclone, flood, volcano, eruption, landslide, landslip, fire, snow, wind, rain, weather, climate or climatic, drought, and erosion. We reviewed a sample of our search results and refined our search criteria to also include the keyword "environmental".

Figure 4
Types of natural hazard found in the appropriation data

The table lists the hazard types we found in the appropriation data. If an appropriation did not contain a specific hazard type, we categorised it as “Unknown”.

Type of natural hazard Criteria Notes
Bushfire Contains the key word “bushfire”.
Climate change related Contains the key words “climate” or “climatic”. This is an indirect association. Climate change is not necessarily related to a specific hazard but could lead to a natural hazard.
Earthquake Contains the key word “earthquake”.
Earthquake and tsunami Contains the key words “earthquake” and “tsunami”. Although a tsunami can also be caused by a volcanic eruption or an underwater landslide, we did not identify any references to these key words in the appropriation data.*
Environmental Contains the key word “environmental”. This is an indirect association. Environmental does not necessarily indicate a hazard but might relate to hazard response or risk reduction.
Erosion Contains the key word “erosion”. This is an indirect association. Landslips and landslides can be caused by erosion.
Flood Contains the key word “flood”.
Weather Contains the key word “weather” This is an indirect association. Natural hazards can be caused by weather events.

* For more about tsunamis, see Reference – Tsunamis on the National Geographic website: www.nationalgeographic.com.

Specific hazard event

31
We also categorised appropriations according to the specific natural hazard event (including the year when the event occurred) where this was possible (Figure 4). Where it was not possible to categorise an appropriation under a specific hazard event, it was categorised as “Unknown”.

32
We included expenditure associated with international hazard events (Figure 5) because we thought it reasonable to assume that the financial assistance New Zealand provides to other countries would be reciprocated when major events occur here.

33
We expected to see a number of other hazard events specified in the appropriation data – for example, Taranaki and Auckland storms in 2019 and ex-tropical Cyclone Gita in 2018 – but we did not find them.

Figure 5
Criteria for events and the key words that returned results

The table outlines the criteria used for each category, and the events and key words found in the appropriation data.

Event Criteria Key words that returned results that were then categorised
2009 Victoria bushfires Contains key words of the event. Victoria and bushfire

Australia and bushfire
2011 Queensland floods Contains key words of the event. Australia and flood
2010/11 Canterbury earthquakes Contains key words of the event. Canterbury and earthquake

Canterbury and rebuild

Christchurch and earthquake

Canterbury and recovery
2011 Great East Japan earthquake Contains key words of the event. Japan and earthquake/tsunami
2016 Kaikōura earthquake Contains key words of the event. Kaikōura and earthquake
2017 Edgecumbe flood Contains key words of the event. Edgecumbe and flood

6: An example from 2019/20 is available on the Treasury website. The Treasury, B19-expenditure-data at www.treasury.govt.nz.

7: Departmental expenditure is directly incurred by departments. Non-departmental expenditure is administered by departments but incurred on behalf of the Crown.

8: The Department of Internal Affairs is also working with the New Zealand Institute of Economic Research to develop a better understanding of government department expenditure on response and risk reduction of natural hazards. We have consulted with both entities on our approach to this analysis. 

9: Resources: Previous emergencies at www.civildefence.govt.nz.