Appendix 1: Our audit methodology

Our methodology combined existing evidence from the Ministry of Health’s (the Ministry) documentation and data with new evidence collected through semi-structured interviews and a survey of District Health Boards (DHBs) and Primary Health Organisations.

Using existing evidence – review of Ministry documentation

We reviewed strategic documents relating to how the Ministry had approached implementing the recommendations of the WAVE Report to understand what action the Ministry and the sector had taken and why. The documents included presentations used by the Ministry to communicate its approach to the sector, the Ministry’s annual reports, progress updates given to the Health Committee and overarching strategies such as The New Zealand Health Strategy, The Primary Health Care Strategy and The New Zealand Disability Strategy. We also reviewed available literature on information management and information technology capability in New Zealand compared to other countries.

We reviewed Ministry documents to understand how each of the initiatives had evolved and progressed. The documents we reviewed included Ministry business cases, implementation plans and progress reports for each initiative.

Using existing evidence – analysis of Ministry data and reports from District Health Boards

Where Ministry data was available to quantify progress and the effect of initiatives, we gathered and analysed it. For example, we analysed Ministry data on the trend in the number of duplicate National Health Index numbers created by District Health Boards (DHBs) as an indicator of the effect of the National Health Index upgrade programme. We also used Ministry data on use of the Health Intranet and reviewed surveys conducted on behalf of the Ministry to quantify general progress with information management and information technology capability in the sector. The surveys included one by the Royal New Zealand College of General Practitioners giving the proportion of general practitioners using electronic practice management systems for recording clinical data.

To quantify progress with electronic discharge and referral capability in DHBs, we analysed the reports on progress with implementing the WAVE Report recommendations that DHBs were required to submit to the Ministry as part of the key performance indicators in their annual plans. We also analysed the strengths and weaknesses of DHBs’ plans for improving their information management and information technology capability as identified by the Ministry through its review of DHBs’ first Information System Strategic Plans.

Collecting new evidence – semi-structured interviews

We conducted around 30 semi-structured interviews, mainly between April and June 2005. Interviews covered principal stakeholders within the Ministry and throughout the sector, and listed in the table below. Our framework of questions covered awareness of the WAVE Report and how it had been implemented so far, what progress had been made, the effect that the initiatives were having and how information management and information technology improvements were being taken forward.

In the Ministry of Health we interviewed:

• the Director-General;
• the Deputy Director-General Corporate and Information, Deputy Director-General Sector Policy and Deputy Director-General Clinical Services; and
• staff within the Corporate and Information Directorate, including leaders, advisors and project managers from the New Zealand Health Information Service responsible for strategy and implementing the key initiatives.

In the sector we interviewed:

• the Chairperson of the Health Information Standards Organisation and the Health Network Governance Board;
• Chief Executive Officers, Chief Information Officers and Planning and Funding Managers within District Health Boards;
• Chairpersons, Chief Executive Officers, General Managers and Information Systems Managers within Primary Health Organisations;
• representatives of The Royal New Zealand College of General Practitioners and some general practitioners;
• representatives of the Independent Practitioners Association Council;
• staff of the Accident Compensation Corporation;
• representatives of HealthLink;
• representatives of the New Zealand Health IT Cluster representing organisations interested in health IT (software and solution developers, consultants, health policy makers, health funders, infrastructure companies, health care providers, and academic institutions); and
• members of the WAVE Advisory Board.

Most of the interviews were with 2 or more interviewees and were held in the offices of the stakeholders in Wellington, Auckland, Christchurch, or Greymouth. Each interview lasted between one and 2 hours. The key points were noted and reviewed for all interviews, to make links and draw out themes.

Collecting new evidence – survey of District Health Boards and Primary Health Organisations

During April and May 2005, we conducted an electronic survey of all DHBs and the Ministry, and all 75 established Primary Health Organisations. Our survey was targeted at managers responsible for enhancing information management and information technology, not health practitioners. The purpose of the survey was to gather their opinions on whether implementing the recommendations from the WAVE Report was resulting in more effective use of health information.

For DHBs, 2 copies of the survey were sent out in each case to the Chief Information Officer and the Planning and Funding Managers. One copy of the survey was sent to either the Chairperson, Chief Executive Officer, or General Manager of each Primary Health Organisation.

The survey included a mixture of “closed” questions requiring a yes, no, or don’t know response and “open” questions allowing a free text response, covering opinions on:

• the strategic steps taken by the Ministry;
• the key initiatives progressed;
• the effect so far of activities under the strategic steps and key initiatives;
• monitoring and reporting of progress (DHBs only); and
• continued implementation of information management and information technology improvements.

As well as the Ministry’s response, we received a response from 20 of the 21 Chief Information Officers in the DHBs. At the time of our survey, Northland DHB did not have a Chief Information Officer.

We also received a response from 18 of the 21 Planning and Funding Managers in DHBs, including 7 joint responses on behalf of both the Chief Information Officer and the Planning and Funding Manager. There was a very high degree of consistency between the opinions expressed by Chief Information Officers and Planning and Funding Managers, and we have presented the Chief Information Officer responses in this report.

Just under half (47%) of Primary Health Organisations responded to the survey. Many did not complete the survey as they had limited awareness of the WAVE Report, the strategic steps and the key initiatives. Also, many of those who did respond submitted a response compiled by their managed services organisation, through which they contracted for information services. There was a slight response bias towards more recently established, smaller Primary Health Organisations from the South Island, including one response on behalf of 13 Primary Health Organisations.