Auditor-General's overview

In 2001, the then Minister of Health launched the Primary Health Care Strategy (the Strategy), which the Government regarded as introducing the most significant changes to primary health care in more than 50 years.

The Strategy sought to achieve many goals, which were set out in its vision statement:

People will be part of local primary health care services that improve their health, keep them well, are easy to get to and co-ordinate their ongoing care.

People will be part of local primary health care services that focus on better health for a population, and actively work to reduce health inequalities between different groups.

It was estimated that carrying out the Strategy could take five to 10 years. It is a large and difficult task, which involves participation by many primary health care stakeholders.

My staff audited how the Ministry of Health (the Ministry) has monitored progress toward the Strategy's goals. Overall, the Ministry needs to review its measures to ensure that it can assess progress toward all of the goals in the Strategy's vision statement. Once it has that progress information, it needs to report it publicly and regularly in a consolidated report.

The Ministry of Health's approach to the Strategy

Through the Strategy, the Ministry has introduced major structural and funding changes to the health sector. About 80 primary health organisations (PHOs) have formed to provide primary health care services through more than 1,000 contracted providers (such as general practices and general practitioners). More than four million patients are enrolled in PHOs. The Ministry has phased in significant increases in primary health care funding, and the funding is no longer based on a fee-for-service model. Most of the new funding was phased in from 2002/03 to 2007/08; more than $3.2 billion was spent on primary health care funding during that period.¹

Because of the size and scale of change, including the complexities involved with negotiating agreements with many parties, the Strategy described a "stepwise, evolutionary" approach to implementation. Initially, the Ministry did not set out what would be achieved and by when, apart from the directions and actions described in the Strategy.

In 2005/06, the Ministry and district health boards (DHBs) recognised the need to shift their focus from implementing structural and funding changes to achieving the Strategy's wider goals. A consultation process produced a "Joint Work Programme" report that described the situation in 2001 and 2005, and described about 100 outcomes to be achieved by 2010. The Ministry and DHBs are leading projects to achieve the Joint Work Programme's outcomes. The projects' progress is reported, although not publicly. The Ministry and DHBs do not report explicitly on progress toward the 100 outcomes.

Some differences the Strategy has made

There are indications of improvements in primary health care. An evaluation report by the Ministry concluded that, from 2001/02 to 2004/05, lower cost access to primary health services had improved consultation rates for most patient groups.

More recently, the Ministry has been reporting improvements in performance for a broader range of health indicators as evaluation and health survey data becomes available. The Director-General of Health tells me that he is getting good feedback about the Strategy's achievements from the health sector. New Zealand's performance compares well against international primary health care indicators.

My staff have not assessed or verified the Strategy's achievements. The audit focused on what information the Ministry was collecting and reporting to assess progress toward each of the Strategy's goals. My staff expected the Ministry to use the information collected to maintain progress and ensure that the Strategy's goals would be achieved.

How the Ministry has monitored the Strategy

Once the initial implementation effort was over, the Ministry put significant resources into a range of monitoring and evaluation initiatives. The Ministry has collected and reported a lot of information about the changes brought about by the Strategy. The information has included the effect on people's health, the number of PHOs and the number of people enrolled with them, some aspects of PHOs' and DHBs' performance, the phasing in of funding for the Strategy's subsidies and initiatives, and independent evaluations of the Strategy's implementation.

The Ministry has reported this information many different documents, including its annual reports, the Director-General of Health's health and independence reports, reports to Ministers of Health, and reports to Cabinet.

The Director-General of Health informed me in August 2008 that the Ministry is developing advice to Government to establish a comprehensive performance management framework for the next phase of the Strategy. Part of the framework involves aligning the indicators used to monitor PHOs' performance with the health targets used to monitor DHBs' performance. Until then, the current monitoring of PHO and DHB performance will continue, alongside the schedule of independent evaluations and periodic national health surveys.²

More coherent information needed

The Ministry needs to organise its monitoring and evaluation to better assess and report progress against all the goals inherent in the Strategy's vision statement.

The Ministry's approach to reporting information is fragmented, which makes it difficult to get a full and clear picture about the progress that has been made toward achieving the Strategy's goals. After my staff had brought the information together, it was difficult to decide its significance because reported achievements are not always set in the context of expected results.

The Ministry is already aware that there are some gaps in the information being collected. For example, there are gaps in indicators for community involvement and self-management. The Ministry intends to monitor in more depth the management of long-term diseases. The Ministry might identify further gaps, duplication, or measures that need to be modified when it prepares a more comprehensive monitoring framework.

More complete information needed

The Ministry needs to bring together all the existing information it has into a single report to provide as full and clear picture as possible for the health sector, Parliament, and the public about the progress made toward the Strategy's goals. The Ministry should produce these consolidated reports regularly, and tailor their content more appropriately for the audience.

Importance of well-designed measures

Being able to report meaningfully on performance, particularly for major initiatives, is a core part of public sector accountability.

Well-designed measures can function as effective drivers of change. A set of measures that covers all important aspects of a strategy and is well designed from the outset can help to support changes occurring in all the areas where change is needed or wanted.

Agencies that collect and report information in some areas but not others create a risk that Parliament and the public will perceive that a strategy's implementation and results are also uneven, and potentially ineffective and inefficient. This may not be the case, but the perception is difficult to counter without good information.

By reporting only changes, reports are inevitably silent about any changes that should have occurred but have not. If reported changes are not compared with starting positions and anticipated results, then important achievements may not be recognised. Changes that were intended to occur but have not may not be identified. Problem areas that should be recognised may remain undetected.

In June 2008, I published a discussion paper about performance reporting - The Auditor-General's observations on the quality of performance reporting. In it, I encouraged public entities to think carefully about the outcomes they are working towards and to explain in their external reporting the reasons for what they are doing, the focus of their reporting, and the rationale for - and relationships between - the various elements and measures.

I do not expect entities to measure and report on everything. Rather, the aim should be to provide a coherent and reasonably complete picture of overall performance, through a mixture of financial and non-financial information. That same message is relevant here.

I thank the Ministry's staff for their helpful co-operation during the audit.

K B Brady
Controller and Auditor-General

6 October 2008

1: The First Contact subsidy and increases to the Pharmaceutical subsidy account for $2.2 billion. Other subsidies and initiatives account for a further $1 billion (see Appendix 3).

2: On 12 September 2008, the Minister of Health, Hon. David Cunliffe, made a speech about primary health care to communicate decisions recently made by Cabinet. He said: "Looking further forward I want to see existing performance initiatives aligned and a comprehensive and closely aligned performance management system developed across DHBs, PHOs and providers. This will ensure performance management is more consistent between DHBs, PHOs and providers and that it is more focused on measuring outcomes, rather than inputs."