Part 4: Reporting information about the Strategy's progress

4.1
In this Part, we comment on:

• what the Ministry has done to report on the Strategy’s progress;
• the gaps in what the Ministry has done; and
• what needs to be done to improve the Ministry’s reporting.

Reporting information about progress

4.2
The Ministry reports changes as they occur. There are many reporting methods, which we have organised into three groups – formal accountability documents and regular reports, other publicly released documents, and confidential reports (see Figure 3).

Figure 3
Examples of the methods the Ministry uses to report on the Strategy

Source: Office of the Auditor-General.

* The reports were provided to Cabinet in November 2004, April and December 2006, and June 2008. The November 2004 report was publicly released on 4 April 2005 and is titled Primary Health Care Strategy: Monitoring its achievements, SDC(04)174.

4.3
The Strategy information most often reported has been about the:

• number of PHOs;
• numbers of patients enrolled with PHOs;
• phasing in of primary health care funding subsidies and initiatives; and
• amounts that patients pay in fees.

4.4
The Ministry also regularly reports on health outcomes in its accountability documents, which include indicators influenced by primary health care services.

Reporting about PHOs’ performance

4.5
The Ministry has access to, and is able to analyse, information from the PHOs’ reports under the PHO Performance Programme. PHO Performance Programme reports are meant to be published 15 months after PHOs join the programme.

Reporting about DHBs’ performance

4.6
The Ministry publishes quarterly summary reports on its website about DHBs’ performance against primary health indicators. This provides a summary of year-to-date progress. Reports are also available for DHBs’ health targets, by DHB and for each three-month period. Annual reports about progress against the targets will be included in the Director-General of Health’s Health and Independence reports. The Ministry does not publish summaries of the DHBs’ written narrative reports.

Reporting the results of independent evaluations

4.7
Figure 4 lists the evaluation reports published so far, which are available from the Ministry’s website. The Ministry planned to publish a further 15 quantitative, qualitative, and economic reports and analyses in 2007 and 2008. These had not been published when we wrote our report. The Ministry advised Cabinet that the first of these reports could be published from about September 2008.

Gaps in reporting information

Unpublished information about the PHO Performance Programme

4.8
The first reports about PHO performance were due to be published after 31 March 2007, but this has not happened.¹ The Strategy intended PHOs to be publicly accountable for the quality of their services, so reports about PHO performance need to be published. Because the Ministry, DHBs, and PHOs have enough confidence in the data for PHOs to be eligible for performance payments after six months in the programme, we suggest that reports about PHOs’ performance should be published after six months, rather than 15.

Fragmented information

4.9
Information collected about the results of the Strategy’s implementation is reported in multiple documents (see Figure 3). There is no single report that periodically collates and reports the information collected about the Strategy’s progress toward the goals inherent in the Strategy’s vision statement.

Figure 4
List of published evaluation reports about the Strategy, to March 2008

Nine reports have been published between December 2003 and March 2008.

Source: Adapted from information provided by the Ministry of Health.

4.10
The Ministry considers that the four annual reports to Cabinet combined the up-to-date findings from evaluations, research, and analysis into a single report. However, the Ministry has also identified that:

As there is currently no single report bringing together [primary health care] data, it is difficult to see trends across indicators, e.g. that a DHB may be performing well on indicators relating to child health … but not indicators relating to … services to adults.

Bringing the current indicators together into reports that enable a DHB to compare their performance against national averages ... can be done, but at considerable effort.²

4.11
Rather than continue with the current resource-intensive system, the Ministry has decided that it would be a better investment to improve the information systems (through Primary Health Care Strategy: Key Directions for the Information Environment, August 2007) so that reporting can occur more easily.

4.12
In our view, it has been, and is, possible for the Ministry to systematically report changes under the Strategy, even though it does not have a comprehensive set of measures covering each of the goals in the Strategy’s vision statement. For example, the Ministry could have produced and published reports about action taken to fulfil the Six Key Directions (and corresponding actions) or Five Priorities for Early Action. The Ministry has reported on some aspects of each, but no reports have listed and reported against all the directions, actions, and priorities.

4.13
Although a lot of information is collected and reported, it is difficult to get a full and clear picture about the progress made because the Ministry has not systematically reported against all six goals inherent in the Strategy’s vision statement.

4.14
It is useful for reports covering single topics or shorter periods, such as reports about smaller evaluation studies, to be published as they become available. The Ministry should continue to do this. However, periodically bringing information from all the different sources together would provide a rounded assessment of the changes that have occurred.

4.15
There has been no public reporting about progress against the Joint Work Programme’s outcomes. The Ministry has reported progress about discrete projects to the Minister and DHBs, but these do not constitute reports about the whole programme’s progress.

No communications and reporting strategies for the Strategy

4.16 The Ministry does not have a communications strategy for reporting about the
Strategy that:

• identifies audiences for reports; and
• ensures that reports are understandable and relevant for each audience.

4.17
Many of the Ministry’s reports (especially, but not only, the evaluation reports) are technical and densely written. We accept that technical data needs to be available. However, because the Ministry publishes most of its reports widely and publicly, plain English interpretations or summaries of technical information would be useful.

4.18
Except for reporting to Cabinet, the Ministry does not have a clear schedule setting out what information about the Strategy it will report and when, or when it will incorporate occasionally updated information (such as the results of health surveys) into its accountability reports (see Figure 3).

Problems with information systems

4.19 In 2001, the Strategy identified that:

… accurate and useful information about enrolled populations and their health needs is critical to quality as well as to the successful adoption of a population health focus in primary health care services.

… the development of further information initiatives will be a key priority for the Ministry of Health, DHBs and PHOs.

… all parties need to work together to ensure that accurate and useful information is collected and shared … building a standardised primary health information infrastructure.³

4.20
We referred to the Ministry’s plans to address this issue in paragraph 4.11. The plans, in principle, seem sensible to us.

Difficulty finding basic information and deciding its significance

4.21
We had difficulty finding some fairly basic information in published reports about changes under the Strategy. Once we had the information, we had difficulty deciding its significance. We give some examples in Figure 5.

Figure 5
Examples of how inconsistent public reporting and a lack of measures make it difficult to understand the significance of the information being reported

* Ministry of Health (2006), The Annual Report 2005/06 including The Health and Independence Report, Wellington, page 121.

What the Ministry needs to do

4.22
We have identified areas where the Ministry could improve its reporting of the Strategy’s progress. It could:

• produce consolidated progress reports about the Strategy; and
• publicly report the results from the PHO Performance Programme.

Produce consolidated progress reports about the Strategy

4.23
To get more value out of the information the Ministry already has, it should produce consolidated reports summarising, from existing information, what it knows about progress toward the goals inherent in the Strategy’s vision statement.

4.24
The reports could also include other relevant information, such as short summaries of any evaluation reports that have been published since the last report. The reports should explain why progress is ahead, on, or behind schedule, and how the Ministry is using the information collected to maintain progress and ensure that the Strategy will be successful.

4.25
The reports could include commenting on progress with the Six Key Directions and 40 corresponding actions, and progress with the Five Priorities for Early Action.

Publicly report results from the PHO Performance Programme

4.26
The Ministry should ensure that information about the performance of PHOs is reported as soon as possible. It was intended that reports about PHOs’ performance would be published after the PHOs had been in the programme for 15 months. However, we suggest that the programme be amended so results are published once PHOs are eligible for performance payments. In our view, if the data is reliable enough to be used for performance payments, then the data should be published in some form of report about the performance of PHOs.

1: The Ministry tells us that this is being negotiated with PHOs and their providers.

2: Ministry of Health (2007), Health Report Ref. No.: 20070693: Review of the Primary Health Care Strategy Monitoring Environment, Wellington.

3: Hon. Annette King (2001), The Primary Health Care Strategy, Wellington, page 25.