Part 4: Reporting information about the Strategy's progress

Ministry of Health: Monitoring the progress of the Primary Health Care Strategy.

In this Part, we comment on:

Reporting information about progress

The Ministry reports changes as they occur. There are many reporting methods, which we have organised into three groups – formal accountability documents and regular reports, other publicly released documents, and confidential reports (see Figure 3).

Figure 3
Examples of the methods the Ministry uses to report on the Strategy

Examples of the methods of reporting
Accountability documents and regular reports
  • Statements of intent
  • Annual plans
  • Annual reports
  • The Director-General of Health’s annual Health and Independence reports, which are reports about the state of the public health
  • Reports giving and analysing the results of periodic New Zealand Health Surveys
Other publicly released documents
  • Quarterly reports about DHBs’ performance
  • Evaluation reports prepared by contracted entities and commissioned by the Ministry
  • Press releases by the Ministry and the Minister
  • Speeches and presentations by Ministry staff to conferences or meetings
  • Responses to questions asked by members of Parliament and select committees
Confidential reports (unless public release is approved)
  • Four annual reports to Cabinet (due in December, 2004-2007*)
  • The Ministry’s weekly reports to the Minister
  • Reports as needed from the Ministry to the Minister

Source: Office of the Auditor-General.

* The reports were provided to Cabinet in November 2004, April and December 2006, and June 2008. The November 2004 report was publicly released on 4 April 2005 and is titled Primary Health Care Strategy: Monitoring its achievements, SDC(04)174.

The Strategy information most often reported has been about the:

  • number of PHOs;
  • numbers of patients enrolled with PHOs;
  • phasing in of primary health care funding subsidies and initiatives; and
  • amounts that patients pay in fees.

The Ministry also regularly reports on health outcomes in its accountability documents, which include indicators influenced by primary health care services.

Reporting about PHOs’ performance

The Ministry has access to, and is able to analyse, information from the PHOs’ reports under the PHO Performance Programme. PHO Performance Programme reports are meant to be published 15 months after PHOs join the programme.

Reporting about DHBs’ performance

The Ministry publishes quarterly summary reports on its website about DHBs’ performance against primary health indicators. This provides a summary of year-to-date progress. Reports are also available for DHBs’ health targets, by DHB and for each three-month period. Annual reports about progress against the targets will be included in the Director-General of Health’s Health and Independence reports. The Ministry does not publish summaries of the DHBs’ written narrative reports.

Reporting the results of independent evaluations

Figure 4 lists the evaluation reports published so far, which are available from the Ministry’s website. The Ministry planned to publish a further 15 quantitative, qualitative, and economic reports and analyses in 2007 and 2008. These had not been published when we wrote our report. The Ministry advised Cabinet that the first of these reports could be published from about September 2008.

Gaps in reporting information

Unpublished information about the PHO Performance Programme

The first reports about PHO performance were due to be published after 31 March 2007, but this has not happened.1 The Strategy intended PHOs to be publicly accountable for the quality of their services, so reports about PHO performance need to be published. Because the Ministry, DHBs, and PHOs have enough confidence in the data for PHOs to be eligible for performance payments after six months in the programme, we suggest that reports about PHOs’ performance should be published after six months, rather than 15.

Fragmented information

Information collected about the results of the Strategy’s implementation is reported in multiple documents (see Figure 3). There is no single report that periodically collates and reports the information collected about the Strategy’s progress toward the goals inherent in the Strategy’s vision statement.

Figure 4
List of published evaluation reports about the Strategy, to March 2008

Nine reports have been published between December 2003 and March 2008.

Report title Summary purpose
Primary Health Organisations: The first year (July 2002 – June 2003) from the PHO perspective, December 2003 To describe PHOs, experiences of PHO implementation, and perceived strengths and weaknesses of PHOs so far.
Evaluation of the Implementation and Intermediate Outcomes of the Primary Health Care Strategy, May 2005 To understand how PHOs and their member providers were responding to the intermediate outcomes of the Strategy, including the effect on reducing health inequalities. To describe the implementation of the Strategy with a specific focus on PHOs.
Review of the Implementation of Care Plus, August 2006 To answer three broad questions: Was Care Plus reaching those individuals with high primary health care need? What were the effects of the Care Plus programme? What were the best ways of organising and delivering Care Plus services?
Improving Access to Primary Health Care: An evaluation of 35 reducing inequalities projects, January 2007 To understand what types of approaches were successful in reducing inequalities and why.
Evaluation of the Primary Health Care Strategy: Practice Data Analysis 2001-2005, September 2007 To examine changes, for different population groups and funding models, between 2001/02 and 2004/05, including changes in the amount patients were paying and how they related to policy objectives, changes in the use of primary health care services, whether more patients were being seen by nurses, and changes in the pattern of ACC claims.
The Evaluation of the Eleven Primary Health Care Nursing Innovation Projects, September 2007 To describe the establishment of the innovations and how well they had achieved the expected outcomes, identify the factors contributing to success, draw lessons from the overall evaluation to help others enhance the role of primary care nurses, and disseminate the results.
Intersectoral Community Action for Health (ICAH) Evaluation, March 2008 To assess whether the programmes had a positive effect on health and disability outcomes, particularly those for population groups experiencing worse health outcomes; identify critical success factors for the ICAH projects; and assess the process and outcomes of one sub-project in each ICAH programme.
Primary Mental Health Initiatives Interim Report, December 2006, October 2007 To report on primary mental health initiatives and innovations projects in 41 PHOs. The final report was due in June 2008.
Key Directions for the Primary Health Care Strategy: case study report and composite success model, June 2007 To report a series of case studies about the use of information in primary health care services to improve clinical practice, organisational performance, and health outcomes. The aim was to draw on the experience from some case studies to create a "composite success model" that would guide future information planning and support learning across the health sector.

Source: Adapted from information provided by the Ministry of Health.

The Ministry considers that the four annual reports to Cabinet combined the up-to-date findings from evaluations, research, and analysis into a single report. However, the Ministry has also identified that:

As there is currently no single report bringing together [primary health care] data, it is difficult to see trends across indicators, e.g. that a DHB may be performing well on indicators relating to child health … but not indicators relating to … services to adults.

Bringing the current indicators together into reports that enable a DHB to compare their performance against national averages ... can be done, but at considerable effort.2

Rather than continue with the current resource-intensive system, the Ministry has decided that it would be a better investment to improve the information systems (through Primary Health Care Strategy: Key Directions for the Information Environment, August 2007) so that reporting can occur more easily.

In our view, it has been, and is, possible for the Ministry to systematically report changes under the Strategy, even though it does not have a comprehensive set of measures covering each of the goals in the Strategy’s vision statement. For example, the Ministry could have produced and published reports about action taken to fulfil the Six Key Directions (and corresponding actions) or Five Priorities for Early Action. The Ministry has reported on some aspects of each, but no reports have listed and reported against all the directions, actions, and priorities.

Although a lot of information is collected and reported, it is difficult to get a full and clear picture about the progress made because the Ministry has not systematically reported against all six goals inherent in the Strategy’s vision statement.

It is useful for reports covering single topics or shorter periods, such as reports about smaller evaluation studies, to be published as they become available. The Ministry should continue to do this. However, periodically bringing information from all the different sources together would provide a rounded assessment of the changes that have occurred.

There has been no public reporting about progress against the Joint Work Programme’s outcomes. The Ministry has reported progress about discrete projects to the Minister and DHBs, but these do not constitute reports about the whole programme’s progress.

No communications and reporting strategies for the Strategy

4.16 The Ministry does not have a communications strategy for reporting about the
Strategy that:

  • identifies audiences for reports; and
  • ensures that reports are understandable and relevant for each audience.

Many of the Ministry’s reports (especially, but not only, the evaluation reports) are technical and densely written. We accept that technical data needs to be available. However, because the Ministry publishes most of its reports widely and publicly, plain English interpretations or summaries of technical information would be useful.

Except for reporting to Cabinet, the Ministry does not have a clear schedule setting out what information about the Strategy it will report and when, or when it will incorporate occasionally updated information (such as the results of health surveys) into its accountability reports (see Figure 3).

Problems with information systems

4.19 In 2001, the Strategy identified that:

… accurate and useful information about enrolled populations and their health needs is critical to quality as well as to the successful adoption of a population health focus in primary health care services.

… the development of further information initiatives will be a key priority for the Ministry of Health, DHBs and PHOs.

… all parties need to work together to ensure that accurate and useful information is collected and shared … building a standardised primary health information infrastructure.3

We referred to the Ministry’s plans to address this issue in paragraph 4.11. The plans, in principle, seem sensible to us.

Difficulty finding basic information and deciding its significance

We had difficulty finding some fairly basic information in published reports about changes under the Strategy. Once we had the information, we had difficulty deciding its significance. We give some examples in Figure 5.

Figure 5
Examples of how inconsistent public reporting and a lack of measures make it difficult to understand the significance of the information being reported

Example 1
It was not a straightforward task to produce Figure 6 in Appendix 1 from published reports, even though the information about the number of PHOs and number of enrolled patients is some of the information most often reported. We needed to go through several documents to find the PHO numbers, and they were not reported for the same date each year. Enrolment figures were reported for different dates. We asked the Ministry to provide figures as at the end of June each year.
Example 2
The Ministry's Annual Report 2005/06 mentioned that, by April 2006, there were 81 PHOs with a combined enrolled population of just under four million people, which is about 95% of the total population.* We could not discover from the reports whether 95% was the maximum potential enrolment expected then or ever. For example, we are aware that, in some locations, people are waiting to register with a general practice, and therefore to enrol with a PHO.
Example 3
The Ministry's Annual Report 2005/06 said the new subsidies had reduced financial barriers to accessing primary health care services. Almost 70% of PHOs (56 of the 81 PHOs) had shown higher use of primary health care services by high-need groups compared with non-high-need groups.* However, without any targets, it is not clear to us whether 70% was a good result for 30 June 2006.

* Ministry of Health (2006), The Annual Report 2005/06 including The Health and Independence Report, Wellington, page 121.

What the Ministry needs to do

We have identified areas where the Ministry could improve its reporting of the Strategy’s progress. It could:

  • produce consolidated progress reports about the Strategy; and
  • publicly report the results from the PHO Performance Programme.

Produce consolidated progress reports about the Strategy

To get more value out of the information the Ministry already has, it should produce consolidated reports summarising, from existing information, what it knows about progress toward the goals inherent in the Strategy’s vision statement.

The reports could also include other relevant information, such as short summaries of any evaluation reports that have been published since the last report. The reports should explain why progress is ahead, on, or behind schedule, and how the Ministry is using the information collected to maintain progress and ensure that the Strategy will be successful.

The reports could include commenting on progress with the Six Key Directions and 40 corresponding actions, and progress with the Five Priorities for Early Action.

Recommendation 2
We recommend that the Ministry of Health regularly produce consolidated reports – the first by 30 June 2009 – about progress toward the Primary Health Care Strategy’s goals and summarise how the information collected is being used to ensure that the Strategy will be successful.

Publicly report results from the PHO Performance Programme

The Ministry should ensure that information about the performance of PHOs is reported as soon as possible. It was intended that reports about PHOs’ performance would be published after the PHOs had been in the programme for 15 months. However, we suggest that the programme be amended so results are published once PHOs are eligible for performance payments. In our view, if the data is reliable enough to be used for performance payments, then the data should be published in some form of report about the performance of PHOs.

Recommendation 3
We recommend that the Ministry of Health ensure that performance reports about the primary health organisations that have been in the Primary Health Organisation Performance Programme for more than 15 months are promptly written and published.

Recommendation 4
We recommend that the Ministry of Health work with district health boards and primary health organisations to review, by 30 June 2009, the Primary Health Organisation Performance Programme so that performance results are published once primary health organisations are eligible for performance payments.

1: The Ministry tells us that this is being negotiated with PHOs and their providers.

2: Ministry of Health (2007), Health Report Ref. No.: 20070693: Review of the Primary Health Care Strategy Monitoring Environment, Wellington.

3: Hon. Annette King (2001), The Primary Health Care Strategy, Wellington, page 25.

page top