Part 3: Collecting information about the Strategy's progress

Ministry of Health: Monitoring the progress of the Primary Health Care Strategy.

3.1
In this Part, we comment on:

3.2
We expected that the Ministry would have considered how it would measure progress towards the Strategy’s goals. A recent document produced by the World Health Organization shows that others share this expectation of the health sector:

Whereas policy-makers in the past often reformed without critically evaluating their efforts, they now need to define expectations, track resources and demonstrate outcomes. Performance measurement makes possible a structured assessment of how health systems are doing and flags up what can be done better.1

The Ministry’s monitoring and evaluation framework

3.3
Once the initial implementation effort was over – enabling PHOs to form and setting up the new subsidies and funding arrangements – the Ministry put significant resources into a range of monitoring and evaluation initiatives. Some early or relatively specific projects have been completed. Others have yet to report, because they are substantial studies tracking long-term changes.

3.4
Three years after the Strategy’s launch in 2001, the Ministry outlined a monitoring and evaluation framework to report on the Strategy’s implementation and outcomes. The framework had two main parts: monitoring PHO performance, and commissioning a series of large and small independent evaluations of the Strategy’s implementation and outcomes.

3.5
The framework was broken up into individual projects, which were reported separately.

3.6
During 2006/07, the Ministry carried out a project, focused on DHB accountability and performance against the Strategy, to:

  • document the current monitoring framework;
  • complete a gap analysis of current monitoring reporting and evaluation approaches; and
  • produce a “development pathway to support ongoing development of the monitoring framework, including an implementation plan”.2

3.7
The result of the project was a new framework. The new framework linked the existing primary health care indicators used to monitor DHBs’ and PHOs’ performance (see paragraphs 3.10-3.14) to the New Zealand Health Strategy. It showed gaps for areas such as community involvement, self-management, care co-ordination, and improved access (including patients’ use of services and the fees they pay), which are important parts of the Primary Health Care Strategy.

3.8
The Ministry also identified that it could improve the use of the existing indicators by bringing them together to compare population groups at a district and lower level. This had not yet been done because of the substantial effort it took to collate and analyse the information from several databases (see paragraphs 4.10 and 4.11). The Ministry made plans for improved reporting that included producing a single report for each DHB of its performance against all the indicators it is measured against. A single report would enable all DHBs and the public to see which DHBs perform better overall, and if some DHBs have strengths or weaknesses in some areas. Reports consolidating DHBs’ performance have not yet been produced.

The Joint Work Programme

3.9
In 2005/06 (the fifth year of the Strategy’s implementation), the Ministry and DHBs shifted their focus from establishment to concentrating more on achieving “the delivery aims central to the Strategy”.3 This resulted in the Joint Work Programme, which describes about 100 outcomes to be achieved by 2010 based on assessments against starting positions in 2001 and 2005.4 At higher levels, the Joint Work Programme is organised into five goals, 11 themes (with sub-themes), and four work streams. Projects to implement the Joint Work Programme are organised in keeping with the Strategy’s Six Key Directions (see Figure 2).

Monitoring of PHOs’ performance by DHBs and the Ministry

3.10
The Ministry introduced a programme to monitor PHO performance in 2005. It was a programme that PHOs could voluntarily join. Responsibility for monitoring PHOs’ performance has since been devolved to DHBs. PHOs, in turn, are to monitor and help improve their members’ performance. A few of the indicators being monitored have target dates for their achievement. Other indicators require incremental annual improvements to be made.

3.11
Under the PHO Performance Programme, information has been collected since 1 January 2006 for 29 PHOs, and all PHOs had joined the programme by 1 January 2008. PHOs report their performance quarterly to their DHB. After PHOs have been in the programme for six months, they are eligible to receive performance payments that are based on progress against the programme’s indicators.

3.12
The Ministry plans to phase in two more sets of indicators to focus PHOs’ attention in priority areas. These indicators will focus on long-term conditions such as cardiovascular disease and diabetes. The second set was to have been introduced in mid-2007, but this date was amended to 1 July 2008. The Ministry now plans to introduce the second set of indicators in October 2008. There is no date set for phasing in the third set of indicators.

Monitoring of DHBs’ performance by the Ministry

3.13
As all devolved primary health care is funded by DHBs, monitoring DHBs’ performance is one method of assessing the performance of all primary health care providers (not only PHOs), and any benefits from better co-ordination between primary health and hospital services. DHBs’ Crown Funding Agreements with the Minister contain expectations for DHBs’ performance.

3.14
The Ministry has increased its monitoring of DHBs’ performance. The number of primary health indicators has grown from one in 2002/03 to 11 in 2007/08. Some of the primary health indicators have target dates for their achievement. Some of the indicators within the nine health targets for DHBs introduced on 1 July 2007 are influenced by primary health care. The Ministry also requires DHBs to submit written narrative reports about specified community issues affecting primary health care within their districts (such as Māori participation in PHOs).

Independent evaluations commissioned by the Ministry

3.15
The Ministry is managing a portfolio of independent evaluations of the Strategy, focusing on three areas:

  • the Strategy’s implementation;
  • the effect the Strategy has had on the delivery of primary health care services; and
  • changes in the population health.

3.16
The evaluations’ findings (and sometimes interim findings) are reported as they become available, so they can be used to improve the Strategy’s implementation.

3.17
The central part of the evaluation portfolio is the "Evaluation of the Implementation and Intermediate Outcomes of the Primary Health Care Strategy", which began in mid-2003 and is due to be completed in June 2009.

3.18
We discuss in Part 4 the nine published reports and 15 studies yet to be published.

Gaps in collecting information

Monitoring and evaluation framework

3.19
The Ministry’s monitoring and evaluation frameworks are yet to set out a comprehensive set of measures to guide progress for each of the Strategy’s goals. Therefore, it is difficult to assess the Strategy’s progress. However, it is clear from the Ministry’s reports that improvements have occurred in some areas.

3.20
The Ministry told us it did not set measures from the outset (particularly milestones with due dates) because, in the first few years, it wanted to focus on setting up PHOs and phasing in the new funding arrangements. It did not know how quickly PHOs would be formed. Its “stepwise” approach took account of the need to negotiate agreements between many parties – the Ministry, DHBs, PHOs, other providers, and national professional bodies. The Ministry made decisions for the next one or two years based on the status in the current year.

3.21
People reading the Ministry’s many reports must try to match the reported information against the Strategy’s vision statement, directions, actions, and priorities, and try to work out for themselves if progress is satisfactory. The problem then is that each reader judges the information against their own (rather than the Ministry’s) expectations, or is left asking “So what does this mean?”

3.22
For example, the annual consultation rate for people aged 65 years and older increased from 7.2 to 8.8 visits from 2001/02 to 2005/06. While acknowledging the improvement, without an expectation being set it is difficult to know how much more improvement (if any) is needed, and by when, to achieve the Strategy’s goals.

3.23
In our view, the Ministry would benefit from an overall framework that organised its measures to produce a coherent, comprehensive, and useful picture of progress towards the goals inherent in the Strategy’s vision.

Aligned indicators for measuring progress

3.24
The indicators used for parts of the Strategy are not aligned with the indicators used in other primary health programmes. The Ministry has recognised this and reported that:

Work is … underway to get alignment between the PHO Performance Programme indicators descriptions that are also indicators in Get Checked and in the Health Targets. … [A] minimum [data] set for primary health care would assist by … ensuring standard definitions in key areas …5

3.25
It is easier and more efficient to collect, analyse, and report information when the indicators used for measuring the progress of different primary health care programmes are aligned. It means comparisons of the results from different programmes are more valid.

What the Ministry needs to do

3.26
Setting measures is fundamental to assessing progress. It is difficult to assess progress against the Strategy’s vision if achievements are not analysed and reported within the context of expected results.

3.27
The Ministry uses some measures it has about primary health care to report against the New Zealand Health Strategy. These measures and other existing measures could be brought together to report on the Primary Health Care Strategy’s progress. This would also show where there are further information gaps that should be filled, and the Ministry could explain what it intended to do to fill those gaps.

3.28
The Strategy said it might take 10 years before improvements in public health would be realised. There are about three years from the publication of this report until 2011, which will be 10 years after the Strategy’s launch. The Ministry should set measures for what it expects to be achieved by 2011, and beyond 2011 if the Strategy’s goals have not been achieved by then. It should collect and report information about progress against the measures. The Ministry should explain how it will use the information to maintain progress.

3.29
The Ministry needs to set clear measures (as we define them in paragraph 1.9) for the six goals inherent in the Strategy’s vision statement. Once this is done, the information currently collected about PHOs’ and DHBs’ performance may need to be amended. We are aware of the need not to overload the sector with reporting requirements. We expect the measures to be manageable and meaningful.

3.30
In our view, the Ministry could consider using the Joint Work Programme to help set measures for the Strategy. When we wrote this report, the Ministry’s measures and reporting did not cover the outcomes of the Joint Work Programme. We are not suggesting that all 100 outcomes have their own measures. The outcomes represent the most detailed specification of the changes the Ministry and DHBs want to make. Measures could be set for some or all of the Joint Work Programme’s goals, themes, or work streams (see paragraph 3.9).

Recommendation 1
We recommend that the Ministry of Health review its measures to ensure that they are complete and that it can assess progress toward all of the goals in the Primary Health Care Strategy’s vision statement for the next three years.

1: Figueras, J., McKee, M., Lessof, S., Duran, A., and Menabde, A., (2008), Health systems, health and wealth: Assessing the case for investing in health systems, page ix, WHO European Ministerial Conference on Health Systems, World Health Organization, Estonia.

2: Ministry of Health (2006), Clinical Services Directorate Business Plan 2006-07.

3: Feek, C and McKernan, S (2006), Primary Health Care Strategy Implementation Programme 2006-2010: Working Document for Sector Engagement – March 2006, Ministry of Health, Wellington.

4: Feek, C and Clarke, C (2006), Primary Health Care Strategy Implementation Work Programme 2006-2010: The Next Steps, Ministry of Health, Wellington. The publication was withdrawn from the Ministry’s website in September 2007, awaiting revision. A revised programme had not been published as at 16 September 2008.

5: Ministry of Health (2007), Health Report Ref. No.: 20070693: Review of the Primary Health Care Strategy Monitoring Environment, Wellington.

page top