Part 7: Service performance information of district health boards – the accountability framework

Central government: Results of the 2008/09 audits.

7.1
In this Part, we report our conclusions from our review of the service performance information and associated systems and controls for the 21 district health boards (DHBs). We carried out our review as part of our 2008/09 annual audits. In drawing our conclusions, we acknowledge the extensive work that the sector is doing on accountability documents and the accountability framework, and recognise that the DHB sector has work in place to address weaknesses in its service performance reporting.

7.2
We also summarise the DHB sector's planning and accountability framework and consider whether the complexities of the framework, and other underlying issues, may be contributing to the low quality of the DHBs' service performance information, and the statements of intent (SOIs) in particular.

7.3
We make three recommendations for helping to improve the accountability framework that DHBs must operate under, which we would expect to result in improved accountability reporting within the general purpose accountability documents (that is, the SOI and the annual report).

Audit work during 2008/09

7.4
In the 2008/09 audit of each DHB, we reviewed the SOI for 2009-12. The SOI includes the statement of forecast service performance (forecast SSP), which sets out the performance measures and targets to be reported in the year-end Statement of Service Performance (the SSP), which is subject to our audit opinion.

7.5
This year, and for the first time, we issued a grade for entities' service performance information and associated systems and controls. The grade is based on our review of the reporting of prospective information in the 2009-12 SOI and supporting systems and controls, and of service performance information reported in the 2008/09 SSP. We explain our reporting and associated grading system in Part 2 of this report. To help ensure consistency in the grading of SOIs, we also reviewed our auditors' findings, conclusions, and proposed grade. We reported our conclusions about the individual DHBs' service performance information to the DHB's Board and management, the Minister of Health (the Minister), and to Parliament.

7.6
The results reported below apply to the documents reviewed at the time of the 2008/09 audit. Since that time, we are aware of further work by both the Ministry and the DHBs to improve DHB accountability documents. This includes revised guidance from the Ministry for the DHB SOIs and District Annual Plans (DAPs), a review of the DHB performance measurement framework to reduce the reporting burden, more focus on Government priorities, and the advent of new requirements for Performance Improvement Action, for all Crown entities.

7.7
It remains to be seen how successful these will be in addressing the following issues. We note, however, that revised guidance and reduced amount of reporting will be effective only to the extent that the resulting reports give a meaningful and complete picture of DHB service performance.

Poor grades for district health boards

7.8
We graded all DHBs' service performance information and associated systems and controls as "poor/needs improvement", at the lower end of our assessment scale. While the presentational quality varied considerably, a consistent feature of the SOIs was that they did not identify clearly or comprehensively the DHBs' services, and the quality of the reported performance measures was poor. In our view, DHBs' 2009-12 SOIs are not yet as well developed as similar accountability documents in other parts of the public sector.

7.9
We took into account the following factors, evident throughout the health sector, in giving a grade of "poor/needs improvement":

  • a failure to distinguish between the various elements of performance;
  • poor specification of outcomes;
  • a lack of main measures for outcomes;
  • poor specification of outputs;
  • a lack of coverage of the range of DHB services beyond those focused on in the national health priorities; and
  • a lack of direct measures of service delivery, especially the (almost total) lack of service quality measures.

What good performance reporting is supposed to achieve

7.10
To meet the expectations set out in the Crown Entities Act 2004, generally accepted accounting practice, and guidance from the Treasury and the State Services Commission, SOIs and annual reports (including their forecast and historical SSPs) should present a clear, logical, integrated, and cohesive performance story. The reports should exhibit the qualities of relevance, reliability, comparability, and understandability.

7.11
The elements of performance and their associated measures should be presented in an informative context. The links between outputs, impacts, and outcomes (the "intervention logic") should be clear and understandable, and their relevance to the DHB's strategic priorities should also be apparent. Performance reports produced to a satisfactory standard should provide a basis for assessing the DHB's efficiency (of service delivery) and effectiveness (in achieving intended outcomes). The link between financial information and good quality non-financial performance information should provide a basis for assessing cost-effectiveness.

7.12
However, we have concluded that the quality of SOIs for the DHB sector in these respects is poor and has shown insufficient improvement in recent years, despite quite extensive work throughout the sector to improve the quality of SOIs. It is as yet too soon to see if the more recent initiatives in the sector and by the National Health Board business unit in the Ministry of Health will make a difference to the quality of non-financial performance information.

7.13
Reported performance objectives are often vague and disconnected. We do not see a unified performance story within the accountability documents that:

  • plans the DHB's outputs and makes it clear what impacts and ultimately outcomes they are expected to achieve;
  • reconciles long-term strategic priorities, the Minister's priorities, national health priorities, and local communities' specific health profiles, needs assessments, and initiatives; and
  • identifies clearly and unambiguously the services the DHB is accountable for, including services delivered on its behalf by other providers.

The risks in reporting performance information poorly

7.14
There are risks associated with poor reporting of performance information. DHBs need to consider the wider implications for business planning and management of having effective and efficient service performance reporting. The risks to the sector are not confined to getting poor grades, receiving an adverse audit opinion, or providing misleading information about its activities.

7.15
In our view, good accountability processes should be an integral part of how an entity manages its day-to-day business. The low quality of SOIs may reflect underlying problems with the sector's accountability framework. It may also reflect poor integration of the DHB's business planning and management processes with accountability reporting, an important risk to effective and efficient management and to good accountability.

7.16
Considering these risks, we have revised our Standard for issuing audit opinions on SSPs.1 We intend to apply the revised Standard to audits of DHBs during 2012/13.

7.17
We recognise the work that DHBs are doing both individually and collegially, assisted by the monitoring department (the Ministry of Health) and the central agencies (the Treasury and the State Services Commission) to improve the quality of SOIs. DHBs have sought advice from their auditors and from this Office, too. However, in our view, the quality of SOIs is also affected by the complexity of the DHB accountability framework. DHBs are required to produce more accountability documents than other Crown entities – notably a district strategic plan (DSP), district annual plan (DAP), and an SOI. An intention of the Crown Entities Act 2004 was that Crown entities would produce good quality general purpose information for Parliament and the public. In our view, in their current form none of these DHB accountability documents (individually or collectively) fulfil this intention.

DHBs' planning and accountability framework

7.18
DHBs are required to prepare a larger number of planning and prospective performance reports, as part of their accountability framework, than do other Crown entities:

  • Each DHB is responsible for identifying and providing for its district's health needs and is required to prepare district strategic and annual plans (signed off by the Minister and the Ministry of Health) that are consistent with New Zealand's health and disability strategies. Both documents are quite detailed.
  • Each year, the DHB also publishes an SOI, a higher-level, less detailed statement about how the DHB intends to address the health needs of its district.
  • Actual performance achieved against the performance measures and forecast targets in the SOI are reported in the annual report, which includes the SSP upon which the auditor must issue an audit opinion.

District Strategic Plan

7.19
The DSP is required by section 38 of the New Zealand Public Health and Disability Act 2000 (the Act) and must be made available to the public. It is a medium-term plan (covering five to ten years) that outlines how the DHB plans to fulfil its objectives and functions. It must be reviewed at least every three years.

7.20
As well as expressing the DHB's strategy for meeting the local health needs of its district, the DSP must also reflect the overall direction of health and disability strategies. Section 38(3)(a) of the Act requires that, in developing or amending the DSP, DHBs must:

  • assess the health status of their respective populations;
  • identify any factors that may adversely affect this status;
  • assess the population's needs for health services; and
  • determine the contributions those services are intended to make towards the health outcomes sought.

7.21
The approach required for preparing a DSP reflects the usual reporting framework in the public sector and is compatible with the performance reporting framework required by the Crown Entities Act 2004.

District Annual Plan

7.22
The district annual plan (DAP) is also a public accountability document. It must not be inconsistent with the DSP. It must include the intended outputs of the DHB for the forthcoming year (together with the amount of any capital expenditure required) and how they relate to the DSP, as required by section 39 of the Act. It is the basis for the DHB's funding agreement with the Crown. The DHB's SOI and Crown Funding Agreement for the year must be attached to the DAP. The DAP does not have to duplicate any output information that is in the SOI.2

Statement of Intent

7.23
Section 42 of the Act requires DHBs to prepare performance reports in accordance with Part 4 of the Crown Entities Act 2004. Sections 141 and 142 of the Crown Entities Act prescribe the content of the SOI, and Sections 151 and 153 prescribe the contents of the annual report, including the SSP.

7.24
Crown entities, including DHBs, are required to prepare an SOI to cover the medium term, namely the forthcoming financial year and at least the two following years. The SOI's purpose is to promote public accountability and, as such, it must include the following performance information:

  • background information about the DHB, its operating environment, and the nature and scope of its functions and intended operations;
  • the specific impacts, outcomes, or objectives that it seeks to achieve or contribute to, including how they might relate to government policy directions;
  • how the DHB intends to perform its functions and conduct its operations;
  • how the DHB proposes to manage its organisational health and capability; and
  • the main financial and non-financial measures by which the future performance of the DHB may be judged.

7.25
Extra information required for the first financial year of the SOI includes a forecast SSP. The forecast SSP must:

  • describe the classes of outputs the DHB proposes to supply;
  • provide measures and forecast targets for output delivery;
  • identify the expected revenue and expenses for each class of outputs; and
  • comply with generally accepted accounting practice.

7.26
The measures and forecast targets provided in the forecast SSP are to be reported against in the end-of-year SSP, and audited.

Annual report

7.27
The annual report must include an SSP. In essence, the SSP must:

  • be prepared in accordance with generally accepted accounting practice;
  • describe each class of outputs supplied during the year;
  • include, for each class of outputs, the standard of service delivery achieved compared with the forecast targets set out in the forecast SSP; and
  • include, for each class of outputs, the revenue earned and output expenses incurred compared with the forecast expected levels set out in the forecast SSP.

7.28
Section 156 of the Crown Entities Act 2004 requires the Auditor-General to audit the SSPs of Crown entities.

The Ministry of Health's review of accountability documents

7.29
The Ministry of Health reviewed DHB accountability arrangements in 2007. The review sought to reduce compliance activity and to sharpen focus on performance improvement. In addition, the review sought to:

  • assign appropriate responsibility for planning and decision-making to a mature DHB sector by minimising prescriptive compliance requirements;
  • simplify accountability processes and documents, clarifying the linkages between documents and ensuring that only meaningful and relevant information is collected; and
  • sharpen the focus of the DHB accountability arrangements on improving performance and on the reporting of performance in a vital few areas.3

7.30
In October 2007, the Ministry produced its report, Accountability: A better way of working together (Report of the DHB Accountability Arrangements Review). The report signals the need for change, observing that, "The DHB sector has now matured and it is time to shift the focus from capacity building to improving performance". It comments that the accountability arrangements need updating to reflect this shift and that the changes envisaged "aim to give stakeholders a clear picture of DHB and system performance overall".

7.31
In the report's summary of major changes, it lists several changes to annual accountability and planning arrangements. The changes focus on the DAP, not the SOI, although a subsidiary comment on the DAP mentions further consideration to be given to moving to a single, annual accountability document (instead of producing both a DAP and an SOI).

7.32
Although the report makes passing reference to the SOI, its focus is the DSP and the DAP. References to the SOI are to the high-level outcomes and objectives, with no reference to the output-oriented forecast SSP, which is a critical component of the SOI. Indeed, the report appears not to acknowledge the role of the SOI in output reporting, seeing this as the exclusive domain of the DAP.

7.33
Neither is reference made in the report to the role of the SSP within the annual report, which constitutes the independently verified, historical, and public account of a DHB's performance.

7.34
It appears to us that, at the time of writing the report, the Ministry of Health did not focus on the SOI or the annual report as primary accountability documents to Parliament. Its primary focus was the DAP. This is not surprising. The DAP is the funding and contractual management mechanism between the Ministry of Health and DHBs and, in the Ministry's view, DHB boards focus on the DAP as a key means for effective and efficient governance. The Ministry's 2007 report gives little consideration to the extent to which DHB accountability requirements are complementary and how they interact to minimise effort while providing an appropriate basis for funding, performance, and accountability needs.

7.35
Two years on, there have been several recent and major changes within the health sector. In response to the recommendations of the Ministerial Review Group at the end of July 2009, the Government has established the National Health Board within the Ministry (see Part 6, paragraphs 6.12-6.13). We have been told that, during the next 12 months, the National Health Board will review the DHB planning, reporting, and monitoring framework and activities.

7.36
The purpose of this review is to move towards a more integrated process that links planning, accountability, and performance management in the DHB sector. We have also been told that:

  • changes have already been made, and the Ministry has signalled legislative changes to support effective performance accountability of DHBs within changing national, regional, and local responsibilities; and
  • the Ministry acknowledges that the SOI and the DAP should be seen as complementary documents, both of which matter under the current accountability arrangements.

7.37
We outline below the issues that any such changes will, in our view, need to address.

Issues with the accountability framework for district health boards

7.38
We are concerned at indications that the poor quality of the external performance reports may reflect more serious, underlying problems with the health sector accountability framework and, possibly, the planning and management arrangements for DHBs' services.

7.39
In our view, there are two major factors in the poor quality of reporting in SOIs:

  • the large number of accountability documents (which we have discussed in detail above); and
  • the diminished status of the SOI in comparison with the DAP.

Primacy of the District Annual Plan for funding purposes

7.40
Several DHBs have described the SOI to us as an "add-on" to the suite of accountability documents, relegated in importance, and produced as a compliance exercise instead of as a document at the centre of the DHBs' strategic and operational performance management system. This appears to be reflected in the lack of emphasis given to the SOI (and in particular the lack of mention of the forecast SSP) in the Ministry of Health's Accountability report.

7.41
The Crown Funding Agreement is the funding mechanism for DHBs. It is based on the DAP and, to that extent, the DAP is of greater importance to DHBs than the SOI. Because it contains the information by which Crown funding is determined, it tends to be a fairly detailed document. The DAP generally contains a greater volume of information, more extensive level of detail, and greater use of technical terms and specifications than we would expect to see in an SOI. A DAP will typically contain volume and case-weight information for various clinical procedures as well as an extensive list of objectives relating to inputs, capability and relationship building, processes, reporting, and other administrative tasks. It also sets out output and outcome information. It is intended to be primarily an "internal" document, used as a management tool for DHBs and to provide accountability to the Minister.

7.42
By contrast, the SOI is an external general-purpose accountability report. In practice, it is prepared primarily for Parliament and usually derived from the DAP, with little acknowledgement that it is the primary prospective performance report for external users, including both Parliament and the public. While the sector appears to acknowledge its parliamentary audience, it is less obvious that SOIs also need to be prepared with the public in mind.

7.43
Consequently, there is little incentive to ensure that the SOI has the quantity and quality of performance information required to achieve its legislative purpose.

7.44
Yet the DAP is unable to fulfil the important accountability role intended for the SOI. There is no expectation that the DAP will be reported against – there is no statutory requirement to report historical (that is, actual) performance against the DAP. By contrast, the SOI contains the forecast SSP – the blueprint for the SSP, which is intended to be the historical and public account of a DHB's performance, verified by the audit opinion. For this reason, the SOI performs the primary function of laying out the performance expectations of DHBs, against which actual performance is formally reported and verified.

7.45
We consider that the information currently included by DHBs in the SOI (or the DAP), taken either individually or when read together, would fail to meet the requirements of the Crown Entities Act 2004 and generally accepted accounting practice to provide information necessary to assess the DHBs' service performance. This is because that information does not, generally speaking, cover all the services that the DHB is responsible for, and because the quality of reporting of outputs and associated performance measures is poor. Consequently, the SSP based on the SOI would also fail to meet the expected standard of reporting.

7.46
In our view, attention needs to be given to the health sector accountability framework as a whole, and in particular to:

  • the integration of business planning, management, and reporting systems;
  • meeting a variety of information needs in the most effective way; and
  • accountability beyond the individual DHB, on a regional and national basis.

Accountability information as part of integrated management of health services

7.47
The lack of integration of the planning and accountability documents leads to further questions about the extent, or lack of, integration in the business planning process itself:

  • How well are the planning and management arrangements integrated throughout each DHB?
  • How well are the planning and accountability documents integrated with what is actually being planned and managed within DHBs? Do the documents reflect how DHBs manage their business?

7.48
To be effective, and to avoid compliance burden, service performance information needs to be systematically integrated with the DHB's business management processes.

7.49
It has been suggested to us from within the DHB sector that integrating service performance planning and management with financial planning is a challenge. This challenge, together with the vague, incomplete, and disjointed performance stories in the SOIs, suggests that planning and accountability documents may not be fully reflecting the breadth of service delivery or a sufficient range of the key performance indicators that are integral to successfully managing service delivery.

7.50
Misalignment and lack of integration of planning documents is likely to result in waste. If the systems for producing non-financial performance information are not integrated and streamlined, then there is likely to be unnecessary duplication of effort. In our view, consideration needs to be given to:

  • reviewing the range of planning and accountability documents and, where necessary, to streamlining the reporting requirements; and
  • integrating the systems that capture, collate, and report non-financial performance information.

7.51
Ideally, an integrated system would capture all of the financial and non-financial information necessary for management's day-to-day running of DHBs, and customised internal and external performance reports (including DAPs, SOIs, and annual reports) would be produced from that common system.

Responding to the need for a variety of accountability information

7.52
The recent publishing in daily newspapers of the performance of the 21 DHBs against the six national Health Targets, and of data on sentinel events, is evidence there is public interest in, and demand for, certain performance information, not necessarily in the form of annual reports.

7.53
DHBs use means other than the required accountability documents, including community newspapers and their websites, to tell their public of progress in certain health areas that are of particular public interest.

7.54
The Ministry of Health and DHBs need to identify the most effective and appropriate way to report to their various audiences, including and beyond the current public accountability documents, such as the SOI and annual report. Different reports will be needed for different purposes:

  • special-purpose, internal reports for DHB management monitoring and decision making;
  • general or special-purpose reports from DHBs to the Ministry;
  • general-purpose, annual external accountability reports to Parliament and the public; and
  • special-purpose reports to the public, when timely.

Accountability beyond the single DHB

7.55
Public sector accountability arrangements, to a large extent, assume individual entity accountability. This raises some issues in the health sector, where 21 DHBs have been set up to cover the whole country, with funding based on population numbers, but with some services provided across DHB boundaries, by region, and nationally. Variability in demand, and scarcity and uneven distribution of health personnel, add to the complexity of service delivery and increase cross-boundary accountabilities. Various funding and management mechanisms are applied to overcome service delivery issues. However, the accountability requirements continue to assume individual DHB responsibility.

7.56
Responses to the recent report of the Ministerial Review Group will put greater pressure on cross-boundary accountability issues, by requiring more regionalisation of (and in some cases, national) planning and service delivery. The Ministerial Review Group acknowledged that this would require changes to governance and support arrangements.4 In our view, it also requires changes to accountability arrangements.

7.57
There is also scope for cross-DHB measures and standards to allow for a wider view to be taken, of the health system as a whole as well as DHB by DHB. The commonality of services among DHBs provides scope for significant common performance measures to be used, to enable such system-wide scrutiny. We are aware that the Ministry has done much work in this area. Agreement on commonality of measures is yet to be reached in many areas, something that should, in our view, be progressed as soon as reasonably possible.

Our recommendations

Recommendation 1

7.58
We recommend that the Ministry of Health review, in consultation with the district health boards, the range of planning and accountability documents in the health sector to:

  • identify options for statutory reporting requirements that are more effective and efficient;
  • incorporate within the accountability framework provision for:
    • better accountability for services that are planned and delivered across district health board boundaries; and
    • clarifying the target audience for various forms of reports and then more appropriately meeting the needs of the users of those reports.

Recommendation 2

7.59
We recommend that district health boards, working with the Ministry of Health:

  • investigate and assess the extent to which non-financial performance systems (for planning, managing, and reporting service performance and its impacts) are integrated, and the extent of duplication or other inefficient use of resources; and
  • improve the effectiveness and efficiency of individual district health boards' performance planning, management, and reporting systems if these are not well-integrated or are inefficient, having regard to any requirements by the Ministry to streamline reporting requirements (as described in Recommendation 1).

Recommendation 3

7.60
We recommend that the district health board sector and the Ministry of Health continue their efforts to improve external performance reporting in the sector, including:

  • adopting performance measures and standards throughout the sector that better enable an assessment of the performance of the health system; and
  • better identifying, distinguishing, and linking the health services delivered (outputs) and the impact of those services on health status.

1: AG-4 (Revised): The Audit of Service Performance Reports, 2009.

2: Section 39(2)(b) of the Act exempts from inclusion in the DAP any output information (and proposed funding) included in the SOI for that year.

3: Ministry of Health (December 2008), 2009/10 DAP Guidelines: Questions and answers.

4: Meeting the Challenge: enhancing sustainability and the patient and consumer experience within the current legislative framework for health and disability services in New Zealand, Report of the Ministerial Review Group, 31 July 2009 ("the Ministerial Review Group report"), page 5.

page top