District health boards: Quality annual reports

1 Introduction

This paper supplements District health boards: Learning from 2010/2013 Statements of Intent,¹ which we published in February 2011. The underlying principles in that paper remain and are being used to improve public reporting by district health boards (DHBs).

A DHB’s performance framework (including performance measures and targets) is set out in the Statement of Intent (SOI) components of its annual plan. This is the basis for the DHB’s annual report about the services it provides and the effect of those services on the community.

In this paper, we discuss characteristics of annual reports that we consider are necessary for good accountability.

Our intention is to help DHBs to consider whether their annual reports present complete and meaningful performance information about their services and about the impacts of those services on the health of people in their community.²

It is timely to focus on achieving good quality performance reporting in DHBs’ annual reports. For the 2012/13 annual reports, auditors will examine closely whether the performance framework (primarily outcomes, impacts, and outputs) is meaningful, and whether the Statement of Service Performance (SSP) fairly reflects actual service performance for the year.

Fundamentally, annual reports should show whether DHBs spent their money and delivered services as they had planned at the start of the financial year. They should also be clear about whether the right mix and level of services is being delivered – including an assessment of the impacts of their services and their progress towards desired outcomes.

The annual reports should show how well services (outputs) are delivered. A balanced, rounded picture of service delivery covers various dimensions of performance, such as the quantity, timeliness, and quality of service delivery, and discloses financial information for each group of services. Appropriate performance information is relevant, reliable, and understandable. It can be compared over time, and compared with the performance of similar entities.

Our assessment of the 2010/11 annual reports

The following symbols indicate how we consider the 2010/11 annual reports rate on each aspect. The purpose is to show DHBs at a glance the aspects that they could well focus on.

2 Describing performance

An informative overview

An overview gives the opportunity to highlight particular aspects of performance, key performance achievements, and where the DHB did not perform that well during the year. It can also highlight new initiatives or different approaches planned for future years. An informative overview could, for example, discuss:

• What is the DHB’s high-level assessment of its performance during the year (service delivery) and its progress towards meeting its strategic objectives and priorities (including as formally captured in its outcome and impact statements)?
• What are the strategic-level implications of this analysis?

Progress towards achieving desired changes in community health

DHBs should assess and report on the impacts of their services and/or the contribution of their services and other activities towards achieving better health for their community. In effect, this shows whether the DHB is making progress towards medium-term goals.

The annual report should specifically identify the outcomes and/or impacts that the DHB was aiming to contribute to or influence – as set out in its SOI. Reporting progress on these should be based on the “main” measures of future performance identified in the SOI – that is, those measures with targets for the full three years of the SOI, and typically relating to health impacts and/or outcomes.³ The annual report should clearly identify which of the reported measures are the “main” measures.

Where the DHB’s performance against outcome and/or impact measures is not re-assessed every year (for example, because of cost or the availability of periodic data), the results of the most recent measurement should be disclosed.

Longer-term trend analysis and evaluation are particularly important where improvements are likely to be evident only over time, such as smoking rates and the incidence of cancer. Useful evaluation is likely to include a discussion about how service delivery has affected the results, including any significant changes
planned to services as a consequence of the results.

Graphs are useful for showing trends in performance.

As well as identifying and reporting on their main measures of non-financial performance, DHBs must also report on their main measures of financial performance.4 DHBs usually discuss whether they have broken even or the amount of their deficit.

Statement of service performance

Describing output classes and outputs

Output classes group similar services together. Although there is a standard set of output classes for DHBs, it is still important to describe each output class in the annual report.

Describing the outputs within the output class also makes the service performance information clearer.

Revenue and expenditure information

Section 142(2)(b) of the Crown Entities Act 2004 requires revenue and expenditure information for each output class. This provides an indication of the relative expenditure across the output classes, and whether there is under- or over-expenditure in certain areas.

DHBs often discuss the need for financial sustainability and various initiatives (for example, related to productivity and changes in emphasis in service delivery). This more-detailed analysis of the financial information, showing the relative costs of significant services where transformational changes are planned over time, increases the value of reporting.

3 Reporting on service delivery

Not assessed.

All measures and targets from the SOI are reported against

The service performance measures relate to the first year of the SOI and are set out in the forecast Statement of Service Performance (SSP). All measures and targets identified in the forecast SSP must be reported against in the SSP in the annual report. Where DHBs cannot report against measures, they should clearly state the reason why, and explain how they plan to remedy this.

Although not under the DHB’s control, demand-driven data can provide useful context – for example, by indicating the scale of services being delivered. For these measures, DHBs should:

• disclose that the performance results are a reflection of demand;
• where the level of demand differed significantly from that forecast, be clear about any significant consequences of that – for example, on the timeliness or quality of services; and
• be clear about how the DHB is trying to influence demand, where this is the case, such as demand for Emergency Department services.

Clearly presenting targets and results

Clear presentation of targets and results is essential to allow readers to compare actual performance with planned performance. DHBs should comment on variances in performance.

As with reporting on outcome and/or impact measures, graphs are useful for showing trends in performance.

Understanding and explaining the results in the SSP

As a minimum, DHBs need to comment on measures where targets were not achieved. There will also often be other instances where it is beneficial to comment on the results rather than leaving readers to make their own interpretation.

Interpreting the performance results and communicating the wider performance story can help readers better appreciate, for example:

• what the measure, or groups of measures, demonstrate about performance;
• the underlying activities that affected the result; and
• the implications for the service in future years (for example, whether new initiatives or approaches are now planned as a result of the performance).

The commentary could be at the level of individual measures, outputs, or output classes – whatever helps the reader to make an informed assessment of the DHB’s performance and services. The performance information should be concise and balanced, and emphasise matters according to their significance.

The reader should be able to clearly see the relationship between relevant narrative in the front sections of the annual report and the performance information in the SSP.

Providing a well-rounded picture of performance

The forecast SSP should have outlined the significant services in each output class and presented the core performance measures that the DHB sees as the most important and likely to most meaningfully represent output delivery. The measures should have addressed a reasonable range of dimensions of performance (such as quality, quantity, and timeliness). They should also have been linked to impacts/outcomes, with associated performance measures.

A robust performance framework, with appropriate performance measures in place, helps the DHB to understand its results. The DHB also needs to understand and explain any gaps or relevant changes in direction or services that may have emerged during the year.

To ensure that the annual report fairly reflects actual performance, DHBs might need to expand their narrative commentary or consider including supplementary performance measures. Where supplementary measures are used, or existing measures are improved, we would expect the DHB to explain what it has done and why.

4 Providing comparative information

Baseline and trend data provided to allow comparisons over time

Providing baseline data from the previous year(s) is important contextual information about the DHB’s performance.

A minimum of three years of baseline data is likely to be appropriate for outcome and impact performance measures – where change will often be visible only during the medium to longer term.

Benchmarking

Benchmarking is useful for showing how a DHB has performed compared to other DHBs and/or compared to best practice in a particular aspect of its performance. It can also give the DHB the chance to highlight its own priorities and particular circumstances.

DHBs have to report on common sets of performance measures to meet other requirements – such as the national health targets and measures specified by the Ministry of Health.5 Benchmarking information should be readily available where such measures are also included in DHBs’ SSPs.

International benchmarking

DHBs that use international benchmarking information as part of their performance monitoring could report this information where it would add value.

5 Reliability

All performance measures need to be supported internally by detailed definitions and appropriate systems to collect the relevant data. This should not be a significant issue if DHBs base their external performance reports on the information that is used to guide their internal management.

1: Available on www.oag.govt.nz

2: There are additional matters required in the annual reports, and they are set out in section 151 of the Crown Entities Act 2004.

3: This is required by section 141(1)(f) of the Crown Entities Act 2004.

4: This is required by section 141(1)(f) of the Crown Entities Act 2004.

5: Ministry of Health, DHB non-financial monitoring framework and performance measures, Wellington.