Part 5: Monitoring and evaluating the effectiveness of the Programme

5.1
In this Part, we discuss the Ministry's progress in:

• monitoring the results of the Programme; and
• evaluating the effectiveness of the Programme.

5.2
Most of the Programme's changes to improve the management of sickness and invalids' beneficiaries were implemented by September 2007. The Client Management System (to provide case managers with more support for decision-making) and the new health services were not in place by September 2007. Although the work on these two initiatives was well advanced at the time of our audit, they were yet to be fully implemented.

5.3
The Ministry told us that it thought that it could take up to five years – that is, until 2012 – for the effects of the Programme to be fully realised.

Our overall findings

5.4
The Ministry's assessment of the results and effects of the Programme was limited. For example, there had been significant changes to how case managers managed sickness and invalids' beneficiaries, including providing them with additional specialist health and disability advisors. However, the Ministry had carried out little monitoring of the effect of those changes.

5.5
The Ministry expected the Programme's outcomes to include:

• reducing the number of people coming onto sickness and invalid's benefits;
• increasing the number of people moving off sickness or invalid's benefits and into work; and
• making savings in sickness and invalid's benefit expenditure while continuing to provide social and financial support for people with barriers to work.

5.6
Although the Ministry was closely monitoring changes in sickness and invalid's beneficiary numbers, it was unable to attribute the changes in the numbers to the Programme. The Ministry had identified outcome goals for the Programme that would help it establish this link. However, at the time of our audit, the Ministry had not measured its achievement against those goals.

5.7
The Ministry was not gathering information to enable it to report whether the projected annual savings of $49 million of expenditure on sickness and invalids' benefits, expected by 2010/11, were likely to be achieved.

5.8
The two recommendations we make in this Part focus on the need for the Ministry to better evaluate and report on the Programme's effects on beneficiary numbers, expenditure, and other intended outcomes.

Background

5.9
In approving the wider Programme in December 2006, Cabinet noted that the Ministry would monitor and evaluate the whole Programme (for all types of benefit) to establish:

• how well the initiatives⁶ operated in practice;
• where feasible, the effect the Programme as a whole, and the key initiatives where appropriate, had on outcomes; and
• an evidence base that supported the design and implementation of new services.

5.10
Cabinet directed the Ministry to report back to the Joint Ministers⁷ on the main operational and outcome findings from the research, evaluation, and monitoring of the Programme. The reporting was to occur through:

• bi-annual Ministerial updates beginning September 2007; and
• the Sustainable Employment Annual Reports from February 2008.

5.11
When Cabinet approved the delivery of additional health services in April 2007, it noted that, once fully implemented, the Programme was:

…. estimated by a panel of international experts to generate savings of $49.0 million a year, compared to current forecasts, in Sickness Benefit and Invalid's Benefit expenditure.

5.12
Before Cabinet approved the Programme, the Ministry provided the Cabinet Policy Committee with its Research, Evaluation and Monitoring Plan (the Evaluation Plan), which identified the following outcomes for the Programme:

• improving labour market participation for beneficiaries already in work (including increasing the hours of employment for part-time employees);
• reducing the number of people coming onto a benefit;
• increasing the number of beneficiaries moving into full-time employment, and the speed with which they made that move;
• increasing part-time employment;
• increasingly moving people into sustainable employment;
• meeting working-age benefit population targets;
• making savings in benefit expenditure;
• people continuing to have access to social assistance; and
• people receiving a level of income appropriate to their circumstances.

Monitoring the results of the Programme

Regional staff were monitoring changes in the numbers of people on the sickness benefit and invalid's benefit, and carrying out some analysis of the reasons for the changes. They were not monitoring how well the initiatives were operating in practice, or the extent to which the expected outcomes were achieved.

5.13
Monthly targets for numbers of sickness and invalids' beneficiaries were set at a national level, and for regions and individual service centres. Targets were generally set to reduce the numbers of people on the sickness benefit and stabilise the number of people on the invalid's benefit.

5.14
Ministry staff closely monitored the numbers of sickness and invalids' beneficiaries, and service centres and regions reported regularly to Work and Income's national office, the Ministry, and to the Minister on their achievement against the targets. Service centres and regions also monitored reasons why the targets were not achieved, and prepared strategies to achieve the targets.

5.15
We expected the Ministry to monitor how well the initiatives were operating in practice, and the effectiveness of the changes. Monitoring the numbers of sickness and invalids' beneficiaries provided only a rudimentary measure of how well the initiatives were operating.

5.16
Total numbers do not identify or explain movements between the sickness and invalid's benefits, and they do not show whether people have moved off the benefit into work or for some other reason – for example, going overseas. Also, at a service centre level, numbers go down when beneficiaries move and are assigned to another service centre. Therefore, monitoring numbers does not provide enough information to guide the ongoing development and operation of the Programme.

5.17
Although the Ministry had created new positions to help front-line staff, there had been little monitoring of the extent to which these new people were used or the effect that their work was having. For example, the regional health advisor and regional disability advisor positions were established to help case managers assess eligibility and identify appropriate services for beneficiaries. There was little monitoring of the number of cases referred to the services or the outcome of the referrals.

5.18
In monitoring the effectiveness of the changes, Ministry staff were not collecting data to establish or measure the extent to which the outcomes (see paragraph 5.12) were achieved. Therefore, staff were not able to tell us whether:

• labour market participation for sickness and invalids' beneficiaries was improving;
• there were fewer people coming onto the sickness benefit;
• the number of sickness and invalids' beneficiaries (where appropriate) moving into full-time employment was increasing, and the rate of any such increase;
• the number of sickness and invalids' beneficiaries participating in part-time work, and the amount of part-time work undertaken, was increasing; or
• the anticipated savings in benefit expenditure were being achieved.

5.19
In our view, the Ministry (including regional and service centre staff) needs to better monitor the effect of the changes introduced as part of the Programme. The monitoring needs to occur in the following two areas:

• how well the initiatives are operating in practice – this includes the use and usefulness of the new positions, and the extent to which case managers are carrying out comprehensive case management; and
• how well the Programme is achieving its outcomes.

5.20
The Ministry needs to consider whether the measures are complete and appropriately reflect the progress expected in relation to sickness and invalids' beneficiaries. For example, measuring the number of beneficiaries completing courses or increasing the amount of part-time work that they are doing could also be appropriate measures for sickness and invalids' beneficiaries.

Evaluation of the Programme

The Ministry had carried out only limited evaluation of the effect of the September 2007 changes for sickness and invalids' beneficiaries.

5.21
The Centre for Social Research and Evaluation (CSRE), the unit within the Ministry responsible for social sector policy research and evaluation, prepared an Evaluation Plan that was considered by the Cabinet Policy Committee in late 2006. The unit was responsible for carrying out the research and evaluation set out in the Evaluation Plan and for reporting the results to the Joint Ministers.

5.22
The Evaluation Plan covered the whole programme. It noted that a supplementary Programme Health and Disability Research and Evaluation Plan focusing on the effects of the changes for sickness and invalids' beneficiaries would be prepared by February 2007. The Ministry told us that this plan was not prepared because other work priorities took precedence.

5.23
The Evaluation Plan included some evaluation of the effect of health and disability initiatives for the period 2006/07 to 2009/10. However, the evaluation was very narrow, and focused on identifying the other health services required and evaluating the mental health services, rather than the effectiveness of all of the Programme changes. For example, the effectiveness of the new specialist positions and of comprehensive case management were not discussed in the Evaluation Plan.

Bi-annual evaluations

5.24
In keeping with the December 2006 Cabinet decision (see paragraph 5.10), the Ministry provided two bi-annual evaluation briefings to Joint Ministers – the first in November 2007 and the second in June 2008.

5.25
The bi-annual reports provided little evaluation of the effectiveness of the Programme's changes for sickness and invalids' beneficiaries. The reports were about the Programme as a whole, and the November 2007 report specifically excluded the changes from the scope of that evaluation.

5.26
The second report (for the period from November 2007 to 31 March 2008) included some analysis of the implementation of the Programme's changes. For example, the report noted the proportion of sickness and invalids' beneficiaries assessed using the new medical certificate, and commented on progress in establishing mild to moderate mental health services contracts (which was slower than anticipated because they were in place in only four regions, rather than the planned 11 regions). The report also noted the number of referrals to the regional health advisors and regional disability advisors, but did not look at the results of the referrals or the effect they had. The report identified growth in the numbers of sickness and invalids' beneficiaries participating in work training and employment-related services, but did not explain why that growth had occurred. We consider that the report provided very limited information on the effect of the Programme on intended outcomes for sickness and invalids' beneficiaries.

5.27
The Ministry started to prepare a third process evaluation report for the Joint Ministers that focused on the Programme's changes for sickness and invalids' beneficiaries. The objective of the process evaluation was to understand how the new engagement and planning requirements, medical assessments, information collection processes, and specialist health and disability roles were operating in practice. The report was neither finalised nor given to the Joint Ministers because of changing priorities after the 2008 change of government.

Achieving the savings target

5.28
In April 2007, the Ministry had quoted estimated annual savings in sickness benefit and invalid's benefit expenditure of $49 million by 2010/11, compared with forecasts current at that time. The estimate was based on realising all the benefits from the Programme's changes, and was subject to a large number of assumptions, including the availability of health services purchased through the Innovation Fund. The Ministry did not indicate what this fiscal saving would represent in terms of a reduction in beneficiary numbers.

5.29
The Ministry was not formally required to report against its savings estimate until 2010/11. However, at the time of our audit in 2008, the Ministry was not monitoring progress towards achieving the expected savings. It was not in a position to make a preliminary estimate of how likely the expected savings were to be achieved.

5.30
We note that funding for the additional health services has since been withdrawn, and the Ministry will need to take this into account when assessing the level of annual savings.

Achieving the numbers targets

5.31
In March 2007, the Ministry noted in a report to the Government that it expected the Programme to have a significant effect in reducing the number of people receiving a sickness benefit or invalid's benefit, by moving them into work (where appropriate).

5.32
Between September 2007, when the Programme was extended to sickness and invalids' beneficiaries, and December 2008, the combined number of reported sickness and invalids' beneficiaries increased by 5.6% (see Figure 6). The number of invalids' beneficiaries increased by 6.7% and the number of sickness beneficiaries increased by 3.9%.

Figure 6
Changes in beneficiary numbers between September 2007 and December 2008

5.33
The number of invalids' beneficiaries increased from 78,268 in September 2007 to 80,082 in December 2007, with a minimal increase in sickness beneficiary numbers in this same quarter. The Ministry's economic and fiscal updates for 2008 partly attributed increases in the number of beneficiaries transferring from the sickness benefit to the invalid's benefit to the changes introduced as part of the Programme in September 2007. In particular:

• using the new medical certificate, case managers became responsible for determining whether an applicant would receive a sickness or invalid's benefit. (This was previously decided by medical practitioners for the sickness benefit, and designated doctors for the invalid's benefit.) The analysis in the updates noted that some case managers might have incorrectly assumed that the beneficiary qualified for the invalid's benefit if the two-year reassessment period was ticked by the medical practitioner; and
• medical practitioners completing the medical certificate were not clearly distinguishing between the likely duration of the illness or disability and its effect on the beneficiary's ability to work.

5.34
Overall, we consider that the lack of adequate and ongoing monitoring means that the Ministry does not know whether the Programme is achieving the outcomes intended. The Ministry is unable to make well-informed adjustments to the Programme to achieve those intended outcomes.