Part 5: Measuring early results and learning from them

5.1
In this Part, we discuss:

• the analysis of results from the case management approach trials;
• some findings from current performance data;
• some concerns about isolating the effect of work-focused case management; and
• whether the Ministry has good ways to measure results and can learn from them.

Summary of our findings

5.2
Early results from the trials of intensive case management services are mixed. The results for sole parents are good. Most service centres showed at least a small improvement in outcomes for clients from work-focused case management, but only a few service centres can show that the improvement is statistically significant.

5.3
The Ministry is doing well to help people to find work, and is meeting its targets for the time those people stay in work.

5.4
The Ministry is finding it hard to meet its performance standard for client engagement. Clients missing appointments are a major contributing factor, and the Ministry needs to act to improve appointment attendance rates.

5.5
The Ministry has a robust method for evaluating the effect of work-focused case management. However, it has not yet looked in detail at how operational and practice variations between sites affect results.

5.6
Mostly, the Ministry knows about the reasons for the mixed results and has reasonably advanced plans to do something about most of them. The Ministry has plenty of good information to act on, and it is getting better at using that information consistently.

Analysing the results of the case management approach trials

The Ministry’s analysis shows that both work-search support and work-focused case management can make a difference, in the right circumstances.

5.7
In February 2014, the Ministry analysed the results of trials of work-focused case management and work-search support. The trials took place between October 2012 and July 2013. Case management includes a one-to-one relationship with a case manager as well as employment services, such as employment subsidies.

5.8
This analysis produced useful information about the cost-effectiveness of each approach and the client groups for whom that approach is most effective. The assessment included an analysis of the return on investment for each approach and a detailed analysis based on “amenability scores” of different groups. Using this information should enable the Ministry to target its resources towards the client groups that will get the best results.

What the analysis showed

Key fact
Early results showed sole parents succeeded in getting part-time work more quickly.

5.9
Early results showed that work-focused case management was effective in supporting sole parents with part-time work obligations, achieving an average reduction of 13 days in benefit duration after 44 weeks in the service. Consequently, the Ministry added 18,000 sole-parent clients to the work-focused case management stream.

5.10
After 32 weeks, the main effect of both work-focused case management and work-search support was that clients’ time receiving a benefit was reduced. Compared to general case management, clients in the work-focused case management stream were receiving a benefit for a shorter duration – 13 days, or 5.3%, less.²³ The effect of work-search support was less, at nine days.²⁴ Neither service reduced the likelihood of clients returning to a benefit.

5.11
Work-focused case management continued to increase the likelihood of clients no longer receiving a benefit, up to the end of the analysis period. However, the work-search support service made no difference once clients had been in that service for more than 35 weeks.

5.12
The analysis also concluded that:

• Both work-focused case management and work-search support are cost-effective interventions. Work-search support “breaks even” (the point at which savings become equal to the extra cost of that form of case management) after 24 weeks, and work-focused case management breaks even after more than a year.²⁵
• The Ministry expects work-focused case management to break even after one to one-and-a-half years.
• Work-focused case management and work-search support are more effective for some clients than others, based on amenability scoring.
• The break-even period for both is shorter than for other employment services and programmes.
• Work-focused case management was also effective in supporting sole parents with part-time work obligations, achieving an average of 13 fewer days on the benefit.

5.13
The Ministry is using the results of the analysis to refine its streaming rules – for example, by increasing the number of sole parents allocated to a work-focused case manager.

The analysis has limits

5.14
The Ministry recognised that there were limits to the analysis. In particular:

• The pilot sites might not be representative.
• The analysis compared work-focused case management and support to general case management, rather than to the model it replaced.
• The reduced time on benefit might not translate to an increase in employment.
• The analysis did not examine whether resources invested in these more intensive approaches came at the expense of other clients.
• The analysis of costs and benefits did not include broader fiscal and social costs.

5.15
When we started our audit, the February 2014 report was the only analysis available, although another was anticipated towards the end of the year. The Ministry is incorporating evaluation into its business processes, but it is taking some time to turn the evidence into reports.

Supporting people after they move into work

The Ministry is meeting the targets it has agreed with the Government for the time former clients stay in work after leaving Jobseeker Support.

5.16
The Ministry produces much performance data. Its performance framework has 65 targets spread between five results areas. We looked at how the Ministry is supporting people to move off a benefit or not to have to claim a benefit in the first place. In 2013/14, there was a slight decline in the proportion of clients who get work before requiring a benefit²⁶ (from 39.7% in the previous year to 37%).

5.17
The number of people who had been claiming a job-seekers benefit continuously for more than 12 months fell from 74,599 at June 2013 to 67,531 at June 2014 (see Figure 8).

5.18
We also looked at the Ministry’s performance in “Improving Employment Outcomes – Service Provision”. The Ministry established a base line for some new performance standards in 2013/14. The new standards include the proportion of people who participate in work-focused interventions and other measures relating to the time former clients remain off a benefit after getting a job.

5.19
At the end of June 2014, the Ministry had met or exceeded its performance standard in six out of nine areas and only narrowly missed the other three.

5.20
During our audit, we asked staff about how they supported people through the transition from receiving a benefit to working. People new or newly returning to the workforce can often face early barriers to staying in work. If they do not receive support, they can quickly return to receiving a benefit.

5.21
The Ministry refers some clients to its existing “in-work support” service (provided by third parties). However, this service is not available to all clients because not enough places are funded. Many others have less formal support. Some case managers keep in touch with clients, but this is not an official policy. Others put the client into “benefit suspension”. This enables a quicker return to receiving a benefit if the job does not work out, but could delay clients being able to claim working tax credits. This is also not official policy.

5.22
The Ministry has advanced plans to trial additional in-work support services in February 2015 for up to 6000 clients who move most often between working and needing to receive a benefit. The Ministry’s evidence suggests that there are sound investment reasons in helping to sustain people’s employment. If people stay employed for more than two years, they have a 94% chance of remaining without benefit support.

Keeping clients engaged with case management

The Ministry is finding it hard to meet its performance standards for client engagement. Clients missing appointments are a major contributing factor, and the Ministry needs to improve the appointment attendance rates.

5.23
The Ministry produces a regular, internal performance summary. This contains measures – for each of the case management streams – on:

• client engagements;
• client appointments;
• client outcomes; and
• client wait times.

5.24
The Ministry expects that at least 95% of clients in work-focused case management will have contact with their case manager at least once every 28 days. Figure 6 shows that, in the six months leading up to July 2014, the Ministry was finding it hard to meet its target.

Figure 6
Percentage of work-focused case-managed clients with at least one contact in the previous 28 days

Source: Ministry of Social Development.

5.25
The performance data for appointments suggests why the Ministry might be struggling with this target. Figure 7 shows the percentage of scheduled appointments for work-focused interviews that clients did not attend. The average figure for all regions is about 24%. This equates to about 21,000 appointments every four weeks.

Key fact
Clients do not turn up for around 24% of appointments.

5.26
The Ministry does not assign a cost to these missed appointments because staff can use the time to catch up on other work or help in general case management. However, in 2013/14, the Ministry invested $126 million to provide work-focused case management for just over 84,000 clients. If clients do not turn up for their appointments, it is a lost opportunity for the Ministry to help them get a good outcome. This could have a flow-on monetary effect, such as the client remaining on a benefit for a longer period. As noted in paragraph 3.12, time spent on one-to-one case management correlates most closely with improved outcomes.

5.27
The Ministry encourages its case managers to increase the contact they have with clients, but the number of “no shows” for appointments remains stubbornly high. This suggests a wider issue that case managers alone will not be able to resolve.

Figure 7
Percentage of scheduled work-focused appointments that clients did not attend, January to July 2014

Source: Ministry of Social Development.

5.28
We reviewed the files of several clients, some of whom regularly missed appointments. The Ministry has powers to apply sanctions to clients who do not meet their obligations, which includes attending scheduled appointments. We saw letters warning people that they were at risk of a sanction. In each instance, the person contacted the Ministry, and the Ministry lifted the sanction.

5.29
The legislation is clear that clients must carry out some recompliance activity to have a sanction lifted. Sometimes the client met this requirement by making another appointment, but in some instances the client failed to attend that appointment too.

5.30
Staff told us they found it hard to manage people who continually missed appointments but did just enough to prevent having their benefits stopped. Staff cited the law, which says that, if there is doubt, the decision should be in favour of the client. However, we did not find anything to this effect in the legislation, only in the Ministry’s guidance for staff.

5.31
In our view, this is a matter that the Ministry’s senior leaders should pay more attention to. Client obligations are an important part of the welfare reforms. Because no region has a rate of less than 20% of “no shows”, we consider that the cause is systemic and not something that will change by encouraging staff. We heard that the Ministry has other powers under the law that staff are finding challenging to use in practice, such as compulsory drug testing for some job seekers or requiring some parents to meet social obligations.

5.32
This is not an easy aspect of the case manager role. We consider this is a good time to review whether staff have enough management support to work appropriately with clients who find it hard to meet their obligations.

5.33
We know that staff would rather encourage clients to attend appointments than apply sanctions. We consider that the Ministry should investigate how it can use innovation to reduce missed appointments. For example, clients told us that they would appreciate text reminders of their appointments, and this happens routinely in many other organisations with appointment systems.

5.34
The Ministry told us that text-messaging capability was on the list of projects IT services were working on as a matter of priority. Once in place, that capability would partly address our third recommendation.

Recommendation 3 We recommend that the Ministry of Social Development reduce the number of appointments that clients miss by: improving how it communicates with clients about appointments; encouraging other innovation to increase the percentage of appointments kept; reviewing the ways in which clients can comply again after failing to attend an appointment; and increasing management support and coaching for staff to help them manage clients who routinely miss appointments.

Measuring the effect of work-focused case management

The Ministry has put in place a robust method to evaluate the difference that work-focused case management and work-search support are making for clients. Most service centres are showing at least a small improvement, although some increases are too small to be statistically significant. The Ministry has not yet identified the extent to which variations in practice influence the effectiveness of the service.

5.35
The third set of data we looked at was the comparison between the outcomes achieved for people in the work-focused case management stream and those placed in the control group. Nationally, there is a statistically significant improvement in the outcomes for people who receive both a benefit and work-focused case management, compared to the outcomes achieved for the control group.²⁷ Locally, most of the sites show at least a small improvement in outcomes (people moving off their benefit sooner).

5.36
This is promising, but our analysis of the Ministry’s data shows that only nine of 90 service centres show a statistically significant positive change. Partly, this is because these centres are larger sites, with more people participating, so it is possible to be more certain about the significance of the changes being observed.

5.37
Two centres are close to Auckland (Glenfield and Tamaki), and three are in the greater Wellington region (Wellington, Wanuiomata, and Otaki). The remainder are in Linwood (Christchurch), Greerton (Tauranga), Kamo (Whangarei), and Nelson. None of these areas is rural, and all have good access to one or more main centres of employment.

5.38
For clients with a heath condition, injury, or disability, the results in the original 23 trial regions are less positive. The difference at a national level for this stream is smaller than for people in the work-focused case management stream, and only two sites – Glenview (Hamilton) and Naenae (Lower Hutt) – show a statistically significant positive effect. There is no positive effect at nine of the 23 sites.

5.39
We consider that it is too early to draw firm conclusions from these early results. However, the Ministry has put in place a robust method to evaluate the difference work-focused case management and work-search support are making for clients.

5.40
The Ministry has been using the new case management approach for only a year. Some variation was to be expected until the new methods become routine. The Ministry has not yet identified the extent to which unintended variations in practice (see Part 4) influence the effectiveness of the service. It intends to monitor continuously how future changes to practice or service design affect the outcomes for clients.

Delivering Better Public Services results

5.41
In 2012, as part of its Better Public Services programme, the Government set targets for 10 aspects of public services. The Ministry has a target to:

Reduce the number of people continuously receiving Jobseeker Support for more than twelve months by 30 per cent, from 78,000 in April 2012 to 55,000 by June 2017.

5.42
At the end of June 2014, the number of clients in this category had reduced to 67,531. The trajectory since June 2010 and the June 2017 target are shown in Figure 8.

Figure 8
Number of working-age clients receiving Jobseeker Support benefit continuously for more than 12 months, June 2010 to June 2014

>

Source: Ministry of Social Development.

5.43
We cannot draw any conclusions about whether this performance puts the Ministry on track to deliver its 2017 target. This is mostly because we do not know whether it will become progressively harder for the Ministry to achieve further reductions, and the Ministry does not publish quarterly interim targets.

5.44
For example, although the number of people on Jobseeker Support has been declining at a faster rate than the 2012 valuation projected, more people are moving out of the “work ready” categories and into other categories with lesser, or no, work obligations.

Monitoring changes in the valuation of the welfare system

5.45
The valuation is producing some useful data and insights, but it is still developing. In 2011, the Ministry and the Treasury commissioned the baseline valuation of the welfare system’s future costs. The baseline figure was $78.1 billion as at 30 June 2011.

5.46
By 30 June 2012, the valuation of future liabilities had increased to $86.8 billion. Some factors had increased and some had decreased, adding a net $8.5 billion.

5.47
Of the factors that increased, the valuation showed that $13.4 billion was because of changes in the discount rate²⁸ and increases in inflation. In the 2012 report, the actuaries said:

Changes to inflation and discounting assumptions will have a significant impact on the valuation figures from year to year. However, these are outside the control of [the Ministry]. For this reason we separate the change in the valuation attributable to these items from other impacts to the valuation.

5.48
The actuaries reported that predicted costs reduced by $3 billion, mainly because of a lower number of clients than expected. The result was also partly attributable to the mix of clients receiving a benefit. A main contributor to this was thought to be the probable effect of policy and operational changes related to earlier welfare reforms in September 2010.

5.49
The June 2013 annual valuation is the most recent. It includes the period since the first two stages of welfare reforms and the early trials of work-focused case management, including:

• improving training and education for young people aged 16 or 17, young parents aged 16 to 18, and youth not engaged in employment or training, from August 2012;
• the new work obligations for some clients, including sole parents with dependent children between 5 and 13 years old, from October 2012; and
• trials of the new service delivery model in 23 service centres.

Key fact
The 2013 valuation of welfare system future liabilities was $76.5 billion, down from $78.1 billion in 2011.

5.50
The Ministry expected that the future liability would decrease from $86.8 billion as at 30 June 2012 to $83.9 billion as at 30 June 2013, based on 2012 forecasts. The actual future liability as at 30 June 2013 was $76.5 billion, or $7.4 billion lower than expected.

5.51
Of the $7.4 billion, $1.8 billion was because of a lower number of clients than expected. This was mainly because lower than expected numbers claimed Unemployment, Domestic Purposes, and Widow’s Benefits (all three benefits now fall under the Jobseeker category). The actuaries considered that “policy and operational changes probably contributed to this reduction in [future] liability – in particular, the revised work requirements in the October 2012 reforms and the piloting of the new service delivery model”.

5.52
The investment approach is producing some useful data and insights, but it is still developing. The actuaries note that, to their knowledge, the application of an investment approach to a social welfare system is a world first.

Using results to inform decision-making

The Ministry has a good range of methods to collect and analyse information, and was able to explain the mixed results we found. It had reasonably advanced plans to do something about most of them.

5.53
Based on our overall findings in this Part, we concluded that the Ministry is well placed to measure the results arising from the changes it has made.

5.54
The Ministry has changed the way in which information and data is analysed. As well as building its analytical capacity within the national office, it has strengthened the arrangements for analysis in the regional offices. The new arrangements are leading to greater consistency of reporting, which in turn enables better analysis. This means that the Ministry is able to make comparisons and those comparisons give good insights as to why results can be different between the regions.

5.55
The Ministry has worked hard to build evaluation into the new model – for example, the annual valuations. It is constantly learning about what is working well, and not so well, by using control groups and trials to measure outcomes and effects.

5.56
The Ministry’s performance in this area so far is good. It is building the analytical capability and capacity of its Knowledge and Insights team. Analytics is complex, and some of the techniques in use – for example, predictive analytics – need specialist skills that are scarce.

5.57
The Ministry’s investment strategy for 2014/15 brings together what it has learned from the trials and the first year of operating the new service delivery model. This strategy describes the Ministry’s plans, at a high level, for the changes it will make to operational services. The investment strategy addresses the challenges the Ministry has identified, and the opportunity for extending successful activities. The challenges include many of the matters we have raised in this report.

5.58
We saw some of the strategy addressed in new business cases that had been prepared and submitted for approval to the Work and Income Board. These business cases included costs, expected benefits, timings, and risks.

5.59
The Ministry requires that business cases and project designs include impact measures. For example, the Ministry has identified three high-level outcomes that it seeks from the trials it is planning for 2014/15. These outcomes are to:

• improve client outcomes, by clients becoming independent or less dependent on benefits;
• generate savings, by producing a positive return on investment by June 2017; and
• reduce the expected future cost of benefits paid.

5.60
Identifying intended outcomes in advance is good practice, because it means that the Ministry is in a good position to analyse the results of trials. It is likely that some of the trials might not produce the anticipated benefits, but this approach gives the Ministry good information on which activities do not provide a good return before it invests more heavily.

---

23: Subject to a margin of error of +/- 0.5 weeks at a 95% confidence level.

24: Subject to a margin of error of +/- 0.4 weeks at a 95% confidence level.

25: Work and Income cost savings only – not wider costs.

26: People wanting to apply for some kinds of benefits might have to undertake some pre-benefit activities first (such as attending a seminar). This measure counts those people who do pre-benefit activities but get a job, so do not need to claim a benefit after all.

27: This was a snapshot of cumulative effects taken at 490 days into the new service delivery model.

28: Accountants use discounted cash flows to estimate future purchasing power. Decreases in the discount rate means it becomes more expensive to pay for something later, because the value of money decreases over time.

page top