Part 10: A focus on continuous improvement

10.1
In this Part, we discuss:

• how the Department evaluates and monitors the effectiveness and quality of its programmes and interventions, and how it uses that information to make improvements;
• international benchmarks and where New Zealand is doing well; and
• how the Department gets feedback about the experiences of offenders and stakeholders.

Summary of our findings

10.2
The Department has a robust and internationally recognised method for monitoring the delivery, and evaluating the effectiveness, of its programmes. The Department uses the results of reviews and evaluations to improve its strategy and programmes. In our view, the Department should be commended for the importance it places on monitoring and evaluation.

10.3
International comparison is challenging because other jurisdictions have different measures and ways of reporting. Despite this, New Zealand seems to be doing well and leading the way in some aspects.

10.4
The Department has a range of approaches for getting formal and informal feedback from offenders and stakeholders. However, the Department could benefit from being more consistent in the way it collects the experiences of offenders and stakeholders.

Evaluating the effectiveness of reducing reoffending

10.5
The Department defines effectiveness as meaning that both the rate of reoffending and the seriousness of reoffending are decreasing. There is a growing awareness in the Department that it takes time to move away from crime. This is why the Department also measures effectiveness by looking at desistance or a change in an offender's former pattern of behaviour. This indicates that offenders are on a pathway to stopping reoffending.

10.6
The Department has prepared methods to evaluate the effectiveness of its programmes. It has commissioned studies to evaluate its methods, including an external review as part of its expenditure review (see paragraph 2.30). The review team was impressed by the standards of professional integrity that the Department sets and by the quality of its programmes. In particular, the review team was impressed with the work on the employment and work programmes. The team maintained that the reported reductions in recidivism were more than adequate when compared to what has been cited in international research findings.

Using the recidivism index and rehabilitation quotient to measure effectiveness

10.7
The recidivism index uses two primary measurement approaches. The first approach measures reoffending rates throughout the entire population of offenders managed in a year by age, gender, ethnicity, gang affiliation, type of crime committed, type and length of sentence, and prisoner security classification.

10.8
The second approach is called the rehabilitation quotient. This calculates the effectiveness of programmes by comparing a group of offenders who have completed a programme with a control group of offenders who have not. For example, if the rate of reimprisonment for untreated offenders was 35% and the corresponding rate for offenders who had completed the programme was 25%, there would be a 10 percentage point reduction in the rate of reimprisonment between treated and untreated groups. Figure 12 would show this as a percentage-point change of -10.0.

10.9
Several factors can affect the success of rehabilitation interventions. It can be difficult to separate out what has been successful, especially if offenders have completed several programmes. However, by comparing treated and untreated offenders, the Department can show that its programmes are successful.

Figure 12
Results for rehabilitation programmes and interventions, 2012/13

* Results are statistically significant below the 95% level of confidence threshold but are highly likely to be indicative of effectiveness.
**Results are statistically significant at the 95% level of confidence. Source: Department of Corrections.

10.10
Figure 12 shows that participation in rehabilitative activities generally has a positive effect on reducing reoffending. For example, offenders who complete a rehabilitation programme in a special treatment unit are reimprisoned at a rate 12.9 percentage points lower than offenders who have not completed a programme.

10.11
The Department told us that rehabilitation quotient scores of around -10.00 are the "gold standard" internationally. However, this reduction is infrequently achieved when programmes are routinely delivered to large volumes of offenders.

10.12
The rehabilitation quotient also shows which offender groups the programmes work best for. For example, results for the drug treatment units in 2009/10 showed that they were particularly successful for Māori participants. Conversely, some programmes are not producing good results for youth. This is reflected in the recidivism index, which shows that youth are reimprisoned and reconvicted at significantly higher rates than older offenders. The Department acknowledged the difficulty of achieving good results with youth. It told us that it knows it needs to improve the quality of programmes delivered to youth.

Evaluating and reviewing the effectiveness of programmes for Māori

10.13
The Department is committed to reducing the reoffending rates of Māori:

As Māori offenders make up over half of the offender population, we need to succeed with Māori in order to reduce reoffending overall.

10.14
The Department has set an even more ambitious target to reduce reoffending by 30% by 2017 for offenders in the Māori focus units. The target is 5 percentage points higher than the overall target for the general prison population.

10.15
The Department's expenditure review highlighted some concerns about the purpose and delivery of the tikanga Māori programmes. The review recommended re-evaluating the purpose of the programmes and relaunching an improved programme. The Department found that there was not a lot of evidence to support tikanga Māori as a rehabilitative programme. However, the "word on the ground" was that they are "pretty good" and that offenders and staff love the programme. The Department has reviewed the purpose of tikanga Māori programmes and recognises that Māori offenders benefit by improving their sense of cultural identity and values. The Department now uses tikanga Māori programmes as short motivational programmes to encourage further participation in proven rehabilitation programmes, such as medium-intensity rehabilitation programmes or Mauri Tu Pae.

10.16
The Department's results show that Mauri Tu Pae is doing well. The programmes are similar to medium-intensity rehabilitation programmes but include a specific Māori cultural perspective that builds on the foundation provided by the kaupapa of the Māori focus units and tikanga Māori programmes. We consider that this is a good example of an evidence-based approach that incorporates the importance of cultural values.

Monitoring the quality and delivery of programmes

10.17
The Department monitors the quality of programme delivery to maintain the effectiveness of rehabilitation programmes.

10.18
As discussed in paragraph 2.29, the Department based its approach to managing offenders on international best practice and the standard RNR principles. It monitors the quality or integrity of its rehabilitation programmes to ensure that the delivery of those programmes adheres to these principles.

10.19
The Department's quality monitoring method is based on the internationally recognised Correctional Programme Assessment Inventory. The external review team reviewed the Department's method as part of the expenditure review in 2011.

10.20
The Department follows general principles of monitoring, such as reviewing paperwork, including pre-programme assessment documents, observations, recorded and reviewed programme sessions, peer reviews, and interviews with programme facilitators. However, the extent to which offenders' feedback is collected is not clear.

10.21
The Intervention Design and Delivery Team reviews medium-intensity rehabilitation programmes. The Chief Psychologist's team reviews rehabilitation programmes in special treatment units. There appeared to be less monitoring of Mauri Tu Pae. We note that Mauri Tu Pae is based on the proven medium-intensity rehabilitation programme and that the amount of monitoring might not be a problem. However, if the results for Mauri Tu Pae drop below an acceptable level, increasing the amount of quality monitoring might be one area for the Department to look at.

10.22
The Department monitors how programmes are delivered and evaluates how effective its programmes are. In our view, the Department should be commended for the importance it places on monitoring and evaluation. Although we have identified areas where there might be room for improvement, overall, we consider that the Department monitors and evaluates the delivery of its programmes robustly and to a high standard.

Examples of using information to implement improvements

10.23
We saw many examples of how the Department uses its reviews of its effectiveness in reducing reoffending to make improvements.

10.24
For example, as a result of the expenditure review, the Department has placed more focus on education and training.

10.25
The expenditure review also carried out a stocktake of the Department's rehabilitation and reintegration interventions. It examined the outcomes of the interventions against their costs. The review identified which programmes delivered the best value and the most positive, measurable results. The review identified interventions that do not work and recommended replacing them with ones that are proven to work or could be delivered more cost-effectively. The review made 11 recommendations, most of which have been carried out. For example, the Department has extended the range of alcohol and drug treatments to match need and has stopped using faith-based interventions in prisons.

10.26
At a programme level, the results of monitoring and evaluation are used to redesign and redeliver programmes. For example, a youth programme was piloted in Counties Manukau. At the end of the pilot, it was evaluated by the programme facilitator and psychologist. The programme had not been a success. The reviewers made some recommendations to improve the programme, including shortening the length of the programme and not running it during the Christmas period. The evaluation was peer reviewed by a senior psychologist in the National Office. The Department decided to run the programme again with the recommended changes. It would be reviewed again before deciding whether to set it up nationally.

Benchmarking performance internationally

10.27
International comparison is challenging because other jurisdictions have different measures and ways of reporting. For example, Australia reports at a state level. New Zealand is one of the few jurisdictions to have set a target for reducing reoffending. Scotland used to have a target to reduce reconviction by 2% but no longer does. New South Wales has a target to reduce the rate of reoffending by 5% by 2021.

10.28
The Department participates in two international benchmarking exercises. The first is with the eight Australian states. Each year, a "Comparative Analysis" report is produced that reveals relative performance on a number of critical indicators. These include prisoner assaults, deaths in custody, participation in prison employment, relative operating expenditure, and recidivism. Secondly, New Zealand participates in the International Roundtable for Corrections Excellence benchmarking exercise, which includes eight European countries, including Scotland, Ireland, Norway, and Belgium. The benchmarking exercise produces performance information about indicators similar to those used in the New Zealand and Australia exercise.

10.29
New Zealand seems to be doing well and is leading the way in some aspects. As mentioned in paragraph 10.6, an international review team found that New Zealand's professional standards of following an evidence-based approach set a benchmark. The review team stated:

We contend that one would be very hard pressed to find another state or provincial [Department of Corrections] in Australia, Canada, and the US or Federal prison system … that meets the standards set forth by [the Department].

10.30
Further, we were told that nowhere else in the world assesses risk for both prison and community settings (DRAOR and SDAC-21). The Department told us that New Zealand was the first country to include responsivity factors in a risk assessment tool for prisons.

10.31
New Zealand has the advantage of doing things that other jurisdictions are unable to do because of the smaller scale of our corrections system and because we have only one jurisdiction and one legal system. For example, having a national database gives the Department a comparative advantage over other countries. The database enables it to find data on any offender, such as the computerised risk register for sex offenders. In comparison, the Department told us that, in Australia, it can be difficult to track offenders across state lines.

10.32
Another area where the Department seems to be leading the way is reporting results on effectiveness in its annual report. We are not aware of any other country or jurisdiction that does this. New Zealand also appears to be the only country to publish results at a wider system level and report on programmes delivered at multiple prisons. Other countries that publish effectiveness tend to focus on small, niche, and specific programmes that show good results. The Department does not just publish good results but also results that show no improvements or small increases. For example, the Department's 2011/12 annual report showed that there was no difference between treated and untreated groups of prisoners in the medium-intensity rehabilitation programmes.⁸

Gathering feedback from offenders and stakeholders

Offender-centric approach

10.33
The offender-centric approach means providing offenders with rehabilitation and reintegration interventions that meet their individual circumstances and that help them to not reoffend. In Scotland, an audit on reducing reoffending pointed out that the best way to ensure that the needs of offenders is being met is to ask offenders what worked for them. Staff and offenders we spoke to agreed, because this approach encourages support.

How the Department gets feedback from offenders

10.34
The Department has several ways to get feedback from offenders. We attended a community meeting in the Kia Marama therapeutic unit at Rolleston Prison. The meeting involved all 60 offenders in the unit, prison staff, psychologists, the Chaplain, and volunteers at the prison. The meeting was formal and minutes were kept. These meetings are held once a week. All members are given the opportunity to speak and voice any concerns they have. This allows the community to set its culture, self-manage and regulate, and improve respect for others. As one offender pointed out, it allowed the group to deal with little infringements before they escalated into big problems.

10.35
We attended another unit that holds daily meetings. Offenders told us that making staff available like that meant they can raise issues and get them addressed. They said that staff were helping them work towards their release: "They don't turn you away."

10.36
We attended several programmes and had the opportunity to talk with offenders and facilitators both individually and in focus group settings. We were often accompanied by the Regional Commissioners, who told us that they found it valuable hearing feedback from offenders and facilitators. One Regional Commissioner told us that they often visited and spoke with offenders, but it was not clear to us how regular or consistent this practice is.

10.37
We heard concerns that not all offenders were getting the right programmes to meet their needs and that the system has a "one size fits all" approach. The alcohol and drug treatment programmes are an example of this. The Department acknowledged that it was not meeting the needs of all offenders who need treatment (see paragraph 3.31). Having a systematic approach to capture offenders' experiences could provide opportunities to signal early where the Department might not be meeting needs and identify ways to improve how programmes are delivered.

How the Department gets feedback from stakeholders

10.38
The Department has formal and informal methods to get feedback from stakeholders. For example, members of the Department's senior leadership team meet with unions, judges, the Parole Board, and non-governmental organisations such as the Salvation Army.

10.39
The Department gets feedback on public perception through two surveys. First, the Department purchases a report from a private research company that conducts an annual survey of the general public, seeking views about trust and confidence in a range of government agencies. Second, the Ministry of Justice surveys public perceptions of the justice system and sector agencies. The survey includes questions about the Department and whether it is seen to be effective.

10.40
The Department gets feedback on the support it provides in court, and the quality and timeliness of pre-sentence and prosecution reports. Feedback from the Parole Board and judges resulted in a programme designed to streamline the provision of advice reports and make them more efficient and consistent.

10.41
Community probation managers lead the Department's relationship with district courts at the district level. Other probation staff also meet with local courts and judges to get feedback. They also pick up some complaints from judges in the media.

Taking a more consistent approach to collecting feedback

10.42
The Department has a mix of approaches for getting formal and informal feedback from offenders and stakeholders. In addition to surveying the general public's perceptions, we consider that the Department could adopt a more consistent approach to capturing the experiences of offenders and stakeholders.

10.43
For example, the Department could use a tool such as the Common Measurements Tool survey, which measures client satisfaction by using a common set of questions. The survey measures the six main factors that influence New Zealanders' satisfaction with the quality of services they receive from public service entities. The most important factor is meeting expectations. The other factors link directly to the offender-centric approach the Department uses and the principles of the Act – for example, "ensuring fair treatment of offenders" and taking into account their cultural background, ethnicity, and language. Other public service entities in New Zealand have adopted the methodology. By using a common set of questions, the Department will be able to compare its performance with other justice sector entities, such as the Police and courts, that use the Common Measurements Tool.

8: The 2012/13 medium-intensity rehabilitation programme results show a 3.8 percentage point reduction in reimprisonment between treated and untreated prisoners (see Figure 12).