Part 4: Monitoring the New Zealand Police's work programme

4.1
In this Part, we set out our findings about the mechanisms the Police have for monitoring¹ the:

• progress of their work programme to implement the Commission's recommendations; and
• outcomes from their work programme, including identifying any value added to the Police's functions and responsibilities.

4.2
Effective monitoring can help the Police to keep the work programme on schedule. It also enables continued informed amendments to the work programme as progress is made. It can also help the Police to know what the programme has achieved.

Our overall findings

4.3
The Police have a robust system for monitoring progress in implementing the Commission's recommendations, and for reporting this information publicly and to stakeholders.

4.4
The Police need to ensure that they have arrangements for monitoring the outcomes of their responses to the Commission's findings, in terms of the service levels and cultural changes of the type signalled in the Commission's recommendations. While these are not easy things to monitor, the Police need to do more work on this. We have made two recommendations about this work.

4.5
The Police and SSC have told us that the next survey of the Police to assess organisational health is planned for 2010. The survey will be the primary means to assess organisational health, as recommended in the Commission's recommendation R51. The Police and SSC are planning to repeat the survey about every 12 months after 2010, up to and including 2016/17. We encourage them to decide on a survey tool that is appropriate and cost-effective, for the purpose of meeting the requirements of the Commission's recommendation R51. We do not have a view at this time on the appropriateness or cost-effectiveness of the survey tool used to date.

Monitoring progress in implementing the Commission's recommendations

The Police have taken a project management approach to monitoring progress in implementing the Commission's recommendations.

Monitoring within the work programme

4.6
The Police have a project management application to monitor progress in implementing the Commission's recommendations. We discussed this application in paragraph 3.11. The application has been very helpful in monitoring the Police's progress with the Commission's recommendations.

4.7
The application allows a user to record and view the progress of the project for any given recommendation, as well as a summary of progress for all recommendations.

4.8
The Police use the information recorded in the application for reporting their progress with the Commission's recommendations. Progress against the Priority 1 recommendations is reported monthly to the steering committee.

4.9
Information from the application is also used for quarterly reporting of progress to Ministers (of Police and Justice), to other public entities, and to the public through the Police's website.²

4.10
The quarterly reporting of progress to Ministers has been largely descriptive. The reports have focused on progress made rather than the effect achieved. In our view, the reports would be more useful if they included analysis and text that clarifies what the described pieces of work actually mean, or could mean, for changes in the way the Police work and the services they deliver. The reports would also benefit from an assessment of whether the described progress is satisfactory.

4.11
The Police told us that, during the implementation phase of their work programme, they will report progress as we have recommended.

4.12
Communication channels other than the project application are used for informing the wider Police organisation about progress in implementing the Commission's recommendations, or aspects of them. These channels include newsletters, the Commissioner of Police's blog, and the Police's internal bulletin board.

4.13
Some of these communications refer to specific projects (for example, the Code of Conduct, new disciplinary process, and ethics training) rather than the Commission's findings or a specific Commission recommendation. The Police told us that they consciously do this because some staff view the Commission's findings negatively rather than as an improvement opportunity. This illustrates the challenges faced when trying to actively involve staff and change the culture within a large and complex organisation.

4.14
Figure 7 summarises the Police's self-reported progress, as at 22 May 2009, in implementing the Commission's 48 recommendations that the Police identified as applying to them. Information on progress with each recommendation for the technical and implementation phases of the Police's work programme is shown. The recommendations are grouped by the Police's workstreams within their work programme. We have not assessed the accuracy of the Police's assessment because our next report will assess what progress the Police have made against each recommendation.

Monitoring outside of the work programme

4.15
As well as the programme monitoring arrangements outlined above, the Police also report and monitor progress in implementing the Commission's recommendations by other means.

4.16
At the time of our audit fieldwork, the Police had scheduled some assurance work for specific Commission recommendations. This work was to be carried out by the Organisational Assurance Group.³ The specific projects and the associated Commission's recommendations are:

• Adult Sexual Assault Review (Recommendations R9, R10, R18, and R19);
• Community Policing Review (Recommendation R57);
• Code of Conduct Review (Recommendations R34, R35, R36, and R38); and
• Capability/Recruitment/ability to service diverse communities (Recommendations R50 and R57).

4.17
To varying extents, each District or service centre's 2008/09 annual business plan includes activities to give effect to the priority recommendations. Progress against these priorities is monitored through the Police's national quarterly and annual performance reporting processes.

Figure 7
The New Zealand Police's reported progress, by workstream, in implementing the Commission's recommendations

Source: New Zealand Police.

4.18
For the Code of Conduct⁴ and the Organisational Health Audit⁵ projects, each District was required to prepare an implementation plan as well as its standard annual business plan.

4.19
The Police also reported progress in implementing the Commission's recommendations in their Statement of Intent 2008/09 – 2010/11 and the 2006/07 and 2007/08 annual reports.

4.20
The Police do not report on progress with the Commission's recommendations in their national monthly management reports. The focus of these reports is on finance, human resource, operational performance, and asset management metrics.

4.21
We were told that the Police anticipate adding some measure of Commission-related process improvements to their six-monthly reviews of District performance. The Police's Organisational Performance Group⁶ carries out these reviews, which are described as involving “analysis of results, interviews, and site visits to identify strengths and opportunities for improvement and practices that work in reducing crime and increasing safety”.⁷

4.22
We encourage the Police to review, and amend as appropriate, their reporting of progress against the Commission's recommendations outside of the work programme reporting mechanisms. In our view, any of the Police's key internal or external reporting documents could reasonably be expected to include some information on progress against the Commission's recommendations.

Monitoring the results of efforts to address the Commission's recommendations

The Police are still developing their approach to monitoring the next phase of their work programme and its results.

Monitoring the next phase in the work programme

4.23
Several of the recommendations in the Commission's report refer to changes in internal or external services delivered by the Police or changes in Police culture. These types of changes can take a long time to achieve.

4.24
Making progress with the Commission's recommendations, according to the Police, “requires a change in behaviour and a shift in the 'hearts and minds‘ of every person employed by Police”.⁸

4.25
Changes in service levels and culture can be difficult to monitor and measure. The Police anticipate taking a different approach to monitoring the next phase of their work programme. This approach may include labelling Commission-related work in non-Commission terms. The Police had not finalised their approach at the time of our audit fieldwork.

4.26
We support the Police thinking about a different monitoring approach during the next phase of their work programme. This is because the “technical phase” was characterised by putting in place system changes to support organisational changes in the Police. Our analysis of the Commission's recommendations shows that a large number of them included aspects related to a corporate document (a policy or procedure) or a process change the Police are to follow.

4.27
We acknowledge that the Police could use some of their existing management or performance measures, where these might provide indications of cultural and other changes. We support this approach because it minimises the amount of additional work for the Police.

4.28
Analysing and identifying the extent to which the Police's responses to the Commission's recommendations have contributed to improvements in policing, as experienced by the public and the Police, is important. This is the value added to the way the Police fulfil their responsibilities and functions. However, identifying this value can be difficult because of the complexity of factors that can influence the Police's performance, beyond their responses to the Commission's findings. We encourage the Police to identify this added value to the extent that is possible and measurable.

The organisational health audit

4.29
Recommendation R51 of the Commission's report states:

The Commissioner of Police should invite the State Services Commissioner to carry out an independent annual “health of the organisation” audit of the police culture (in particular, whether the organisation provides a safe work environment for female staff and staff from minority groups). The need for the audit should be reviewed after 10 years.

4.30
The Police and the SSC decided to use the Gallup Q12® survey, supplemented with additional questions, on the culture and safety of the Police as an organisation to work in (issues particularly referenced in Recommendation R51), as the primary means to assess organisational health.

4.31
The Police and SSC have told us that this decision was made based on private sector evidence from overseas that suggests higher levels of employee “engagement” raises performance and also helps manage costs, such as the costs associated with staff turnover. They also told us that evidence suggests that an “engaged” and committed workforce improves productivity and service delivery.

4.32
The Police and SSC decided to supplement the Gallup survey data with some other organisational health data (including attrition rates, lost-time injuries, sick leave, exit interview results, personal grievances, complaints, and demographics for women and ethnic minorities).

4.33
The first Gallup survey was completed in the second quarter of 2007/08 (October-November 2007). The Police and SSC have told us that the first survey was a pilot, administered to a random sample of 6700 Police staff (4880 of whom responded to the survey). The survey found lower levels of staff “engagement” in the Police than in other New Zealand organisations, including New Zealand state sector organisations.

4.34
The second Gallup survey was completed in the second quarter of 2008/09 (August 2008). The second survey was administered to all staff (about 11,000 people), with an 82% response rate. The survey found an increase in the percentage of “Actively Disengaged” staff (defined by Gallup as being physically present but psychologically absent). It also showed that those in middle and lower roles/ranks were significantly less “engaged” than their senior colleagues. The results also showed that leaders could be more effective in inspiring staff about the future of the Police.

4.35
More positively, the results showed that minority groups (including Asian, Māori, and Pacific staff) were the most “engaged“ in relative terms. The results also showed that women were significantly more “engaged” than men.

4.36
The Gallup Organisation has told us that the results of their second survey showed the perceptions of Police employees had either declined or stayed the same for all additional questions with particular reference to Recommendation R51.

4.37
The Police and SSC have told us that the results of these surveys need to be read in the context of the variability in the level of “engagement” across the state sector and other indicators. For example, public satisfaction with services and confidence in the Police remains high, despite lower-than-desirable “engagement” of some levels in the Police, and staff turnover is very low. Both of these elements are usually associated with a high level of “engagement”. The Gallup Organisation has told us that this inconsistency between the level of engagement and public satisfaction and turnover is to be expected, given the nature of the Police organisation and their interaction and role with the New Zealand public.

4.38
We note that PricewaterhouseCoopers, on behalf of the SSC, has completed two reviews of the Police. PricewaterhouseCoopers commented in the second review:

Given the structural and cultural uniqueness of Police organisations, we wonder about both the appropriateness of [the Gallup Q12® survey] content and the benchmark comparison group. More practically, we have concerns that [the Gallup Q12® survey] is expensive to maintain, given current pressures on Police expenditure.

We wonder whether a simpler but more customised system for securing regular upward feedback would not be helpful at this stage in organisational development. On-line bespoke pulse surveys are used in many organisations as overall health checks, and may provide a more cost effective approach for Police. They also have the advantage of being designed for purpose, taking the pressure off “survey time” and reducing the environmental factors that could allow the results to be downplayed or dismissed.

4.39
In January 2009, the Police and SSC told the Ministers of Justice, State Services, and Police that:

SSC and NZ Police are also in a process of ensuring that the current survey is fit-for-purpose, and providing value-for-money.

4.40
During the Law and Order Committee's 2007/08 financial review of the Police in March 2009, the Police told the Committee that:

We have made a decision to do the next survey next year rather than this year to give ourselves some time to get some actions in place and to make a difference that the staff are looking for. We've got some work to do but we're pleased that we have a good feed-back system from our staff, and we will continue to survey staff. Who we use to do that survey will be the subject of an open tendering process.⁹

4.41
The Police and SSC have been reflecting on the lessons learned from the 2007/08 and 2008/09 Gallup surveys. They have told us that a better fit-for-purpose survey is required. They have also told us that information from the current survey gives some very useful information on “engagement” but it does not provide enough data to address all matters relevant to Recommendation R51.

4.42
The Police and SSC have told us that these data gaps will be addressed and result in a better fit-for-purpose survey. They have also said that particular attention will be given to the wording of survey questions to ensure that the intent of the question is easily understood and not open to misinterpretation by respondents (which happened with one question related to safety in the 2007/08 and 2008/09 surveys).

4.43
The Police and SSC have told us that the next survey to audit the Police's organisational health is planned for 2010 and will be implemented within the first six months of that year (17 to 22 months since the second Gallup survey was conducted). They have also told us that they are planning to repeat the survey about every 12 months after that, up to and including 2016/17.

4.44
In our view, the Police and SSC's planned approach is sensible, and we encourage them to decide on a survey tool that is appropriate and cost-effective for assessing organisational health, for the purpose of meeting the requirements of Recommendation R51. In our view, it is important that any tool used supports the collection of a consistent series of information over time. Without consistency, it will be difficult to meaningfully measure changes over time. We do not have a view at this time on the appropriateness or cost-effectiveness of the survey tool used by the Police and SSC to date.

Citizens' satisfaction survey

4.45
The results of independent surveys of the public's experience and attitude towards the Police are another measure of change within the Police. The Police conducted a citizens' satisfaction survey as part of the Service First initiative¹⁰ between February and June 2008. It involved telephone interviews with 8300 people, nearly half of whom have had contact with police. The Police plan to conduct more of these surveys.

4.46
The results of the survey are publicly available on the Police's website, and the Police reported selected aspects in their annual report for 2007/08. That annual report notes that the “results will also form part of the wider monitoring of satisfaction with the Police as recommended by the Commission of Inquiry into Police Conduct”. The survey focuses on six factors that affect satisfaction with services. These are the extent to which:

• The service experience met your expectations
• Staff were competent
• Staff kept their promises – that is, they did what they said they would do
• You were treated fairly
• You feel your individual circumstances were taken into account
• It's an example of good value for tax dollars spent.

4.47
The 2008 citizens' satisfaction survey found an 80% level of satisfaction with the quality of the Police's service delivery for those respondents who have had contact with Police staff.

4.48
The citizens' satisfaction survey replaces the Police's quarterly Public Confidence and Satisfaction surveys that were conducted from 2001 to 2007.

4.49
For 2008/09, the Police have identified changes in public trust and confidence, and changes in public satisfaction, as success measures as part of their national business plan.

Other measures

4.50
During our interviews with Police staff, we asked them what they thought might be useful indicators (direct or indirect) of change in Police culture, attitudes, and behaviour, other than the organisational survey. Appendix 6 summarises what we were told. We acknowledge that interpreting these indicators may be difficult in some instances.

Complaints

4.51
In our view, complaints against Police staff and the outcomes of these are useful indicators of change within the Police. They do, however, need careful interpretation. We agree with the United Kingdom's National Audit Office observation that:

An effective complaints function is important in keeping people's faith and trust in services and is an essential building block of a high performing organisation. It can also provide the organisation with assurance about the safety and quality of service provision.¹¹

4.52
The Police told us that they are unsure whether the number of recorded complaints is a useful indicator of change. They said that an increase in complaints can reflect more trust and confidence in the Police, and therefore the willingness of people to lay complaints. We agree that a number of factors can influence complaints laid against police. These factors need to be considered when interpreting complaints information.

4.53
Data about complaints against Police staff was one of the potential indicators of change often mentioned by our interviewees. The Police have provided summary information on complaints in their annual reports. They also said that success would include that it were rare for complaints to be upheld.

4.54
Information provided to us by the Police showed that the most common type of complaint was for attitude and/or language issues. We have reproduced, in Figure 8, the information provided to us by the Police for attitude and language complaints, and sexual offending complaints. The information shows that complaints for sexual offending are less than 1% of the total complaints against police. The information also shows that there is variation in the proportion of these complaints upheld between 2005 and 2008.

Figure 8
Complaints against Police staff for attitude and language, and sexual offending 2005/06 – 2007/08*

Source: New Zealand Police.
* We also asked the Police to provide the number of specific sexual harassment complaints received. The number of sexual harassment complaints was relatively small (fewer than 10 each year). The Police did not indicate the level of severity of the complaints in the information provided to us.

4.55
At the time of our audit fieldwork, the Police had approved a business case to implement a new information system to better record, analyse, and report complaints. The Police expect to take most of 2009 to get the system operating. The system has the potential to enable the Police to identify trends in complaints earlier than they are currently able to.

Observations

4.56
As well as indicators of change, many of the Police staff we spoke with described cultural and other changes that they have observed. We have quoted in Appendix 7 some of the comments made to us during our audit fieldwork about the Police culture and changes to it.

4.57
We acknowledge that our interviews were with only a relatively small number of Police staff (less than 1% of all staff). However, there is value in this qualitative information. The Commissioner of Police and others told us that they regard this type of information highly, and that they deliberately spend time “managing by walking around” or in open forum discussions with staff to obtain this type of information.

4.58
We encourage Police management to continue their efforts to observe and receive qualitative information about the Police culture and changes within it.

Integrity and conduct survey

4.59
In response to our draft report, the Police provided us with a copy of a December 2007 integrity and conduct survey of the Police. This was performed by an independent and non-profit American organisation called the Ethics Resource Centre as part of a wider New Zealand State Services Integrity and Conduct Survey. The Police told us that the conduct and integrity survey will be repeated in 2010.

1: We use “monitoring” in its broader sense to include project tracking, reporting results, and evaluating results.

2: See the Commission of Inquiry section of the Police website, www.police.govt.nz.

3: The Organisational Assurance Group provides assurance, evaluation, and risk services for the Police.

4: Recommendations R33, R34, R35, R36, and R38.

5: Recommendation R50.

6: The Organisational Performance Group provides quality improvement, statistical, and performance monitoring services for the Police.

7: New Zealand Police, Annual Report 2006/07.

8: New Zealand Police, National Business Plan 2008/2009.

9: Report of the Law and Order Committee (2009), 2007/08 Financial review of the New Zealand Police, New Zealand House of Representatives, pages 17-18.

10: According to the Police's annual report for 2007/08, Service First is a service improvement programme and a Police priority. “It aims to improve citizens' satisfaction with policing services by using a citizen centred approach to service delivery. In practice this means: … knowing who uses Police services and what is important to them; … providing services that are responsive to these needs … seeking feedback from recipients of Police services about their levels of satisfaction and dissatisfaction … using this information to realign, as necessary, and improve services.”

11: National Audit Office (2008), Feeding back? Learning from complaints handling in health and social care, United Kingdom, page 10.