Part 6: The right data to support decision-making

6.1
In this Part, we discuss the need for NZDF to have the right data and information to guide its approach to Operation Respect. Data is essential to understanding the extent and nature of harmful behaviours occurring, and for monitoring progress towards Operation Respect's outcomes.

6.2
We expected to see a plan to enable the right data to be collected to both understand the issues and measure progress. We also expected that NZDF would have a plan to make that data available to the right people so they could make informed decisions about the programme.

Summary of findings

6.3
Information is not currently collected in a way that allows NZDF to properly understand risk, effectively monitor and measure progress, and hold leaders accountable for progress.

6.4
In our view, NZDF needs a data improvement plan which identifies the data that is needed and sets out how that data will be collected and collated, how data gaps will be filled, how confidentiality will be managed, and who will lead this work. In our view, this must be a priority.

The systems for collecting and maintaining data need improvement

6.5
NZDF has a range of data sources that record some information about harmful behaviour:

• Military Justice and Summary Trial outcomes – records of these proceedings include information about incidents of harmful sexual behaviour that are offences within the Armed Forces Discipline Act.
• Sexual Assault Prevention and Response Advisors (SAPRAs) hold data about restricted and unrestricted disclosures reported to them, including information about the service of the victim/survivor, the rank and service of the accused (only when an unrestricted disclosure is made), the type of behaviour (for example, indecent assault and sexual harassment), and some characteristics, such as whether alcohol was involved.
• PULSE data (from NZDF's annual engagement survey) is centrally held. In 2021, a question was included in the survey that asked people whether they had experienced inappropriate or harmful behaviour in the previous year.
• Complaints data is held in multiple forms and locations. It includes records of complaints raised (that is, bullying complaints).
• Social workers, chaplains, and psychologists all record data that can include information about harmful behaviour (for example, information about whether someone went to a social worker after experiencing bullying).

6.6
However, leaders at all levels told us that they were not able to access the right data to understand risks and monitor progress. Without the right data there is a risk that decisions will not be well informed and interventions will not be well targeted.

6.7
We expected to see work on a plan to improve data and to develop measures to enable progress to be monitored but we did not. Without a plan for monitoring and evaluating progress, it is not clear how NZDF will know whether Operation Respect is making a difference.

Senior leaders do not have access to data that identifies risks and trends

6.8
Leaders at all levels were frustrated with the data they received. Most felt that the data was not enough to help them understand what behaviours were occurring in their areas of responsibility and across the organisation.

6.9
The main sources of data senior leaders accessed to understand harmful sexual behaviour were from the Court Martial and SAPRA data. These sources provide information about the type of reported incidents. Although valuable, on their own they provide only a limited picture of the scale and nature of harm.

6.10
Summarised and anonymised information sourced from SAPRAs is provided regularly to governance groups. However, we did not see evidence that it was used to understand trends. SAPRA data does not provide enough detail about incidents to identify risk areas. There are also limits placed on the details SAPRAs can record and provide to others to protect confidentiality, particularly for restricted disclosures.

6.11
There is a range of other data that could inform the understanding of incidences of inappropriate and harmful behaviour, such as social worker data, chaplain data, and complaints data. However, there is no requirement to bring these sources together, nor systems or processes to easily do so. These forms of data are held in different systems (for example, SAPRA data has been held in a different system to social worker data – although we were told this is changing). This makes it challenging to bring all this data together.

6.12
Some leaders on camps and bases try to gather this information to inform their understanding by, for example, asking those specialist support services to provide them with information about what trends they were seeing. Although valuable, it was inconsistently done and there is currently no easy way to form a view of what is happening across the whole of NZDF.

Gaps in data limit understanding of bullying, harassment, and discrimination

6.13
Senior leaders do not have much visibility of the extent and nature of bullying, harassment, and discrimination due to the way complaints data is managed.

6.14
There are mechanisms to record complaints. All bullying, harassment and discrimination complaints should be recorded in a form called an MD1037. All units are also required to maintain a register that records all administrative complaints they have investigated. However, we heard that data is recorded only for complaints raised to the commanding officer level (or above), and that collating the data is difficult. We heard there is no way to know how many complaints there were across NZDF and how they were resolved. Although Defence Force Order Three (see paragraph 4.53) states that there should be annual audits of complaints data to provide analysis, the last audit was carried out in 2015.

6.15
We were told that incidents would not be recorded unless a personal grievance is raised. Incidents are often resolved informally by the officers, NCOs, or civilian managers they are reported to. There is no obligation for officers or NCOs below commanding officer level to record any details of incidents reported to them, nor is there a system to hold any such records. This means it can be difficult for commanding officers to know what bullying, harassment, and discrimination is occurring in their unit and for the organisation to have a clear picture of where these behaviours are occurring.

Little data is collected on non-criminal harmful sexual behaviour

6.16
There is very little data collected about non-criminal harmful sexual behaviour. Many of these forms of harmful behaviour fall outside the Armed Forces Disciplinary Act. Therefore, they are not subject to investigation and/or recorded in the Summary Trial or Court Martial data.

6.17
Although people might raise complaints about these types of incidents, they are not electronically recorded unless the complaint reaches the commanding officer. Incidents are not always understood as harmful, and the processes for addressing these behaviours are not well understood. We heard that these factors mean incidents are not always being recorded.

6.18
There is currently no centralised place that individuals experiencing harm, or leaders dealing with this type of harm, can record these incidents to ensure that they are used to gain a view of what is occurring across NZDF. Without this data, there is limited visibility of the amount of this type of behaviour in the organisation.

6.19
The gaps in data and information collection present a range of challenges for Operation Respect. It makes it difficult to know where to direct efforts and measure effectiveness. It also makes it difficult to hold leaders accountable because there is no way to monitor how leaders are dealing with incidents of harmful behaviour.

An organisation-wide data improvement plan is needed

6.20
Improving the collection and management of data and information is a long-term investment and will take time. In our view, development of a data improvement plan should be a priority for NZDF and needs to cover:

• what data needs to be collected;
• how to improve the collection of current types of data and close any gaps; and
• how to collate the current types of data.

6.21
Data about harm that has occurred is sensitive. A robust framework is required to ensure that confidentiality and privacy is managed appropriately as more data is collected and collated. This needs to include clear guidance on who can access different forms of data and for what purpose. This is wider than Operation Respect but needs to be improved substantially to inform Operation Respect's work.

6.22
Improvements are also needed to the complaints data. In our view, there needs to be an NZDF-wide system that records complaints and the actions taken in response. Ideally, all complaints brought to command (be they harmful sexual behaviour or bullying) would be recorded in the same system. This data would also ensure that there is a record of those who have engaged in harmful behaviour to inform decisions about career progression.

6.23
Leaders need some data to help them identify emerging issues so they can intervene early. That data needs to be brought together regularly. The data improvement plan needs to set out how these different data sources can be collated and presented to different parts of the organisation.

6.24
The SAPRA data is being moved to a system used by NZDF social workers and chaplains called Profile. We were told that this will allow easier interrogation of data. In our view, this is sensible because it should allow easier collation with other sources of data.

6.25
There are other sources of data that could be anonymised and brought together regularly to help leaders to understand risks and issues, and monitor and measure change. Other data sources that could be useful are:

• exit interview data;
• PULSE data, which includes specific questions on harmful behaviour, that leaders currently find useful;
• complaints data; and
• medical data (psychologist and doctor), where appropriate.

6.26
The data improvement plan needs to identify how NZDF will fill its current gaps in data and information, including how it will record data about non-criminal harmful sexual behaviour and bullying, harassment, and discrimination incidents. In some locations, leaders have tried new methods for anonymous reporting. However, in our view, an organisational approach is required so data can be collated from across NZDF.

6.27
There are a variety of contributing factors that can make harmful behaviour more or less likely to occur. During our audit, we heard about many of these. We saw that leaders and specialist support personnel working in teams or units where harmful behaviour occurs often have a good understanding of what the risks and protective factors are in their units. Although all situations are unique, the lessons learned could be valuable to those in other parts of the organisation.

6.28
Collating information about situations where harmful behaviour has occurred would provide an opportunity to build institutional knowledge over time about the risk factors for harmful behaviour and the effectiveness of different responses. This is not currently happening.

Leadership of data management is needed

6.29
It is clear that senior leaders want to see improvements in data. However, progress has been slow. It was not clear to us who was responsible for improving data collection and use. We were told that it would be sensible for the data management system to be managed centrally but it is currently owned by different parts of the organisation. For progress to be made, there needs to be clear allocation of responsibilities and appropriate governance oversight.

6.30
Establishing new systems and processes for data collection will take time. In the meantime, there are ways that leaders can better understand what is occurring on their camp or base, or in their unit. For example, specialist support personnel working together to provide information about trends they are seeing across their camp or base. This could be a useful way for commanding officers to understand what is occurring until more systematic methods are established. We saw this happening on a few camps and bases.

6.31
One of the best ways for leaders to understand what is going on in their units is by encouraging their people to raise issues through the chain of command. The extent that commanding officers felt this was happening varied and appeared to depend on the ability of those below them to identify harmful behaviour. Commanding officers need to create safe forums where people can speak about these issues, and be clear with their subordinates about the type of information they want to receive, so they can monitor risks and issues in their units.