Data in the public sector

One of the themes guiding our work recently was information and how it is used to improve public services.

As part of that Information theme, we carried out some research to understand the bigger picture of how public organisations were using a particular form of information – the data they hold about the services they provide and the people they serve.1

Guide to the articles

Data leadership imageOur first article describes the leadership arrangements put in place to lead and encourage the smart and effective use of data by public organisations. Those arrangements are referred to as “functional leadership” – they cover many different public organisations and different aspects of managing data well. It’s clear that particular leadership qualities are needed for data to be used to best effect.

Building data capability imageIn our second article, we describe what has been learned about the capability and capacity needed to use data well in public organisations. A wide range of skills are required for mature data management, data analysis, and data use. With finite resources, organisations are often looking in vain for what is often described in the industry as a “unicorn” – one person who has all the technical, analytical, and communications skills necessary to make great use of an organisation’s data. This article describes some of the capability and capacity challenges organisations are grappling with, and solutions people are finding.

Sharing data imgaeThe third article covers the challenges of collaborating to share data. Public sector organisations are increasingly expected to work together, across organisational lines, to design and deliver services. Rather than focus on what an organisation does, the emphasis is increasingly on what New Zealanders need.

To deliver these joined-up services, public organisations need to improve the way they use and share information. We’ve heard about the challenges, but also learned about some myths that can get in the way of progress. We identify some examples of collaboration that went well, and the factors said to be important to success.

Data security imageOur fourth article focuses on data public organisations hold about their own finances, and is based on work we do during our annual audits of public organisations. Specifically, we describe the findings of our work looking at data security during our 2016/17 audits. Although we didn’t find any substantial data security issues, we did find many basic weaknesses in security controls and procedures that increase data security risks. These basic weaknesses are matters we have raised year after year. In our view, they need to be taken more seriously by leaders, governors, and senior managers if data security risks are to be properly managed.


Data helps the government and public organisations – including all the organisations funded by rates or taxes – to understand where to focus their effort, how well services are delivered, and how policy affects people and their lives. It is critical to improving public services.

Better use of data is also required for accountability, and the government’s stated priorities of economic growth, sustainability, and environmental protection. To make the most of it, data needs to be valued, shared appropriately, open where possible, and managed and governed well.

It is essential that public organisations get the balance right between making information accessible and ensuring that there are enough protections and safeguards in place. A security failure, using inaccurate information, or breaching someone’s privacy can lead to a loss of trust and confidence in the public sector. 

What we looked at

We’ve looked at the leadership arrangements that are expected to help public organisations use data and share it appropriately, what helps agencies to use data wisely and well, and what gets in the way of using data to provide New Zealanders with better services.

How we did the work

We looked at the use of data in 20 public organisations, based on what we knew and learned about their use of data analytics. The 20 organisations included, for example, the Association of Local Government Information Management; Callaghan Innovation, Department of Internal Affairs; Inland Revenue Department; Land Information New Zealand; Ministry of Business, Innovation and Employment; Ministry of Health; Ministry of Social Development; Statistics New Zealand; and the Treasury.

In the 20 organisations, we spoke to chief executives, deputy chief executives, assistant commissioners, chairpersons, directors, general managers, the Government Chief Archivist, the Government Chief Privacy Officer, chief information officers, chief data scientists, and data analysts.

We asked these people what “leadership” meant in their roles. We asked about capability (whether they had the right skills available) and capacity (the size of data teams). We asked about how collaboration and data sharing were working in practice. And for each of these topics, we asked people to tell us what they saw as the main challenges and opportunities.

We ran a workshop that posed similar questions. The workshop was attended by second- and third-tier managers from throughout the public sector whose roles involved data (such as business, customer, and predictive analytics). They included lawyers, chief information officers, chief data scientists, general managers, intelligence operators, and information technology specialists.

We reviewed:

  • data initiatives carried out by the Ministry of Health and the Department of Internal Affairs; and
  • documents produced by the Department of Internal Affairs, Statistics New Zealand, and other organisations that provide more general advice and information on data-related matters (such as data sharing, privacy, and security).

We also considered the recommendations we made after our 2016/17 audits about data security controls for financial data.

What we want to achieve

Our aim in carrying out this work was to make a positive difference for those striving to use data to good effect. We knew from our audit work that some data projects had been successful and gone well, and others had struggled.

For data to be used to improve services to the public, the right sorts of leadership arrangements need to be set up, we need public sector workers with the right skills and experience, and we need public organisations to collaborate effectively.

We also need public organisations to take care of the data they hold and keep it secure. We focused on the data we rely on for audit purposes and the need to keep that data secure (from being lost, corrupted, or accessed inappropriately).

We hope that the insights we are sharing through these articles will help those charged with leadership – either in a system-wide leadership role or in an organisation that is trying to use data well.

We thank the people we interviewed for the insights they shared with us, and the work they are all doing on using data to improve services for New Zealanders.

1: Data needs to be analysed and interpreted before it becomes useful information.