Sharing data

Collaboration matters – it’s essential for designing services that put the needs of New Zealanders at the heart of the process, not at the end of it.

Why do public organisations need to share data?

Multiple public organisations can be involved when people move to or from another country, start a family, purchase a property or business, formalise or end a relationship, gain an education, retire, and when they pass away.

Making people deal with one public organisation after another just to get through these predictable and expected life events is less than ideal. The government has recognised that, by working more collaboratively between public organisations and with communities and others, it can design services around people’s needs and keep up with their expectations about easy, sensible, and often mobile services.

Aspects of these “joined up” services are still provided by several different public organisations, but the service user has one online place to go to.

To make these services work smoothly, the data held by government organisations often needs to be available to others – in ways that keep it secure and do not compromise people’s right to privacy.

Why is collaboration difficult?

When we interviewed people working with data or responsible for it, they all supported collaboration and sharing data to improve services – and ideally, outcomes – for New Zealanders. They also said that doing so was often fraught with complexity.

Different levels of data maturity

One of the challenges is about maturity levels, which we discussed in the article on building data capability. If the organisations that need to share data have different levels of sophistication and readiness to manage large data sets, it can cause problems.

For example, data experts need to know that the data they receive will be reliable and up to date, and won’t compromise anyone’s privacy. They need to know that the data they send will be governed and used appropriately, including keeping it secure.

Different types of organisational culture in managing data can also get in the way. A few people we interviewed admitted that part of the reluctance to share data can be due to concerns that other agencies will use data in ways that the “source” agency hadn’t thought of, but probably should have.

Agreement about the right approach

Grappling with data maturity mismatches assumes that people have got past the initial hurdles, which can take years; figuring out and agreeing what can and should be shared, how best to do it, which public organisations will benefit most and therefore which should carry the costs of the work, and how that funding is managed.

Although public sector workers are encouraged and expected to collaborate by very senior public servants and functional leaders, they told us that they can then be deterred by risk-averse managers and leaders in their own agencies:

We have been told to work together – that message just seems to get lost as it goes through the system.

Agreement about what’s allowed

Sometimes, the obstacles aren’t about caution but different interpretations of the policy and legislative settings. Some people told us about the frustration of being certain that sharing particular data in a particular way was entirely appropriate, but failing to convince others at more senior levels. When people have different views about the policy and legislative settings, data experts don’t have the necessary authorisation to collect, share, and reuse data.

One of the difficulties mentioned most often was privacy – people can have different ideas about what’s acceptable from a privacy point of view. People don’t always understand what’s allowed and assume that the legislation is more restrictive than it is.

In a briefing to the incoming Minister, the Privacy Commissioner wrote that the Privacy Act provides ample scope for the sharing of government-held data and information. In his view, the main barriers to sharing are operational – misunderstandings or uncertainty about the law, the inability of IT systems to connect, security concerns, costs, and public organisations having different priorities.

Often, the view that “We can’t do that” is incorrect, and people told us that having agreed principles for sharing data can make it easier to resolve those perception issues.

Some people spoke to us quite passionately about their frustration with public organisations, their own and others, setting limits on how individuals can access and use personal data about themselves:

I use Manage My Health app on the phone. This has my entire health record. … I can’t take my own data and do anything with it beyond emailing my doctor. I can’t bump it to you. I can’t send it. I can’t get my immunisation record and provide it to someone else through this mechanism. However, this is my data. I own it.

And:

We are seeing privacy and security as a constraint but actually it’s an obligation. We already have the obligation. What we don’t have is automation, making people’s data and information available to them.

This reinforces the importance of involving from the very start the people who will use or benefit from a service – when identifying the problem to be solved, through the design stages, to the user testing stage near the end. When people trust that their data will be used as they have agreed, they are likely to be more comfortable with its use.

Agreement about who’s paying

People told us that one of the biggest challenges in collaborating is financial. There is no “cross-entity funding stream” (specific funding set aside) for this sort of data collaboration work.

There used to be such funding: the “Better Public Services Seed Fund” helped fund a few successful collaboration data projects, such as SmartStart. That fund no longer exists.

The main barriers to sharing are operational – misunderstandings or uncertainty about the law, the inability of IT systems to connect, security concerns, costs, and public organisations having different priorities.

Without cross-entity funding streams, people told us that they weren’t sure how other cross-organisation projects might be funded.

Although there are other funds, such as the Justice Sector Fund, they tend to be temporary, annual, and require a lot of effort to set up. Uncertainty about whether such funds will continue isn’t conducive to public organisations collaborating. People told us that funding for collaboration that sits in one particular public organisation is unhelpful when the organisation that receives the benefits has to release funding (and control) to the organisation delivering those benefits.

We were also told that funds designed specifically to foster collaboration also allow for innovation – there can be more testing of ideas on a relatively small scale, without the need for lengthy business cases.

Once collaborative projects are complete, there are difficulties in funding them to continue. Organisations struggle with how best to share costs when the “system” is designed around Votes and output classes for individual public organisations. We heard that:

The big issue is the funding piece. We are funded down into outputs, which are very specific to an organisation. How do you make that funding more about outcomes that force people to work together? Everyone wants to collaborate, there is just no incentive to.

People also told us about unclear expectations and incentives:

My sense is that there is great intent at chief executive level, there is kind of intent at deputy chief executive level, then you get down to specific accountability in my level and it’s difficult. It all comes down to how you’re measured, KPIs. … We need to change what we’re rewarded for.

In our view, more thought needs to be given to how funding mechanisms and accountability expectations can be better designed to support public organisations in working together.

Agreement on the difficult details

Without an agreed set of standards for data, sharing between public organisations can be frustratingly difficult. For example, for organisations to check that they are matching data about New Zealanders accurately, they need certain identifying fields to match.

We’ve probably all had experience of the different approach that online forms can take to address details. Some have separate fields for the house or apartment or unit number; others include that information along with the street name. Some let you write the suburb in, others have a restricted list of recognised suburbs that you select from.

It can take months for public organisations to agree how they will resolve these sorts of data field clashes, because there are time and cost implications in changing the data fields. And they can’t start any meaningful work to improve the services they offer to New Zealanders until these fiddly but essential details are sorted out.

The Chief Data Steward is working on data standards (starting with addresses, x and y co-ordinates, and names) as part of their functional leadership role (see the article on data leadership).

Identifying the “success factors”

Although there are challenges to collaboration, they can be overcome. For example, there have been some notable successes when councils work together, such as the collaboration between councils in Canterbury to share geospatial data. As well as helping the councils, the public can search, view, and interact with spatial information from any computer, tablet, or mobile device. Users can search for properties, see the topography, and locate water services.

We interviewed people involved in three cross-agency projects focused on designing better services for New Zealanders: SmartStart; an end-of-life project Te Hokinga ā Wairua; and a project to tackle smoking rates among young Māori women (the three projects).

We asked those involved about the factors they thought had contributed to the projects’ successes. Here’s what they said matters the most.

Governance and sponsorship

For data-sharing projects to work, the senior executives in each public organisation need to support the work, and there has to be an effective level of governance.

Governing projects can be particularly difficult when it involves more than one public organisation. We were told that having the right people with a can-do attitude at the table is critical to a project’s success.

Principles for managing the data

As well as solid and supportive governance, people told us that they need clear principles for using and managing the data. When the principles are agreed early, it’s easier to work your way through the difficulties that can arise along the way.

Those principles have included, for example:

• data will be collected once and reused many times;
• data will be compiled in real time from existing data sources;
• responsibilities for managing data will be outlined clearly and comply with legislation and standards;
• people’s consent will be actively obtained; and
• wherever possible, data will be open.

Clear and shared vision

All three of the projects we looked at had clear vision and mission statements, and a view of the outcomes they sought. Importantly, that clarity was widespread – people involved in the three projects knew what success was going to look like.

When people talked about “citizen-led design”, they often meant letting New Zealanders tell them what the problem was, not just involving citizens later in the design of a solution. We were told that it was important never to assume that an organisation’s understanding of “the problem” was actually the right problem.

Privacy by design

The three projects were proactive about privacy, seeking guidance and help from sources such as the Office of the Privacy Commissioner and the Government Chief Privacy Officer at the beginning of the projects. Many people we interviewed told us that having privacy experts “at the table” helped to dispel misunderstandings about privacy and legislation.

Privacy by design also meant seeking permission and consent from those who would use the service. For example, the project for young Māori women who smoke was carried out with a mindset of being for and with the people, culturally responsive and reciprocal. For that service, women’s permission is sought at each stage – their privacy isn’t compromised because they actively agree (or not) at each step.

The right team

One of the most important success factors in the three projects mentioned earlier was said to be bringing together the right group of motivated and committed people. That meant the right people from a range of public organisations, the private sector, and non-governmental organisations. The mix of capabilities and disciplines included data analytics, researchers, story tellers, IT architects, legal advisers, privacy experts, and service designers – all of whom had equal importance at the table.

We were told that the teams needed to be demanding, flexible, disciplined, and creative.

Access to existing and reliable data sources

Many people mentioned “the IDI” as a successful initiative. It’s the Integrated Data Infrastructure – sets of data held by Statistics New Zealand and collated from a range of public organisations under strict protocols about confidentiality and anonymity of the data. The IDI lets organisations find and use reliable data so they can make policy decisions based on evidence rather than theory.

The IDI has been described internationally as a success for New Zealand. It is used as an example for other countries to learn from in terms of getting the most from harnessing public sector data.

Many of the people we interviewed talked about using the IDI to share and integrate their data. It holds data about people and households, educational achievements and enrolments, student loans and allowances, travel patterns and migration, health and safety, social benefits, tax, and justice data. The data spans many years and is updated quarterly.

The anonymised data in the IDI is secure, goes through rigorous checking procedures, and the rights to change or amend it are tightly controlled.

That isn’t always the case with sets of data, which we discuss in the next article in this series: what our audits show about data security.