Part 5: Immigration New Zealand needs to use data more effectively

5.1
In this Part, we discuss how Immigration New Zealand could:

• gain deeper insights from the information that it holds; and
• better explain the performance and outcomes of the skilled residence visa system.

5.2
We expected Immigration New Zealand to:

• systematically collect and analyse user information to help it understand what is working well and what is not;
• use the data and insights it gains from collecting and analysing information to improve its performance for applicants; and
• develop performance measures and reporting to help staff, applicants, Parliament, and the public to understand how well it is managing its decision-making about applications for skilled residence visas.

Summary of findings

5.3
Immigration New Zealand collects a wide range of information from its applicants. This allows it to monitor the number of applications for skilled residence visas it is processing and how long it takes to process them. This information is important for monitoring the skilled residence visa system.

5.4
However, Immigration New Zealand could use its information more effectively to understand how well its systems are working and to gain deeper insights about its applicants, their needs, and what employment and place of residence they choose once they are settled.

5.5
Immigration New Zealand could include this kind of information in its annual performance reporting. Including a wider range of data could improve the public's understanding of the skilled residence visa system and Immigration New Zealand's role in managing it.

Immigration New Zealand could gain deeper insights from information it holds

Immigration New Zealand should use applicant data to improve its understanding of whether the system is working as intended

5.6
To understand whether the skilled residence visa system is working as intended, Immigration New Zealand needs to systematically monitor how well applicants move through the system, identify bottlenecks in processing applications, and know whether decisions about visa applications meet policy objectives.

5.7
We could not see that Immigration New Zealand routinely monitors applications or groups of applications to understand whether and why some move through the skilled residence visa system quicker than others.

5.8
For example, Immigration New Zealand does not regularly report on whether the system is working for different groups as intended or achieving policy objectives. In our view, it would be helpful for Immigration New Zealand's reporting to include analysis of application processing information by categories such as nationality, gender, or occupation.

5.9
These insights would allow Immigration New Zealand to check for patterns in how it processes applications. It could use this understanding to improve how the system works for applicants by adjusting its practice as needed.

5.10
To illustrate the potential use of Immigration New Zealand data, we analysed a sample of application data for skilled residence visa applications from September 2022 to December 2023, to identify how occupation affected wait times and outcomes for applications for skilled residence visas.³⁵

5.11
Figure 6 shows that some types of occupations experience faster processing times than others. This was true even when controlling for age, gender, and nationality. For example, from the sample we looked at, we can see that Carers and Aides are consistently being processed at a faster rate than all other occupations, while Design, Engineering, Science, and Transport professionals consistently experience the slowest visa processing times of occupation groups, below the mean processing time of the visas we looked at (of 4.2 months). The width of the bars in Figure 6 gives an indication of the variability in processing times for the different occupation types.

Figure 6
Likelihood of faster visa processing, by occupation type, from September 2022 to December 2023

Source: Immigration New Zealand's data.

5.12
We also saw that some groups of occupations were more likely to be approved than others. For example, although general approval rates were high for the period from September 2022 to December 2023 (a 98% approval rate), we could see that occupations under the category of health professionals had their applications approved at a higher rate than other occupation groups such as ICT professionals. This was true even when controlling for age, gender, and nationality.

5.13
In our view, this type of analysis could help Immigration New Zealand to understand whether its system is working as intended. Even if processing times and outcomes overall seem to be working well, this kind of analysis could help to identify and fix problems in specific areas and to check the effect of any policy changes.

5.14
For example, since we did this work, approval rates have decreased for skilled residence visas.³⁶ The kind of analysis we have done, repeated regularly, would help Immigration New Zealand to understand whether particular factors are causing these changes in approval rates (for example, the introduction of new visa types, new processing instructions, or characteristics of applicants, such as occupation or nationality).

5.15
Publishing this kind of information could also improve the public's understanding of how well the skilled residence visa system is working for applicants.

Immigration New Zealand could use information to better understand why applications are delayed

5.16
In our view, Immigration New Zealand could also use information from the application processing system to better understand bottlenecks or issues in the processing of applications for skilled residence visas.

5.17
When an Immigration Officer needs further information from an applicant and/or a third party before they can continue assessing the application or make a decision about it, the application becomes "non-actionable".

5.18
Immigration New Zealand currently uses the Applicant Management System to assess applications for skilled residence visas. There are standardised ways of recording why an application is non-actionable, called "bring up reason codes" (reason codes).

5.19
A reason code should be accompanied by a "bring up due date" to specify when the application will once again become actionable. These dates help Immigration Officers manage their workloads.

5.20
When we looked at the data, the four reason codes given for information most often missing from applications, from most to least frequently, were:

• further information requested from the applicant;
• a foreign police check (supplied by the applicant);
• a National Security Check; and
• a medical assessment.

5.21
Some non-actionable applications did not have accompanying reasons.

5.22
In our view, reason codes could be used to help Immigration New Zealand to gain further insights about the decision-making process. Using them consistently, and analysing their use, could help Immigration New Zealand to better understand what factors most commonly hold up applications. It could then use this understanding to inform its guidance for applicants on how to improve their applications.

Immigration New Zealand needs to better explain the performance and outcomes of the skilled residence visa system

Annual performance reporting should give the public a good overview of an organisation's performance

5.23
We wanted to see whether MBIE was applying good practice to its annual performance reporting on the work of Immigration New Zealand.

5.24
In our view, good performance reporting is:

• concise, relevant, and understandable;
• focused on what matters to users;
• balanced about what went well and what did not; and
• capable of giving a good overview of the organisation's performance.³⁷

Annual performance reporting about immigration does not consistently focus on the skilled residence visa system

5.25
MBIE reports annually on several measures about visa application processing. The reporting includes measures of applicant satisfaction and the quality of decisions about residence visas. MBIE publishes this information in its annual report and on Immigration New Zealand's website.

5.26
The published performance information for 2022/23 that is relevant to applicants for skilled residence visas included:³⁸

• the percentage of visa applicants satisfied with the overall experience of applying for a visa;
• the quality of residence visa decisions: percentage rated as accurate;
• the average monthly percentage of new registrations entered into the New Zealand Now database whose occupations match those at skill levels 1-3 under the Australian and New Zealand Standard Classification of Occupations;³⁹ and
• the percentage of recent migrants who feel that New Zealand is their home.

5.27
These performance measures provide high-level information about Immigration New Zealand's work from both a process and applicant perspective. They focus on outcomes that matter to users, such as how satisfied people are with the process of applying.

5.28
However, in our view, the information does not show how well the skilled residence visa system is working. None of the current measures have this focus because they do not consider skilled residence as a specific category. The measure of skill levels gets the closest to doing this, but covers people who register their intention to live and work in New Zealand rather than visa applicants specifically.

5.29
We consider that the current lack of reporting specifically about the skilled residence visa system is a weakness, given the significance attached to the skills that this group of migrants is expected to bring to New Zealand. In addition, Immigration New Zealand and the public would be better placed to understand the effects of changes to the skilled residence visa system (see Figure 1) if performance reporting about this part of the immigration system were available.

5.30
In our view, there is significant public interest in how well the skilled residence visa system is meeting its objectives, and the current suite of measures that MBIE reports do not show this aspect of its performance.

Some annual performance measures lack clarity and consistency

5.31
MBIE could also do more to make the measures easier for the public to understand. For example, "quality of residence visa decisions: percentage rated as accurate" lacks context about how quality is managed. This makes it hard for a member of the public who does not have specialist knowledge of the immigration system to understand what is being measured.

5.32
MBIE also did not report on the full year for 2021/22 and 2022/23. Partial year reporting makes it harder for the public to understand whether an organisation's performance is improving.

5.33
Changes in MBIE's measures from year to year also make it hard to take a long-term view, especially if the organisation does not clearly explain the reasons for the change.

5.34
In its 2022/23 annual report, MBIE said that its measure for the average monthly percentage of new registrations entered into the New Zealand Now database (see paragraph 5.26) would change to a raw count of the number of registrations for skilled occupations.

5.35
Both of these measures are potentially relevant to understanding the performance of the skilled residence visa system. However, MBIE did not give readers any contextual information about the measures to explain why it changed the measure or how the two measures compare.

5.36
Presenting its performance information in a way that is relevant to its readers would help MBIE to improve the public's understanding of Immigration New Zealand's work.

Immigration New Zealand needs to report more effectively to the public about the skilled residence visa system

5.37
Immigration New Zealand makes a range of data about the skilled residence visa system available to the public through its website and the Migration Data Explorer database.

5.38
Immigration New Zealand explains that the Migration Data Explorer is a tool that replaces the annual Migration Trends report (see paragraph 5.43). It includes information about current and historical skilled residence visa numbers, visa decisions, and the nationality and occupations of people receiving residence visas.

5.39
Although the tool contains a large amount of information about processing and outcomes, it is not a substitute for meaningful reporting to help Parliament and the public understand how well the skilled residence visa system is working.

5.40
In our view, Immigration New Zealand should interpret this information for the public further to help increase understanding of the performance of the skilled residence visa system.

Immigration New Zealand could improve its reporting on the longer-term outcomes of the skilled residence visa system

5.41
Improved reporting on the longer-term outcomes of the skilled residence visa system would help Parliament and the public better understand whether it is working as intended.

5.42
In our view, MBIE could give the public better information about the effectiveness of the skilled residence visa system by collecting longer-term information about outcomes for people who receive skilled residence visas. This could include asking for information about the occupations skilled migrants currently hold and whether they have stayed in New Zealand in the longer term.

5.43
MBIE published this information for many years. The Migration Trends report provided information about the number of permanent residents remaining in New Zealand after five years as a measure of retention. However, when we carried out our audit, the most recent report available on MBIE's website was for 2016/17.⁴⁰

5.44
We were also able to view other reports about longer-term outcomes for migrants on MBIE's website, including:

• Regional migration trends reports from 2016;
• the report Settling in New Zealand: Migrant survey trends from 2015 to 2019;
• summary results from the 2021 and 2022 migrant surveys; and
• a report on the settlement experience of Pacific migrants to New Zealand from 2012/13 to 2016/17.

5.45
These reports contain useful information about migrant sentiment and trends in migrant behaviours.

5.46
In our view, MBIE should regularly evaluate and publish information from sources such as this in one place, with a specific focus on skilled residence visa applicants. This would help MBIE to improve its understanding of whether the skilled residence visa system is meeting the needs of New Zealand and report that information to the public.

35: The dataset included anonymised data for 13,613 applications for the period September 2022 to December 2023, which represents the period of skilled residence visa processing activity from the re-opening of the border to the beginning of our audit. It includes Straight to Residence Visas, Skilled Migrant Category Visas (including the new Skilled Migrant Category Visa from the opening of the scheme in October 2023), and Care Workforce and Transport Work to Residence Visas.

36: For 2023/24, the approval rate for skilled residence visas was about 93%, according to information taken from Immigration New Zealand's Migration Data Explorer (see immigration.govt.nz).

37: Controller and Auditor-General, Audit New Zealand, and the Treasury (2022), Good practice in reporting about performance, at oag.parliament.nz.

38: When we carried out this audit, MBIE's most recently published annual report was for 2022/23. We have taken the wording of these measures from that report.

39: The New Zealand Now database is a website that Immigration New Zealand provides for potential migrants to register their interest in applying for a visa. The five skill levels under the Australian and New Zealand Standard Classification of Occupations refer to the level of skill that is typically needed to competently perform the tasks of a particular occupation. Level 1 is the highest, and an occupation at this level needs the equivalent of a bachelor's degree or higher qualification to perform.

40: See "Migration Trends report" at mbie.govt.nz.