Part 4: Evaluation of the Industry Partnership programme
4.1
In this Part, we outline:
- the types of evaluation and monitoring of the programme carried out by IRD;
- our expectations of effective programme monitoring and evaluation; and
- our findings.
Evaluation and monitoring
4.2
Originally, IRD anticipated building a full statistical model to monitor the outcomes of the programme, but there was not enough historical data available to make this possible. At the beginning of the programme, IRD did not set quantitative measures of the types of activities to be carried out as part of the programme. These measures were also required for statistical modelling.
4.3
A longer time series of information and some activity data is now available, so preparing a statistical model is more feasible. IRD’s strategy and evaluation documents for the programme indicate the intention to prepare and introduce a full statistical model.
4.4
IRD used external experts to analyse the results of the programme using an alternative to statistical modelling. The experts used an approach called “ratio analysis”. This involved comparing trends in 14 measures of tax compliance (some of which were ratios of one measure to another) for industries in the programme with the trends for similar industries outside the programme. This work was carried out in 2003 and 2004. In 2003, the analysis was done at a regional as well as at a national level.
4.5
IRD also analysed trends in various tax compliance measures during the life of the project component of the programme. There was a move away from this analysis over time because of attribution and data quality issues. These issues are described in paragraphs 4.42-4.48.
4.6
IRD did not attempt to monitor trends in the hidden economy as part of the programme.
4.7
A lot of IRD’s data was “cleaned” as part of the programme. The main data quality issue that required data to be “cleaned” related to inaccurate, missing, or outdated Australia New Zealand Standard Industrial Classification (ANZSIC) codes within IRD’s transactional information system. These codes were used to identify taxpayers in industries covered by the programme. IRD identified these data quality issues at the start of the programme.
4.8
During 2004 and 2005, IRD wrote seven papers that evaluated aspects of the programme and specific initiatives within it. The papers evaluated:
- field team activity;
- external relationship management;
- 10 service delivery initiatives;
- research and issues gathering;
- communications;
- data integrity; and
- field delivery.
4.9
A specific research and issues gathering team existed as part of the programme during 2004 and 2005.
4.10
In 2005, a survey was used to assess perceptions of the programme. Two Industry Partnership industries were surveyed, and there was also a community perceptions survey.
4.11
In 2005, IRD reviewed the systems it had in place to monitor the programme. This work was carried out by its Risk and Assurance section. The review concluded that adequate systems were in place to monitor compliance and the success of the programme.
4.12
In 2006, a high-level summary of lessons learned from the programme for the period 2002-06 was written.
4.13
A measurement and evaluation framework is still to be developed for the Customer Insight Group.
Our expectations
4.14
We expected the programme to be effectively evaluated. In short, this requires the results of the programme to be measured and acted on. Specifically, we expected:
- regular industry, intervention, and programme level-monitoring or evaluation;
- the monitoring and evaluation to show positive results in terms of tax collected, at both the industry and programme levels;
- monitoring and evaluation information to be credible;
- results of monitoring and evaluation to be fed back into the design, implementation, and management of the programme;
- monitoring and evaluation to be linked to the programme’s overall and operational objectives;
- attributable results of the programme to be recorded accurately in appropriate reports and documents; and
- the monitoring and evaluation measures and framework to compare favourably with those used by other tax organisations.
Summary of our findings
4.15
IRD’s learning approach to the programme was supported by monitoring and evaluation of results, and this was a strength of the programme at an industry level and overall programme level. The monitoring and evaluation measures used for the programme were consistent with the practices of IRD’s overseas counterparts.
4.16
Less systematic monitoring and evaluation was carried out of Industry Partnership activities across the five field teams – this limited IRD’s ability to develop specific tools and approaches for dealing with the hidden economy.
4.17
Monitoring and evaluation showed positive results in terms of tax collected, tax compliance, and taxpayers’ perception of IRD, but attributing changes in tax revenue and compliance directly to the programme was difficult. In some instances, IRD needed to qualify reported results of the programme but did not do so.
Monitoring and evaluation
4.18
IRD has done a lot of evaluation and monitoring of the programme and it has recognised that the quality and availability of data have constrained this work. As part of understanding the constraints, it has sought external expert advice and has employed internal review mechanisms independent of programme staff.
4.19
There has been a lot of monitoring and evaluation of the programme to inform IRD’s learning approach to the programme. However, there have been some limitations to this monitoring and evaluation. While there has been a mixture of industry, intervention, and programme-level monitoring and evaluation throughout the programme, there has been limited sub-programme level evaluation (that is, at a regional or individual initiative level). This is an issue that IRD has identified.
4.20
In our view, a relative lack of sub-programme level evaluation and monitoring has limited IRD’s ability to develop specific tools and approaches for dealing with the hidden economy.
4.21
In some cases, the internal scrutiny by staff independent of the programme has been less critical than the evaluation carried out by programme staff. IRD may wish to examine the reasons for this so that any lessons arising from the differences can be used to inform future monitoring, evaluation, and review activities.
Results of the programme
4.22
IRD’s monitoring and evaluation show positive results in terms of tax collected and tax compliance.
Tax payable
4.23
IRD’s analysis for five industry groups in 2002/03 and 2003/04 showed the programme had raised tax return filing rates and lowered tax debt in these industry groups. This was based on comparisons of trends in those industries with trends in other industries. The five industries covered by the approach and the associated comparison industry for each are shown in Figure 5. In all cases, the comparison groups were broader industry groups.
Figure 5
Industries and associated control industries
| Industry covered by the programme | Control industry |
|---|---|
| Painters and decorators | All construction |
| Electricians | All construction |
| Plumbers | All construction |
| Smash repairers | Motor vehicle retailing and services |
| Hairdressers and beauty salons | Personal services |
4.24
On IRD’s behalf, external specialists estimated the tax dollar effects of raised filing rates and lowered tax debt in these five industries. The external specialists estimated that tax payable increased by $5.2 million in 2002/03 and by $4.5 million in 2003/04 because of the programme. They also estimated that, if these same results applied in the 62 industries identified by IRD as having relatively high cash economy risks (not all of which were covered by the programme), there would be extra tax payable of $83 million each year, and tax debt would be reduced by more than $90 million.1
4.25
The external specialists estimated the benefit-cost of the programme’s activities in relation to the five industries to be 6.7 over a decade. In other words, the increased tax payable arising from the programme was estimated as 6.7 times the cost of the programme for these five industries if observed for a decade. This assumed that results observed early in the programme continued in later years.
Tax compliance
4.26
Trends in tax compliance resulting from the programme have been provided in a number of IRD’s annual reports. However, care must be taken when interpreting this trend information. While the information describes the trends observed, the trends cannot be attributed solely to the programme. The information is also aggregated for two or more Industry Partnership industries. The stated trend may therefore not be representative of what happened in any given constituent industry. The reported trends in tax compliance were also subject to data quality issues, as outlined in paragraphs 4.42-4.48.
4.27
In its 2003 Annual Report, IRD identified trends in tax compliance between 30 June 2002 and 30 June 2003. For the first two Industry Partnership industries, there were decreases in debt cases, and increases in debt for which an arrangement was in place with IRD. These trends are shown in Figure 6. The same annual report also noted that there had been improvements in the child support profile for Industry Partnership participants in these two industries.
Figure 6
Trends in tax compliance during 2002/03
| Electricians, and painters and decorators | |
|---|---|
| Debt cases | 19.4% decrease |
| Average age of debt | 8.3% decrease |
| Debt under arrangement | 195% increase |
| Outstanding returns | 7.7% decrease |
| Average age of outstanding returns | 10% decrease |
Source: Inland Revenue Department’s 2003 Annual Report.
4.28
IRD’s 2004 Annual Report provided similar information, showing improvements in the debt profile of electricians, and painters and decorators for the first two years of the programme (2002/03 and 2003/04 financial years combined). This information is shown in Figure 7.
Figure 7
Trends in tax compliance during 2002/03 and 2003/04
| Electricians, and painters and decorators | |
|---|---|
| Debt cases | 23% decrease |
| Average age of debt | 16% decrease |
| Debt under arrangement | 258% increase |
| Outstanding returns | 27% decrease |
| Average age of outstanding returns | 20% decrease |
Source: Inland Revenue Department’s 2004 Annual Report.
4.29
In its 2005 Annual Report, IRD identified trends in tax compliance in the eight industries covered by the programme as at the end of June 2005. The report indicated a reduction in debt cases and outstanding returns in these industries during the life of the programme. The report showed that, between May 2002 and June 2005:
- the value of outstanding tax debt decreased by 5%;
- the number of debt cases decreased by 24%; and
- the number of outstanding returns decreased by 19%.
Audit returns
4.30
The hourly rates of return for investigation of Industry Partnership evasion cases are shown in Figure 8 from 2004 to 2007. For some of this period, the rates of return were below the internal benchmark set by IRD. At the beginning and end of the period, the rates of return exceeded the internal benchmark. Rates of return for investigation of evasion cases within Industry Partnership industries have continued to be recorded by IRD after the end of the programme.2
Figure 8
Hourly rates of return on Industry Partnership programme investigation of evasion cases, 2004 to 2007
* The Department did not provide us with data as at 30 September 2006 or 31 December 2006.
Source: Inland Revenue Department.
Prosecutions
4.31
Information IRD provided to us showed that, between 1 July 2002 and 30 June 2005, it classified 38 of 45 prosecutions of Industry Partnership taxpayers as successful. The same information indicated that these 45 prosecutions were one-third of the total prosecutions taken by IRD during the same period. The breakdown of successful Industry Partnership prosecutions by industry type is shown in Figure 9. Half the successful prosecutions were of agricultural contractors.
4.32
IRD sought opportunities, where possible, to get media coverage of successful prosecutions as part of the programme.
Figure 9
Breakdown of successful Industry Partnership programme prosecutions 2002/03 to 2004/05, by industry
| Industry | Prosecutions | |
|---|---|---|
| Number | % | |
| Bricklaying | 1 | 3 |
| Drainlayers | 1 | 3 |
| Electrical Services | 1 | 3 |
| Hair and Beauty | 1 | 3 |
| Landscapers | 1 | 3 |
| Painters and Decorators | 1 | 3 |
| Transport | 1 | 3 |
| Automotive Repairs | 2 | 5 |
| Plastering and Ceilings | 2 | 5 |
| Carpenters | 8 | 21 |
| Agricultural Contractors | 19 | 50 |
| Total | 38 | 100 |
Note: This includes prosecutions that started before the programme but were successfully completed during the life of the programme. Prosecutions have been included where IRD’s information shows that the prosecution relates to an Industry Partnership industry, the prosecution has been successful, and IRD has categorised the prosecution within one of the 2002/03 to 2004/05 financial years.
Source: Inland Revenue Department.
4.33
The value of tax discrepancies identified by the successful prosecution of Industry Partnership taxpayers up to 30 June 2005 was $6.241 million. This was 55% of the total discrepancies covered by all of IRD’s successful prosecutions over the same period. It is important to note that identification of the value of discrepancies does not mean that all of these discrepancies are recovered.
Industry perceptions
4.34
In 2004, two years after launching the programme, IRD evaluated the effectiveness of its partnership with two industries with which it had the longest relationship. This research, based on a small number of interviews and focusing specifically on the programme, was compared with the results of a broader community perceptions survey which interviewed a sample of 300 members from the two industry groups. The results for the target industries were assessed against the perceptions of small businesses as a broad group, to identify the impacts of the programme.
4.35
This research concluded that the programme had been well received by the two industry groups, and that there were several areas where it had made a positive contribution towards compliance and a change in attitude towards IRD.
4.36
The response from the industry representatives we spoke to was positive. We found evidence that the programme was successful in creating a more positive perception of IRD as approachable and willing to help. The IRD’s partnership approach was also consistent with the desire of the industry bodies to create a fairer and consistent competitive business environment through tax compliance, and to promote the perceived integrity of their members in the community.
4.37
IRD has concluded that the programme “altered the business community’s view of[IRD] and generated much goodwill”.
Other results
4.38
We saw some evidence that people within Industry Partnership industries, but outside the tax system, were brought within the tax system through the programme. We did not see any systematic analysis by IRD of the number of new taxpayers attributable to the programme.
4.39
IRD staff told us that the programme had led to better awareness among taxpayers of a variety of tax matters (such as child support obligations and Kiwisaver). While these matters do not explicitly relate to the cash economy, they are potential benefits of the programme.
4.40
IRD learnt from the programme about the effectiveness of a number of approaches and initiatives for managing relationships with industries and targeting the cash economy.
Data accuracy and analysis
Data accuracy
4.41
IRD had a good understanding of factors that have affected the quality of data within its information systems. This includes the quality of taxpayers’ records and the quality of time and activity reporting against the programme. We have not attempted to verify the individual effect of each of these factors. The factors identified by IRD include:
- time constraint targets for call centres;
- a system that permits the entry of invalid data;
- reduced customer contact with IRD as a whole, resulting in more outdated contact data;
- use of incorrect project codes when opening and closing cases;
- problematic reporting of time spent on programme investigations work;
- an ACC decommissioning project (this affects industry coding); and
- data cleansing.
Trend analysis
4.42
The findings of IRD’s analysis of trends, as reported in public documents, could have been better qualified. This is because the stated information attributes all of the reported changes in a measure (for example, the percentage of programme taxpayers in debt) to the programme when this was not the case.
4.43
Early in the programme, IRD carried out a lot of data cleansing. It included removing inaccurate records of taxpayers. Doing so changed the size of the population of a given Industry Partnership industry within IRD’s information system. For example, during 2002/03 IRD removed 2032 records (about 10%) from the population of electricians, painters, and decorators. These 2032 records were then counted as part of the reduction in the number of electricians, painters, and decorators owing tax debt. There were similar issues with the reported results of the programme in terms of taxpayers with outstanding tax returns.
4.44
Other non-programme effects may also be reflected in the reported results. As the Canadian Revenue Authority has stated about tax compliance behaviour in general:
Compliance is sensitive to many factors, such as perception of government, values held by society, the economy, legislation, as well as the public’s perception of our tax system … measures demonstrate the effectiveness of our approach to fostering compliance with tax laws, but also reflect the willingness of taxpayers to meet their own obligations without our intervention.3
4.45
We were unable to replicate IRD’s historical information because the IRD’s information system is live. This means records are added or removed from it over time. We cannot assess whether the records removed by IRD as part of its data cleansing had the same or similar characteristics to those of the remaining records for taxpayers in Industry Partnership industries. If the removed records had the same distributions of tax debt or outstanding tax returns, then the information made publicly available by IRD about these attributes of the programme overstates the results of the programme.
4.46
IRD was aware of issues with attributing to the programme all of the observed change in an Industry Partnership industry.
4.47
IRD needs to be more careful when quoting the results of a programme publicly in instances where there are influences on the results outside the programme. In these situations, we believe IRD should qualify the data so that any attribution limits are clear. In our view, IRD should give special attention to this as it develops a monitoring and evaluation framework and subsequently reports on the results of the Customer Insight Group’s work.
| Recommendation 2 |
|---|
| We recommend that the Inland Revenue Department identify, as part of reporting on the results of a specific programme or initiative that may be affected by a range of variables, those factors contributing to the results that are not attributable solely to the programme or initiative. |
4.48
IRD has told us that the data cleansing work carried out as part of the programme gave it a better understanding of data quality issues and has influenced IRD’s subsequent data integrity work.
Ratio analysis
4.49
Over time, IRD’s ratio analysis will become less reliable. This is because it depends on similar industry groups against which to compare Industry Partnership industries. The technical term for such a comparison industry is a control group. Because control groups were generally broader groupings of industries aligned with a given Industry Partnership industry, there was a risk that the control groups become “tainted” by the Industry Partnership effects. IRD recognised this risk. We did not find evidence of IRD having tested for “tainting” of control groups for the two financial years for which the ratio analysis approach was used, but it had identified “tainting” over time as a risk.
Using the monitoring and evaluation results
4.50
IRD used a set of high-level programme objectives in key documents. However, analysis in these documents was not always explicitly linked to the high-level objectives. Monitoring and evaluation could have been more clearly linked to the high-level objectives.
4.51
At a sub-programme level, monitoring and evaluation activities were not in a structured framework. IRD attempted to address this issue during the programme by requiring a template to be completed before any field initiative could start. The template required information on how the initiative would be monitored and evaluated.
4.52
There was some evidence of the results of monitoring and evaluation having been fed back into the design, implementation, and management of the programme. For example, suggestions were made to IRD’s business initiatives governance board about changes that could be made to IRD’s organisation-wide business as a result of the lessons learned from the programme.
4.53
IRD reviewed its own feedback mechanisms for the programme. In 2005, IRD’s Risk and Assurance section concluded that “processes observed for identifying improvements to the relationships and providing feedback to enhance future partnerships’ successes are effective”.
4.54
While there has been a lot of monitoring, evaluation, and feedback into aspects of programme design, management, and implementation, IRD is not yet at the stage where it can make definitive decisions about the right mix of education and compliance activities for a given operational situation.
4.55
A number of IRD staff told us that the programme has had a major influence on the design and existence of the new Customer Insight Group. We have not examined the link between the design of the new group and the programme. However, the new group and the programme do have a number of common features, including helping and understanding the needs of customers and enhancing customer relationships.
Comparable tax measures
4.56
IRD’s voluntary compliance model and the programme are similar to approaches taken by the Australian Tax Office, although IRD told us that its programme had a stronger focus on relationships. IRD’s Customer Insight approach is similar to an approach being used by HM Revenue and Customs in the United Kingdom.
4.57
Early in the programme, IRD contacted other tax jurisdictions about how they measured the hidden economy.
4.58
The tax measures used by IRD for the programme are broadly consistent with those used in Australia and Canada. They were also consistent with an approach recommended by the Australian National Audit Office for the Australian Tax Office for work on the cash economy.
4.59
The Australian National Audit Office recommended that the Australian Tax Office measure:
- the underlying movements in revenue collection (controlled for impacts outside the tax authority’s work);
- the change in attitude of industry participants and consumers over time; and
- the tax payments of treated entities compared with those of non-treated entities (that is, outside the programme of work being measured).
4.60
IRD has used all three of these approaches, but has not been able to exclude non-IRD influences on underlying movements in revenue collection when carrying out trend analyses. While IRD was not able to exclude non-IRD influences on revenue collection, it sought and used expert advice. In our view, IRD took appropriate steps to address this issue.
1: These figures exclude any demonstration effects of the programme (that is, the changes in other taxpayers’ behaviour because of contact with the taxpayers targeted by the programme).
2: IRD’s reporting of rates of return takes into account only the additional tax identified for up to and including the next tax period. At the time of our audit, IRD was developing a process for measuring the dollar value of future tax outcomes (that is, outcomes beyond the next tax period).
3: Canadian Revenue Agency, Annual Report to Parliament 2005-2006.
page top