Part 4: The Research for Industry programme
4.1
In this Part, we:
- describe the RFI grant programme; and
- discuss our findings from the applications and grants we audited.
Overview of the Research for Industry programme
4.2
RFI is the largest grant programme administered by the Foundation, accounting for nearly 46% of total grants funding in the 2005-06 appropriations. RFI is part of the Foundation’s wider funding of public good science and technology. Public good science and technology is defined in the Act as science or technology that:
- is likely to increase knowledge or understanding of the physical, biological, or social environment;
- is likely to develop, maintain, or increase skills or scientific or technological expertise that is of particular importance to New Zealand; or
- may be of benefit to New Zealand, but is unlikely to be funded, or adequately funded, from non-governmental sources.
4.3
The purpose of the RFI programme is to increase the competitiveness of New Zealand industries and sectors through strategic research. Recipients of RFI funding are predominantly public sector institutions, including Crown Research Institutes and universities. Other recipients include specialist research companies, and not-for-profit research centres.
4.4
RFI funding is directed into 4 broad areas:
- research, with the main goal of advancing food and fibre-based industries and sectors through innovation;
- research, with the main goal of advancing manufacturing and services industries and sectors through innovation;
- research to improve infrastructure that supports economic development; and
- research consortia that help public/private partnerships to increase private sector investment in New Zealand.
Our RFI audit sample
4.5
We audited a sample of RFI grants directed to the food and fibre industries and sectors, because this area of funding makes up nearly 60% of the total value of RFI grants.
4.6
Between 1 July 2003 and 30 June 2005, 167 RFI (food and fibre) grants were awarded, worth a total of $131 million (including GST). These were in the food and fibre “portfolios” (or subject areas) of biological industries, sector sustainability, and innovative foods. We audited 37 of these grants, equivalent to 10% of the total funding allocated to RFI (food and fibre) grants during the period.
4.7
RFI grants typically span several years, so our audit sample included newly funded applications as well as grants that had been continued from funding rounds before 1 July 2003. Our audit sample is summarised in Figure 9.
Figure 9
Audited Research for Industry (food and fibre) grants
| |
Audit sample | Total* | Sample as % of total |
|---|---|---|---|
| Number of grants | 37 | 167 | 22% |
| Value of grants (including GST) | $12.8m | $131.0m | 10% |
* Total number of RFI (food and fibre) grants allocated by the Foundation between 1 July 2003 and 30 June 2005, and the value of the funding allocation during that period.
4.8
We did not ascertain if the grants in the RFI programme met assessment criteria, because the applications tend to be very complex and scientific in content. However, we did audit the process used by the Foundation for assessing applications, to see whether the process was sound.
The approval process
4.9
The Foundation allocates RFI grant funding in “investment rounds”. There is a contestable process based on a comparative assessment of written applications. The Foundation has limited funding it can allocate to RFI grants in a financial year. In the RFI investment rounds that we audited, there were more than twice as many applications for funding than there were funds available.
4.10
Before an investment round starts, the Foundation publishes a Request for Proposals document for potential applicants.1 The Request for Proposals sets out:
- an overview of portfolios for which applications are requested, and the application process;
- the “investment signals”2 for the particular portfolio; and
- submission instructions, which set out for applicants what information needs to be included in their application.
4.11
The Foundation requires applicants to submit applications by a due date. The documentation provided by the Foundation to applicants was comprehensive and clear. The Foundation’s expectations and decision-making criteria were clearly defined.
Assessment of applications
4.12
Foundation staff first review applications to ensure that the minimum information requirements are met, before the applications are formally assessed by a reference group.
4.13
Reference groups are made up of external advisers to the Foundation, selected for their expertise and knowledge in the portfolios covered by a particular investment round. They do not have funding decision-making powers. They make recommendations to an Investment Sub-committee, which is a sub-committee of the Foundation’s Board. The Investment Sub-committee appoints a reference group chairperson from a pool of nominations put forward by Foundation staff. The chairperson is then responsible for recommending the composition of the reference group to the Foundation. The names of reference group members are published on the Foundation’s website.
4.14
At various stages throughout the application process, applicants may be invited to provide further information or to answer any questions that reference group members or Foundation staff may have.
4.15
Reference group members must:
- understand the assessment criteria;
- read all assigned applications fully, and the executive summary of all other applications to obtain an overview of the pool of applications;
- complete and record scores and comments regarding applications;
- contribute to equitable and defensible decision-making processes; and
- take collective ownership of the reference group recommendations (and the processes used to rank and select applicants).
4.16
The Foundation has clear and well-defined procedures for dealing with confidentiality issues and potential conflicts of interest. All reference group members must sign a confidentiality agreement, and declare any conflict of interest with an application. Applicants can specify special confidentiality requirements for all or part of their application. This can result in a reference group member not receiving an application, or being excluded from deliberations on an application.
4.17
Conflicts of interest are recorded in a conflicts register. This sets out the nature of the conflict and the action taken to address the conflict (such as not taking part in the discussion of the application, or leaving the room when the application was discussed). The conflicts register is included as part of the paper submitted to the Investment Sub-committee, setting out the recommendations of the reference group.
4.18
The reference group assesses each application against a set of criteria (see Figure 10). As noted earlier, because of the complex and scientific content of most RFI-funded projects, we did not determine if grants met the criteria.
Figure 10
Assessment criteria for Research for Industry applications
| Benefit to New Zealand through innovation |
|---|
| Applicants are required to demonstrate how the research will create wealth through direct commercial returns and potential public returns as well as the expected environmental and social returns and risks. |
| Science merit |
| Applicants must demonstrate the rigour, originality, ‘stretch’, and soundness of the methodology of the proposed research. |
| Future human/provider capability |
| Applicants must show how scientific or technological ‘stretch’ develops new skills and knowledge within the team in a way that does not just duplicate skills already in place elsewhere in New Zealand. |
| Users’ capacity to innovate |
| Applicants need to show how their research will improve the ability of research users to understand and manage the benefits of the research to achieve their particular goals. |
| User connections and partnerships |
| Applicants should demonstrate evidence of user commitment to the research through formal or informal partnerships – it is preferable that end-users are involved in the design of the research from its inception. |
| Pathway to implementation |
| Applicants need to demonstrate a clear path to achieving returns and to demonstrate a convincing route to achieving uptake of the results of the research in a way that maximises benefits to New Zealand. |
| Existing delivery capacity |
| Applications must show the ability of the science provider or research consortium to pull together the best team to carry out the research tasks and to implement the results of its research. |
4.19
Each criterion is assessed on a scale of 1 to 7. Typically, a score of one indicates that the application has not demonstrated any worthwhile contribution to research, science, and technology, while a score of 7 indicates an outstanding contribution. A score of 4 indicates that the application has satisfactorily met the major requirements of the criterion. Each criterion is weighted independently, so specific criteria can be emphasised in different portfolios.
4.20
Each application is scored against the criteria, and then considered in relation to the other applications. The assessment is made in terms of a proposed project’s contribution to the overall investment in the area, as well as taking into account the overall balance of the portfolio.
4.21
The reference group agrees a score out of 7 for each of the assessment criteria, to produce an aggregated score for each application. If required, reference groups may get applicants to provide additional information as part of the assessment process. They may also seek an external peer review for any aspect of a proposed project if the reference group is unclear about its scientific merit.
4.22
We are satisfied that the process used to assess applications was sound. The use of clear decision-making criteria and weightings, reference groups, and external peer review (as required) ensured that all applications were rigorously scrutinised, and carefully debated and considered.
Approval of applications
4.23 The reference group makes written recommendations to the Investment Sub-committee of the Board, after all the applications have been assessed and ranked. The Investment Sub-committee has ultimate responsibility for deciding whether a grant is awarded. The papers we examined, that set out to the Investment Sub-committee recommendations of reference groups for the investment rounds, were clear and comprehensive.
4.24
Figure 11 summarises the RFI approval process.
Figure 11
Approval process for Research for Industry grants
Documentation
4.25
Overall, the file documentation for RFI grants was harder to follow than for the TBG and GPSRD grants. This was mainly because:
- identification numbers assigned to RFI applications were not subsequently linked to approved grant contracts; and
- some information about the grants was contained in different electronic databases, which were not consistently linked.
| Recommendation 6 |
|---|
| We recommend that the Foundation for Research, Science and Technology use a consistent file identification and management system for its Research for Industry grant programme, so that applications can be easily linked to approved grants. |
4.26
RFI grants are administered primarily using a paper-based system within the Foundation, with only small amounts of grant or client information stored electronically. This contrasts with the TechNZ schemes (including GPSRD and TBG), which have been designed to be mainly administered electronically.
4.27
However, we understand that the Foundation is seeking capital funding from the Government in 2006 for a project to integrate all of its grant programmes and investment rounds into a single computer-based administration system. An integrated system would help improve the consistency of data collection for all the grant programmes. It should also address our recommendation that the Foundation uses a consistent file identification and management system for all of its grant programmes. An integrated system should also enhance the Foundation’s ability to effectively monitor and evaluate the effect of the grants it awards.
Monitoring Research for Industry grants
4.28
RFI grant recipients are required to submit annual performance reports to the Foundation, describing the progress of funded projects against their contracted objectives. The reports are collated into a report to Parliament each year, and are also published on CD-ROM by the Foundation. This reporting requirement was complied with for all the RFI grants we examined.
4.29
Aside from the annual performance reports, we found little evidence on file of regular contact by the Foundation with grant recipients. However, interviews with Business Managers indicated regular liaison takes place between Foundation staff and RFI grant recipients.
4.30
We note that the Foundation has recently changed how it interacts with its grant recipients. The change has included a move from annual monitoring of grant recipients to quarterly exception-based monitoring. All new RFI grants since June 2005 require quarterly reporting, while existing contracts are being progressively transferred to quarterly reporting as they are renewed. The shift to quarterly reporting is prudent, given the large amount of funding involved with RFI grants. It should also strengthen the Foundation’s monitoring of the RFI programme.
4.31
Features of the new quarterly reporting system for RFI grants are summarised in Figure 12.
Figure 12
Performance and reporting requirements for RFI grants*
| Requirement | Description |
|---|---|
| Quarterly reports | Exception-based reporting, covering all critical performance indicators. |
| Fourth quarterly annual report (to be completed by 31 July of each year) | To cover all critical performance indicators and:
|
| Statistical information | Including basic profiling data as reasonably required. |
| Miscellaneous | Information that would enhance the
Foundation’s understanding of the work
programme. Any other additional reporting requirements as specified in individual contracts between the Foundation and grant recipients. |
* Since June 2005 for new RFI grants, and progressive introduction for other existing grants.
Evaluating the Research For Industry programme
4.32
A range of studies and research has been undertaken in recent years that includes the evaluation of aspects of RFI grants. However, the RFI programme as a whole has not been evaluated. Aspects that have been evaluated include:
- RFI output class evaluation case studies by Foundation-commissioned consultants in 2001 and 2002;
- an evaluation of RFI-Manufacturing by the Ministry of Research, Science and Technology in 2003; and
- various portfolio evaluations undertaken by an evaluation unit of the Foundation between 2002 and 2005.
4.33
Although the Foundation and the Ministry of Research, Science and Technology both undertake or commission evaluation work, their respective roles and responsibilities were unclear to us. We are not aware of any formal agreement between the 2 entities regarding evaluation work.
| Recommendation 7 |
|---|
| We recommend that the Foundation for Research, Science and Technology liaise with the Ministry of Research, Science and Technology to define clearly their respective roles and responsibilities for evaluating grant programmes administered by the Foundation. |
1: In some cases, the Foundation may also require applicants to submit a Registration of Interest to help determine the likely level of funding requested, and the breadth of research ideas covered.
2: Investment signals specify, at a detailed level, the research priorities requested by the Foundation for a particular portfolio or group of portfolios.
page top