Part 3: Assessing proposals and business cases, and managing conflicts of interest

Ministry for Primary Industries: Managing the Primary Growth Partnership.

3.1
In this Part, we discuss:

how PGP was set up;
how PGP proposals and business cases were assessed; and
how the Ministry managed conflicts of interest.

Our expectations

3.2
We expected that the Ministry would support the IAP to assess proposals and business cases. In particular, we expected to find that the Ministry had:

set clear expectations for potential PGP applicants;
ensured that assessments were based on clearly understood criteria;
established clear and consistent processes for assessing proposals and business cases; and
recorded openly and transparently the rationale for decisions about successful and unsuccessful proposals and business cases.

3.3
We also expected that the Ministry would effectively manage conflicts of interest when they arise.

3.4
To assess whether the Ministry's performance met our expectations, we reviewed documents about programme assessments and decisions to see whether criteria had been applied consistently and assessments recorded transparently. We analysed the process from proposal to the signing of the contract in the six programmes we reviewed. This included reviewing programme proposals, business cases, and minutes of governance group meetings. We reviewed correspondence between the IAP and the Ministry, and we discussed PGP processes and expectations with Ministry staff.

Summary of findings

3.5
The Ministry moved quickly to set up programmes with industry and, during early funding rounds, did not assign enough staff to support the introduction of PGP.

3.6
The Ministry followed the steps in the assessment process that Cabinet defined. However, the conditions and criteria for PGP funding were loosely defined during early funding rounds and were open to interpretation.

3.7
Over time, understanding of how to apply the conditions and criteria to assessments evolved. By 2012, the Ministry had assisted the IAP in reaching a view about how to apply some PGP conditions and criteria and made other improvements to better manage PGP.

3.8
In practice, the IAP's decisions were stated clearly. However, the Ministry did not always clearly and comprehensively record the reasons for the IAP's decisions, including how proposals and business cases met the conditions and criteria for PGP funding. Our review of six programmes showed one example where, in our view, Cabinet's requirement for KPIs in business cases was relaxed. The Ministry did not record clear and comprehensive explanations of why this requirement was relaxed.

3.9
The Ministry has suitable policies and practices to identify and manage any potential conflicts of interest with its staff, members of the IAP, and members of PSGs.

Moving quickly to set up the Primary Growth Partnership

3.10
When Cabinet approved the setting up of PGP, it noted that, in order for PGP to operate in "a timely and pragmatic manner", the governance of PGP will be "relatively light" and will make maximum use of existing mechanisms to deliver investment programmes.

3.11
The Ministry moved quickly to set up programmes. It assessed 69 proposals in the first two years of operation and has received 105 proposals to date.

3.12
In 2012, the Ministry commissioned an independent review (the 2012 review)⁷ to identify potential improvements to PGP processes. The 2012 review acknowledged that the evolution of PGP in its first two years was a notable achievement. However, it concluded that PGP was introduced without "adequate preparatory work in the areas of policy and practice development". The 2012 review stated that, although the Ministry was addressing this, the lack of support had led to some problems, including "ad hoc and inconsistent practices".

3.13
For example, capacity for PGP was initially inadequate. Support for introducing PGP was shared by up to 20 policy analysts, who also had other responsibilities in the Ministry. In 2013, the Ministry commissioned an internal learnings review of PGP (the 2013 learnings review)⁸ to identify how the management of PGP could be improved. The 2013 learnings review noted that Ministry staff felt that they were "helping out", which meant that staff did not feel accountable for the delivery of PGP or always follow proper process.

3.14
In our view, the Ministry did not adequately support the introduction of PGP. This is likely to have affected the Ministry's ability to adequately support the IAP to assess programme proposals and business cases during early funding rounds.

3.15
The Ministry underestimated the size and complexity of commercial negotiations. A theme to emerge from our interviews was that it was sometimes challenging for the Ministry to achieve an equitable balance of skills and experience on PSGs between the representatives of the Ministry and the representatives of the industry partner.

3.16
Over time, the Ministry built capacity and capabilities to support PGP. For example, in 2013, the Ministry set up a PGP Directorate to help manage PGP. The Ministry has also sought an equitable balance of skills and experience by appointing senior Ministry managers to PSGs.

3.17
Additionally, there are independent chairs for some programmes appointed in conjunction with industry partners – for example, FarmIQ and PSH. For other programmes, the Ministry has appointed senior industry specialists as Ministry representatives on PSGs, including a chair on, for example, the PSG for NZSTX. Representation is different for different programmes, depending on the complexity of the programme, the state of the relationship between the partners, and how the partners want to operate together.

Assessing proposals and business cases

3.18
Our review of the six programmes showed that assessments of proposals and business cases followed the steps in the process that Cabinet approved in 2009.

3.19
In practice, the Ministry set up informal working groups known as Business Case Development Groups (BCDGs) of Ministry and industry co-investor representatives to prepare business cases. These BCDGs had no formal authority or formal meetings. Cabinet intended for PSGs, made up of industry representatives and Ministry staff, to be set up to prepare programme business cases.

3.20
The Ministry preferred the BCDGs because they allowed the industry partner to lead the preparation of business cases. The groups also gave the Ministry some distance for assessing completed business cases. The Ministry felt that using BCDGs to prepare business cases addressed Cabinet's intention for the Ministry to prepare business cases with industry partners.

The conditions and criteria for PGP funding were loosely defined during early funding rounds and were open to interpretation

3.21
Until 2012, proposals had a particularly low success rate. The 2012 review attributed the high failure rate of proposals to their low quality. The 2013 learnings review found that there was a lack of understanding in the Ministry about PGP, its intent, and its limitations. The IAP also told us that they were strong in judging intended outcomes of programmes and ensuring that key criteria were met, such as programmes being truly innovative.

3.22
Ministry staff told us that the Ministry failed to explain to industry applicants what PGP was in simple, understandable terms. The Ministry also failed to explain the nature of the partnership that industry was going to be part of.

3.23
In 2011, the Ministry and the IAP identified that helping applicants understand the nature of PGP and what they were looking for in proposals remained a significant challenge. We were told that different interpretations of what the criteria for PGP funding meant contributed to a wide variety and quality of applications.

3.24
The responsibility for helping industry with programme applications was unclear. When the Ministry and the IAP discussed who should help industry with proposals, there was concern that the IAP offering too much help might stifle the ideas and innovation of participants. The IAP and the Ministry decided that it was better for the Ministry to work with applicants. They considered that the IAP becoming too involved in the process could compromise its decision-making.

3.25
By June 2012, the Ministry had revised the conditions and criteria for PGP funding. It published the revised criteria and included them in written guidance to industry before subsequent funding rounds. Until the Ministry provided that clarification, it would have been difficult for industry participants to gain a full understanding of what was expected during early funding rounds. In our view, until June 2012, the conditions and criteria for PGP funding were loosely defined and open to interpretation.

Changing understanding of how to apply the conditions and criteria for PGP funding

3.26
The IAP is made up of representatives who provide the commercial knowledge and expertise to assess proposals and business cases.

3.27
The IAP considers all PGP proposals received. To ensure that programmes respond to market demand, the Ministry does not identify preferred areas for investment.

3.28
The IAP makes recommendations to the Director-General on whether business cases should be approved, based on how well the business cases meet the conditions and criteria that Cabinet approved (see paragraphs 2.19 to 2.21).

3.29
IAP minutes show that, on occasion, the IAP asked the Ministry to assist them in reaching a view about how to apply some programme conditions and criteria in certain circumstances. In particular, the IAP asked for assistance in deciding how to apply "in-kind spending", "public good", and "additionality" to its assessments of some proposals and business cases.

3.30
The Ministry discussed the IAP's questions at subsequent IAP meetings. The Ministry assisted the IAP in reaching a view about these matters in the first three years while programme proposals and business cases were assessed.

3.31
The IAP found the concept of additionality to be particularly difficult to apply. In the context of PGP, the Ministry's interest in additionality is about showing the need to invest public money and avoiding giving the appearance that it is subsidising industry by showing that funded programmes are additional to the usual work of the industry partner.

3.32
In May 2010, after three funding rounds had closed, the IAP was still asking the Ministry to clarify additionality and how to interpret and apply it in terms of the inputs and outputs of the programmes.

3.33
In May 2010, the IAP considered a paper that the Ministry had prepared about additionality. The IAP stated that additionality is not testable and is easier to describe than to measure. In our view, there are ways in which additionality can be measured. Over time, the Ministry has sought to assist the IAP in reaching a view about what is meant by additionality and how it can be assessed. The IAP told us that there needs to be some flexibility around how additionality is assessed to reflect different circumstances and that there are good, clear "ground rules" for judging additionality.

3.34
Our review of IAP minutes showed that the IAP last asked for assistance with deciding how to apply conditions and criteria to its assessments in December 2011. This suggests that, over time, the IAP's understanding of how to apply conditions and criteria has improved.

3.35
The Ministry considered advice from the various reviews and evaluations of PGP, addressing recommendations incrementally while managing PGP. For example, the Ministry accepted and addressed 51 of the 63 recommendations made in the 2012 review. In our view, this was necessary.

3.36
Of the remaining 12 recommendations, the Ministry noted that seven were out of scope or reflected current practice. The Ministry disagreed with the other five recommendations.

3.37
Other actions included setting up the dedicated PGP Directorate in 2013 to manage PGP. The Directorate enabled the Ministry to better focus its efforts (such as engaging with applicants and industry partners) and to manage critical PGP elements (such as process improvements and interpretation of programme conditions and criteria).

Clear assessment decisions were reached, but reasons for decisions were not always clearly and comprehensively recorded

3.38
Our review of proposal and business case assessment documents showed that the IAP reached clear decisions about whether to approve proposals and whether to recommend business cases for approval to the Director-General. Assessment documents also included clear statements about whether, in the IAP's judgement, the proposal or business case met the criteria for PGP funding.

3.39
Programmes proposals and business cases are assessed against PGP criteria. This is reflected in various documents, including the Ministry's letter to applicants summarising the IAP's assessment of proposals, the Ministry's advice to the IAP on business cases, and the Ministry's advice to the Director-General following the IAP's assessment of business cases. What is not clear from the minutes of IAP discussions and the IAP's letter to the Director-General following their assessment of business cases is how the IAP have considered advice on the criteria and taken alignment with the criteria into account in their assessments to reach their decisions.

3.40
In our view, current documentation of IAP discussions and decisions needs to be improved to provide a clearer, more comprehensive, and easier-to-follow trail connecting how PGP criteria have been considered by MPI and the IAP in informing decision-making. This will improve transparency and accountability.

3.41
The 2012 review recommended using the programme criteria as a quantitative tool. It also recommended that the Ministry develop scores or weighting for each criterion, to make IAP decisions more robust and transparent.

3.42
The Ministry did not agree to the recommendation for a quantitative tool to support assessments because it did not want to make the assessment process too prescriptive. It wanted ideas and discussion to continue to flow.

3.43
In our view, it is important to take account of the differences between proposals. Innovation is not a "paint by numbers" exercise. It is reasonable for the IAP to decide that a programme is worthy of funding even if, in its opinion, the programme does not meet all criteria equally, because different proposals and programmes have different objectives.

3.44
For example, environmental benefits appeared to feature more heavily in the PSH programme proposal than in the NZSTX proposal. The IAP considered that both programmes were eligible for PGP funding.

3.45
However, it is important to clearly record consideration of all of the conditions and criteria used to make a decision. This provides more transparency to the industry partners and other applicants, and enables accountability.

3.46
We expected to see clear and consistent processes for assessing programme proposals and business cases. When processes were not applied consistently, we expected the Ministry to clearly record the reasons for the inconsistency.

3.47
For one of the programmes we reviewed, the IAP recommended the business cases for approval with, in our view, underdeveloped KPIs. KPIs are a Cabinet requirement and indicate performance against milestones, which in turn indicates progress towards intended outcomes.

3.48
In its briefing on the funding approval for the Dairy Value Chain business case, the Ministry recommended that the Director-General approve government investment in the programme. The Ministry also noted that the business case "provides a good feel for what the programme will deliver but [the Ministry] will be seeking more clarity around robust and meaningful milestones and KPIs" should the business case progress to the contracting phase. In our view, the level of underdevelopment of the KPIs when the business case was approved was inconsistent with Cabinet's instructions. There might have been reasons for this. However, the Ministry did not clearly and comprehensively record the reasons for working differently with these industry partners.

3.49
Contract negotiations took nine months to complete and the initial programme contract contained KPIs in the form of milestones and achievement measures. In our view, some of these KPIs were still underdeveloped. Since the start of the programme, additional KPIs have been identified and models have been developed that seek to demonstrate the path between the programme investment and the eventual benefits expected. The Ministry told us that it is working with industry partners to further improve some of the KPIs for this programme.

3.50
The Ministry told us that it expects programme objectives and intermediate outcomes to be well defined before PGP contracts are finalised. This includes having measures of progress towards outcomes for the first few years of programmes. All contracted programmes are required to have these measures, which are published on the Ministry's website. However, the Ministry considers that it is not always cost-effective to develop longer-term KPIs in advance of programmes starting, as measures are sometimes dependent on future activity. In our view, this practice seems pragmatic and sensible.

Recommendation 1
We recommend that the Ministry for Primary Industries improve current documentation of Investment Advisory Panel discussions and decisions so that there is a clear, comprehensive, and easy-to-follow trail connecting how Primary Growth Partnership criteria have been considered by the Ministry and the Investment Advisory Panel in informing decision-making that aligns programmes with Primary Growth Partnership objectives.

Managing conflicts of interest

3.51
New Zealand's small population and the interwoven nature of our primary industries and the people involved mean that conflicts of interest related to PGP or the respective programmes are likely. The Ministry has a duty to effectively manage conflicts of interest when they arise.

3.52
The Ministry has policies and practices in place to identify and manage any potential conflicts of interest with its staff, the IAP, and the PSGs.

Managing conflicts of interest of IAP members

3.53
IAP minutes show that, at the start of every meeting, IAP members would declare any conflicts of interest they might have. The IAP would discuss these possible conflicts and decide how best to manage them. This practice has been applied consistently.

3.54
We saw an example of an IAP member declaring a conflict of interest before the IAP considered a proposal. The IAP agreed that the member would contribute to discussions about the proposal but not participate in decision-making.

3.55
We saw another example where an IAP member declared a conflict of interest and left the room on three occasions while discussions about the proposal took place.

3.56
In 2010, the then Minister of Agriculture and Forestry wrote to us asking for advice on the way that the Ministry was managing conflicts of interest of IAP members within PGP. We found the Ministry's systems to be suitable and offered advice about how it could strengthen its systems to make them more transparent. We encouraged the Ministry to make conflict of interest systems and situations public, to ensure that there is greater transparency. The Ministry now publishes a register of IAP members' interests on its website and updates this as interests are declared.

Managing conflicts of interest of PSG members

3.57
The PGP Operational Policy Statement on Programme Governance (the policy) includes details about managing conflicts of interest of PSG members. The policy states that members must declare any potential conflicts of interest at the outset of the programme, and at any stage during the programme should conflicts of interest arise.

3.58
The policy provides for either partner to object to the appointment of a member if it believes, on reasonable grounds, that the member has a conflict of interest. Reasonable grounds for considering that a PSG member has a conflict of interest include whether a member:

has links to a competitor organisation of the industry partner;
poses a risk to the reputation of the programme; or
is not considered to be a suitable person to be a member of the PSG.

3.59
In our review of the six programmes, we saw no examples where PSG members needed to declare a conflict of interest.

Managing conflicts of interest of Ministry staff

3.60
The Ministry's code of conduct refers to its organisational conflict of interest guidelines. The conflict of interest guidelines provide Ministry staff with guidance on how to identify and manage conflicts of interest.

3.61
During our audit, the Ministry indicated that it would adopt a new practice to support its management of conflicts of interests relating specifically to PGP. The Ministry will require members of the PGP Directorate to review any conflicts of interest before allocating responsibilities for working on newly received proposals.

3.62
In our review of the six programmes, we saw no examples where Ministry staff members needed to declare a conflict of interest.

7: Broom, F. (2012), Primary Growth Partnership – Process and Management Recommendations.

8: Ministry of Agriculture and Forestry (2013), Primary Growth Partnership – Building the plane while flying it.

page top