Part 4: Risk management, contracting, and evaluation

4.1
In this Part, we outline our expectations of the Ministry's risk management, contract management practices, and evaluation of the effectiveness of professional development initiatives. We then describe our findings and conclusions for each of these areas, and discuss how the Ministry learns from its information.

Our expectations

4.2
We expected the Ministry to have:

• arrangements for effectively managing risks to the provision of professional development initiatives;
• systems for managing and monitoring contractors and ensuring that the provision of professional development initiatives is satisfactory; and
• mechanisms for evaluating the effectiveness of professional development initiatives for teachers and providing feedback on lessons learned to support continuous learning.

Summary of our findings

4.3
We identified the following strengths:

• the frequency and scale of the Ministry's evaluation of professional development initiatives; and
• the Ministry's systems for comparing the delivery of professional development initiatives against the Ministry's contracted expectations.

4.4
We identified the following areas for improvement:

• The Ministry does not systematically use the full range of information it collects to actively identify risks, or to identify the wider lessons to be learned from professional development initiatives.
• The risks and risk management activities were not fully documented in the contract files that we examined.

4.5
We also identified that, for one large initiative, Extending High Standards Across Schools, the Ministry has proposed developing and implementing an overall evaluation strategy, but this strategy had not been written at the time of our audit.

Risk management

4.6
The Ministry invests large amounts of public money in professional development for teachers. To help ensure that its investment results in improved outcomes for students, it is important that the Ministry identifies potential risks that could reduce the effectiveness or efficiency of professional development initiatives and identify, where possible, how the risks might be managed.

4.7
Responding to risks can be complex, given the number of stakeholders involved in providing professional development initiatives for teachers. Nonetheless, the Ministry is responsible for the overall effective provision of those professional development initiatives, and must manage any risks appropriately.

4.8
We looked for evidence of risk identification and communication among relevant stakeholders and for evidence of risk registers, or equivalent documents, detailing the risks to professional development.

Managing high-level risks

4.9
Much of the high-level risk identification we saw within the Ministry took place through internal meetings and correspondence. We did not see any high-level risk registers, although the Ministry does have a database of risk indicators for schools. Therefore, it is unclear whether information on high-level risks was shared among, or accessible to, Ministry staff responsible for managing risks associated with individual professional development initiatives or contracts. The Ministry has identified the need for "a more strategic approach to the integration and use of system-level information in strategic decision making".¹

4.10
There are several high-level risks that the Ministry or stakeholders in professional development have identified, including:

• limited access by some teachers to professional development initiatives because of geographic isolation or financial constraints;
• limited capability (skills and knowledge) of providers and teachers to implement an evidence-based approach to professional development and learning;
• limited capacity of providers to provide initiatives because of difficulties in attracting suitable staff ; and
• schools over-committing to professional development initiatives, or taking up professional development initiatives sufficiently similar to other initiatives that little or no additional value is gained.

4.11
We have also identified the geographic mobility of teachers as a risk because of targeting professional development at a school level. The Ministry believes that it is managing this risk to the extent it can.

4.12
The Ministry has made some recent efforts to better understand and manage risks to professional development initiatives. For example, the Ministry has identified a need for a more flexible approach to centrally-funded teacher professional development programmes, with a balance between centrally-funded priorities and school-identified needs. It supports initiatives to develop teachers with the potential to be school principals. The Ministry also funds its INSTEP initiative to strengthen the capability of teachers to use assessment data to support student learning. In our view, it needs to strengthen and continue those efforts.

Limitations of an evidence-based approach

4.13
The Ministry believes that students' outcomes can be improved through effective teaching and that the effectiveness of professional development initiatives can be measured using student achievement information.

4.14
Although an analysis of student achievement information can identify areas in need of improvement, relationships between the professional development received by teachers and students' achievement are complex. The performance of students can be influenced by a range of factors and circumstances. While we support an evidence-based approach, we consider it important that the Ministry recognises any limitations of its evidence-based approach.

Contract management

Managing contract risks

4.15
Managing individual contract risks is one of the means by which the Ministry can reduce the risks to professional development. The Ministry's risk management policy requires risk analysis and monitoring to be built into all phases of the contracting process, including planning, selection of provider, contract negotiation, contract management and review, and re-negotiation. The Ministry has a comprehensive suite of templates to support staff to do this.

4.16
We saw varying risk assessment practices within the Ministry. For example, when the Ministry was considering an extension and a possible expansion to one professional development initiative, it identified the risks of extending the initiative and possible strategies to mitigate those risks. In contrast, for another initiative, a report on funding applications for that initiative found that none of the applications considered risk. The report recommended that the Ministry issue an application template that includes a section on risks, such as staff turnover, and how the risks will be handled. We support this recommendation.

4.17
When we asked the Ministry for evidence of its risk analysis and monitoring throughout the contracting process, we were told that risks were considered but may not be documented in the contract files.

4.18
We saw limited evidence of risk registers, or other equally specific documentation, recording risks at the contract level. This limits the Ministry's ability to identify and share information about risks among service providers, and within and across particular teacher professional development initiatives. It may also limit the Ministry's ability to effectively manage risks.

4.19
The Ministry's contract management guide describes the reporting of risks as a "critical component of the contracting cycle". Although we do not recommend an inappropriately resource-intensive approach, we expected the Ministry to follow its risk management policy and we expected to see documented evidence of this. For example, we expected to see evidence that the Ministry had identified risks to each project or contract, assessed the likelihood and impact of each risk, and identified strategies to reduce each risk.

4.20
In our view, if risks are documented and managed effectively, important lessons can be learnt throughout the contracting process and shared among contract managers and policy advisers at the Ministry. This can help maintain a knowledge base of lessons learned and can prove to be a valuable resource, especially if there is a high staff turnover at the Ministry or among professional development service providers.

4.21 The Ministry has identified "variable quality of in-service teacher education practice"² as a risk for professional development.

4.22
In its response to the draft of this report, the Ministry noted that it would be asking its staff responsible for professional development contracts to review the risk management guidelines and templates and their use. The Ministry also noted that it would be training staff and managers to ensure greater compliance with risk management guidelines, and to ensure that risks are identified and actively managed. The Ministry also said that it would like its staff to consider a process for raising concerns about risks, to allow information about risks among service providers or within particular professional development areas to be shared and managed.

4.23
We would consider this recommendation to be implemented if we saw documented evidence of active and considered risk management within the contract files.

Monitoring contractor performance

4.24
We looked for systems to reduce the risk of failure by ensuring that the provision of professional development initiatives was satisfactory to the Ministry. We also looked for feedback and review mechanisms to support Ministry staff and enable providers to learn from others' experiences.

4.25
The Ministry has processes in place to match contractor performance with contract expectations. For example, contractual agreements between the Ministry and the professional development providers contain clear guidance on expected outputs, milestone reporting requirements, and expected time frames for the provision of the professional development.

4.26
In our review of contract files, we saw service agreements with clear performance measures and monitoring expectations. For example, most providers are required to submit between three and five milestone reports to the Ministry each year. These reports include expenditure and budget information. We observed some variability in the number of milestone reports required for the initiatives we reviewed. Many of these reports were written but some were oral. The Ministry provides feedback on these reports. Some providers have found oral reporting to be more effective than written reporting and would like to see a change to oral reporting, with summary documents to focus on important and emerging issues.

4.27
We also saw that the Ministry is increasingly focusing on outcomes in its monitoring and its milestone reporting requirements. Changes in student achievement outcomes are becoming more commonly used as indicators. We were told that in 2008 the Ministry will require evidence of a shift in student achievement for the literacy professional development initiatives that it funds.

Evaluation of professional development initiatives

4.28
Evaluation is an assessment of the strengths and weaknesses of a specific initiative, including, where possible, an assessment of the actual results of the initiative.

4.29
The frequency and scale of the evaluation of Ministry-funded professional development initiatives for teachers is a strength of its work (although we have not benchmarked the Ministry's level of expenditure on evaluation against expenditure on evaluation in other social sectors). Evaluation is essential if the funding and provision of professional development for teachers is to be informed, on an ongoing basis, by the evidence of what is effective and efficient.

4.30
During our fieldwork, we saw examples of evaluations having been planned, commissioned, or carried out for some of the initiatives we looked at. These included the School Support Services initiatives. We also saw evidence that the Ministry had evaluated a number of past teacher professional development initiatives.

4.31
However, for Extending High Standards Across Schools, which is a large initiative, developing and implementing an overall evaluation strategy has been proposed but the strategy has not yet been written. In our view, it is important that this strategy is written, and that the evaluation is carried out.

4.32
The Ministry has identified some limitations on its ability to evaluate the overall effectiveness of professional development provision.

Learning from a wide range of information

4.33
The Ministry collects a large amount of risk and other information each year from schools and professional development providers. This can be gathered in formal and informal ways; for example, it may be formally documented in milestone reports, school charters, or planning documents or obtained more informally from the regular interactions of Ministry staff with schools.

4.34
Schools, providers, and the Ministry compile and analyse this information, and compliance costs can be high for all stakeholders. We were told that, in some cases, schools viewed reporting requirements as compliance mechanisms rather than tools for improvement, and that providers have questioned the usefulness of reporting to the Ministry.

4.35
Despite the range of information the Ministry collects from the education sector, we have seen limited evidence that the Ministry uses this information as systematically and consistently as possible to generate collective knowledge across providers and initiatives. For example, the Ministry could make a greater effort to compile information from milestone reports, school charters, and planning documents to actively assess and reduce risks and inform decisions about its investments in the professional development of teachers.

4.36
The Ministry has acknowledged that it could make better use of the large amount of information that it collects, but that the costs and benefits of any additional work would need to be considered.

4.37
We would consider the recommendation to be implemented if the Ministry could provide us with examples of issues being combined and compared from the multiple sources of information it has.

1: Ministry of Education (2007), Learning Policy Frameworks Project Plan: Professional Learning and Development Provision.

2: Ministry of Education (2007), Learning Policy Frameworks Project Plan: Professional Learning and Development Provision.