1027 - Empowering next generation evaluators with a step-by-step Terms of Reference checklist: a workshop with practical exercises on how to commission and respond to evaluation proposals.

Abstract Information: The relationship between a good Terms of Reference (ToR) and a high-quality evaluation has long been recognized; a strong ToR is crucial to evaluations being well designed, budgeted, managed, and meeting stakeholder needs. Development of Mercy Corps’ Final and Mid-Term Evaluation ToR Checklists included a review of dozens of evaluation ToR guidance from a range of implementing, donor, and academic agencies. About half of these resources were just slight variations of the same content and so were not used. Among the 27 retained, 16 did not include checklists; they lacked the precision needed for entry and mid-level monitoring and evaluation staff – and for those involved in commissioning evaluations - to produce a best-practice ToR. 11 included a checklist, all of which were thoroughly reviewed and found to be an improvement over guidance alone but lack some needed content and did not include clearly defined steps and guidance. Most were based on agency-specific policies and procedures and used agency-specific terms and language, making them less flexible for use by other agencies. Mercy Corps’ mid-term and final evaluation checklists have now gone through many months of testing and improvement. They provide the ToR content and step-by-step tips needed to develop high-quality ToRs by less-experienced evaluators, and non-evaluators such as program managers and finance/procurement department staff. These checklists are flexible and so can be used “as they are” or can be easily adapted for any organization’s use. Mercy Corps has found that the use of these checklists are good capacity-building tools. This half-day workshop aims to empower next-generation and mid-level evaluators to write a best-practice ToR. It is also relevant for non-evaluators who commission and/or use evaluations. After briefly presenting Mercy Corps’ ToR checklists used for mid-term and final evaluations, the workshop will facilitate group work to identify how sample ToRs can be improved using the checklists.

Relevance Statement: High-quality evaluations are, among other things, (A) accurately budgeted, (B) excellently managed, (C) characterized by good client-provider communication, (D) well-designed, and (E) conducive to high-standard, learning-optimized evaluation products. A strong ToR has long been recognized as critical to each of these elements throughout the evaluation process, from the preparation stage all the way through to delivery of findings. Development of Mercy Corps’ Final and Mid-Term Evaluation ToR Checklists included a review of the evaluation ToR guidance that is readily available on the web. An initial scan of many web sites and documents from a range of implementing, donor, and academic agencies was conducted. About half of these resources were just slight variations of the same content and so were not used. Among the 27 retained, 16 did not include checklists and were found to include good messaging but lacked the precision needed for entry and mid-level monitoring and evaluation staff – and for those involved in commissioning evaluations - to produce a best-practice ToR. 11 included a checklist, all of which were thoroughly reviewed. These provided neither clearly defined steps nor clear guidance and were not granular enough to ensure the inclusion of all required content. Most of them were also based on agency-specific policies and procedures and used agency-specific terms and language, making them less flexible for use by other agencies. Mercy Corps’ mid-term and final evaluation checklists have now gone through many months of testing and improvement. They provide the granularity – via step-by-step guidance - needed to develop high-quality ToRs, and they include tips to help less-experienced evaluators, and non-evaluators such as program managers and finance/procurement department staff, make choices among options. These checklists are flexible and so can be used “as they are” or can be easily adapted for any organization’s use. Mercy Corps is now socializing the use of these checklists internally and has found that they are also good capacity-building tools. During this half day workshop, participants will be organized into small groups taking the role of an evaluator who wants to submit a proposal to conduct the evaluation. Each group will be given a sample ToRs to read. As they read, each group will write down clarifying questions they would send to the organization that released the solicitation. Groups will present the question they had to other groups. Afterwards, facilitators will present the Mercy Corps checklists and describe how they are used and why they are needed to design, budget and manage evaluations. Then, each group will use the checklists to review the same ToR, discussing how the checklist facilitated ToR review and the extent to which it uncovered weaknesses not identified in the questions they had previously written. Facilitators will be circulating to provide assistance during both exercises. Each group will present their impressions of the usability and usefulness of the ToR checklist. A Q&A session will conclude the workshop. Next generation (up to mid-level) and other non-evaluators that develop and use ToRs are the ideal participants for this workshop.