Transcription
MAR/APR 2019 NO. 26i -eval Flash newsThe ILO Evaluation Office is pleased to share the twenty-sixth edition of i-eval Flash news.Through this triannual electronic bulletin, we provide readers with updates, news and informationon publications and upcoming events related to evaluation.Investing in knowledge managementEVAL to launch its new Knowledge Sharing Platform:i-eval Connectby Guy Thijs, DirectorIN THIS ISSUEMESSAGE FROMTHE DIRECTOR - 1INNOVATION &RESEARCH - 2NEWS FROM THEDEPARTMENTS - 4NEWS FROM THEREGIONS - 6EVALHIGHLIGHTS -7LEARNINGACTIVITIES &EVENTS - 9For many years, the ILO Evaluation Office (EVAL) has been actively promoting and investingin improving the quality and use of evaluation knowledge systems. This should not come as asurprise. Evaluation is, after all, a critical means to generate knowledge in the organization; theactual uptake of evaluation findings is the ultimate aim of the whole process. There are manycritics of knowledge management. Often, it is described as overwhelmingly optimistic andunlikely to provide a return on investment.A few examples in EVAL show, however, that investing in knowledge management can beworthwhile in the short to long run. Take for example EVAL’s dashboard, i-eval Discovery(launched in 2016), which has received widespread internal and external acknowledgement asan innovative vehicle for encouraging the uptake of evaluation results and recommendations;or the Automated Management Response System (AMRS) to evaluation recommendations(introduced in 2018) which, so far, has initiated over 60 online management responses,allowing EVAL to generate, compile data and keep track of management responses forindependent evaluations in an efficient and transparent manner. Furthermore, the EvaluationDocument Library (also introduced in 2018), which uses SharePoint technology, has led to afully functional and well-structured electronic records management system for the wholeEvaluation Office.For EVAL, the next step in its knowledge management journey is to now formally launch itsnew knowledge sharing platform, called i-eval Connect. It provides a forum where all ILOofficials in the field and at headquarters can share ideas and information through a largenumber of interactive modules. The platform contains an evaluation consultant roster, amission report facility, a resource centre featuring EVAL products and services, an “ask aquestion” facility, a calendar of evaluation related events, a community of practice module andmany more features. Initially aimed at providing a platform for ILO evaluation officers and theover 150 trained ILO evaluation managers and internal evaluators, the aim of the platform is togradually involve all ILO colleagues and constituents interested in evaluation-related matters.by GuyThijs, DirectorILO EVALILO EVALILO EVALUATIONeval@ilo.org
MAR/APR 2019 P. 2i -eval Flash newsInnovation & ResearchPlanned and ongoing studiesSynthesis review: Supply chains 2010-19Completed studiesA brief history of evaluation in the ILOAs the ILO celebrates its100th anniversary thisyear, the Centenary is anopportunity to celebratethe ILO's achievementsand to reaffirm the ILO'sposition as theauthoritative globalorganization for the worldof work. Formal evaluationin the ILO does not extendback one hundred years.Nonetheless, EVAL seesthe Centenary as an opportunity to reflect on the historicalevolution of the ILO’s evaluation function.Historically, evaluation dates back to the 18th century.However, modern day evaluation is a relatively recentphenomena. It has taken place in various implicit formsthroughout history, such as part of public administration, toinvestigate and evaluate educational and socialprogrammes. In the 1940s and 1950s, internationalprogrammes in areas, such as in health and ruralcommunity development, involved substantialexpenditures and there was a need for the documentationof results. Throughout the 1960s, evaluation was thusincreasingly used in the US, Canada and Europe toassess such programmes.The need for evaluation in the ILO goes back to the sameperiod of time during which modern day evaluation began.In 1967, the third resolution adopted by the InternationalLabour Conference at its 51st Session called for theselective evaluation of technical cooperation programmeswith the participation of employers' and workers'organisations. One of the conclusions of the study is thatmany of the issues that were of concern to the ILO in themid-1960s when the evaluation function began are stillimportant today.The 105th Session of the International LabourConference (ILC) in 2016 noted that failures in globalsupply chains (GSC) contribute to decent work deficits,such as in wages, working time and occupational healthand safety. The ILO Evaluation Office (EVAL), incollaboration with the Office of the Deputy DirectorGeneral for Policy are commissioning an independentstrategic review of existing ILO developmentcooperation programmes in GSCs and related work. Thestrategic review, using EVAL’s established methodology,will review evaluations and studies (both published andunpublished) related to GSCs from 2010-19. The resultswill serve as inputs into the Office’s mid-term report tothe Governing Body on the progress of the GSCprogramme of action, due in November 2019. Thesynthesis review will be available on EVAL’s websitelater this year.Synthesis review: ILO’s social protection(social security) interventions 2012-2018Since 2012, EVAL has been regularly contributing to therecurrent discussions by preparing companion pieces torecurrent reports to enhance organizational learning. Todate, EVAL has produced synthesis reviews for recurrentdiscussions on Social Dialogue (2013 & 2017);Employment (2014); and Social Protection (LabourProtection) (2015). During the discussions on the followup to the resolution on Advancing social justice throughDecent Work: Framework for recurrent discussions, thiswas recognized as a good practice and formalized aspart of the drafting process for recurrent discussionreports (GB.331/INS/3, para 7).This year, EVAL is supporting the RecurrentDiscussions on Social Protection (Social Security) 2020by providing a synthesis report which reviews lessonslearned from ILO interventions in social protection. Thepurpose is to contribute to organizational learning and toprovide effective guidance to constituents of the Officeon their future work on social protection by bringingtogether lessons from past interventions. The synthesisreview will build and expand upon an available synthesisreview (2012-16) that was conducted as part of the highlevel evaluation of ILO’s strategy and actions forcreating and extending social protection floors (201217). The findings from the synthesis review will be usedby the working group that is drafting the recurrent
MAR/APR 2019 P. 3i -eval Flash newsdiscussion report for discussions during the 109thSession of the International Labour Conference in 2020.It will also be published in the form of a working paperlater this year.Rolling quality appraisals of projectevaluationsThe ILO’s new evaluation policy and strategy identifyquality control as a continued important task for EVAL toundertake. The ILO uses a layered approval process inwhich EVAL assures the quality of independent projectevaluations. This process is complemented by ex-postquality appraisals (QA) mechanisms that are conductedby external appraisers. Since 2008, a total of eightquality appraisals have been conducted of independentevaluation reports that have been submitted to EVAL.The results of the last QA (2017) found that more than90% of the reports assessed met the minimum level ofoverall quality. Despite the positive results describedabove, analysis showed that, with regard to evaluationmethodology and efforts to mainstream gender intoevaluations, there is ample scope for improvement.EVAL has evolved its QA process and has engaged afirm to conduct QAs on a rolling basis.The contractor will use the rating protocols to obtain aQA score for each report. The QA will, essentially,validate EVAL’s review of project evaluation reports. Inaddition, the consultants will send a short questionnaireto the concerned evaluation manager to determinecompliance with evaluation policy guidelines. Such anapproach would enable EVAL to detect quality issuesquickly and to take immediate action to address them.The quality appraisal will be completed by the endof the year.Decent work results and effectiveness of ILOoperations: Meta-analysis of developmentcooperation evaluations on an ongoing basisEVAL will soon launch a rolling schedule for analyzingevaluation reports to provide up-to-date decent workresults assessments of development cooperationactivities. This will allow us to provide timely informationin each Annual Evaluation Report. The current study willcarry out a one-off analysis of project evaluations for2017-18 and then institute a rolling schedule of analysisof project evaluations as and when completed.Transformative approach to evaluationsin ILOWork on a transformative approach to evaluations isprogressing. Based on the first phase which resulted inThink Piece No. 14, a new guidance note is beingprepared on how to align evaluation frameworks,methods and processes to the ILO’s normativestandards and social dialogue mandate and context.The new guidance note will also reflect on theimplications of the SDGs as well as the desire of the ILOto undertake more strategic cluster evaluations. Theinternal study on the principles, processes and practicesof cluster evaluations have been completed. Pilots fortesting are currently being identified under the typologiesgiven in Table 1, using information from the study, theevaluation planning module in i-eval Discovery andrecent consultation with regions and departments onintegrated evaluation planning.Table 1. Possible typologies for clustered evaluations
MAR/APR 2019 P. 4i -eval Flash newsImpact evaluation – Review facility andcommunity of practiceEVAL continues to offer support to technicaldepartments, regions, countries and projects in the ILOto carry out evaluations and assessments that focus onthe level of impact. Departments and regions canvoluntarily submit evaluation proposals, designs andreports for EVAL’s technical experts to review throughits Impact Evaluation Review Facility. Please contactEVAL IMPACT@ilo.org if interested in using the facility.Communities of practice sessions provide another formfor review as part of the ongoing experience sharing inILO on impact evaluation. EVAL organizes these to theextent that departments are interested in such sessions.The ex-post quality analysis of a sample of impactstudies and evaluations will soon be launched. As partof updating the guidance notes and tools within the ILOEVAL Policy guidelines for evaluation, the guidance noteon impact evaluation will be updated and enriched withthe outcomes of the ex-post quality analysis.New Books and Articles onEvaluationEvaluation Failures:22 Tales of MistakesMade and LessonsLearnededited by Kylie HutchinsonSummary from SagePublishing, “EvaluationFailures: 22 Tales ofMistakes Made andLessons Learned is acandid collection of storiesfrom seasoned evaluatorsfrom a variety of sectors sharing professional mistakesthey have made in the past, and what they learnedmoving forward. As the only book of its kind, editor KylieHutchinson has collected a series of engaging, real-lifeexamples that are both entertaining and informative.Each story offers universal lessons as takeaways, anddiscussion questions for reflective practice. The book isthe perfect companion to anyone working in theevaluation field, and to instructors of program evaluationcourses who want to bring the real world into theirclassroom.”News from the DepartmentsEnterprises DeparmentImproving learning and knowledge generationthrough evaluations(photo credit: AMA Innovation Lab, University of California, Davis)The ILO Enterprises Department seeks to promotesustainable enterprises for innovation, growth, more andbetter jobs. To achieve this mission, its efforts focus onstrategic areas of work: from business developmentservices, to cooperatives and creating enablingenvironments for sustainable enterprises, to name afew. Evaluation supports the department by providinginsight into what works and what does not in order tocontinuously improve projects and programming. Forinstance, evaluation results, such as findings and goodpractices, have informed future project design, inaddition to their implementation and monitoring ofactivities. During 2018, over 10 enterprise-relatedevaluations were conducted in accordance with ILO’sevaluation policy. More opportunities for learning areexpected as there are more than 20 planned evaluationsin the pipeline for 2019.One notable evaluation featured in this article thatprovided learning opportunities was the independentfinal evaluation of the project “A Global Action Network(GAN) to advance agricultural insurance” that wascompleted in 2018.The project purposeAgricultural development is one of the most powerfultools to end extreme poverty, boost shared prosperityand feed a projected 9.7 billion people by 2050. In 2016,65% of poor adults worked in the agricultural sector,accounting for one-third of global gross-domesticproduct (GDP). While there have been many initiativesin developing countries to support inclusive growth topull the rural poor out of poverty, challenges remain tocurb farmers’ exposure to risks, such as drought andflooding. Small-scale farmers have limited access to
i -eval Flash newsfinancial services and are thus unable to secure funds totake advantage of new technologies and marketopportunities due to financial institutions’ resistance toagricultural lending.The project, awarded by the United States Agency forInternational Development (USAID), sought to respondto such challenges for a four year period, beginning in2014. The project’s objective was to address gaps inagriculture insurance in order to advance practitionerknowledge on how to better provide agricultureinsurance solutions.Evaluation findingsThe project successfully brought together leaders inagriculture insurance, academicians and practitionerswith diverse backgrounds and affiliations to discuss keyissues focused on agriculture insurance. The knowledgethat was generated through these discussions wastransferred into knowledge products, includingpublications. Lessons learned and good practicesdeveloped under various work streams weredisseminated to the broader insurance community. TheGlobal Action Network (GAN) responded to the need forstrengthening coordination and collaboration among theagriculture insurance community and advancing globalknowledge on the topic.Project stakeholders noted that additional key playerswould need to be involved in the network’s WorkingGroup Meetings in the future to better enablegovernments to scale up agriculture insurance. Thefeedback also raised the importance of furtherstrengthening collaboration with local initiatives, theimportance of pilot tests, integrating quality standards,tools, methodologies, guidelines and emerging goodpractices that have been developed or identified by theGAN. Identifying this gap and using it as an opportunity,stimulated the creation of a Peer Learning Platform(PLP) for policy makers on agriculture insurance.Learning opportunities related to future project designshowed the importance of providing technical assistanceto countries through a holistic approach. This includesnot only capacity building activities, but alsostrengthening local coordination with key stakeholdersand local initiatives. The evaluation also called attentionto the need for closer association among stakeholdersand the monitoring of outcomes.MAR/APR 2019 P. 5Moreover, stakeholders suggested that it would havebeen useful to involve aggregators representingemployers and workers, such as the farmers’ unions orfarmers’ associations. This could reinforce futuretripartite dialogue on key agriculture insurance issuesand would contribute to accelerating the adoption ofbetter agriculture practices supported by the GAN.Strengthening local coordination with existing initiativesand key stakeholders, including tripartite stakeholders, isessential to create relevant synergies and to betterutilize locally available funds.The evaluation’s lessons learned have assisted in finetuning project based activities which can contribute tothe sustainability of project activities and results. Theseinclude: assistance and coordination in definingagriculture insurance quality standards and other keyissues to be integrated in specific development/countryprogrammes, and; the provision of technical assistanceand coordination for their implementation throughimproved design, delivery and distribution models.RecommendationsThe evaluation recommends that the GAN meetingsand activities continue and reinforce activities in focuscountries while exploiting synergies for acomprehensive knowledge management process andsustainability plan. This would allow to further developglobal innovative knowledge and to test and apply itinto local practice and policy. The evaluation alsosuggested that future projects of this nature shouldhave solid project management tools, methodologiesand results-based management (RBM) guidelines.This would encompass regular outputs and outcomesmonitoring and a stronger reporting mechanismbetween project donors, ILO and focus countries.To view the full report, please see i-eval Discovery.
i -eval Flash newsNews from the RegionsAfricaRenewed drive to build evaluation capacity andsupport organizational learningThe Africa region has one of the largest developmentcooperation portfolios in the ILO. This is reflected by thehigh number of evaluations that are undertaken eachyear: over one hundred evaluations were conducted inthe past five years. Many interventions are related toyouth employment, skills development, enterprisedevelopment and gender equality, to name a few. Thisyear promises to be a busy one for the region: over 40evaluations are scheduled in 2019 alone although thatnumber is expected to drop because of carry-over to2020. Nevertheless, all these evaluations present apromising period ahead with important learningopportunities. In mid-November 2018, a new RegionalEvaluation Officer (REO) took post in the regional officefor Africa (ROAF). The REO is responsible forreinforcing the work of evaluation in the region, inaddition to overseeing the regular management andoversight of all self, internal and independentdecentralized evaluations. This article summarizes someof the more salient evaluation-related activities that areupcoming in the region.In November 2018, the ILO’s Governing Body approvedEVAL’s rolling work plan for 2019 which included anindependent evaluation of a cluster of ILO Decent WorkCountry Programmes (DWCP) that have beensupported by the ROAF. This high-level evaluation(HLE) will offer an opportunity to assess the ILO’sperformance in delivering decent work results in one ofthe more dynamic sub-regions. The HLE will bepresented to the ILO’s Governing Body during its lastSession of 2019. In consultation with the RegionalOffice, it was decided that the main focus of theMAR/APR 2019 P. 6evaluation would be the ILO’s programme of support forthe Southern African Development Community (SADC)and to selected SADC member-countries of Lesotho,South Africa, Tanzania and Madagascar from 20142019. The HLE will offer an opportunity to developevaluation capacity in the region while, at the same time,examine the results of the ILO’s renewed evaluationcapacity building activities to support constituents andorganizational learning.Several lines of action are being implemented in theregion to strengthen evaluation capacities. First, theROAF, in collaboration with EVAL and ITC-ILO, ishosting its 10th edition of the Evaluation ManagerCertification Programme (EMCP), which will take placein northern Africa in October 2019. The EMCP waslaunched in 2013 to upgrade the quality of evaluationmanagement and to expand the pool of qualifiedcandidates. Over the years, 51 colleagues from theAfrica region have participated in the EMCP training and38 have been certified as ILO evaluation managers.Expanding the pool of ILO certified evaluation managerswill support the region to undertake its substantialevaluation-related work. The ROAF also promotes theexchange of evaluation managers with other regions,such as Asia & the Pacific and headquarters toencourage further opportunities for mutual learning.In developing the new DWCP for Burundi, the ROAFand EVAL are currently implementing the newevaluability diagnostic instrument that supports the ILOin analyzing the linkages between Decent Work CountryProgrammes (DWCPs), the Sustainable DevelopmentGoals (SDGs), and their monitoring and reportingcapacity. The instrument is pivotal for supporting theILO’s work by enhancing the design of DWCPs and toensure robust monitoring and evaluation to report onresults. To date, the instrument has been pilot tested inArgentina, Suriname, Iraq and Sri Lanka with the help ofconcerned regional and country offices. With the ILO’songoing initiative to support the development of a newgeneration of DWCPs that reflect the aspiration of ILOconstituents to achieve social justice through decentwork, and position them as effective vehicles forimplementing the 2030 Agenda, the instrument is tosupport the formulation of a theory of change and M&Eplan in a manner that illustrates ILO country-levelcontribution to the SDGs.The use of evaluation by constituents is an area ofspecial attention in the context of the SDGs, not only asa tool to track indicators, but also as a mechanism to
i -eval Flash newsreflect on the “how” and “why” that attempt to explain theachievements and pitfalls of policies in reducinginequalities. The involvement of ILO’s tripartiteconstituents in the national reviews is critical to ensurethat sufficient attention is paid to Decent Work inreporting on the SDGs (e.g. SDG 8 and other relevantSDGs).In an effort to strengthen the monitoring and evaluation(M&E) of projects and programmes to demonstratedecent work results, the ROAF is working on reinforcingcapacities as a key basis for more relevant independentevaluations that inform planning and implementation ofevidence–based policies. The development of guidelinesand conducting trainings for programme officers andconstituents on evaluating the Decent Work Agenda inthe SDG era, in addition to generating technical adviceon the implementation of Comprehensive M&EStrategies (CMESs) will also be mainstreamed to fosterthe use of evaluation results by constituents.EVAL has been advocating for the development of bettertheories of change in an effort to improve projects andprogrammes for many years. For instance, arecommendation put forth in the Annual EvaluationReport (AER) 2014-15 stated, “Given the importancethat the ILO places on results-based management,greater focus should be placed at the project designphase on ‘getting it right’ insofar as development of thetheory of change, log frame, performance indicators,measurement strategies and M&E plans are concerned.The ILO should require ‘hands-on’ support from relevanttechnical experts (either internal or external) for all highvalue project proposals and reject those that are not upto standard.”[1] The AER for 2015-16 raised the sameissues, but in the context of the contribution to the SDGs,“ILO will need to review its theories of change anddevelop new theories where none exist. It will then benecessary to review measures of success against therevised/updated theories of change so as to ensurealignment with the SDGs and the Decent Work Agenda.“Alignment” means that the SDGs will be incorporatedinto all aspects of major ILO interventions, at both globaland national levels.”[2][1] ILO. Annual Evaluation Report 2015-16, p. 18.https://www.ilo.org/wcmsp5/groups/public/---ed mas/--eval/documents/publication/wcms 532853.pdf[2] ILO. Annual Evaluation Report 2014-15, p. 22.https://www.ilo.org/global/docs/WCMS 415198/lang-en/index.htmMAR/APR 2019 P. 7The ROAF remains committed to supporting resultsbased management through evaluation. By buildingorganizational capacity to conduct and manageevaluations, applying theories of change to interventionsand by using EVAL’s evaluability diagnostic instrumentto support the ILO’s new generation of DWCPs, will allhelp pave the way for stronger learning opportunities inorder to deliver maximum results to the people theorganization serves.EVAL HighlightsLaunch of a new knowledgemanagement system: i-eval ConnectEVAL is pleased announce the forthcoming launch (May2019) of its new knowledge sharing platform, called ieval Connect, in an effort to build a community ofevaluation professionals. Accessible to ILO staff theworld over, and later to external members, i-evalConnect serves as unique space for current andprospective evaluation practitioners to share experience,information, and to find solutions to common evaluationrelated challenges. i-eval Connect hosts variousinteractive facilities that encourages exchanges betweenmembers:Capacity development: Hosts information on futuretraining sessions, material, online forums and morefor ILO staff, constituents and consultants.Consultant roster: Enables members to access over300 evaluation consultants (and prospective) whomhave conducted one or more evaluations for the ILO.Calendar of events: Features evaluation-relatedworkshops, meetings, conferences, etc., that occur inthe field and at headquarters.Mission reports: Provides access to a dynamic list ofmission reports that are key to unlockingorganizational memory.
i -eval Flash newsCommunities of practice: Designed to host moderateddiscussions on a variety of topics. A report will becreated to summarize discussions once they close.Ask a question: Colleagues are encouraged to sendquestions and comments to EVAL officials based ontheir areas of expertise.i-eval Connect, together with i-eval Discovery and theAutomated management response system, representnew efforts to help bring the knowledge management ofevaluation results into the next centenary.Streamlined cooperation fordecentralized evaluationsOver the years, EVAL and the Department forPartnerships and Field Support (PARDEV) havecollaborated on delineating responsibilities andcoordination of ILO evaluations. An operating workflowfor decentralized evaluations has been revised to reflectprocedural changes that emerged from the newevaluation policy and strategy (such as clusteringevaluations and their earmarked resources) and inPARDEV's business processes. In light of the GBapproval of procedural modifications in the reporting ofself-evaluations for projects with a value below USD500,000, EVAL and PARDEV have jointly released a newFinal Progress Report template that now includes amandatory section on self-evaluation results for suchprojects.MAR/APR 2019 P. 8(CPOs) and relevant SDGs; conducting trainingactivities for employers’ and workers’ organizations inmonitoring, evaluation and reporting on SDGs; andenhancing coordination with national counterparts andUnited Nations Coordination Teams (UNCTs) to developclearer linkages from immediate needs-basedinterventions to support longer-term SDG goals.EVAL and the ILO Regional Evaluation Officer (REO) inthe Arab Region participated as speakers in the ILOWorkshop on South-South and Triangular cooperationon Mainstreaming SDGs in the Arab Region held inBeirut from 3-4 April, 2019. It aimed to foster informationand good practice sharing among countries inpreparation for the 2019 Voluntary National Reports.EVAL provided an overview office’s efforts inmainstreaming SDGs into evaluations of decent work(including through the evaluability diagnostic instrumentand training programme for Constituents on evaluationand the SDGs), and showcased how this informationcould be used by constituents in preparing annualcountry reporting on SDG progress relevant to decentwork. A summary of the main findings from the recentimplementation of the EDI in Iraq was presented,together with an outline of future initiatives in the regionon evaluation and the SDGs.Supporting the SDGs through anevaluability diagnostic instrumentAs reported in our previous newsletter, EVAL developedan evaluability diagnostic instrument (EDI) to supportenhanced planning for the implementation, monitoringand evaluation of the objectives put forth in theSustainable Development Goals (SDGs). The EDI hasalready been implemented in Sri Lanka, Suriname,Argentina and Iraq with the support of concernedregional and country offices.The EDI provided recommendations to these offices onparticular strengths, gaps or challenges that are relatedto the planning, monitoring, evaluation and reporting onDecent Work Country Programmes (DWCPs) in syncwith the SDGs. Some examples of concrete follow-upactions to country offices relate to: developing aparticipatory planning strategy for a DWCP theory ofchange that is reflective of the SDGs; nominatingan internal lead to facilitate discussions to refine theTheory of Change; developing indicator frameworksthat include intermediate progress milestones andimpact indicators for Country Programme OutcomesEVAL presenting at the South-South and TriangularCooperation on Mainstreaming DW in the SDGs in BeirutNew service: Making sense ofevaluation results through NVivoEVAL welcomes all ILO officials to access NVivo softwareto undertake qualitative analyses of evaluation reports.NVivo supports researchers to synthesize rich andcomplex qualitative information. Its results can be used toinform evaluations, meta-studies and other researchreports, as already practiced by EVAL. Colleagues arewelcome to use this software in EVAL’s multi-purposeroom 7-118, at ILO headquarters in Geneva. Contacteval@ilo.org for more information.
Document Library (also introduced in 2018), which uses SharePoint technology, has led to a fully functional and well-structured electronic records management system for the whole Evaluation Office. For EVAL, the next step in its knowledge management journey is to now formally launch its new knowledge sharing platform, called i -eval Connect.
