WIXLIB

ACRONYMS AND ABBREVIATIONSATLASUNDP’s enterprise resource planning programmeBDPBureau for Development PolicyCCACommon Country AssessmentCCFCountry Cooperation FrameworkCPAPCountry Programme Action PlanCPDCountry Programme DocumentM&EMonitoring and EvaluationMDGMillennium Development GoalMfDRManaging for Development ResultsMYFFMulti-year Funding FrameworkNHDRNational Human Development ReportOECD-DAC Organisation for Economic Co-operation and DevelopmentDevelopment Assistance CommitteeOSGOperations Support GroupPRINCE 2Projects in Controlled Environments(a process-based method for effective project management)RBARegional Bureau for AfricaRBAPRegional Bureau for Asia and the PacificRBECRegional Bureau for Europe and CISRBLACRegional Bureau for Latin America and the CaribbeanRCAResults and Competency AssessmentROARResults-oriented Annual ReportRSCRegional Support CentreSURFSub-regional FacilityUNCTUN Country TeamUNCDFUN Capital Development FundUNDAFUN Development Assistance FrameworkUNFPAUN Population FundUNIFEMUN Development Fund for WomenUNVUN VolunteersVDAVirtual Development AcademyA C R O N Y M S A N D A B B R E V I AT I O N Sv

EXECUTIVE SUMMARYINTRODUCTIONThis report sets out the findings of an evaluation ofthe adoption and use of results-based management.The evaluation has focused on the organizationalstrategy, vision and expectations of the results-basedmanagement approach; the design, implementation and use of the system to operationalize thisapproach; as well as the results of this effort. Thescope of the study covers the period 1999-2006,all geographic regions, and the adoption of resultsbased management at the programme, country,regional and corporate levels.An inception report prepared by the evaluationteam proposed a theory-based methodology. Thetheory identifies a causal process with five keyelements. This five-stage process provides thestructure for enquiries:nnnnnSetting a strategic framework to describe thedesired resultsDeveloping programmes aligned to thestrategic results frameworkMeasurement and analysis of the actualresults achievedUse of information to improve design anddelivery of programmesReporting on performance as part of theaccountability processInformation was gathered through interviewswith staff in UNDP and other United Nationsorganizations at headquarters; review of UNDPdocumentation and that of other organizations;visits to five country programmes (in Argentina,Egypt, Indonesia, Moldova and Zambia), whereinterviews were held with a wide range ofstakeholders in the United Nations system,government, civil society and other developmentpartners; and through an electronic survey ofUNDP staff in country offices. The choice ofcountries did not include conflict-affectedEXECUTIVE SUMMARYsituations, a specification in the original terms ofreference, and this limits the ability to generalize,in the conclusions and recommendations, aboutthe use and impact of results-based managementin those contexts. A full list of people consultedin all locations is presented in the report.THE EVOLVING DEVELOPMENT CONTEXTThe 1990s marked a shift from aid flows beingdetermined by geopolitical considerations tohaving a focus on the promotion of sustainablehuman development. That shift was accompaniedby a steady decline in official developmentassistance and increasing pressure to demonstratethe effectiveness of aid. UNDP adopted resultsbased management with the clear intent of reversingthe declining resource base, assuring predictabilityof programme funding and demonstrating aperformance focus to donors.Since then, the role of UNDP has changed frommainly funding and implementing downstreamactivities to emphasizing upstream activitiesinvolving advocacy, policy support and capacitystrengthening, using national execution as thepredominant mode of delivering assistance. Thishas been reinforced by the changing environmentfor development cooperation. The increasingneed for country-based joint assistance strategies,as was emphasized by the Paris Declaration, hasled to new aid modalities, such as direct budgetsupport and sector-wide approaches. The decline inresources has been overcome, and programmaticservices delivered by UNDP increased from 2 billion in 1999 to 4.36 billion in 2005.INTRODUCTION OF RESULTSBASED MANAGEMENTFor results-based management to be successful,organizations need to develop and nurture aculture of results where enquiry, evidence andvii

learning are valued as essential to good management. The use of results information in managingand informing decision-making is usually seenas the main aim of introducing results-basedmanagement. Managers are expected to: understand why projects and other activities contributeto the outcomes sought—the theory of change, setmeaningful performance expectations, measureand analyze results, learn from this evidence toadjust delivery and modify or confirm programmedesign, and report on the performance achievedagainst expectations. When results-based management was introduced in UNDP, it was seen asinvolving all the features previously listed, andthe importance of building a ‘culture of results’was recognized.In many respects, results-based management wasa logical continuation of management reformsfrom the 1990s. A large number of tools andsystems within UNDP are now associated withresults-based management, but these emergedwithout a guiding design over a 10-year period ofinnovation, redesign and change, which has attimes been unsettling for country office staff.Results-based management is not without itscritics. In trying to set clear, concrete objectivesand targets, political scientists argue that resultsbased management can conflict with the need tokeep objectives sufficiently imprecise to gainwidespread support. Another criticism is thatmany of the developmental results sought byUNDP cannot easily be measured. As a result,results-based management forces measurementand reporting of less important results, especiallyoutputs. When an organization overemphasizesany set of performance indicators and targets,the staff tend to become preoccupied with thoseindicators and targets rather than the wider results.It is important to take these concerns intoaccount when developing and, especially,managing the results-based management regime.GOALS AND PROGRAMME FOCUSOne of the most visible elements of the resultsbased management approach was the adoption ofviiimulti-year funding frameworks (MYFFs), whichhave strategic goals designed to help focus theprogramme and improve communication withexternal stakeholders. Alignment of countryprogrammes with strategic goals was furtherpromoted by a shift in focus from project outputsto outcomes. In a parallel initiative, a series ofAdministrator’s business plans introduced plansto change the culture of the organization and putforward the ‘balanced scorecard’ as a tool toreport against a broad range of physical andfinancial indicators of operational change.This evaluation has found that the goals ofUNDP in the strategic frameworks have changedin presentation, but the underlying areas ofwork have remained almost the same as before.The focus areas under the goals have been rationalized and simplified, but it is hard to identifysubstantive change to the scope of activities at thecountry level. Managers and staff in countryoffices believe that the MYFFs have helped tobring focus and improve positioning and advocacy.They have been a positive tool in conjunctionwith the reformed United Nations DevelopmentAssistance Framework (UNDAF) and countryprogramme document to foster dialogue aboutcountry results. However, their effect on countryportfolios has been limited to encouraging theremoval of ‘outlier’ activities. Projects have justbeen mapped to the new frameworks.The reasons for these limited effects are several.The MYFF sub-goals and service lines were verygeneral and were not implemented with strictlymonitored ‘hard boundary’ rules about whatcould and could not be justified under them.Intended outcomes for country programmes alsotended to be vague, rather than rigorous statementsforcing consideration of what projects would bestcontribute to their achievement. In other words,outcomes were defined to cover the existingportfolio rather than drive decisions as to howthe portfolio should be refined and developed.Results-based management assumes that managershave the flexibility to allocate programmeresources to maximize results. That assumptiondoes not generally hold true for UNDP. CoreEXECUTIVE SUMMARY

programme resources, the one area where managershave some financial flexibility, may have increasedduring the evaluation period, but core programmeresources have become a progressively smallerpercentage of total programme funds. Countryoffices have thus become more dependent onnon-core programme financing, which isrelatively inflexible, and often find that the coreprogramme resources are only sufficient to act asseed money.Flexibility to change country office organizational structures has the potential to foster closerlinkages to results. Development and investmentin thematic practice areas was intended to fosterskills. Twenty-five offices have appointedmonitoring and evaluation (M&E) specialistsand 10 have an M&E unit. There is evidence thatcountry offices organize their staff for delivery ofresults, with many structuring programme staffteams around broad programme areas. TheResults and Competency Assessment for staff hasthe potential to foster a results and accountabilityculture. But in practice, mobilization of resourcesand delivery are more powerful drivers of individualperformance among programme staff thanachievement of outcome level results. In its currentmodality, the Results and Competency Assessmentdoes not support managing for results.MONITORING AND ADJUSTMENTThe results-oriented annual report wasintroduced to monitor country programmes, butit has become primarily a tool for reporting tosenior management—of little use for countryoffice or regional management—and is viewed asthe data source for the annual MYFF report tothe Executive Board. That report, in turn, iscriticized by Board members as being too vagueand containing little substantive detail on results.With the advent of results-based management,the focus of results shifted to outcomes, but apartfrom the results-oriented annual report, nospecific tools were developed to help monitorresults. The ATLAS system is steadily gaining inimportance, but its primary focus is financialmanagement and project monitoring.EXECUTIVE SUMMARYThe removal, in 2002, of many mandatoryproject monitoring tools and procedures wasconsidered a risk, but it was done in the hope thatcountry offices would continue with projectmonitoring as well as moving into outcome-levelmonitoring. The evidence suggests that the movemay have led to a decline in project-level M&Ecapacity in some country offices, but it hasstimulated the creation of diverse M&Eapproaches in others—especially where there is astaff member dedicated to M&E. While someprogress has been made in country officestowards monitoring outcomes, approaches failto explain how projects are contributing toprogramme outcomes.Evidence from countries visited also shows thatcountry offices face challenges in definingoutcomes that are owned by other partners,although the UNDAF/country programmedocument/country programme action planprocess is a help, as are joint assistance strategies.This reflects the reality that many partnergovernments and donors have their own M&Esystems, which may or may not use results-basedmanagement. The clearest indicator of this is theabsence of joint outcome evaluations and thelimited use by UNDP of M&E evidence derivedfrom other systems of partners.The oversight and management roles of theregional bureaux do not focus on results. Formalpoints of interaction with country offices areconcentrated in the development of new countryprogrammes, but the interaction focuses onprocess conformity rather than on substance.Thereafter, the engagement is generally driven byresource mobilization and delivery. Routineinteraction between country offices and therespective bureaux takes place through deskofficers in the regional bureaux, who are animportant link in understanding countryperformance but are overworked, junior in status,and often lack resources and technical skills.In order to address shortcomings in the resultsoriented annual report, the regional bureauxhave approached information management indifferent ways, often drawing on the newix

information tools. But the quantitative information in ATLAS and other tools is not accompaniedby qualitative information that explains performance. Some regional bureaux argue that the shiftin focus to outcomes has not been matched byM&E tools to measure performance. The currentsystem is dominated by self-reporting throughthe results-oriented annual report and theResults and Competency Assessment, withoutadequate mechanisms for periodic validations ofquality.support results-based management. Roles andresponsibilities are generally clear, but countryprogramme outcomes and indicators are notsubject to quality assurance and there is littleindependent validation. Individual targets in theResults and Competency Assessment are selfselected and are often applied retrospectively andpoorly linked to incentives. Despite the intendedshift to managing for outcomes, individual staffremain tied to a project orientation and accountability for outputs.Overall, the evaluation finds that there is littleevidence to indicate that results are being usedsystematically to inform adjustments to thecountry portfolios.There is no evidence that the Resident Representative/Country Director is held accountable formanaging for outcomes, and there is considerablescepticism within UNDP over whether this isfeasible, despite evidence of moves towards such anapproach in sister organizations such as UNFPA.EVALUATION, LEARNINGAND ACCOUNTABILITYThe adoption of results-based management in1999 was followed, in 2002, by the introductionof outcome-level evaluations. Adoption ofoutcome-level evaluations in the countries visitedhas been slow. In general, they have been underbudgeted and poorly timed to influence thecontent of country programme documents.Independent evaluation is an important elementof results-based management to validate performance. Yet country evaluations conducted by theUNDP Evaluation Office have emphasizedlearning over accountability and have notmeasured performance against stated intentions.Validation occurs only through infrequent audits.However, the role of evaluation was strengthenedin 2006, when the Executive Board endorsed anew evaluation policy.Little use is made of results for the purpose oflearning at the country level, and staff would liketo see more time allocated to this process. AtUNDP headquarters, the Bureau for DevelopmentPolicy’s (BDP) practice architecture is used fortechnical support, but this architecture has poorlinkages to build on lessons emerging fromevaluations and has produced few productsclearly tailored to business processes.The UNDP accountability framework does notxMANAGING FOR RESULTSIN THE WIDER CONTEXTUNDP works in a multilateral context, in whichits mandate emphasizes the centrality of nationalownership and the role of UNDP in buildingnational capacity. This implies the need to workthrough national systems, where feasible. UNDPprogrammes invest significant attention insupporting national systems for tracking progresstowards the Millennium Development Goals andother development results. But there is littleevidence of engagement with national planningand results/performance systems at the sector orthe programme level.This suggests two important things. First,UNDP has missed opportunities to harmonizewith the results-based management approachesof national partner governments. This reducesthe scope for national capacity development andfor enhancing national ownership. Second,UNDP has not fully considered the implicationsof its results-based management approach onbroader United Nations reform initiatives such asthe Paris Declaration and the MonterreyConsensus, which emphasize closer alignmentand harmonization with governments anddevelopment partners.EXECUTIVE SUMMARY

The shift in strategic direction and the newresults framework brought early gains to theprogramme through a clearer expression ofUNDP roles and functions. Staff have foundstrategic frameworks to be valuable in discussionswith governments and development partners.But the goals in the results frameworks for thetwo MYFFs and the draft strategic plan are toobroad to help UNDP focus support in areaswhere it has a comparative advantage. Thestrategic objectives have not been used to definewhat is and what is not appropriate for countryprogrammes to support.Sharpening the focus in this regard would requirea change in the relationship between regionalbureaux and country offices. The focus wouldhave to shift away from oversight of processesand resources, to a greater emphasis on substantive content. The low level of core funding andhigh reliance on non-core funds by many countryprogrammes means that the managementspotlight needs to be on the degree to whichResident Representatives manage to refocus inthis situation.In the inception report for this evaluation, a tableof benchmarks based on international performancestandards for results-based management wasintroduced as an evaluation tool. The reportassesses each benchmark, drawing on thefindings of the present study and relevantindependent studies. Unfortunately, in theirpresent form, it is not possible to use thebenchmarks to assess the status of UNDPrelative to other United Nations organizations orto the wider population of public-sector organizations that use results-based managementapproaches. But progress has been made in mostareas. Of the 21 categories assessed, two areassessed as ‘fully achieved’, 16 as ‘partiallyachieved’, and three as ‘not achieved’. The largenumber of benchmarks assessed as ‘partiallyachieved’ reflects the positive work of UNDPtowards creating an architecture to manage forresults, but it also reflects that too many elementsof the approach are not functioning effectively.EXECUTIVE SUMMARYCONCLUSIONSOverall, UNDP has established a cycle of settingand revising corporate goals, introduced improvedoffice systems to manage project finances,institutionalized the need to report on corporateand individual performance, and raised awarenessabout results throughout the organization.Conclusion 1: The experience of UNDP withintroducing results-based management issimilar to that of other organizations.UNDP was one of the first United Nationsorganizations to move to a results-based management approach. Review of the literature stronglysuggests that the experience of UNDP did notdiverge significantly from that of many otherpublic-sector organization

As the report points out,results-based management is a journey not a destination and results-based management is just one piece in the complex set . 2.2.6 Adjustment and learning 17 2.2.7 Accountability 17 2.2.8 Guidance and capacity building for results-based management 18 3. Findings from the Evaluation 21