WIXLIB

Effective EvaluationHow can collaborative groups avoid positional debates and circular discussions?Collaborative group participants can be frustrated by timeand resources reallocated from on-the-ground work tomeetings where disputes are raised but often not resolved.Groups that have completed collaborative planning andmonitoring are frustrated further when lessons learned arenot captured and used to inform ongoing work. Mutuallearning and evaluation can help address these frustrationsby reframing disagreements as questions to be answeredand providing structured methods to address them.Disagreements about management needs and practicesoften are based in deeply-held values and different socialand political views. Such disagreements are not resolvedthrough technical analyses and rational planning processes.They require mutual understanding of different worldviewsand political realities as well as the best available scientificand technical information. These value-based differences,along with complexity of interrelated ecological, social,and economic systems, mean management decisions willnever be clear-cut.Concepts from organizational and social learning,adaptive management, and utilization-focused evaluationcan help groups constructively address planning andmanagement questions. Organizational learning providesguidelines for purposeful and collective learning frompractical experience. Social learning emphasizes informationsharing and deliberation across organizations and amongpeople with different experiences and perspectives. Adaptivemanagement aims to reduce uncertainty and improvemanagement by monitoring programs and projects andcomparing results to management objectives. A utilizationfocused approach to evaluation produces concrete,actionable recommendations that can feed directly intopolicy, planning, or management decisions. Designing andconducting evaluation to produce clear recommendationshelps ensure that results will be used.Multiparty LearningWhen people with different backgrounds and perspectivesjointly review information, assess current conditions, raisequestions about observed and proposed activities, andshare their experiences and expertise, they build both theircollective knowledge and their capacity to evaluate theirindividual and collective work.Why Do It Build understanding of different perspectivesMake tacit knowledge explicitIdentify agreed-upon issues and challengesLocate and use relevant technical andpractical information and expertiseChallenge ingrained beliefs and practicesResolve conflictsIdentify innovative solutionsBuild shared knowledge and reduce relianceon a few individualsMultiparty learning and evaluation tools includecollaborative planning and joint fact-finding.Collaborative PlanningMany organizations and agencies use an interdisciplinaryteam approach to planning. However, involvingexternal stakeholders throughout the planning processis less common. Collaborative processes such as multistakeholder and multidisciplinary landscape andwatershed assessments help build understanding andtrust among participants as they collectively evaluateconditions and identify management objectives beforeplanning begins. Ideally, collaborative groups alsoparticipate in identifying alternative management actionsand discussing how best to implement those actions.Although final decision making authority lies with one ora few entities, other stakeholders are more likely to acceptthose decisions if they were substantively involved in thediscussions that led up to the decision. The process canbe time consuming, but collaboratively developed plansoften benefit from the additional expertise and innovationsbrought by people outside the core planning team andcollaboratively planned projects are less likely to bedelayed by appeals, objections, or litigation.Joint Fact-findingJoint fact-finding is a strategy to help stakeholders workthrough disagreements. The goal is to clearly separateagreed-upon issues from those still in dispute and reportfindings in summary documents that can be used toinform management or policy decisions. By synthesizingClosing the Feedback Loop: Evaluation and Adaptation in Collaborative Resource Management 5

Shared Sacrifice: Blackfoot Drought Response PlanThe Blackfoot Drought Committee was establishedin 2000 to address declining fisheries and waterallocation in the Blackfoot River Basin in westernMontana. The planning process brought togetherprivate landowners, state and federal agencies,and conservation groups to develop a solutionto the fisheries decline by better managing wateruse during drought years. The Blackfoot DroughtResponse Plan, based on the goal of watershed-widerestoration of fishery resources and “shared sacrifice”among all water users, grew out of these discussions.The Drought Response Plan is tiered to morethan 100 individual drought management plans thatlay out how and where irrigators will reduce theirwithdrawals. Participating senior water rights holdersvoluntarily reduce their water use when flows at themouth of the river drop below pre-set thresholds.Those with rights junior to the state’s instream flowright legally can be required to shut down operations,but under the Drought Response Plan the state hasagreed to the individual management plans thatgive users more flexibility in how and where theywill reduce withdrawals. The Drought ResponsePlan also limits fishing at predetermined lowflow and high-temperature thresholds. A DroughtCommittee with representatives from the BlackfootChallenge, Trout Unlimited, Montana Fish, Wildlife,and Parks [FWP], U.S. Fish and Wildlife Service,the Montana Department of Natural Resources,irrigators, and outfitters was created to oversee planimplementation.Drought Committee members point to twofactors that make the plan effective: users wereinvited to help craft the management plan, and thestate has always been willing to work with users toseek alternatives to shutting down operations. Thestate retains its authority to enforce water rightsand close areas to fishing, but generally relies onthe concept of shared sacrifice rather than formalenforcement to maintain compliance.The Blackfoot Drought Committee was established in 2000 to address declining fisheries and water allocation in the Blackfoot river basinin western Montana. Photo by Ron Pierce, Montana Fish, Wildlife, and Parks6 Closing the Feedback Loop: Evaluation and Adaptation in Collaborative Resource Management

available information, participants avoid the challenge oftrying to choose between different reference materials or“dueling scientists.” The process of developing a summarydocument focuses the group on problem-solving ratherthan debate.Process1. Convene a team of people with differentperspectives on the issue2. Agree on the nature of the problem and questionsthat need to be answered3. Identify and select qualified experts to assist theteam4. Work with the experts to refine the questions andagree on methods for answering them5. Identify and review relevant information,including technical and scientific documents6. Write a summary report synthesizing what hasbeen learned, including any outstanding questionsor disagreementsIt is important to start by clarifying the questionsthat need to be answered, to focus data gathering ondecision-relevant information and avoid reviewing thefull range of available information on a topic of interest.It is also important to reach agreement on appropriateinformation sources at the outset, so that all participantswill accept the fact-finding results. Experts consulted mayinclude not only resource specialists with scientific andtechnical knowledge but also policy decision makers andother stakeholders who bring information on politicalconstraints and financial feasibility into the discussion.Evaluation StrategiesAlthough collaborative groups rarely undertake indepth program evaluations, they can use principlesfrom utilization-focused evaluation to produce concrete,actionable recommendations that feed directly into policy,planning, or management decisions. Collaborative groupsalso use rapid review processes to identify lessons learnedand recommend changes. After-action reviews provide asystematic way to quickly capture lessons learned frommanagement actions and recommend more effectivepractices. Regularly scheduled program and processreviews are used to assess and recommend adaptations tobroader policies and programs.A Way Forward: Lower Dolores Working GroupThe Lower Dolores Working Group in Coloradoused joint fact-finding to address participants’disagreement over the condition of native fisheriesand minimum flow needs for whitewater boating.In an effort they called “A Way Forward,” thegroup hired three fisheries experts to individuallysynthesize and interpret available data onnative fish conditions, then collectively developmanagement options for addressing conditions andtrends.The group agreed at the outset that theywould accept the experts’ collective synthesis ofthe science, but that the scientists’ managementoptions would be further evaluated for social,economic, and legal feasibility. A subcommittee,including agency decision makers andcounty elected officials, environmental grouprepresentatives, and water users, discussed andrefined the management options in terms of legaland fiscal feasibility and water users’ needs. Thegroup then separated the options into those thatcould be implemented immediately and those thatwould require fundraising or policy changes.The Lower Dolores Working Group used joint fact-finding to addressdisagreement over the condition of native fisheries and minimumflow needs for whitewater boating. Photo courtesy of the LowerDolores Working GroupClosing the Feedback Loop: Evaluation and Adaptation in Collaborative Resource Management 7

Why Do It Identify best practicesCorrect unintended effectsImprove programs, plans, and managementpracticesProvide concrete, actionablerecommendations for improved actionBuild capacity to reflect on practice and togive and receive critical feedbackUtilization-focused EvaluationThe utilization-focused approach to program evaluationprovides an alternative to punitive performance reviewsand descriptive reports by focusing on improving planningand management. The first step in a utilization-focusedevaluation is to determine how the results will be used.This is done by asking key stakeholders, particularlythe expected end users of the results, to identify howand why they would use evaluation results. Involvingintended users helps ensure that they understand and feelownership in the evaluation process and findings, whichmakes them more likely to use the results. Once expectedusers and uses have been identified, stakeholders canidentify appropriate evaluation questions and methods foranswering them.Questions to Focus Evaluation What is the activity or decision that we want toinfluence?Why do we care (what’s the issue)?Who is responsible for making the decision?Who is responsible for implementing thedecision?What data and findings are needed to supportdecision making?What other factors will affect the decisionmaking?How and when will decision makers andimplementers use the evaluation results?Answering these questions helps focus the evaluationon useful results and change and change mechanisms.Methods for answering these questions are selected basedon their ability to produce useful results.8 After-action ReviewThe after-action review (AAR) is a technique for rapidlyreviewing management actions in terms of desiredoutcomes and implementation realities. In an AAR,people involved in or affected by a management activitydiscuss what happened in terms of intended resultsand unexpected events, isolate key lessons learned,and recommend changes to improve effectiveness. In acollaborative resource management context, participantsin the review should be those directly involved in projectplanning, implementation, and monitoring.To encourage open sharing, the AAR facilitatorshould begin by establishing that it is not a critique orperformance evaluation. Discussion should be grounded inmutual respect, without judging people or their individualactions. Participants should be asked to keep individualcomments confidential and participate freely, regardless ofprofessional hierarchies. While individual input should bekept confidential, it is important to have a clear record oflessons learned and recommended actions.Despite the name, AARs do not have to be performedat the end of a project or action. “Before-action reviews”can be used to make sure everyone agrees on theimplementation process and desired outcomes and toreview lessons learned from past actions. After-actionreviews can be used periodically during planning orproject implementation to evaluate progress and discusswhether strategies or operations should change.Questions to Guide an After-action Review What did we set out to do?What did we actually achieve?What unexpected things happened?How did we respond?What went well?What could have gone better?What should we do next time?What additional information do we need?Periodic Process and Program ReviewsRegularly scheduled multi-stakeholder reviews areimportant for evaluating social and policy structures andprocesses as well as management actions. These reviewscommonly occur in annual meetings where stakeholdersrevisit program and policy strategies and goals. As withAARs, process and program reviews should involve peoplewith decision making authority as well as those activelyinvolved in implementing decisions. These meetings canencourage critical reflection on program purposes andunderlying assumptions as well as effectiveness.Closing the Feedback Loop: Evaluation and Adaptation in Collaborative Resource Management

Multiparty Field Reviews: Central Oregon Partnership for Wildfire Risk ReductionThe Central Oregon Partnership for WildfireRisk Reduction (COPWRR) uses a type ofafter-action review to evaluate implementationand outcomes of forest restoration and fuelsreduction projects. Participants representcontracting foresters, environmental groups,and resource management agencies, includingproject interdisciplinary team members. Thegroup evaluates projects using forms thatdescribe the project’s original purpose andneed, management objectives, silviculturalprescriptions, and best management practices.Participants visit three to five treatmentunits and discuss how well the project metdesired outcomes, what factors arose duringimplementation that affected the process oroutcomes, what adaptations were made alongthe way, and lessons learned that the groupwould like to see carried forward into futureprojects. Through back-and-forth discussionand debate, the group develops a shared projectevaluation and recommendations for futuremanagement, which then become part of thegroup’s permanent record.Members of the Central Oregon Partnership for Wildlife Risk Reductiongather during a field trip. The group often conducts multiparty field reviewsto discuss how well the project met desired outcomes and share lessonslearned to carry forward into future projects. Photo by John ChinnockAnnual Meetings: Blackfoot Drought CommitteeThe Blackfoot Drought Committee implementsits Drought Response Plan through an iterativeprocess of reviewing monitoring indicators,communicating with plan participants,implementing the plan, and reviewing andrefining plan implementation. The committeemeets monthly in winter and early spring andweekly in late spring and summer to reviewwater temperature and flow data and trends anddecide when to put the plan into effect. Eachfall, the committee holds a year-end meetingwhere monitoring indicator data, drought planparticipation, amount of water conserved, andoutreach activities are reviewed and possiblechanges discussed. Over time, technical assistancehas been expanded, trigger response requirementshave been revised, and additional levels of responsehave been added as a result of the annual reviews.Closing the Feedback Loop: Evaluation and Adaptation in Collaborative Resource Management 9

“Multiparty evaluation and sharedlearning are necessary to interpretresults and provide an assessment ofwhat works, what needs to change, andhow it should change.Adaptive management is an iterative process, not aone-time change in response to monitoring results. Eachtime management is changed the cycle begins again,with identification of expected outcomes of the revisedmanagement actions followed by monitoring and evaluation.Over time, this ongoing process builds understanding of howthe system works and leads to better management based onthat improved understanding.Adaptive ManagementAdaptive management is a structured process of evaluatingresource management actions and using what is learnedto adjust future management. The adaptive managementcycle begins with a clearly defined objective and anexpectation that a proposed action will lead to thatobjective. The action is then taken, results are monitoredand evaluated, and a decision is made whether or notto change management. Using agreed-upon goals andobjectives as the basis for evaluating results distinguishesadaptive management from simple trial-and-error

Closing the Feedback Loop: Evaluation and Adaptation in Collaborative Resource Management 3 Collaborative resource management grew out of a desire to overcome gridlock and better address multiple needs and concerns. In the last 30 years, collaboration has come a long way toward building understanding and trust among formerly conflicting .