WIXLIB

Evaluation GuideFundamentals ofEvaluating PartnershipsEvaluation and Program Effectiveness Team . Division for Heart Disease and Stroke Prevention

Fundamentals of Evaluating Partnerships

AcknowledgmentsThis guide was developed by the Centers for Disease Control and Prevention, Divisionfor Heart Disease and Stroke Prevention under the leadership of Susan Ladd and JanJernigan, working in collaboration with Tim LaPier.State Heart Disease and Stroke Prevention Programs were invaluable in thedevelopment of this Evaluation Guide. We thank:Carrie Daniels, Oklahoma State Department of HealthSusan Mormann, North Dakota Department of HealthMario Rivera, Colorado Department of Public Health and EnvironmentAbha Varma, North Carolina Department of Health and Human ServicesNamvar Zohoori, Arkansas Department of HealthWe encourage readers to adapt and share the tools and resources in the Guide in orderto meet their program evaluation needs. For further information, contact the Divisionfor Heart Disease and Stroke Prevention, Applied Research and Evaluation Branch atcdcinfo@cdc.gov or (770) 488-2560.Suggested citation:Centers for Disease Control and Prevention. Evaluation Guide: Fundamentals ofEvaluating Partnerships. Atlanta:U.S. Department of Health and Human Services; 2008.

Table of ContentsIntroduction to the Evaluation Guides . iProgram guidance on partnerships . 1Outline of the process . 3Use and user: how will the evaluation results be used and by whom? . 5Engage stakeholders . 6Describe the partnership . 7Partnership logic modelPartnership stage of developmentFocus the evaluation design . 9Determine the evaluation questionsPrioritize evaluation questionsEvaluation designGather credible evidence . 11Justify conclusions . 12Ensure use and share lessons learned . 13Example: evaluation plan for evaluating your partnership . 14Increase the success of your evaluation . 16Tools . 17Additional resources . 18AppendicesAppendix 1: Sample Evaluation Questions and Methods . 20Appendix 2: Partnership Evaluation Plan Template . 22Appendix 3: Partnership Membership Assessment Tool . 25Appendix 4: Processes of Partnership Operation . 29Appendix 5: Evaluation Content by Stage of Development . 30Appendix 6: Sample Meeting-Effectiveness Survey . 31Appendix 7: Coalition Effectiveness Inventory . 32Bibliography . 37

Introduction to the Evaluation GuidesPurposeThe evaluation guides are a series of evaluation technical assistancetools developed by the CDC Division for Heart Disease and StrokePrevention (DHDSP) for use by state and local health departments.The guides clarify approaches to and methods of evaluation, provideexamples specific to the scope and purpose of health departments, andrecommend resources for additional reading. The guides are intended tooffer guidance and a consistent definition of terms. The guides are alsointended to aid in skill building on a wide range of general evaluationtopics while recognizing that health departments differ widely in theirexperience with, and resources for, program evaluation.The guides supplement existing program guidance and programevaluation documents. As they are developed, guides are posted onthe DHDSP Evaluation Resources website (https://www.cdc.gov/dhdsp/evaluation resources.htm). Health department staff are encouraged toprovide feedback to the Applied Research and Evaluation Branch on theusefulness of the guides and to suggest additional guide topics.ResourcesConducting partnership evaluation requires both staff and fiscalresources. Before planning such an evaluation, it is necessary to firstidentify funds in the program budget and staff who can lead the work.It is not unusual to dedicate 5-10% of a project budget to evaluation.Assistance with budgeting can come from discussion with colleagues inthe health department and contracting offices about the costs of previoussimilar evaluation activities.Partnership evaluation is a good collaborativeactivity for state and local chronic disease programs, who can sharedevelopment and implementation costs. State colleagues may alreadyhave partnership evaluation tools or strategies they would be willing toshare. Partners may also have evaluation staff that could help plan andconduct evaluation activities.Universities and Prevention Research Centers (http://www.cdc.gov/prc/) are also good evaluation resources. Check for evaluation classes orprograms that require class projects, a master’s thesis, or an internship.Student energy and faculty leadership on these projects make for awinning combination. Ask about consulting services or communityservice projects as well.The American Evaluation Association is an association of professionalevaluators that is “devoted to the application and exploration of programevaluation, personnel evaluation, technology, and many other formsof evaluation” (http://www.eval.org). American Evaluation Associationaffliates are located throughout the United States. Check with a localaffliate for potential resources.

Develop an Evaluation PlanThis guide applies the CDC Evaluation Framework(https://www.cdc.gov/eval/framework/) toevaluating your partnership. The Frameworklays out a six-step process for the decisions andactivities involved in conducting an evaluation.While the framework provides “steps” for programevaluation, the steps are not always linear; somecan be completed concurrently. In some cases, itmakes more sense to skip a step and come backto it. The important thing is that all the steps areaddressed. The steps and a brief description of eachare listed below. Each is described in more detailon the pages that follow. Sections of the guide arelinked to this outline and the CDC framework by a“bubble” graphic in which the highlighted bubbleidentifies the corresponding point in the framework.As you work through the next sections of the guideand begin planning your partnership evaluationactivities, remember to add evaluation questionsand methods to an evaluation plan. Additionalguidance and a template are provided in the“Developing an Evaluation Plan” guide located onthe DHDSP Evaluation Resources website. Theelements of the evaluation plan to be identifiedthrough this planning process are:Evaluation questions.Indicators – measures needed to answer theevaluation questions.Data sources.Data collection methods.Determine how the evaluation results will be used and by whom. Identifyresources available for the evaluation, including money, staff time, andexpertise. Begin developing an evaluation plan.1. Identify and engage evaluation stakeholders. Plan for how they will beinvolved in, and will contribute to, the evaluation.2. Describe the partnership’s members, activities, products, expectations, andoutcomes. Develop a logic model to depict the partnership’s theory of change(i.e., how activities will accomplish goals). Identify the stage of developmentof the partnership. Identify contextual factors that will impact effectiveness.These will be helpful in developing evaluation questions.3. Brainstorm and then finalize a list of questions the evaluation will answerrelated to effective processes, partnership activities, and expected outcomes.These will form the basis of an evaluation plan.3Use & UserStakeholdersDescribeFocus4. Determine how you will answer the evaluation questions by identifyingindicators, data sources, how you will collect data, and a timeline for datacollection. Identify who is responsible for seeing that the work gets done. Pilottest tools. Collect the data.Evidence5. Enter and check the data for errors. Analyze the data in a way that willmake sense to the program partners. Interpret the data to reflect the currentcontext. Consider and document factors that may affect or bias the findings.Compare findings with benchmarks or with what others have found.Justify6. Distribute and use evaluation results. Report often along the way. Tailor theformat and the mechanism of reporting to the specific audience.Use & ShareEvaluating Partnerships

Time frame for evaluation activities.Data analysis.Communicating results – to whom and in whatformat.Lead person responsible for overseeing the work.As you make decisions, information can be added toa table similar to the following:Objective:Evaluation Indicators/Questions ysisCommunicateResultsLeadA completed example of an evaluation plan isprovided on page 14. A blank planning template isprovided in this document as Appendix 2.Evaluating Partnerships4

Improve and guide partnership activities.Evaluation can be used to assess partnershipinterventions and activities so that successfulstrategies can be supported and replicated.Determine whether goals or objectives havebeen met. Achieving objectives provides a senseof accomplish to members and demonstratesto funders that the partnership is a goodinvestment.Promote the public image of the partnership. Apartnership with a positive public image may findit easier to recruit new members, retain existingmembers, secure additional resources, gainaccess to needed data, etc.Partnership leadership.Partnership organizers.Partnership members.Use & UserStakeholdersFunders.People affected by partnership activities.Potential partners.The evaluation sponsor (such as the partnershipfunder or leader) should work with the evaluatorto ensure that the intended use and users of theevaluation are agreed upon. The evaluator willuse this information to direct and focus evaluationactivities, set timelines, and select communicationstrategies.Build capacity for evaluation within thepartnership. People unfamiliar with evaluationmay be uncomfortable with the idea of “beingevaluated.” However, engaging partnershipmembers in evaluation may help reduce this“evaluation anxiety”. Engaging partners inevaluation tasks may increase their appreciationof the usefulness of evaluation and providepartners with evaluation skills they can apply tothe partnership or their own organization.5DescribeExamples of potential users of the partnershipevaluation include:FocusImprove the functioning and productivity ofstate partnerships. Evaluation can identifypartnership strengths and areas for improvementin operating processes, structure, planning, andactivity implementation.The intended user of an evaluation will influencemany aspects of the evaluation as well, includingthe prioritizing of evaluation questions and howevaluation results are communicated. Identifyingeffective communication strategies early in theevaluation process facilitates planning especiallywhen multiple stakeholders are involved andmultiple communication methods are needed.Evaluating PartnershipsEvidenceSome potential uses of partnership evaluationinclude:Evaluating partnerships can be resource intensive;therefore, it is critical that mutual uses and benefitsof such an effort be clearly understood by allinvolved. Otherwise, partners may see evaluationas taking time away from the “real” work of thegroup.JustifyBefore any other evaluation planning takes place,the purpose of the evaluation and the end user ofthe evaluation should be clearly understood. Thesetwo aspects of the evaluation serve as a foundationfor evaluation planning, design, and interpretationof results. The purpose of an evaluation influencesthe identification of stakeholders for the evaluation,selection of specific evaluation questions, andthe timing of evaluation activities. If evaluationfindings are intended for use in funding or planningdecisions, the evaluation activities will have to betimed to meet that expectation.Provide accountability to funders and partners.Accountability applies to not only achievingresults, but managing resources. It also appliesto valuing the partners’ time and opinions.Use & ShareUse and User: How Will TheEvaluation Results be Used andby Whom?