WIXLIB

Chapter I Background and Overview of the GuideThe SEBM project also included an in-depth process study that collected rich information from threeAmeriCorps-funded grantees and their partners about their experiences in scaling evidence-basedinterventions. The process study generated insights about the approaches that grantees and their partnerstook to scaling, as well as the challenges and facilitators they faced with different facets of scaling(Anderson et al. 2020; Eddins et al. 2020; Jones et al. 2020; Needels et al. 2020).To improve scaling readiness and build upon the lessons learned from the process study, AmeriCorpscommissioned this guide to disseminate the knowledge gained from the SEBM project. This guideprovides recommendations that funders can use to identify which of the interventions they fund areeffective, enhance their knowledge base on scaling those interventions, and help them pursue scaling. Awide range of grantmakers, including governmental agencies and philanthropic funders, can use thisguide to learn how to systematically review evidence of effectiveness of the interventions they fund, aswell as assess how ready their grantees or funded organizations might be for scaling activities.A. Scaling Programs with Research Evidence and Effectiveness (SPREE) ProcessTo deepen AmeriCorps’ knowledge base on scaling evidence-based interventions, Mathematica designedand implemented the SPREE process under the SEBM project. The SPREE process provides acomprehensive methodology for funders to use when they want to help organizations scale effectiveinterventions (Maxwell and Richman 2019). SPREE was grounded in rigorous research standards forevidence of effectiveness and implementation science research that identified how organizations mightscale interventions so they improve desired outcomes for a larger number of participants (Miller et al.2006; National Implementation Research Network [NIRN] 2018). SPREE was further informed andrefined by the lessons learned when applying the SPREE process to a group of organizations that scaledinterventions using funding received from AmeriCorps.The SPREE process has two distinct parts (Figure I.2):1. Effective intervention. The first part ensures that the intervention to be scaled is likely to producedesired outcomes and is therefore worthy of being scaled. It is the base for preparing for scaling. Forthe SEBM project, the first part—the blue portion of Figure I.2—was designed to identifyAmeriCorps-funded interventions with evidence of effectiveness, which we defined as those that metthe project’s effectiveness standards and produced favorable findings on the majority of targetedoutcomes among participants receiving the intervention. We often refer to interventions that haveprovided evidence of effectiveness as interventions that work. For more information about thestandards used to determine whether interventions had evidence of effectiveness, see Richman andStreke (2020).2. Readiness to scale. The second part of the process—the green portion of the Figure I.2—is intendedto ensure that the effective intervention and the organization implementing it are ready to scale. TheSPREE process specifies five conditions that should be present for an intervention and organization tobe ready to scale. Having these five conditions in place before scaling helps foster fidelity during andafter scaling, which helps ensure that the intervention achieves at least the same positive outcomes forparticipants as it did before scaling. The SPREE process is important for guiding scaling because bothfidelity and effectiveness often deteriorate as participants, personnel, and resources increase (Larsonet al. 2017). The first three conditions of the SPREE process indicate that an intervention is ready toscale, and the last two indicate the organization is ready to scale it.Mathematica3

Chapter I Background and Overview of the GuideFigure I.2. Scaling Programs with Research Evidence and Effectiveness (SPREE) processSuccinctly, the SPREE process provides a framework that funders can use to understand whether theinterventions and organizations they fund demonstrate the conditions for successful scaling—meaningpositive outcomes for a large number of people—are in place. The conditions for successful scalinginclude both offering an effective intervention and having the procedures and processes in place to ensurethat it can be implemented as intended after scaling (Figure I.3). If either part is missing from the formula,funders’ goals of improving outcomes for a larger number of people may not be achieved.Figure I.3. Improving outcomes formore peopleB. Purpose of this guideDrawing on the knowledge generated from applying theSPREE process to assessing AmeriCorps’ fundedorganizations and interventions during the SEBM project,this guide provides five key recommendations that will helpfunders identify which of their funded interventions areeffective, enhance their knowledge base on scaling them,and pursue scaling. Recommendations are as follows:1. Identify or build evidence of effectiveness for theintervention2. Prepare to scale: Ensure the intervention is ready forscaling3. Prepare to scale: Ensure the organization is ready toscale the intervention4. Prepare to scale: Anticipate and plan for challengesSource: Adapted from Bertram et al. (2001).Mathematica5. Support the intervention and organization during scaling4

Chapter I Background and Overview of the GuideFor each recommendation, the guide frames theissue and describes what might need to be done toimplement the recommendation. Examples from theexperiences of AmeriCorps grantees show how theissues have played out in the field. We also providespecific examples of what funders can do to helpsupport these recommendations. For eachrecommendation, we also list resources developedby this project to help funders and their granteeorganizations build evidence that their interventionsare effective, get ready to scale, and addresschallenges and support scaled interventions.Considerations as you read thisguide The recommendations and resourcescontained in this guide weredeveloped with a focus on AmeriCorpsgrantees’ use when scaling theirinterventions; however, they may alsobe useful to a broader audience offunders. The recommendations and resourcesare based on successful scalingprinciples that have emerged fromresearch.Although recommendations are general and canapply to different types of scaling, some resources When tailoring resources, granteesmight need to be tailored based on the type ofshould ensure that the generalscaling that is relevant for a funder and its granteeprinciples in which the resources areorganization (Fixsen et al. 2005). The SEBMgrounded are still applicable andproject has focused on three types of scaling: (1)relevant.expansion, (2) replication, and (3) adaptation.Expansion extends the intervention to more peoplein the same target population in the same location.Successful expansion requires the intervention and the organization to serve a larger number ofparticipants than before with the same service quality and in a manner demonstrating fidelity to theintervention model’s design. Replication extends the intervention to the same target population, but in anew location. Successful replication requires the intervention and the organization to maintain servicequality and fidelity to the intervention in the new location. Adaptation extends the intervention to a newtarget population. Successful adaption requires the organization to change the intervention in a way thatmaintains beneficial outcomes for the population being served.Mathematica5

This page has been left blank for double-sided copying.

Chapter II Recommendation 1: Identify or Build Evidence of Effectiveness for the InterventionII. Recommendation 1: Identify or Build Evidence of Effectiveness forthe InterventionBefore an intervention is scaled to reach a largernumber of people, funders should first determine ifthe intervention has demonstrated effectiveness forthe population it initially targets. Funding thescaling of an intervention is of value only if theintervention has been shown to improveparticipants’ outcomes. Although funders and theirgrantee organizations might desire to scale anintervention based on observed improvements inparticipants’ outcomes, funders and grantees shouldconfirm that the improvements are due to theintervention and not other factors. By eitherreviewing evidence produced by third-partyevaluators and unbiased sources, or helping granteeorganizations conduct their own rigorousevaluations, funders should seek to establish whichinterventions produce beneficial outcomes for theirtarget populations before scaling thoseinterventions.A. What needs to happenWhether seeking to identify or build evidence fortheir intervention’s effectiveness, funders can useresearch and data to obtain a thoroughunderstanding of how the intervention operates,identify what improvements are needed to make iteffective, and show how it affects the participantsbeing served. Five steps can guide how fundersshould approach this issue: Make sure grantee organizations havedefined their interventions. Research andevaluation can help funders and their granteeorganizations continuously improveinterventions in addition to providing evidenceof its effectiveness. A well-articulated theory ofchange is central to this effort because itdefines an intervention and builds anunderstanding of how intervention activitieschange participant behaviors in ways thatimprove outcomes. It provides a road map thatidentifies the human, financial, and physicalresources (inputs) required to implement theMathematicaBuilding evidence: The experienceof one AmeriCorps granteeParent Possible, an AmeriCorps grantee,has participated in the design andevaluation of a possible adaptation toHome Instruction for Parents of PreschoolYoungsters (HIPPY), a home visitingintervention for parents of young children.Parent Possible and the interventiondeveloper, HIPPY USA, recently beganpilot testing an adaptation of theintervention called HIPPY for Friends,Families, and Neighbors in two sites inColorado.Although HIPPY is typically delivered toparents of young children, under thisadaptation, home visitors deliveredHIPPY activities to nonrelative homebased childcare providers, as well asfriends and family members who providecare for children while their parents areworking. Typically, home visitors met withcaregivers who watch two or threechildren in their homes and sharedactivities that the caregivers couldperform with the children in their care.Home visitors delivered 15 activities in 30weeks, offering one activity every otherweek (as opposed to 30 activities weekly,as the evidence-based HIPPY curriculumrequires).Parent Possible personnel reported someearly implementation successes andchallenges in piloting this adaptation. Forexample, one site made additionalmodifications to this adaptation of HIPPY,by meeting with caregivers as a grouprather than one on one. This facilitatedconnections among the caregivers, whichcaregivers found to be beneficial,according to grantee personnel. If HIPPYUSA chooses to disseminate thisadaptation more broadly, a granteepersonnel member said that the granteemight make recommendations to HIPPYUSA to incorporate a networking aspectinto the adaptation for the caregivers.7

Chapter II Recommendation 1: Identify or Build Evidence of Effectiveness for the Interventionintervention’s critical components (activities)that result in services provided (outputs) thatultimately increase the desired outcomes.Funders might want to require including atheory of change framework in its solicitationsfor funding and review the framework to ensurethe components are well-defined and theoverall theory of change appears to be sound.How can funders help buildevidence of effectiveness?Funders have a vested interest inensuring that interventions are likely tohelp participants improve their outcomes.When funders make decisions aboutwhich applications for funding they willsupport, they could ensure that adequateweight is placed upon whether or not thefunding applicant demonstrates that theintervention to be implemented or scaledhas evidence of effectiveness. To fosterknowledge about interventions withevidence of effectiveness, fundingannouncements could highlightclearinghouses that provide informationabout effective interventions, such as theU.S. Department of Education’s WhatWorks Clearinghouse (WWC), the U.S.Department of Labor’s Clearinghouse forLabor Evaluation and Research (CLEAR),the U.S. Department of Health andHuman Services’ Home Visiting Evidenceof Effectiveness (HomVEE) project andPathways to Work EvidenceClearinghouse, and the Pew CharitableTrusts’ Results First ClearinghouseDatabase. Funders also could considerdedicating portions of their funding tosupport the building of evidence abouteffectiveness, if they do not already doso. Build the groundwork for the evaluation. Afunder can help build research evidence and useit to make decisions if the funder has a cultureof measurement, learning, and evaluation inplace. Such a culture requires leadership,management, and staff to develop a commonunderstanding about the value of measurementand evaluation in decision making and to agreeon what constitutes evidence of an effectiveintervention (Austin and Claassen 2008a).Having such a culture also strengthensgrantees. Although some grantees might havean established culture of learning that includesmeasurement and evaluation, others mightrequire a cultural change to accommodateevidence-based decision making. Funders canprovide grantee organizations with funding orother support to help them understand howevaluations can be used for improvement aswell as decide whether to continue anintervention (Austin and Claassen 2008b). Devote resources for grantee organizationsto conduct evaluations. Evaluations requirefinancial and personnel resources. Although there may be trade-offs between dedicating the fundsnecessary for an evaluation and providing resources for implementation, using information from anevaluation—ideally, conducted by an unbiased, third-party evaluator—for data-driven decisionmaking increases the likelihood that the funder’s goals are achieved for producing beneficialoutcomes for program participants. Recognizing that evaluations take financial, personnel, and otherresources (such as subcontracts to third-party evaluators), funders should seek to devote sufficientresources to support organizations in conducting evaluations. Promote or encourage use of rigorous evaluation methods. Most interventions have testimoniesfrom participants stating how the intervention changed their lives, but such anecdotal evidence doesnot provide convincing evidence that the intervention caused those changes. To demonstrate that theintervention caused improvements in participants’ outcomes, evaluation research must meet industrystandards for rigor and objectivity. Rigorous research studies that establish causal impact provide thestrongest evidence about whether a program improved the outcomes that it intended to improve.Mathematica8

Chapter II Recommendation 1: Identify or Build Evidence of Effectiveness for the InterventionFunders can develop and provide information, resources, and guidelines to help organizationsunderstand the standards for producing such rigorous research. Facilitate continuation of the cycle, as needed. If an evaluation does not demonstrate that theintervention is effective, the theory of change can help the funder and grantee organization identifywhy that might be the case and how the intervention can be improved. After making changes toimprove the intervention, funders should again engage in supporting evaluations to establishintervention effectiveness. Research and evaluation can be viewed as part of an ongoing, cyclicalprocess of service delivery, assessment, and adjustment to ensure that resources are used effectivelyin efforts to improve participants’ lives.B. Resources from the SEBM projectThe SEBM project developed five resources to help funders and their grantee organizations engage inresearch and evaluation to continuously improve interventions and demonstrate their effectiveness:“Evidence of Effectiveness in AmeriCorps-Funded Interventions” (Richman et al. 2020). This reportsummarizes three years of Mathematica assessments of AmeriCorps’ State and National and SIF grants toidentify which AmeriCorps-funded interventions had demonstrated evidence of effectiveness, defined byindustry standards for evaluation rigor and objectivity.“How to Fully Describe an Intervention” (Maxwell 2019). This practitioner’s guide was designed tohelp organizations thoroughly describe their intervention and communicate the following to potentialfunders or stakeholders: (1) the intervention model as it was designed, (2) the intervention as it isimplemented, and (3) the parts of the intervention and the population that were evaluated and itseffectiveness.“What Makes for a Well-Designed, Well-Implemented Impact Study” (Anderson and Maxwell2019). This practitioner’s guide was designed to help organizations ensure that their evaluators producehigh quality impact studies that provide evidence about whether an intervention is effective and whether itis worth scaling to reach more people.“Baseline Equivalence: What It Is and Why It Is Needed” (Anderson and Maxwell 2018). Thispractitioner’s guide was designed to help organizations work with researchers to design an impact studywith baseline equivalence—that is, a study with two groups of individuals that are (from a statisticalperspective) identical to each other except that one group, a treatment group, can receive the intervention,whereas the other group, a comparison group, cannot.“Making the Most of Data” (Friend and Needels 2019). This practitioner’s guide highlights issues fororganizations to consider when using, collecting, and managing their data.Mathematica9

This page has been left blank for double-sided copying.

Chapter III Recommendation 2: Ensure the Intervention is Ready to ScaleIII. Recommendation 2: Ensure the Intervention is Ready to ScaleImplementation science literature points to threeconditions that an intervention must have to enableit to be scaled with fidelity—that is, with strongcorrespondence between the intervention asintended and as implemented. First, the interventionneeds to have its core activities clearly specified sothey can be implemented in a manner that willachieve beneficial impacts on participants’outcomes that are at least as large as thoseidentified through prior rigorous research about theintervention’s effectiveness. This first condition—specificity of the intervention activities—should beaccompanied by a clear description of the targetpopulation to be served by the intervention (thesecond condition) and by the supports needed toensure that the intervention is implemented withfidelity (the third condition).A. What needs to happenFunders should critically assess whether anintervention is ready for scaling before scaling it.Doing so can help with strategic planning, such asmaking funding decisions or understanding whatadditional supports and resources granteeorganizations may need to prepare for scaling.Specifically, they should ascertain whether thegrantee organization’s intervention is wellspecified, has a well-defined target population, andhas adequate implementation supports in place.The intervention must be well-specified. Thisincludes having a clear understanding of thefollowing:Specifying and monitoring anintervention: The experience of anAmeriCorps granteeThe Child Abuse Prevention Council(CAPC), an AmeriCor

wide range of grantmakers, including governmental agencies and philanthropic funders, can use this guide to learn how to systematically review evidence of effectiveness of the interventions they fund, as . 2006; National Implementation Research Network [NIRN] 2018). SPREE was further informed and