Transcription
THE COUNTER-NARRATIVEMONITORING &EVALUATION HANDBOOKLouis ReynoldsHenry Tuck
AboutThe Institute for Strategic Dialogue (ISD) is a London-based ‘think and do tank’ that has pioneered policyand operational responses to the rising challenges of violent extremism and inter-communal conflict.Combining research and analysis with government advisory work and delivery programmes, ISD hasbeen at the forefront of forging real-world, evidence-based responses to the challenges of integration,extremism and terrorism.About this HandbookGiven the proliferation of violent extremist content online in recent years, developing effective counternarratives — messages that offer a positive alternative to extremist propaganda, or deconstruct ordelegitimise extremist narratives and challenge extremist ideologies — is an increasingly necessaryalternative to online censorship. The evaluation of counter-narrative campaigns, in order to assess impact,build on best practice and raise the quality of counter-narrative campaigning is vital to this effort.This Handbook has been funded with support from the European Commission. It was created by theInstitute for Strategic Dialogue (ISD) to help anyone looking to evaluate their counter-narrative campaigns,and is intended as a beginner’s guide for those with little or no previous experience of counter-narrativecampaign evaluation. This publication reflects the views only of the authors, and the European Commissioncannot be held responsible for any use which may be made of the information contained therein.It takes readers through the main stages of designing and implementing an effective counter-narrativecampaign evaluation. It can also be used alongside ISD's freely available online Counter-narrative Toolkit,which can be found at www.counternarratives.org, The Counter Narrative Handbook, which can be foundat s/2016/06/Counter-narrative-Handbook 1.pdf, andThe Impact of Counter-Narratives report, which can be found at /2016/08/Impact-of-Counter-Narratives ONLINE 1.pdf.Our advice is based on ISD’s experiences in creating, running and evaluating in-house campaigns suchas Extreme Dialogue, and collaborating with independent content-creators, from civil society and NGOcampaigners to young activists, to amplify their counter-narrative messages through training, networkingand campaign support. This Handbook therefore focuses on civil-society, youth or NGO-led online counternarrative campaignCo-funded by the Preventionof and Fight against CrimeProgramme of the EuropeanUnion
About the authorsLouis Reynolds is a Policy & Research Coordinator at the Institute for Strategic Dialogue. Louis worksacross the Institute’s educational and research programmes. Formerly a Researcher at the cross-partythink tank Demos, Louis holds a Masters in Intelligence and International Security from King’s CollegeLondon and a BA in War Studies from Birmingham University.Henry Tuck is a Policy & Research Manager at the Institute for Strategic Dialogue. Henry manages theInstitute’s educational programme, Extreme Dialogue, which aims to enhance critical thinking skills andresilience among young people, and is involved across various projects to counter extremism online.He holds a Masters in International Conflict Studies from Kings College London and a BA from DurhamUniversityAcknowledgementsThe authors would like to thank Kelsey Bjornsgaard, Lucie Parker and Christopher Stewart for their invaluable support throughoutthe process, and Zahed Amanullah and Jonathan Birdwell for their editorial input. All errors and omissions are the sole responsibilityof the authors.Copyright 2016 Institute for Strategic Dialogue, All rights reserved. Any copying, reproduction or exploitation of the whole or anypart of this document without prior written approval from the Institute for Strategic Dialogue is prohibited.This material is offered free of charge for personal and non-commercial use, provided the source is acknowledged. For commercialor any other use, prior written permission must be obtained from the Institute for Strategic Dialogue. In no case may material be sold,altered, or rented. The Institute does not generally take positions on policy issues. This views expressed in this publication are thoseof the authors and do not necessarily reflect the views of the Institute for Strategic Dialogue or those of the European Commission.
Table of ContentsINTRODUCTION2 23Why is measuring the impact of counter-narrative campaigns important?What this Handbook presentsGLOSSARY6EVALUATION METHODS10 10101213151819192020202121Goals and objectivesMetricsAwarenessDemographics and advertising toolsEngagementWebsite engagement metricsImpactQualitative and quantitative methodsSentiment analysisGeo-location analysisOnline surveysIn-depth interviewsFocus groupsRISKS AND ETHICS24 24242425252525262626 Campaign risks Vanity metrics Poor targeting ViralityEthical risks Measuring negative outcomes Risk to audiences and campaigners Sustained engagement and safeguarding planning Data protection Anonymity in M&E reportingMONITORING AND EVALUATION FRAMEWORK30 303132333333Stage 1: Preparatory planningStage 2: ScaleStage 3: DemographicsStage 4: InteractionsStage 5: Additional researchStage 6: Cost and time
Metrics considerations table35EVALUATION CASE STUDIES40 404345Extreme DialogueAverage MohamedExitUSATOOLKIT50 Simple toolsAdvanced toolsComplex tools505152FURTHER READING54 545455Counter-Narrative Campaign Evaluation Case StudiesCounter-Narrative Theory and Best PracticeOnline Safety and Engagement
INTRODUCTION
IntroductionWhy is measuring the impact of counter-narrative campaigns important?The promotion of narratives legitimising extremist ideology, justifying the use of violence,and sowing division within societies is critical to the success of violent extremist groups.Counter-narratives can play a vital role in combatting extremist propaganda, promotingpositive alternatives to extremist messages, and deconstructing and delegitimisingextremist narratives. In the last few years civil society counter-narrative campaigns havebecome a core part of national and international counter extremism strategies.As internet usage has grown around the world, the digital commons has become a vitalspace in which to explore, develop and contest political, social and cultural positions.Young people especially now spend increasing amounts of time online, primarily throughsocial media. Extreme groups have proved adept at harnessing the potential of theinternet to radicalise and recruit. As a result, online counter-narrative campaigns havebecome increasingly important to ensure that online communications by extreme groupsand their supporters do not remain uncontested.Social media presents a practical and cost-effective way to deliver counter-narrativecampaigns to vital groups. It allows campaigners to reach individuals who are becominginvolved in or are part of extremist organisations, to spread awareness to broaderaudiences, to reach out to parents and teachers, and to otherwise engage people in thefight against extremism.Yet social media is not just a practical, accessible medium. The online spaces used byyoung people are contested ground, where a range of actors, not least extremist groupsand individuals, seek to influence young people’s values, perspectives and actions. If civilsociety fails to challenge extremism online, we not only lose an opportunity to positivelyinfluence young people, but also make it more likely that extremists will be able to do so.While the number of online counter-narrative campaigns has increased, those applyingeffective Monitoring and Evaluation practices (M&E) remains limited. Unfortunately, manycampaigners do not evaluate their campaigns, or conduct a limited evaluation after thefact, as opposed to considering M&E throughout the design and delivery phases. Thereare many factors that can deter civil society campaigners from undertaking effectiveevaluations, from tight delivery timeframes and a lack of evaluation expertise, toinsufficient public or private sector support or funding.This lack of effective M&E, particularly amongst smaller civil society campaigners, meansthat we have limited knowledge about the effectiveness of many counter-narrativecampaigns, and little consensus around what works and does not. It also means thatmany powerful campaigns do not always receive the necessary long-term funding orsupport.Counter-Narrative M&E Handbook 2
To make the most of limited resources, governments and other funders are increasinglyinsisting that effective M&E is embedded throughout all counter-narrative campaigns,and restricting continued funding to organisations that can effectively demonstrate thatthe activities they undertake have a measurable impact. All organisations involved incounter-narrative campaigning must therefore be capable of undertaking effective M&E.Effectively measuring the impact and outcomes of a counter-narrative campaign onlineis difficult. Correlation – a connection between two factors, for example the launching ofa counter-narrative campaign and a reduction in the profile of an extremist group in thesame time period – is not the same as causation, where one factor can be shown to haveinfluenced the other. Causally connecting your project activities to often abstract andcomplex goals, such as reduced interest in extremist narratives, is a significant challenge.Successful evaluation requires forward planning, fixed goals and objectives, and clearcriteria for success.While evaluating counter-narrative campaigns can be a challenge, counter-narrativecampaigners should not be put off. Even simple M&E efforts can provide useful insightsand conclusions. Moreover, effective M&E does not necessarily require significantexpertise in research or evaluation. This Counter-Narrative M&E Handbook is thereforedesigned to help civil society campaigners design and execute effective M&E for theirown campaigns, understand the impact they are having, and improve the effectivenessof future campaigns.The Counter-Narrative Monitoring & Evaluation HandbookThe aim of this Handbook is to provide civil society campaigners with the knowledgeand skills they need to undertake high quality counter-narrative campaign evaluations.It presents model frameworks and case studies of successful counter-narrative M&E, aswell as a range of useful tools and resources that counter-narrative campaigners can useto support their M&E efforts, including: A description of key metrics and how to interpret them.An explanation of various research methods that can be used in online M&E.Guidance on potential risks and ethical considerations for counter-narrative M&E.A flexible M&E framework that can provide a starting point for evaluation planning.Three case study examples of high quality counter-narrative campaign evaluations.A glossary of key M&E terms.An index of useful online M&E tools.This Handbook is part of a larger package of counter-narrative resources for civil societycampaigners offered by ISD: 3 For a beginners guide to counter-narrative campaigning, designed for those withlittle or no previous experience, see our 2016 publication The Counter-NarrativeHandbook.For a step-by-step guide to creating a counter-narrative campaign, frequently askedInstitute for Strategic Dialogue
questions about campaign development, a best practice guide and case studyexamples of successful counter-narrative campaigns, go to www.counternarratives.org.For a case study-based examination of the impacts of social media counter-narrativecampaigns, see our 2016 report The Impact of Counter-Narratives.Counter-Narrative M&E Handbook 4
GLOSSARY
GlossaryAntagonistic sustained engagements:where someone repeatedly disputes ordismisses the content or campaign to otherusers (or campaigners), and/or expressesextreme views or offensive slurs.Clicks: the number of times people haveclicked on your ad or a link in your posts.Constructive sustained engagements:where someone comments more thanonce in a positive manner in an onlineAudience retention: the amount of your conversation about the content, campaignvideo your viewers watch, expressed as or issue.either the average amount of time viewedor the percentage still watching at a given Counter-narrative: a message thatpoint. May also be referred to as the “drop- offers a positive alternative to extremistoff” rate.propaganda, and/or aims to deconstructor delegitimise extremist narratives andAwareness: the extent to which your propaganda.content is seen by the desired audiences(including impressions, reach and video Downstream: a campaign aimed at thoseviews.at-risk of becoming extremists, viewingextremist content, or actively participating inAwareness metrics: metrics that indicate online extremist communities or networks.the number of people exposed to acampaign (e.g. impressions, reach or video Engagement:interactionbetweenviews), and demographic information (e.g. audience members, or with campaignersage, gender or geographic location) that themselves, which could be positive orprovides insights as to whether the right negative and help provide insights intoaudience was reached.reactions to a campaign.Boosted posts: advertised posts or tweets Engagement metrics: metrics that showthat have been promoted to appear in how much, and in what manner, peopleselected audiences’ news feeds.interacted with a campaign’s content, socialmedia accounts or websites (includingBounce rate: the number of people that visit video retention or drop-off rates, numbersyour website and then leave again without of likes, comments or shares).moving past the landing page. This is mostfrequently expressed as a percentage.Exit rate: the percentage of users wholeave your website from a given page,Call to action: a call to action asks audiences rather than clicking a link or button onto do something immediately in response your page and continuing to more of yourto your campaign, and makes it clear why website or content.it is important. This could be as simple asasking the audience to share your video, or Impact: a measurable change in audiencea bigger ask such as volunteering their time attitudes, behaviour or knowledge (onlineto support your campaign or organisation. or offline), ideally constructive, that can beCounter-Narrative M&E Handbook 6
attributed to exposure to, or engagement Reach: the total number of people thatwith, counter-narrative content.received an impression of your post or adon their screens or newsfeeds.Impressions: The number of times contentor adverts appear on a user’s screen online. Sustainedengagements:on-goingAvoid over-relying on impressions as a interactions between audience members ormetric of true reach or impact, as people with campaigners themselves. These coulddon’t necessarily take notice of every ad be positive or negative and help providethat appears on their screen.an insight into reactions to a campaign.Impression frequency: the number of timesyour ad or post appeared on an averageindividual’s screen. Use this metric toensure you’re not over-exposing the samepeople to your campaign.Upstream: a preventative campaign aimedat a broader, but still specific audience,with the intention of building resilienceto extremist narratives or propaganda, orincreasing knowledge and awareness ofradicalisation, recruitment or online safety.In-built analytics: analytics servicesavailable on social media platforms that Views: the number of times a video isallow you to monitor your campaign’s watched or played.reach and levels of audience engagement,and help determine whether your onlineobjectives were met.Metrics: different social media or websiteanalytics services will offer different typesof data or ‘metrics’. There are a vast rangeof different metrics that can help youunderstand who you reach, how well youengage your audience, and the impactyour campaign is having.Natural Language Processing: theapplication of artificial intelligence (AI) tohuman language – words, phrases andsentences that convey meaning – in orderto observe patterns and analyse content.Organic growth/reach: audience reach orengagement on websites or social mediaplatforms generated from searches, and/oras a result of unpaid campaign strategiesand tactics.Promoted content: videos, posts or tweetsthat have been promoted through paidadvertising to selected audiences on socialmedia or search platforms.7 Institute for Strategic Dialogue
Counter-Narrative M&E Handbook 8
EVALUATION METHODS
Evaluation MethodsThis chapter explains the metrics and methods you might use to evaluate an onlinecounter-narrative campaign.Goals and ObjectivesThe very first thing to consider in the planning phase of a counter-narrative campaign iswhat its overall goal is, and within that what objectives might be set for the campaign.Setting a clear goal will keep your campaign true to its original purpose. Setting a series ofobjectives, related to the overall goal of the campaign, will provide a series of quantifiablemilestones against which to measure the success of a campaign: Objectives should be specific. They should relate not to abstract ambitions such as‘engage young people’, but quantifiable measures of a desired effect, such as ‘have1000 online conversations with young people through comments or direct messages.’ They should also be measurable, with campaigners confident before the inception ofthe campaign that they will be able to discern, from available metrics and evaluationactivities, whether or not they were achieved. Finally, they should be realistic. Considering a campaign’s time-span, budget,intended audience size and available resources, as well as the performance ofprevious campaigns, can help campaigners decide what objectives are realistic.MetricsThe monitoring and evaluation of online counter-narrative campaigns centres on themetrics drawn from the websites and social media platforms used during your campaign.There are hundreds of different metrics that can be observed and analysed in manydifferent ways as part of the M&E process. Deciding which metrics matter, and how youwant to analyse them, is an important part of M&E planning and should be establishedbefore a project starts.Ultimately, the key to understanding whether your campaign met its goals or not is tohave a clear understanding of what success and failure would look like, and to decidewhich metrics are best placed to help you judge the eventual outcomes of the campaign.Counter-Narrative M&E Handbook 10
Social media metrics can be drawn into two broad categories, awareness and engagement: Awareness is the total number of people who view your campaign content (whethervideos, ads, posts, websites or other digital content), and who those people are. It isimportant to consider awareness metrics which help you understand the characteristicsof your audience: how, when and where they were exposed to your campaign, andwhich parts of the campaign reached them. Engagement is defined as the volume and types of interaction between audiencemembers, campaigners or campaign materials. Engagements can include everythingfrom likes and shares to email responses, and can be positive or negative. The numberand nature of engagements can help campaigners understand their audience’sreactions to the campaign or its content.These metrics can be combined and analysed to build a comprehensive picture of acounter-narrative campaign's performance, and help campaigners understand the impactthey are having. Impact is a measurable change in the audience’s knowledge, attitudes or behaviourthat can be attributed to exposure to or engagement with counter-narrative content.Awareness and engagement metrics, when properly analysed, can be broughttogether to help evaluators understand the impact of their campaign. Additionalindicators, such as evidence of offline action, or the qualitative evaluation of onlinecomments, can contribute to the overall impact picture. The measurement of impact– and ultimately the answer to the question of whether a campaign succeeded orfailed – will be defined by the goals and objectives set at the very start of the project.In all counter-narrative campaign M&E, awareness and engagement are critical tounderstanding impact. However, the relative importance of each metric, and the particularmetrics evaluators focus on, is dependent on the type of counter-narrative campaign.For example, a downstream counter-narrative campaign – one designed for thoseindividuals who are at higher risk of joining extremist groups – might place a greateremphasis on metrics that measure sustained engagement. An upstream counternarrative campaign – focused on a broader audience – might by contrast concentrateon achieving greater awareness. If a campaign is intended for a particular audience, itwould be appropriate to consider not only awareness in general, but the awareness ofthe specific audience groups which are the focus of your campaign.Presented below is a summary of each type of metric, how they are presented throughsocial media data, and what considerations should be made when using them forcampaign analysis and evaluation.11 Institute for Strategic Dialogue
AwarenessThere are two types of awareness metrics. The first relates to the scale of the audience:how many people saw or interacted with your campaign in any way. The seconddescribes the nature of the audience in demographic or characteristic terms, such asgender, location, age, device type and so on.It is important to note that while it is possible to observe broad demographic characteristicsof your audience through analytics tools, it is not possible to observe the demographicdetails of individual users. This protects the privacy of individual social media users.There are a number of metrics through which to consider awareness that are often vitalto counter-narrative M&E.Impressions occur when your content, often a post or an ad, appears on someone’sscreen. Total impressions represent the most basic and broad measure of youraudience. In fact, it is entirely possible that an impression can take place withoutan individual even noticing your content. For this reason, it’s important not to focustoo much on impressions, which are often overvalued in M&E campaigns as a largenumber which is relatively easily generated.For paid campaigns however, examining the demographic breakdown of yourimpressions is an effective way of checking whether your ad targeting is accurate– whether your content promotion is creating impressions with the right groups.Impressions can also provide a baseline measurement of whether your content isattracting an audience. A high number of impressions but a relatively low number ofclicks on your content might indicate that your content is not appealing, while a lownumber of impressions and a relatively high number of clicks might suggest that it isjust not reaching enough people.Impressions is a term used by Facebook, Twitter, YouTube and other social mediaplatforms, and can be measured through their content promotion and advertisingtools. However, Facebook, Twitter and YouTube each offer slightly different definitionsof an “impression”. For Facebook an impression includes the number of times thecontent from your page is displayed, for example in a newsfeed or side-bar; forYouTube, a ‘video impression’ is counted when the content or ad appeared as an instream advertisement, and is different to a ‘thumbnail impression’, which is countedevery time someone sees your video ad displayed with your video thumbnail; andfor Twitter, an impression is defined as the number of users who see a PromotedCampaign’s content.Counter-Narrative M&E Handbook 12
Twitter Ads Manager impressions chartReach is the total number of people who receive an impression of your content ontheir screen. Total reach will be lower than the total number of impressions, as somesocial media users will receive more than one impression of your content.Reach is a term used by Facebook, Twitter, YouTube and other social media platforms.It can be measured through Facebook’s content promotion and advertising tools,and through AdWords, the system used by YouTube. It can also be measured throughthird party applications and services on Twitter, but it is not included in their standardadvertising or promotional tools.Impression frequency is the number of times an individual saw your content over adefined period of time. If the content you are promoting is being delivered throughads, impression frequency can help you understand whether you are spending toomuch money too quickly, or whether your targeting criteria is too narrow. If you findyour impression frequency is very high, this might be the case. Too high an impressionfrequency can over-saturate individual users, irritate your target audience, or leavepeople feeling unfairly singled out by your campaign.Views refer to the number of times a video is watched or played. What counts as aview varies across social media platforms. For example, on Facebook (where a videois automatically played without sound on a user’s news feed) or Instagram, a videois counted as ‘viewed’ if it is watched for 3 seconds or more, while on Twitter (whichalso employs auto-play) the same rule applies as long as a video is 100 per cent ona user’s screen for those 3 seconds. For YouTube a view is only counted if a videois watched for 30 seconds or more. These varying definitions mean that views arerepresentative of a more or less substantive audience interaction with the content ondiffering platforms. This is one of the reasons why viewer retention or drop-off rates,explored in the engagement section of this chapter, can be valuable.13 Institute for Strategic Dialogue
The table below presents an overview of how different social media platforms definedifferent metrics:METRICFACEBOOKIMPRESSIONSFacebook definesimpressions as thenumber of times apost from your pageis displayed in a newsfeed or sidebar.REACHVIEWSTWITTERYOUTUBEYouTube definesOn Twitter, the numberimpressions as theof impressions isnumber of times your defined as the numbercontent or ad appearsof times a promotedin an in-stream or intweet appears in adisplay advertisement.user’s feed.Facebook definesOn YouTube’sreach as the number of advertising functionpeople who receivedAdWords, reach isimpressions of a page determined as ‘uniquepost.viewers by cookie.’This means that reachis calculated by thenumber of uniquecookies (which storeindividual’s preferencesand useful informationon their browsers) thatview a video.There is no way tomeasure reach throughTwitter’s in-builtanalytics, and thereforeno Twitter definition ofreach. However, reachcan be calculated orapproximated throughthird-party tools, suchas Tweetreach orHashtracking.For Facebook, a view For YouTube, a view ofof a video counts if the a video is only countedvideo runs on autoplay if that video is watchedfor more than thirtyfor a minimum of threeseconds. For videosseconds. Facebookunder this length, aautoplay is muted,view is counted asbut even if the viewerdoesn’t click to engage a percentage of thevideo’s length.audio, it still counts asa view. Facebook alsodisplays metrics forvideo views at 25%,50%, 75% and 95%,as alternative waysof counting views orcalculating drop-off.For Twitter, a view iscounted when a videoplays for three seconds,with the condition thatthe video is 100% visibleon a user’s screen for atleast three seconds.Counter-Narrative M&E Handbook 14
Demographics and advertising toolsAdvertising tools can also provide an indication of the demographic breakdown of the audience being served content. These types of insights can help an organisation understand who is respondingto their content, and to consider designing different content for more specific demographic groups(e.g. a specific video designed for parents and promoted specifically to parents through advertisingtools).Demographic and location data will not however reveal individuals within your audience. For example, although Google Analytics enables demographic data to be collected on certain users, it alsoapplies limits on the availability of age, gender and interests data under certain circumstances. Athreshold may be applied to prevent the possibility of inferring the demographics or interests of individual users. If this threshold is not met then some information is withheld for that category resultingin an incomplete picture of the data.Facebook Ads Manager campaign demographics displayEngagementEngagement metrics demonstrate the extent to which individuals interact with a campaign’scontent, social media accounts, campaigners or websites. Metrics like impressionsor reach, used for understanding awareness, provide no indication of whether anysubstantive engagement has taken place. Engagement metrics, many of which will befamiliar – such as likes, shares, retweets or comments – do provide an indication of thequality and frequency of interactions between audiences and a campaign. Combinedwith awareness metrics they contribute to an evaluator’s understanding of the impact ofa campaign.15 Institute for Strategic Dialogue
Some metrics provide evidence of a more substantial engagement than others, andcertain types of counter-narrative campaigns will prioritise certain types of engagement.For example, a campaign seeking to reach the widest possible number of people for anawareness raising effort might place more value on likes and shares than comments orother forms of engagement. In contrast, a campaign seeking to engage people currentlyinvolved in or at particular risk of becoming involved in extremist groups might place ahigher value on comments or direct message interactions.In general however, actions that require greater effort
The Counter-Narrative Monitoring & Evaluation Handbook The aim of this Handbook is to provide civil society campaigners with the knowledge and skills they need to undertake high quality counter-narrative campaign evaluations. It presents model frameworks and case studies of
