WIXLIB

O F F I C E O F T H E A S S I S TA N T S E C R E TA RY F O R H E A LT HHow to Conduct Qualitative AnalysisJane Y. Choi, Ph.D.ResearcherHHS Office of the Assistant Secretary for HealthOffice of Population AffairsMarch 2020

Agenda Overview of qualitative analysisSteps to conduct analysisGeneral guidelines for reporting your findingsQuestionsResourcesO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH22

Overview of qualitative analysis Tool to answer research questions by finding patterns in qualitative dataQualitative data come from a range of sources such as: ObservationsInterviewsFocus groupsOpen-ended responsesCurricular materialsUsed to: Understand details and nuance of how programs were implementedUnderstand why program was (not) able to meet target outcomes or goalsUnderstand the viewpoints and experiences of those involvedO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH33

Metaphor that represents the process of qualitative analysisO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH44

Steps to analyze qualitative dataO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH55

Example studyResearch questions: What were key challenges the program team faced when implementing thecurriculum? What curricular or implementation modifications were made? Did these modifications mitigate or address the implementation challenges?Data sources Focus groups with participants (students at partner organizations) Interviews with program team (facilitators)O F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH66

1. Refine and focus on the research questions Review initial research questionsRefine your research questions (if needed)Use your research questions to guide your analysis and reporting Make sure you know your research questions inside and outContinually refer back to your research questions while conducting analysesO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH77

2. Get data ready to be codedTake your data and create clean data files (e.g., transcripts, detailed notes) thatcan be coded Raw data are difficult to analyze The person who collected the data is the best person to clean data files Recommend creating files close to when data were collectedName prepared data files in systematic wayO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH88

Example 2.1: Prepare data filesWhat were the key challenges the program team faced when implementing thecurriculum?Raw notes from facilitator interviewClean dataSometimes the discussion questions are toobabyish for the students. Although it may justbe content too young cuz the question vocabis too hard sometimes for them. Like “Why doyou show affection to someone?” The kidsdon’t know the word “affection”, but they thinkit’s silly to explain why you give someone ahug.Sometimes the discussion questions are toobabyish for the students. Although it may justbe the content of the questions is too youngbecause the question vocabulary issometimes too hard for them. Like “Why doyou show affection to someone?” The kidsdon’t know the word “affection”, but they thinkit’s silly to explain why you give someone ahug.O F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH99

Example 2.2: Prepare data file namesBoydCharter Facilitator Interview 2019-11(Nov) MidProgram.docxExample template of a file name:SiteName RespondentType DataSource Date (Whether data collected pre-, mid-, or postprogram).docxO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1010

3. Get data ready to be coded, continuedTake your data and create clean data files (e.g., transcripts, detailed notes) that can becoded Raw data are difficult to analyze The person who collected the data is the best person to clean data files Recommend creating files close to when data were collectedName prepared data files in systematic wayAdd these prepared data files into a database Recommend creating database using a qualitative software program Only include data sources that will answer the research questionsO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1111

3. Develop a coding systemCoding system is a set of codes rooted in research questions A code is a concept or label used to assign meaning to the data Code data (apply codes to data) to group similar data togetherDevelop codes using two methods1.2.Codes are verydetailed and specificExample codes: Difficulties engagingparticipants Low attendanceCodes that are derived from prior research, focus of the research question, OPA corethemes Can generate codes through a brainstorming session with project director, frontline staff,and evaluatorCodes derived as you go through the data Evaluator (and coders) develop codes as they read through the data Keep track of when you create new codes to apply them to already-coded dataThe coding system should include codes organized by research question and a definition ofeach codeO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1212

Example 3: Coding systemRQ1: What were key challenges the program team faced when implementing thecurriculum?CodesDefinitionWhen created?Scheduling and timing Issues scheduling or issues with the timing of the Prior to codingsessions or programUnresponsiveparticipantsParticipants not responsive to the curriculum,lessons, etc.Prior to codingVocabulary toochallengingVocabulary is too difficult for participants tounderstand1/15/2020Issues with literacy/textsParticipants not interested in curriculum thatrequires reading and writing. Participants don’twant to engage with long text passages.1/22/2020O F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1313

4. Code dataIf you will have multiple coders, train coders on the research questions, how touse the software, the coding system Benefit of having multiple coders – able to assess reliabilityDo a quick read of data to get a full picture of topics coveredTo code, read data carefully and assign codes as you go Assign codes that accurately represent each section of the data Should use multiple codes on the section of data, as needed The same information can be used to answer multiple research questionsWhile coding, keep the research questions in mindO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1414

Example 4: Coding dataO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1515

5. Group and regroup dataAfter coding, you may need to group the data from related codes together Codes may be too specific and more meaningful when they are grouped by theme Prior to grouping coded data, read the data in each code to ensure they are relatedto data in the other, related codesMay need group coded data once, twice, or moreO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1616

Example 5: Grouping similar codes Curriculum not appropriate for ageCurriculum not culturally/linguistically appropriateVocabulary too challengingIssues with literacy/textsTroublesome examples in curriculumO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1717

6. Determine findings from coded dataDeveloping key findings is necessary to meaningfully answer the researchquestionsTo develop findings you might consider doing a combination of the following: Examine the extent to which codes affect a large number of facilitators, sites,participants Evaluators determine what a “large number” isConsider if there are codes that are particularly influential or importantO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1818

6. Determine findings from coded data: What is a finding?Determine findings by choosing the codes with data that best answer theresearch questions Evaluators have to define how they will determine what findings to report (decisionrules) It is critical to document how you will determine what are findings andsystematically apply themYou will create the process for determining findings after codingNote: The process of determining findings requires multiple readings of coded dataO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH1919

Example 6: Developing findingsResearch question 1: What were key challenges the program team faced when implementing the curriculum?No. ofrespondents(of 5)No. ofcohortsaffected(of 10)No. ofstudentsaffected(of 80)Scheduling and timing115Unresponsiveparticipants3560Participant engagement in lessons (discussions,activities) is part of the program modelDisruptions to sessions3445Impediment to implementationLate arrivals000Early departures1110Curriculummisalignment2230CodesO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTHSubstantive reasonsWhile critical to coordinate to implement thecurriculum, issues did not affect implementationDevelopers stated participants must be part of thebeginning of each sessionThe program model states students should receive allthe content, but students made up the contentCurriculum must be aligned to target population2020

Example 6: Developing findingsResearch question 1: What were key challenges the program team faced when implementingthe curriculum?No. of studentsSubstantive reasonsCodesaffected (of 80)While critical to coordinate to implement the curriculum,Scheduling and timing5issues did not affect implementationUnresponsiveParticipant engagement in lessons (discussions,60participantsactivities) is part of the program modelDisruptions to sessions45Late arrivals0Early departures10Curriculummisalignment30O F F I C EO FT H EASSISTANT SECRETARY FOR HEALTHImpediment to implementationDevelopers stated participants must be part of thebeginning of each sessionProgram model states students should receive allcontent, but students made up what they missedCurriculum must be aligned to target population2121

7. Report findingsAnswer your research questions by describing the patterns you found fromcoded dataClearly state the significance of findings Should be able to respond to “why does this matter?”Use language that accurately reflects the methods usedO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH2222

Example 7.1: Causal languageThe program did notpositively affect participants’reduction strategies for riskybehavior because theystruggled to understand thecurriculum’s difficultvocabularyO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH2323

Example 7.1: Appropriate language for qualitative findingsParticipants reported theydid not understand reductionstrategies for risky behaviorbecause they struggled tounderstand the curriculum’sdifficult vocabularyO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH2424

7. Report findingsAnswer your research questions by describing the patterns you found fromcoded dataClearly state the significance of findings Should be able to respond to “why does this matter?”Use language that accurately reflects the methods usedPlay to strengths of qualitative data to help audience better understand orconnect to findings Use illustrative quotes or examples; background information and context; samplewritings, drawings, imagesO F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH2525

Example 7.2: Findings write-upA key challenge the program team faced when implementing the curriculum was themisalignment between aspects of the curriculum and students’ needs.One type of curriculum misalignment was that the language was too advanced for students. Twoof the five facilitators stated that they spent at least 10 minutes every hour-long session goingover complex vocabulary—such as “brusque” and “anachronistic”—that were only tangentiallyrelated to the content. The time spent teaching definitions reduced the time for mentorshipactivities that students reported were most valuable because they could connect the densecontent to their own experiences. As one student reported, “The best part is talking in [ourmentorship] group. That’s when I’m like Oh that kind of convo happened to my friends and me,too.”Facilitators addressed this challenge by O F F I C EO FT H EASSISTANT SECRETARY FOR HEALTH2626