WIXLIB

11High school students struggled as they learned remotely with limitedaccess to school-based college counselors. Many colleges closed inperson tours and operations. Fewer Texas students completed theFAFSA in fiscal year 2020, decreasing by 2.2% from 218,081 at the end ofAugust 2019 (USDOE, 2022). All these factors created even more urgencyto help students apply to college while keeping them safe from COVID.Fortunately, when the pandemic hit, the THECB was prototypingthe Virtual Advising Project (ADVi), which reaches students 24/7with “critical and timely information to support their postsecondaryjourney.” The ultimate goal of the project is to help more Texans “accesshigher education and complete their credentials at Texas colleges anduniversities” (THECB, 2021). ADVi, short for “advisor,” is a text-based,artificially intelligent chatbot that sends text messages about collegegoing information, suggests action steps students can take throughoutthe process, and responds to incoming questions from users. The ADVichatbot (or bot) can automatically answer many questions texted bystudents and connect them to ADVi advisors, a team of professionalcollege advisors, when questions cannot be addressed by the bot.ADVi started as a partnership with The University of Texas at Austinin 2019. It was then moved to the THECB and made available to allTexas students in the fall of 2020. During the 2020-21 academic year,most ADVi users opted-in to messaging from ADVi when they startedan ApplyTexas college application.

12The Urban Education Institute at The University of Texas at SanAntonio (UEI) established a research partnership with the THECB’sVirtual Advising team in the fall of 2020, thanks to generous fundingsupport from the Greater Texas Foundation. Over three years, theUEI team will collaborate with the THECB team to facilitate rigorouscontinuous improvement cycles to strengthen ADVi’s effectivenessand engagement with students.This research brief provides findings from the project’s first and secondwave of student interviews, which focused on getting feedback fromADVi student users about their user experience, their understandingof the college-going process, and access to resources outside of ADVi.It also provides findings from focus group interviews held with ADViadvisors to understand the strengths and challenges of implementationand identify areas of improvement.

1313MEET THE USERSIn design-based research, user archetypes, also referred to as personas,are created to spark a sense of empathy in the people who designprograms for the eventual users. With empathy, programs designershave been found to more effectively design programs to meet theneeds of their users.Researchers identified three user archetypes to schematically describethe population of potential ADVi users based on the qualitative researchfindings generated from student interviews. They also validated thepersonas by discussing them with ADVi advisors.Personas include Freddy (pg. 14), Manny (pg. 15), and Sara (pg. 16). Aside-by-side of these personas is available in Appendix A on page 54.Each of these personas has unique attributes and biographies. Butwhat primarily differentiates them is their level of reliance on ADVi tohelp them apply to college.

14FIGURE 1: HIGH ADVI RELIANCE ARCHETYPEMeet Freddy.“First-gen, High level of reliance on ADViADVi helped me complete my collegeapplications more than any person inmy life. I wish I had ADVi at the startof high school.”Freddy’s Goals Become a medical doctor Marry and have kids Own a home with a big yardFreddy’s Frustrations Not starting to plan for college untilmy junior year in high school Not being able to play on my highschool baseball team because I neededto workHigh School Academics Ranks in top 10% of graduating class Highest math credit was pre-calculus Completed an AP biology courseCollege Finances No family college savings Estimated family contribution 0 Estimated annual unmet need 13,500Freddy’s Experience with ADVi Does not remember signing up for ADVi ADVi was his primary source of info andadvice Would not have known about ADVi butfor starting an Apply Texas applicationwith the help of his favorite teacher Most of his questions were on FAFSAand finding strong pre-med programs He was happy to ask questions and seekhelp from ADVi advisors but neededprompts and remindersFreddy’s BioGraduated from SAISD’s BrackenridgeHS in 2020, where his favorite class wasbiology. Worked at a nursing home afterschool for 2 years of high school. He is theoldest of four siblings and will be first inhis family to graduate from high school.

15FIGURE 2: MEDIUM ADVI RELIANCE ARCHETYPEMeet Manny.Continuing-gen, Medium level of reliance on ADVi“ADVi helped me stay on track. It wasmy personal college advisor 24/7.”Manny’s GoalsManny’s Experience with ADVi Be a computer software developer Travel Help my familyManny’s Frustrations Not knowing which dual credit courses would transfer and advance me inmy degree program All the personal info needed forFAFSAHigh School Academics Ranks in top 10% of graduating class Highest math credit was AP CalculusAB Completed 2 AP & 2 dual credit coursesCollege Finances No family college savings Estimated family contribution 5,000 Estimated annual unmet need 8,500 Does not remember signing up for ADVi ADVi and high school counselor wereprimary sources of college-going help Connected to ADVi through ApplyTexas Mostly had FAFSA & scholarship questions Benefited greatly from reminders Was happy to ask questions of the ADVibot because it never made him feel likehis questions were dumb and it helpedhim prep for his meetings with his highschool counselorManny’s BioGraduated from NISD’s Clark HS in 2020,where his favorite class was a computerscience dual credit course at UTSA. Playedhigh school and club soccer and rancross-country. Raised by his single momwho earned a BS and MPA and serves as aprofessional in local government.

16FIGURE 3: LOW ADVI RELIANCE ARCHETYPEMeet Sara.“Continuing-gen, Low level of reliance on ADViI knew I was going to Texas A&M aslong as I can remember. It’s whereboth my parents went.”Sara’s Goals Earn a bachelor’s degree in businessfrom TAMU-College Station Start my own marketing businessSara’s Frustrations Wish I could start at College Stationmy freshman year instead of transferring from another school in the TAMUsystem Need help planning efficient transferHigh School Academics Ranks in top 25% of graduating class Highest math credit pre-calculus Completed 4 AP coursesCollege Finances 529 college savings Estimated family contribution 29,000 Estimated annual unmet need 0Sara’s Experience with ADVi Remembers signing up for ADVi basedon her brother’s suggestion. Asked questions of ADVi bot occasionally, mostly relied on high school counselorsand TAMU recruiters. Benefited from ADVi bot’s remindersand encouragements. ADVi was not an important source ofcollege info and advice.Sara’s BioGraduated from NEISD’s Reagan HS in2020, where here favorite experience wasleading the journalism club. She is theyoungest of two and hopes to join herbrother at TAMU-College Station, whereher parents first met at bonfire.

17FreddyFirst-gen, High level of reliance on ADViFreddy is the most reliant on ADVi of the threepersonas. As a first-generation college-goingstudent, he did not begin planning for college untilhis senior year when his favorite teacher beganasking him about his college plans. Freddy alsoattends a high school that has a below-averagesense of college-going culture, as seen by its belowaverage college enrollment rates.Freddy began using ADVi after his favorite teacherMannyFreddy benefited greatly from the ADVi bot andADVi advisors. He relied heavily on ADVi advisorsto help him and his mom complete his FAFSAapplication. He also used it to learn about collegeswith strong pre-med programs.Continuing-gen, Medium level of reliance on ADViManny relies on ADVi more than Sara but needsit less than Freddy. As a continuing-generationcollege-going student, he has received a lot ofadvice about preparing for college from his mom.He also attends a high school with a strong team ofcollege advisors and college-going culture.However, between his year-round, packed scheduleof athletic events and dual credit courses onthe campus of a local four-year college, Mannybenefited greatly from reminders sent by the ADViSarahounded him to start his ApplyTexas application.But for ApplyTexas and Freddy choosing to opt-into ADVi messaging, he would not have learned ofADVi at all.bot that kept him on track with his college andfinancial aid applications. He also appreciatedADVi’s words of encouragement, which made himthink of ADVi as a “cool mentor.”Like Freddy, Manny began using ADVi after hestarted his ApplyTexas application. The choice toopt-in to messaging in ApplyTexas was responsiblefor connecting Manny to ADVi. But for this feature,he would not have received the benefits of ADVi.Continuing-gen, Low level of reliance on ADViSara relied on ADVi the least as compared to Freddyand Manny. As a continuing-generation collegegoing student, her college plans were set as earlyas fourth grade. She has always planned to followin her parents’ footsteps by becoming an Aggie atTexas A&M—College Station. In fact, she had hopedto join her older brother who started his collegeeducation at TAMU two years earlier.Unfortunately, Sara did not qualify for automaticadmissions based on her class ranking and wasonly admitted under the TAMU System AdmissionsProgram, which meant she needed to start hercollege journey at a regional college of TAMUbefore having the opportunity to transfer toTAMU-College Station. Her parents and highschool counselors helped her begin researchingher options and planning her freshman year ofcoursework to minimize any loss of credit duringthe transfer process.Sara signed up for ADVi before starting herApplyTexas application on the advice of her olderbrother. What she liked most about ADVi were thereminders sent by the ADVi bot that kept her frommissing key college application deadlines. She alsowas grateful for ADVi’s encouraging words, thoughshe sometimes felt like she received too manymessages.

18DESIGN IMPLICATIONSHow can we improve Freddy’s ADVi experience?FreddyFirst-genHigh level of reliance on ADVi Use a preexisting process, such as school enrollment, or TSI ordual enrollment participation, to provide the option to enrollstudents into ADVi starting in ninth grade. Provide prompts to help students select their high schoolcourses and extracurricular experiences to support their collegeand career goals. Begin educating students about the ApplyTexas applicationprocess prior to senior year. Add more content about scholarship opportunities.How can we improve Manny’s ADVi experience?MannyContinuing-genMedium level of reliance on ADVi Use a preexisting process, such as school enrollment, or TSI ordual enrollment participation, to provide the option to enrollstudents into ADVi starting in ninth grade. Add content that helps students select dual credit and AdvancedPlacement courses that align with their college plans. Add more content about scholarship opportunities. Create ADVi training videos for high school counselors toencourage the integration of ADVi into their high-touch, schoolbased strategies.How can we improve Sara’s ADVi experience?SaraContinuing-genLow level of reliance on ADVi Add a feature that estimates a student’s class ranking startingin sophomore year so that there are no surprises at the time ofapplying to college. Add more detailed content to help students plan for efficienttransfers. Create ADVi training videos for high school counselors toencourage the integration of ADVi into their high-touch, schoolbased strategies.

1919THEORY OF CHANGEResearchers relied on ADVi advisors to develop a schematic frameworkthat describes ADVi’s theory of implementation and theory of clientresponse, as shown in figure 4 on page 20. A theory of implementationidentifies the essential activities that program operators willimplement in theory. A theory of client response identifies how clientsare expected to respond to programmatic activities in theory.Together, these theories are referred to as a theory of change. They helpidentify (1) where planned activities and client responses fail to meetexpectations, (2) where processes are underdeveloped, and (3) whereprocesses could be streamlined. Their development also benefittedfrom insights offered by students.

20FIGURE 4: THEORIES OF IMPLEMENTATION AND CLIENT RESPONSETheory of ImplementationTheory of Client ResponseTHECB markets ADVi to schools andstudentsSchools promote ADVi with their studentsADVi advisors conduct outreach to schoolsto promote the use of ADViApplyTexas application allows students toopt-in to ADViStudents start anApplyTexas application and choose toopt-in and enrol intoADViStudents text theword “COLLEGE” to512-829-3687Mainstay receives a batch of studentcontacts from ApplyTexas and feeds theminto ADViADVi bot begins sending messagesADVi bot sends student question orrequest to communicate with an ADViadvisor to a general email accountADVi advisors share the tasks of checkinggeneral email box, distributing queriesamong themselves, and logging queriesinto a tracking documentADVi advisors receive student queries intheir assigned folder, research answers toeach question, and respond by emailADVi advisorsupdate trackingdocumentSystem turns emailto text message andsends to studentStudents open and read ADVi messages,ask questions of ADVi bot, and take a stepin their postsecondary college and careerplanStudents ask ADVi bot to communicatewith an ADVi advisor by responding to aprompt sent by the ADVi bot or by texting#followup to the ADVi botStudents open and read message, askanother question for an ADVi advisor, andtake another step in their postsecondarycollege and career planStudents complete ApplyTexas application,financial aid applications, or another finalstep in their postsecondary college andcareer plan

21ADVI ADVISORSADVi’s theory of change begins at the top ofFigure 1, where the THECB markets ADVito students and schools. This activity is alsocomplemented by school outreach conductedby ADVi advisors. From these activities,schools promote ADVi to their students andstudents sign-up for ADVi.A major program activity is the design of theApplyTexas application to require students toopt-out of ADVi if they do not want to receiveADVi messages. This choice architecture allowsmany students to be included in the ADViprogram. However, before they begin receivingmessages from ADVi, a batch of student datamust be transferred to Mainstay and thenMainstay must input that information intoADVi. This process is expected to be largelyautomated.Once student contact data is entered intoADVi, the ADVi bot sends an introductorytext message to students. This messageexplains what ADVi is about and sharessome instructions. Following this message,the ADVi bot regularly sends messages tostudents. From these messages, students areexpected to become more informed about thecollege-going process, ask questions about theprocess to ADVi, take steps toward completingtheir college application and financial aidapplication, and grow in confidence about theirability to complete the college-going process.Students are regularly asked if they wouldlike a question of theirs to be escalated up toan ADVi advisor for answering. If they agreeto communicate with an ADVi advisor, theirquestion or concern is sent to a general emailbox overseen by the ADVi advisor team.ADVi advisors will review incoming messages,record them on a tracking document, anddistribute the questions into their assignedfolders. This is a shared set of tasks.ADVi advisors read their assigned studentqueries, research information to respond toeach query, and then draft a response to eachstudent by email. ADVi advisors then updatethe person managing the tracking documenton the status of each student query.The system turns the ADVi advisors’ email intoa text message and sends it to the appropriatestudent. From these ADVi advisor messages,students are expected to become moreinformed about the college-going process,ask more questions, take more steps towardcompleting their college application andfinancial aid application, and further grow inconfidence about their ability to complete thecollege-going process.ADVi advisors helped identify a few importantdistinctions between the ADVi bot’s servicesand those that ADVi advisors provide. TheADVi bot quickly answers a wide range ofcollege questions 24/7. In contrast, ADViadvisors take longer to respond but go deeperin searching for answers to questions that areunique to a student. The ADVi bot and ADViadvisors complement each other and serveas a one-stop-shop for answering students’complete list of college-going questions.With repeated questions answered, timelyreminders, and encouragement, studentsincrease their completion rate of collegeapplication and financial aid application,respectively.

2222RESEARCH DESIGNResearchers used a qualitative research design to understandthe implementation and user experience of ADVi and identifyopportunities for improvement. They conducted individual interviewswith 60 students who used ADVi. They also facilitated two rounds offocus group interviews with ADVi advisors. Researchers saved andtranscribed audio recordings for later analysis using Zoom, a videoconferencing application. During the first two years of the study,researchers met with THECB staff weekly or bi-monthly to coordinatedata gathering and share their emerging findings.Researchers shared their interpretations and then synthesized theirfindings into the first draft of this report. They then shared the firstdraft of the report with ADVi advisors and supporting staff for theirfeedback and to give them an additional opportunity to add to thereport. After reviewing the final round of staff feedback, researchersproduced this final report.

23Study SampleResearchers used a study sample of 60 students who used ADVi. Theyindividually interviewed these students in two waves. They alsointerviewed eight advisors during two focus group meetings.Wave One of Students. Thirty students comprised the first wave ofstudent interviews. These students where all located in Bexar Countyand intended to enroll in college in fall 2021. Twenty-eight were highschool seniors. One was a 2020 high school graduate who took a yearlong break from formal education. And one was looking to re-enrollin community college after taking time off to care for her family.Twenty-six students had already applied for college, and all but onehad received acceptance letters. The other four students were still inthe process of applying.Of the 30 students, 70% learned remotely during their senior year dueto COVID-19. Six, or 20%, were learning in a hybrid environment, somein person and some online remote learning. Only three participants, or10% of this w

process. Deliver more personalized information on scholarship opportunities and college programs. Improve ADVi's efficiency and ability to send more personalized information by having it remember how students answered its previous questions. Content Add information on preparing for the application process in 11th grade or as early as