Transcription
DEVELOPING AND USING A LOGIC MODEL FOR EVALUATION ANDASSESSMENT OF UNIVERSITY STUDENT AFFAIRS PROGRAMMING:A CASE STUDYbyJeff CooperMasters in Business Administration, Katz School of Business, University of PittsburghSubmitted to the Graduate Faculty ofthe School of Education in partial fulfillmentof the requirements for the degree ofDoctor of PhilosophyUniversity of Pittsburgh2009
UNIVERSITY OF PITTSBURGHSCHOOL OF EDUCATIONThis dissertation was presentedbyJeff CooperIt was defended onMarch 31, 2009and approved byBill Bickel, PhD, Professor of EducationDennis Galletta, PhD, Professor of Business, Katz School of BusinessMaureen McClure, PhD, Professor of EducationCindy Tananis, EdD, Professor of EducationDissertation Advisor: John Weidman, PhD, Professor of Educationii
Copyright by Jeffrey C. Cooper2009iii
DEVELOPING AND USING A LOGIC MODEL FOR EVALUATION ANDASSESSMENT OF UNIVERSITY STUDENT AFFAIRS PROGRAMMING:A CASE STUDYJeff Cooper, MBA, PhDUniversity of PittsburghThis dissertation addresses theory and practice of evaluation and assessment in university studentaffairs, by applying logic modeling / program theory to a case study. I intend to add knowledgeto ongoing dialogue among evaluation scholars and practitioners on student affairs programplanning and improvement as integral considerations that serve mission and vision at thecontemporary university.Insights on the following research questions can help determinetheoretical justifications and forge an inventory of effective evaluation and assessmenttechniques in student affairs.1.How can logic modeling be used to analyze evaluations of student affairsprograms and an overall assessment campaign?2.How might evaluators and planners have enlisted a logic model such as the onedeveloped in this study to enhance the effectiveness of the assessment campaign at the profileduniversity student affairs unit?These questions involve general principles and particular applications of my arguments infavor of using a logic model to analyze a comprehensive assessment campaign, as conducted bya designated student affairs assessment team. Although sets of workable techniques at oneuniversity may not generalize to another campus culture, findings will reveal how one institutioniv
of higher education (IHE) has behaved and responded to new challenges and inputs – in thiscase, greater emphasis on evaluation and assessment to address issues of accountability andcredibility for student affairs. Using logic modeling as the primary heuristic, this study analyzeswhat the university system depicted in case study has accomplished and might haveaccomplished. I also invite readers to join my speculation how using and perhaps customizingthis logic model could guide the unit’s next steps in ongoing assessment. If a logic model worksretrospectively, then perhaps it might function proactively. My hope is that readers finddescriptions and lessons to compare and contrast to their own evaluative practices, adding to theknowledge base and possible consensus about current practices for university student affairsassessment campaigns.v
TABLE OF CONTENTSLIST OF TABLES . ixLIST OF FIGURES . ixPREFACE . xii1.0INTRODUCTION.11.12.0PURPOSE OF STUDY .11.1.1Objectives .21.1.2Context .31.1.3Plan .7REVIEW OF THE LITERATURE.122.1EVALUATION & ASSESSMENT OF STUDENT AFFAIRS.122.1.1Accountability .142.1.1.1 Accreditation .162.1.2Evaluation guidelines .172.1.3Roles of evaluators .202.1.3.1 Politics .232.1.3.2 Buy-in and consensus .252.1.3.3 Data gathering: interviews and focus groups .282.1.3.4 Benchmarking .31vi
2.1.3.5 Meta-analysis .322.1.3.6 Summary of the evaluator’s role .34HEURISTICS.352.23.02.2.1I-E-O and other acronyms .352.2.2Logic modeling .38METHODOLOGY .473.1CASE STUDY METHODOLOGY .473.1.1BUILDING AND APPLYING A LOGIC MODEL .533.24.0Bounding the case .51ANALYSIS .604.1THE CASE .604.1.1Assessment Campaign .624.1.1.1 Team dynamics .664.1.1.2 Student Baseline Outcomes .704.1.1.3 Team activity.724.1.1.4 Summary of activity for first year of campaign.754.1.1.5 Second year activity .784.1.1.6 Concurrent unit-level evaluations .834.1.24.2Post-service reflections.84INTEGRATING CASE STUDY AND LOGIC MODEL .854.2.1e-portfolios (Student Baseline Outcome #7) .874.2.2Global learning (Student Baseline Outcome #4) .984.2.3Pitt Pathway (Student Baseline Outcome #11) . 109vii
4.35.0FINDINGS / META-EVALUATION. 118CONCLUSION . 1255.1LIMITATIONS AND ALTERNATIVES . 1275.2NEXT STEPS . 134APPENDIX A . 139APPENDIX B . 140APPENDIX C . 180BIBLIOGRAPHY . 183viii
LIST OF TABLESTable 1Key to logic model “scores” .10Table 2AEA Guiding Principles .18Table 3Key to logic model “scores” in meta-evaluation .57Table 4Summary of Student Affairs Assessment Team,evaluation reports submitted 2005-06.76Table 5Key to logic model “scores” in meta-evaluation .87Table 6Key to ratings matrix on overall campaign . 118Table 7Assessment Team “score-card” for Outcomes, 2005 – 07 campaigns . 120Table 8Students’ suggestions to improve their Pitt experience . 153Table 9Students’ suggestions to improve Student Affairs . 153Table 10 Students’ assessment of their Pitt experience . 157Table 11 Average ratings by random students of Student Affairs services. 160ix
LIST OF FIGURESFigure 1Project scope .5Figure 2.Timeline for Pitt Student Affairs assessment campaign, 2005-07 .6Figure 3Logic model depiction of assessment cycle .9Figure 4Example meta-evaluation of an evaluation process on one outcome .10Figure 5Astin’s I-E-O model (1991).36Figure 6Distillation of Weiss’ (1997) sample program theory model .39Figure 7WKKF (2004) diagram on how to read a logic model .40Figure 8Program causality.43Figure 9Residence Life evaluations as subset of Student Affairs assessment process . 52Figure 10Logic model depiction of assessment cycle .54Figure 11Logic model depiction of “impact”.55Figure 12Example meta-evaluation of an evaluation process on one outcome .57Figure 13Where an external scan fits into the logic model .73Figure 14Residence Life evaluations as subset of Student Affairs assessment process . 83Figure 15Logic model depiction of assessment cycle .86Figure 16Logic model meta-evaluation of e-portfolio outcome .96Figure 17Logic model depiction of “impact” as knowledge flowsfrom outcome to assumptions .97Figure 18Logic model meta-evaluation of global and diverse opportunities outcome . 106x
Figure 19Logic model depiction of evaluation of Pitt Pathway outcome. 115Figure 20Figure 1 rotated: Logic model depiction of core valuesthat frame Pathway assessment . 117Figure 21Team performance scorecard based on logic model pyramid . 122Figure 22Logic model depiction of “impact”:Outcome-based knowledge reframes assumptions . 126Figure 23Process logic model with status codes per key element . 136Figure 24Logic model depiction of evaluation of mentoring outcome . 143Figure 25Logic model depiction of evaluation of internship outcome . 146Figure 26Logic model depiction of evaluation of Pitt Promise outcome . 149Figure 27Logic model depiction of evaluation of listening outcome . 154Figure 28Flow of knowledge from achieved outcome to future phasesand to other intended outcomes . 156Figure 29Logic model depiction of evaluation of excellence outcome . 158Figure 30Logic model depiction of evaluation of Student Affairs performance outcome . 162Figure 31Logic model depiction of summative evaluation outcome eruption. 165Figure 32A reconfiguration of summative evaluation outcome logic model. 166Figure 33Logic model depiction of evaluation of parents conduit outcome. 169Figure 34Logic model depiction of evaluation of faculty tools outcome . 172Figure 35Logic model depiction of evaluation of alcohol violations outcome . 176Figure 36Logic model for decreased alcohol violations outcome . 177Figure 37Logic model depiction of evaluation of conduct code violations outcome. 179xi
PREFACEI would be remiss in not mentioning perspectives gathered from a professionalappointment in Residence Life, concurrent with my participation in the Social and ComparativeAnalysis program at the School of Educa
Jeff Cooper, MBA, PhD University of Pittsburgh. v of higher education (IHE) has behaved and responded to new challenges and inputs in this – case, greater emphasis on evaluation and assessment to address issues of accountability and credibility for student affairs. Using logic modeling as the primary heuristic, this study analyzes what the university system depicted in case study has .
