WIXLIB

Attendee DirectoryAssessment 201: Advanced Topics in AssessmentUniversity of San Francisco, San Francisco, CANovember, 18 2016First NameLast NameJob TitleInstitutionEmailEllenBurkeFaculty - Landscape ArchitectureCal Poly San Luis PhelanAcademic Assessment DirectorCal Poly San Luis ObispoJoshMachamerDept Chair, Member, Academic Assessment CouncilCal Poly San Luis Obispojmachame@calpoly.eduMarcHorneyAnimal Science FacultyCal Poly San Luis Obispomhorney@calpoly.eduSeanHurleyAgribusiness FacultyCal Poly San Luis Obisposhurley@calpoly.eduDawnJankeDirector, Writing and Rhetoric CenterCal Poly SLOdjanke@calpoly.eduJenniferNguyenStudent Center for Academic Achievement DirectorCalifornia State University East California State University, te Dean of AssessmentCalifornia University of Science and Medicineettarhr@calmedu.orgTinaKandakaiExe. Director Development and AssessmentCharles R. Drew istant to the Dean of StudentsDharma Realm Buddhist Universitythao.phi@drbu.orgStevenSchapiroDean for Academic Affairs, and ALOFielding Graduate tutional Assessment CoordinatorHawaii Community Collegereshelad@hawaii.eduJaronHightower-MillsIR and Assessment AdministratorInternational Technological itutional Research AnalystJohn F. Kennedy Universitykmclemore@jfku.eduHelenaSimonProject Coordinator Program AssessmentJohn F. Kennedy Universityhmarsala@jfku.eduPamelaZimbaAssistant Professor of LawJohn F. Kennedy Universitypzimba@jfku.eduLisaHuttonProfessor and Chair, Legal Studies DepartmentJohn F. Kennedy Universitylhutton@jfku.eduMatthewBruceProject Director of Program AssessmentJohn F. Kennedy Universitymbruce@jfku.eduLauraPalazzoloDeanLincoln Law School of San Josedean@lincolnlawsj.eduLauraAlipoonDepartment ChairLoma Linda Universitylalipoon@llu.eduLeoRanzolinAssociate Dean, School of ReligionLoma Linda Universitylranzolin@llu.eduCarmanWimsattAssessment ConsultantLos Angeles Community College Districtwimsattcy@gmail.comElizabethSturgeonMount Saint Mary's Universityesturgeon@msmu.eduBruceMatthesAssessment Committee Co-Chair & Associate Professor ofEnglishDirector of Integrative StudiesNewSchool of Architecture & h Language Program CoordinatorNewSchool of Architecture and DesignCarmenFernandezPresidentNorthern Marianas rianas.eduJacquelineCheDirector of Institutional EffectivenessNorthern Marianas tment Chair, BusinessNorthern Marianas Collegechavel.green@marianas.eduPaulChoiExecutive Vice PresidentNorthwestern Polytechnic Universitypaul.choi@npu.eduNellyMangarovaChief Academic OfficerNorthwestern Polytechnic ccidental Collegebrighous@oxy.eduOccidental Collegeerogers2@oxy.eduEsterRogersAssociate Dean, Social Sciences / Core Curriculum and StudentIssues; Professor, Phill/Cog PsyDirector of Institutional Assessment and AccreditationEllieKaucherVice President of Academic AffairsPhillips Graduate Universityekaucher@pgu.eduMaryCoffeyAssociate Dean of the CollegePomona Collegemcoffey@pomona.edu

Attendee DirectoryAssessment 201: Advanced Topics in AssessmentUniversity of San Francisco, San Francisco, CANovember, 18 2016First NameLast NameJob TitleInstitutionEmailBenSandersExecutive DirectorReach Institute for School t Mary's College of Californiafsweeney@stmarys-ca.eduSarah BethDempseySaint Mary's College of Californiasdd4@stmarys-ca.eduAliceBaldridgeSaint Mary's College of Californiaamb19@stmarys-ca.eduAeleahSoineSaint Mary's College of Californiaahs3@stmarys-ca.eduJonasWrightAssociate Dean, Academic AffairsSan Francisco Conservatory of Musicjwright@sfcm.eduJenniferSwansonSan Francisco State Universityjmpeters@sfsu.eduJulianavan OlphenAssociate Director, Writing Across the Curriculum/Writing in theDisciplineDirector, Writing Across the Curriculum/Writing in the DisciplineSan Francisco State Universityjvo@sfsu.eduElaineCollinsAssociate Dean College of ScienceSan Jose State or of Computer Science and Director of AssessmentSan Jose State r of Academic AffairsSaybrook am DirectorSaybrook Universitytjackson@saybrook.eduNamiKimAssistant ProvostSaybrook Universitynkim@saybrook.eduDevorahCurtisDirector of Mentoring/Director of MS MBM ProgramSaybrook ierra rofessor - Agriculture, SLO AssistantSierra ssor - Humanities, SLO AssistantSierra rSierra essment SpecialistStanislaus Stateelittlepage@csustan.eduGabrielaRomoStudent Learning Outcomes Assessment ManagerUniversidad de Las emic Programs ManagerUniversity of San Diegolopezja@sandiego.eduTracyBenningAssociate Professor, Environmental ScienceUniversity of San nt Professor of EconomicsUniversity of San retchenCoffmanAssistant Professor of Environmental ScienceUniversity of San FranciscoXavierGomezSr. Instructional DesignerUniversity of San Franciscoxgomez@usfca.eduChristineYoungAssociate Professor, Performing Arts & Social Justice ProgramUniversity of San Franciscocyoung8@usfca.eduShawnCalhounAssociate DeanUniversity of San rofessor of MusicUniversity of San ofessor of StatisticsUniversity of the r an and Student Success CoordinatorWeimar weimar institutercelestine@weimar.eduGeorgeArayaEducation Department Chairweimar institutegaraya@weimar.eduTeriSalvadorDirector of Student Servicesweimar institutetsalvador@weimar.eduGlennHillProfessor of ReligionWeimar Instituteghill@weimar.eduFlaviaJorgePsychology Department ChairWeimar InstituteInstitutelmfjorge@yahoo.com

Aligning Direct andIndirect AssessmentStrategies with OutcomesAmy Driscoll7

Amy DriscollMany traditional examples(tests/exams, cases, projects,performances, essays, tasks,simulations, practicum/internships Signature Assignments Portfolios Capstones Reflections 8

A generic task, problem, case, or project thatcan be tailored or contextualized in differentdisciplines or course contexts.Signatures are defining characteristics thatreveal thinking or practices (Shulman, 2005).Signature assignments have the potential tohelp us know whether student learningreflects “the ways of thinking and doing ofdisciplinary experts”. Course-embeddedassessment Well aligned with LO’s Authentic in terms ofprocess/content, “real world”application May include reflection Collaboratively designed byfaculty9

Ingeneral education whenmultiple courses meet commonrequirements and shared LO’s –provides a common data set toenable documentation of generaleducation LO’s being met.When multiple sections of the samecourse are offered by multiplefaculty with varied pedagogy –enables programs to collectcommon data across the coursesections for program evaluationand review at both graduate andundergraduate levels.10

WhenInstitutional LearningOutcomes (ILO’s) are met invaried programs and departmentsacross the institution – provides acommon data set which enablesthe institution to determinewhether graduates are meetingthe ILO’s Foruse in a foundational courseon a grad or undergrad level,like PSY 500 or Intro toComputer Science to check thatstudents are prepared withprerequisite understandings andskills for success in the coursesthat follow within the program.11

1.2. Faculty review one or more of theagreed upon targeted learningoutcomes and come to a commoninterpretation of them.Faculty use the learning outcomes tobrainstorm possible and alignedtasks, problems, examples, authenticproblems. (these are oftensuggested within the outcomes)EX. Students analyze a issue frommultiple perspectives and form a personalposition of agreement/support or action.Possible tasks:Articulate and analyze an issueIdentify sources of perspectivesDescribe multiple perspectivesDevelop a position statement12

Kinds of calinternationalpolitical3. First draft of the assignment is intentionallygeneric (in context) to allow for multipledisciplines and contexts.4. Assignment is tailored for varied course ordisciplinary contexts.5. All faculty users agree to the use and tocollaborative review of student work samples.6. Faculty engage in conversations aboutexpectations in student work, preferably designa rubric.13

14Promote faculty discussions of studentlearning, pedagogy, assessment (culture oflearning)Provides significant common data sets todocument program or institutional impactEngages students in important learningactivitiesGuides pedagogy especially practice forlearningHas potential for application or transfer toanother department or institution forinformative comparisonsRequire time for developmentMay be translated as rigid or confining ofcurriculum or pedagogyRequires faculty agreement

LO Students design management systems thatinclude staffing, budget, evaluation,organization, and strategic plan. NON-PROFITCOMMUNITY STUDIES, EDUCATIONADMINISTRATIONLO Students develop needs assessments,analyze data, and design community-basedresponses to the findings. PUBLIC HEALTH,SOCIAL WORKLO Students collaborate to produce public artthat reflects community context. ART MEDIA,MULTICULTURAL STUDIESClear educational purpose (not repositories)Students’ participation in selecting contentsusing faculty selection criteriaRegularly evaluated with established criteriaIllustration of growthContinual updatingStudent reflection with potential for synthesisand metacognition15

16For almost any learning experience at bothundergraduate and graduate levelsEspecially appropriate for courses andprograms focused on thinking skills and ondeveloping synthesis and metacognitionFor courses and programs with smallnumbers of studentsFor self-designed programsIt helps students and faculty to look atlearning holistically – to see connections – topresent a composite of different kinds oflearning.It goes beyond achievement of outcomes andlooks at learning over time.It encourages students to be actively involvedin their learning.

It promotes diversity of assessment and canbe a rich focus for discussion amongstudents and faculty.It provides in-depth information forfaculty/staff use in revising courses andprogramsPortfolios require a great deal of time.Portfolios demand careful planning andgradual implementation.Contents of portfolios can be an issue interms of no. of items, length of items, etc.Storage of portfolios is also an issue.Analysis of portfolios as evidence of programsuccess can be time consuming.17

18Cambridge, Cambridge, & Yancey (2009).Electronic Portfolios 2.0 Stylus PublishingT. P. Light, Helen Chen, & John Ittleson.(2012) Documenting learning in E-portfolios:A guide for college instructors. Jossey-BassLinda Suskie (2009) Assessing studentlearning: A common sense guide, 2nd edition.Jossey-Bass See Chapter 13.Offered or required in ¾ of U. S. baccalaureateinstitutionsDiscipline-based Capstones are most commonformatEmergence of Capstones as a senior experiencein the late 1980’s and early 1990’sCalls for assessment that is synthetic, holistic,interdisciplinary and reflective of national goalsThe Boyer Commission (1998) recommendedcapstones for higher education curriculumResponse to the LEAP initiatives to providetransformational experiences

Capstones are a summarizing process with bothlearning and assessment integrated in theproject, problem solving, report, etc. (multipleforms); at graduate level in the form of thesesCapstones are best coordinated, implemented,and evaluated by collaborative groups (all faculty,teams of faculty, employers, community reps,students, alum, etc.)Actual Capstones are designed by students withinput from multiple directions.Arises from program or institutional missionand/or stated program or institution goalsShaped in a way that both assesses andextends the learning of program or institutionJudged or evaluated by a set of outcomesthat are aligned with programmatic orinstitutional goalsDesigned or conducted to be a high-impactpractice for student learning***19

For a programmatic or disciplinary Capstone,the prerequisite is a credible, sustained, andcoherent program with a solid core to bereflected in the Capstone For an interdisciplinary or institutionalCapstone, there must be broad connectionsbetween academic majors and thegoals/values of the general educationprogramNOTE: The interdisciplinary Capstone is morechallenging for both faculty and students. Intentionality Collaboration CurricularFit Professional Standards20

Functions within distinct cultural andorganizational contexts at all levels Practical issues like identifying stakeholders,analyzing resources, admissions and entryrequirements, course length, enrollment size,and course descriptionPortland State University Example todemonstrate the decisions. While Capstone formats or approaches maydiffer, all Capstone courses “will be informed bythe following expectations”: Demonstrated student initiative Significant preparatory work Active inquiry Integration of acquired knowledge and skills Culmination of previous academic work(Schermer & Gray, 2012)21

Must be a creative, inquiry-based learning experience thatdeepens the student’s knowledge and integration of thediscipline. Must be part of an upper-division course of at least 4 unitsand (preferably) taught by ladder faculty. May be completed individually or by a group, provided eachstudent’s contribution is significant, identifiable, and graded. Must culminate in a tangible product that can be archived(electronically) by the department or program for three years. Must be opportunities for students to share capstone projectswith peers; this can occur in class or outside of class.(Lindholm, 2012) Provide opportunities for self-assessment, peerassessment, and assessment by multiplefaculty/staff/community representatives/employers with shared standards, rubrics, etc.Develop simple and iterative progress reportingmechanisms for monitoring work, learning,relationships, engagement, potential for successConsider a hierarchical set of activities thatincentivize more capable students(Hauhart & Grahe, 2015, p. 197)22

Robert Hauhart & Jon Grahe (2015)Designing and teaching undergraduatecapstone courses. Jossey-Bass.B. Catchings (2004). Capstones and quality:The culminating experience as assessment.Assessment Update, 16 (1), 6-7.G. Roberts & T. Pavlak (2002). Designing theMPA capstone course: A structured-flexibilityapproach. Journal of Public Affairs Education,8 (3), 179-191.The reflective process is a majorcomponent of learningReflection provides an opportunity formaking meaning (with new information,ideas, experiences, skills, etc.)May take place individually and/or incommunity with peersMay take oral, written, graphic or otherforms23

24Reflection on the information, curriculumcontent, topics, experiences - substantivewriting or discussion aimed at processing andrevealing understandingReflection on the learning process itself –personal – focused on the learner’s learningexperience with attempts to identify thesignificance, value, meaning of theexperienceBrief or abbreviated form – one minute paper,muddiest point paper (Angelo & Cross’ CATS)Intermediate form – extended and ongoingover course or program; passed back andforth with other; primarily journals, learninglogsExtended form – a component of a portfolio,blends both foci, integrative, may address thelearning experiences of one course, programor the entire college experience.

Content of learning – what have you learned?Context of learning – How does your learningfit in the larger context of life?Learning process – What have you learnedabout how you learned?(Wlodkowski, 1999)Continuous – an ongoing coherent processConnected – integrates past and currentexperiences, experiences with coursecontent, concepts acrosscurriculum/disciplines, and works toempower learners with knowledge about theirlearning25

26Challenging – poses new questions andunfamiliar, even uncomfortable ideas forconsideration, while simultaneously providingsupportContextualized – extends to application ofknowledge and understandings, may be oralor written.The DEAL Model (describe, examine,articulate, and learn) from Ash & ClaytonEyler, J., Giles, D., & Schmiede, A. (1996). A practitioner’s guide toreflection in service learning. Nashville, TN: Vanderbilt UniversityFink, D. L. (2003). Creating significant learning experiences: Anintegrated approach to designing college courses. San Francisco, CA:Jossey-Bass.Ash, S.L., & Clayton, P. H. (2009a). Generating, deepening, anddocumenting learning: The power of critical reflection in appliedlearning. Journal of Applied Learning in Higher Education, 1, 25-48.Ash, S. L., & Clayton, P. H. (2009b). Learning through critical reflection:A tutorial for service-learning students (instructor’s version).Morrisville, NC: East Coast Digital Printing.Brooks, E., Harris, E., & Clayton, P. H. (2010). Deepening appliedlearning: An enhanced case study approach. Journal of Applied Learningin Higher Education, 55-76.*Wlodkowski, R. (1999). Enhancing adult motivation tolearn: A comprehensive guide for teaching all adults. SanFrancisco, CA: Jossey-Bass.

Amy Driscoll Takes inquiry beyond questions of whetherstudent learned or were successful to WHY?HOW? WHEN?Pushes for information about how learningoccurred, confidence in learning, students’perceptions of their own learning and success.Extends interactions related to and discussionsof learningCommunicates to students that their opinionsand issues are importantCan yield unexpected or unanticipatedinformation.27

28Assignment and course gradesRetention and graduation ratesAdmission rates to other institutionsAlum perceptionsStudent self-ratings of their learningEmployer satisfactionAwards, honors, scholarshipsEmployment rates of grads (includingsalaries, positions, etc.) Surveys Interviews Focus Groups

Determine purpose for survey – goals,issues, inquiryCarefully plan for how data/informationwill be used***Determine the population for the surveyby identifying the individuals who willhave the information being soughtSample size will be influenced by type ofinformation needed, use of theinformation, and population of interestSelect type of survey format that will best providethe intended information (frequency, rating scales,open-ended questions, checklist)Carefully design survey questions Avoid confusing or vague wording with clear,straightforward questions Avoid wording that might bias response Allow respondents to not answer questions Watch for order effects (Allen, 1995, 2004)Pilot test the survey before use29

30Determine purpose for interview – goals,issues, inquiryBe clear about how you will use theinformation/data from the interviewsSelect kind – structured (SI), semistructured (SSI) , or unstructured (UI)Develop a brief introductory protocol toexplain purpose, structure, etc. and useuniformly for each interviewUse care to not indicate your personal viewsPlan careful questions and do not interpretthe questions (SI)For probing, begin with topics and questions,and allow the interviewee to vary content andflow (SSI)Determine no. of interviewees using thecriteria of “sufficiency” and “saturation”(Se

Matthew Bruce Project Director of Program Assessment John F. Kennedy University mbruce@jfku.edu Laura Palazzolo Dean Lincoln Law School of San Jose dean@lincolnlawsj.edu Laura Alipoon Department Chair Loma Linda University lalipoon@llu.edu Leo Ranzolin Associate Dean, School of Religion Loma Linda University lranzolin@llu.edu