Transcription
Journaling and Rubrics:The Pleasure and the PainKathleen Pusecker, Office of Educational AssessmentGabriele Bauer, Center for Teaching EffectivenessWinter Faculty InstituteJanuary 20061
Session Goals Reinforce concepts presented by Dr.Marilee Bresciani at the January 4th - 5thsessions. Contextualize the assessment of criticalthinking. Focus: Journaling and Rubrics.2
OutcomesBasedAssessmentCycleAdapted fromPeggy Maki, Ph.D.Marilee J. Bresciani, Ph.D.3. Gather Data4. Analyze Data andInterpret Evidence1. Mission/PurposesGoalsOutcomes2. ImplementMethods to5. Make decisions to improveDeliverprograms and REPEAT CYCLE.Outcomes (e.g.,planning) andMethods toGather Data3
Planning for Learning and AssessmentBanta, T., 2005What general How wouldoutcome areyou know ityou seeking? if you saw it?What will thestudentknow, be ableto do,demonstrate?How will youhelp studentslearn it? (inclass or outof-class)How do youmeasure eachof thebehaviorslisted in #2(criteria)?What aretheassessmentfindings?Whatimprovementsmight bebased onassessmentfindings?4
Multiple Assessment MethodsFocus: Journaling and RubricsTestsPapersWriting AssignmentsLab ReportsOn-line DiscussionGroupsCase Study AnalysisPresentationsBulletin BoardPerformancesBooklets5
What is critical thinking?6
Core Critical Thinking SkillsApplicationMetacognitionSynthesisDecision MakingProblem SolvingCreativityEvaluationAnalysis7Suskie, L. (2005)
Characteristics of Critical Thinking(criteria for assessment of critical thinking) luation ofobservations, andcommunications, andother information sources(Fisher, 2001, p. 14)8
Engaging Critical Thinking (Bloom’s Revised Taxonomy)CreatingGenerating new ideas, products, or ways of viewing thingsDesigning, constructing, planning, producing, inventing.EvaluatingJustifying a decision or course of actionChecking, hypothesising, critiquing, experimenting, judgingAnalyzingBreaking information into parts to explore understandings and relationshipsComparing, organising, deconstructing, interrogating, findingApplyingUsing information in another familiar situationImplementing, carrying out, using, executingUnderstandingExplaining ideas or conceptsInterpreting, summarising, paraphrasing, classifying, explainingRememberingRecalling informationRecognising, listing, describing, retrieving, naming, finding9
JournalingReflective writing, Exploratory writing Develop thinking skills by helping students extend,clarify, and deepen their thinking about the subjectmatter. Aligned with instructional goals and teachingapproach: context-specific. Can serve as:– instructional method and– assessment tool.10
What journaling method (s) are youcurrently using? (Specific course) What works well? What is one challenge/questionencountered?11
Approaches to Journaling Guided JournalContent-specific questions, certain amount of time/weekallocated for writing, typically out-of-class.e.g., How have you seen journaling function in a particularcourse from both the students’ and the instructor’sperspectives? How would J. Bean respond to these issues?Reference: Bean, J. (2001). Engaging ideas. The professor's guide tointegrating writing, critical thinking, and active learning in the classroom.12
Approaches to Journaling (cont.) Semi-structured JournalProvides guidance in helping writers think of things tosay: Generic questions or tasks appropriate to discipline.e.g., How does your personal experience relate to what we’restudying today?e.g., “writing probes” to clarify mathematical thinking:-- What is the problem asking you to do?-- What makes this problem difficult to solve for you?-- Where did you get stuck and why?-- What information do you need to get unstuck?13
Approaches to Journaling (cont.) Open-ended JournalStudents choose how to write about the course and theirexperiences. Guidance: number of pages/week. Servesas record of students’ intellectual journey through acourse.e.g., summarize lectures, respond to students’ questions,provide personal examples, raise questions, expressexcitement about new ideas.14
Form of Journaling:Classroom Assessment Techniques (CAT)Approach designed to help instructors find outwhat students are learning in class and how wellthey are learning it: information about change instudent learning.Both teaching tool and assessment device.Reference. Cross, P., & Angelo, T. (1993). Classroom assessmenttechniques.15
Potential Impact of CATs Learn what and how your students are thinking. Clarify your session/course goals. Get feedback to make mid-course adjustments. Change classroom norms re: student involvement. Help students become self-aware of their learning. Collect data for post-course improvement.16
Examples of Journaling Double-entry Journal (dialogue journal)reflect on course material/readings and reactions / responsesfocus on processSummarize readingsPut material in own wordsInteractive commentary-- ask questions-- disagree-- challenge-- personal experience17
Examples of Journaling (cont.) Contemporary issues JournalRelate course material to current issues (e.g., newspaperreadings) Pro / Con GridAssess advantages / disadvantagesCompare / contrast18
Classroom Assessment Techniques (CATs) provided bySouthern Illinois University, Edwardsville.http://www.siue.edu/ deder/assess/catmain.html Classroom Assessment Techniques (CATs) provided byField-tested Learning Assessment Guide for science,math, engineering, and technology instructors, Universityof phpFaculty across disciplines talk about why they have used a particularCAT in their teaching and examples are provided.19
What methods of assessment have youused with journaling?20
Considerations: Assessment of Journaling Plus, check, minus – quality thinking (evidenceof analysis, reflection, engaged thinking).-- translated into letter grades, weight for finalgrade Ongoing feedback – review sample of students. Content analysis: major themes.21
Pro/Con Grid Concerning RubricsPLEASUREPAIN 22
What is a html#what is a rubric A rubric is a scoring guide that seeks to evaluate a student's performancebased on the sum of a full range of criteria rather than a single numericalscore. A rubric is an authentic assessment tool used to measure students'work.–Faculty-generated–Student Generated: When used as an evaluative toolenable students to be candid and specific in theirevaluations of their own writing, thus supporting whatresearch says about authentic assessment and selfregulated learning (Grierson, Anson & Baird, 2002;Bandert-Drowns, Kulik, Kulik & Morgan, 1991).23
Types of Rubrics Formative-rubric becomes the map to guide a project.Students progress towards a goal and the process oflearning is evaluated. Summative- Once the final project is submitted,summative assessment is accomplished as the rubric isused for a final grade. All rubrics, regardless of their type, include threeessential features: evaluative criteria, quality definitions, and a scoringstrategy (Popham, 1997).24
1-Rubric Design Protocol When it comes to rubrics, the most frequently asked question. is "Do you have rubrics for (insert any subject orskill)"? Our response is "Yes, but it is recommended that you consider designing for the course training. We havelearned that using others' rubrics does not productively impact instruction or student learning. We rely upon thefollowing professional protocol (structured steps) to construct criteria statements, indicators, and to build rubrics.Rubric Design Protocol Gather student work samples. Sort samples into 3 - 4 groups. Record your own descriptive statements.Exemplary, Proficient, Basic, Novice or Beyond Expectations, Meets Expectations, Below ExpectationsCategorize statements into critical performance elements. Write an operational definition of each element. Select the "best match" of student work per each level of performance--i.e.exemplary, proficient, basic, novice. Repeat steps . . . refining your rubric's elements, descriptors, and indicators. Store your rubrics and student work examples at each level to use for instructional, communication, and futureprofessional development purposes. We have found that the use of this "job-embedded professional development protocol" and the close examinationof student work transfers into daily classroom instruction and results in increased collaboration between studentsand teachers on the topics of "quality" and "continuous improvement".http://www.rubrics.com/best practices rubric design.html (2005)25
Rubrics Answer By what criteria will the work be judged? What is the difference between goodwork and weaker work? How can both students and instructorsfocus their preparation on excellence?s26
Why are they used? Focus instruction---intentionally. Guide feedback---descriptively. Characterize desired results---objectively. Operationalize performance standards---purposefully. Develop self-assessment competence---constantly. Involve students---thoughtfully.27
Rubric Components Performance Element: the major, critical attributes which focus uponbest practice. Scale: the possible points to be assigned (high to low). Criteria: the conditions of a performance that must be met for it to beconsidered successful. Standard: a description of how well the criteria must be met for theperformance to be considered "good". Descriptors: statements that describe each level of the performance. Indicators: specific, concrete examples or telltale signs of what to lookfor at each level of the performance. Taken from software company www.rubrics.com28
29
Student Designed Web Page Rubric5101517The pages are unattractive.Text is difficult to read. Thebackgrounds are distracting.The pages appear "busy" or"boring". Text may be difficult toread. The backgrounds aresomewhat distracting.The pages are eye-catching andattractive. Text is easy to read.The backgrounds are subtle andappropriate.The pages are well organizedwith tables. Text spacing andalignment make reading easy.The backgrounds enhance thepage.There are no photos, icons orclip art or they areinappropriate or of lowquality.Photos are blurry or fuzzy; iconsand clip art do not "fit" with thetopic. Too many pictures makethe download time slow.Photos, icons, and clip art areappropriate, of high quality, anddownload fairly quickly.Photos, icons, and clip art areused creatively and may follow atheme.Information is poorly written,inaccurate, or incomplete.Information could be betterwritten and too much informationis given in each section.Information is well written andinteresting to read and ispresented in short sections.Information is creatively writtenand cleverly presented.The user may become lost orlinks may be missing or notworking.The user may become confusedwhen navigating between pages.Some links may not work.Links are consistent and easy tofind so that the user can easilynavigate back and forth throughpages.Links are created with imagesand icons to enhance the textlinks.Partners argue or fight muchof the time and do not shareresponsibilities.Partners have trouble solvingdisagreements; one partner doesmost of work.Partners get along well and shareequally in responsibilities.Partners show respect for oneanother, get along especiallywell and work together on allaspects of the project.Students are often out of areawithout permission and causedisruptions in the lab and otherclassrooms.Students occasionally leave areawithout permission and are louderthan necessary in the lab and inother classrooms.Students stay in their area, talkquietly to their own partner only,and cause minimum disruptionswhile visiting other classrooms.Students are always on task, stayin their own area, and cause nodisruptions when visiting otherclassrooms.Layout / DesignGraphicsInformationNavigation / LinksWorking TogetherFollowingClassroomGuidelinesNote - Add the scores for each category to get a percentage score.30
Rubric ActivityIn 5 minutes, start a rubric thataddresses a critical thinkingskill.31
A Rubric for RubricsA Tool for Assessing the Quality and Use of Rubrics in EducationScoring:0 - 10 needs improvementCriteriaClarity of criteriaDistinction betweenLevelsReliability ofScoringClarity ofExpectations/Guidance toLearnersSupport ofMetacognition(Awareness of Learning)Engagement ofLearners in RubricDevelopment/ Use *16 – 20 solid/good11 - 15 workable1Unacceptable2Acceptable21 – 24 exemplary3Good/Solid4ExemplaryCriteria being assessed areunclear, inappropriate and/orhave significant overlapCriteria being assessed canbe identified, but are notclearly differentiated or areinappropriateCriteria being assessed areclear, appropriate and distinctEach criteria is distinct, clearlydelineated and fullyappropriate for theassignment(s)/courseLittle/no distinction can bemade between levels ofachievementSome distinction betweenlevels is made, but is nottotally clear how wellDistinction between levels isapparentEach level is distinct andprogresses in a clear andlogical orderCross-scoring among facultyand/or students often resultsin significant differencesCross-scoring by facultyand/or students occasionallyproduces inconsistent resultsRubric is not shared withlearnersRubric is shared and providessome idea of the assignment/expectationsRubric is referenced - used tointroduce anassignment/guide learnersRubric serves as primaryreference point for discussionand guidance for assignmentsas well as evaluation ofassignment(s),Rubric is not shared withlearnersRubric is shared but notdiscussed/ referenced withrespect to what is beinglearned through theassignment(s)/courseRubric is shared and identifiedas a tool for helping learnersto understand what they arelearning through theassignment/ in the courseRubric is regularly referencedand used to help learnersidentify the skills andknowledge they aredeveloping throughout thecourse/ assignment(s)Learners are not engaged ineither development or use ofthe rubricsLearners offered the rubricand may choose to use it forself assessmentLearners discuss the design ofthe rubric and offerfeedback/input and areresponsible for use of rubricsin peer and/or self-evaluationFaculty and learners arejointly responsible for designof rubrics and learners usethem in peer and/or self32evaluationThere is general agreementbetween different scorerswhen using the rubric (e.g.differs by less than 5-10% or lessthan ½ r/Rubrics/A%20Rubric%20for%20Rubrics.htm Dr. Bonnie B. Mullinix Monmouth UniversityCross-scoring of assignmentsusing rubric results inconsistent agreement amongscorersDecember 2003
Using the rubric to assess a rubric as aguide, discuss with a partner your rubricdesign. Consider what would you improve? Whatobstacles did you face? What workedwell? What questions remain?33
Questions you sent Have you had yourstudents generate arubric for anassignment beforethey do it? Are there somecommoncharacteristics thatcould make a rubricunworkable?34
Rubric Problems and Pitfalls Failure to adapt to each project Ceiling Effect or limiting effects especially in standardized writingassessments (LaBrant, 1936; Mabry & Kaytner, 1997; Mabry, 1999). Lack of authenticity - evaluative criteria fail to capture importantfeatures of the skill being measured Criteria may be vague Rubric is too long or complex (Popham, 1997). Wolf & Wolf (2002) stress, the goal of rubric based assessment isnot to “wrench a piece of writing in line with standardized rubriccriteria, but to meet and then push beyond the boundaries ofestablished rubrics to take the writing to the next level” (232). Pitfall of the five-point scales being converted to letter grades(Strickland & Strickland, 1998).35
Rubric Help rld.com/a curr/curr248.shtmlCreating Rubrics: Tools You Can Usehttp://www.educationworld.com/a curr/curr248.shtmlKathy Schrock’s Guide for Educators:Assessment Rubrics: sess.htmlProject Based Learning Checklists arhttp://rubistar.4teachers.org/Rubric ubric builder.php3 Rubric Machinehttp://www.thinkinggear.com/tools/Rubrics (Chicago Public deas and Rubrics/Rubric Bank/rubric bank.htmlRubrics for Web Lessons: Background ns.htmTeach-nology Rubrics Generatorshttp://teachers.teachnology.com/web tools/rubrics/Understanding Rubrics rics at-a-glance (PowerPoint Show requires Internet Explorer)http://ettc.lrhsd.org/rubrics.htmRubrics at-a-glance (PowerPoint Show requires Adobe Acrobat Reader)http://ettc.lrhsd.org/rubrics.pdf36
Online Resources andFacilitator Contact Information Kathleen Pusecker – 831-8537 klp@udel.edu Gabriele Bauer – 831-2027gabriele@aip.udel.edu Office of Educational Assessment http://assessment.udel.edu/index.htm Rubrics – Writing Center:http://www.english.udel.edu/wc/staff/ Center for Teaching Effectivenesshttp://cte.udel.edu/eval.htm PRESENTTechnology tools to gather information and feedback.37
References Bandert-Drowns, R., Kulik, C., Kulik, J. & Morgan, M. (1991). The instructional effect of feedback in test-likeevents. Review of Educational Research, 61, 213–238.Bean, J. (2001). Engaging ideas. San Francisco: Jossey-Bass.Cross, P., & Angelo, T. (1993). Classroom assessment techniques: A handbook for college teachers (2nd ed.). SanFrancisco: Jossey-Bass.Grierson, S.T., with Anson, A. & Baird, J. (2002). Exploring the past through multigenre writing. Language Arts,81(1), 51–59.Huba, M., & Freed, J. ((2000). Learner-centered assessment on college campuses. Needham Heights, MA: Allyn& Bacon.Picket, N, Dodge, B. (2001). Authentic Assessment: San Diego State weblessons.htm.Popham, W.J. (1997). What’s wrong—and what’s right—with rubrics. Educational Leadership, 55, 72–75.Rubric Design Protocol (2005). http://www.rubrics.com/best practices rubric design.htmlSchirmer, B.R. & Bailey, J. (2000). Writing assessment rubric.Teaching Exceptional Children, 33(1), 52–58.Sipe, R. (2002) Research, Rubric-Based Instruction/Evaluation,and Strategies for Writers. dfStik, A. (1997). Creating Rubrics Through Negotiable Contracting and Assessment. ERIC les 006.htmStrickland, K. & Strickland, J. (1998). Reflections on Assessment: It’s purposes, methods, and effects on learning.Portsmouth, NH: Boynton/Cook Heinemann.Suskie, L. (2004). Assessing student learning. A common sense guide. Bolton, MA: Anker Publishing.Wiggins, G., (1993, November). Assessment: authenticity, context and validity. Phi Delta Kappan, 75:3, p. 200214.Wolf, S.A. & Wolf, K.P. (2002). Teaching true and to the test in writing. Language Arts, 79(3), 229–240.38
Student Designed Web Page Rubric 5 10 15 17 Layout / Design The pages are unattractive. Text is difficult to read. The backgrounds are distracting. The pages appear "busy" or "boring". Text may be difficult to read. The backgrounds are somewhat distracting. The pages are eye-catching and attractive. Text is easy to read. The backgrounds are .
