Transcription
51- ‐MJA- ‐C001Developing an Evaluation Rubric forHands-on Science Education ActivitiesAn Interactive Qualifying Project submitted to the faculty of WORCESTERPOLYTECHNIC INSTITUTE in partial fulfilment of the requirements for the Degreeof Bachelor of Science by:Scott CornmanGrace HollidayGregory TersoffApril 30, 2013Report submitted to:Professor Michael Aghajanian, AdvisorProfessor Holly Ault, Co-AdvisorChris Krishna-Pillay, CSIRO LiaisonCarly Siebentritt, CSIRO LiaisonThis report represents work of WPI undergraduate students submitted to the faculty as evidence of adegree requirement. WPI routinely publishes these reports on its web site without editorialor peer review. For more information about the projects program at WPI, seehttp://www.wpi.edu/Academics/Projects.
AbstractThe Commonwealth Scientific and Industrial Research Organisation (CSIRO)conducts science education programs in schools across Australia. This projectdeveloped a formal rubric to evaluate the effectiveness with which the hands-onactivities in these programs teach students science concepts in an engaging ways.Through educator interviews and surveys, program observations, and discussions withteachers and students, this rubric was developed and tested. The rubric was found tobe valid in accurately conveying key information about activities, reproducible whenused by different evaluators, and universally applicable to all age groups. The projectteam also provided recommendations for improvements to the rubric and used therubric to inform design recommendations for two new activities.i
AcknowledgementsThe project team has an extensive list of people (and insects) to whom theyowe their gratitude.Firstly, we would like to thank our advisors, Professors Mike Aghajanian andHolly Ault, for their efforts in making sure our project and paper were the best theycould be and for humouring our requests for more feedback on certain sections of thedraft. We would also like to thank our ID2050 professor, Professor Nicola Bulled, forher endless helpful feedback on our project proposal in C-term.We would like to thank all of the staff at CSIRO Education, Victoria whocontributed to our project by taking time to fill out our surveys and speak with us. Wewould especially like to thank the following educators for letting us observe theirprograms and for not stranding us at various train and bus stations around Melbourne.Thank you to Simon, Sarah, Gemma, Kate, Jason, Elke, Claudie, and Mick for lettingthe Americans invade your program sessions and play with the activities. We’d alsolike to thank Ava for always being there to answer both the front door and all of ourquestions.Additionally, we want to apologize to Cath for insinuating that she had neverseen an icicle. No one told us you were originally Canadian. We’re sorry!We would like to acknowledge our stick-bug friends who kept us companythrough most of our long days in the Green Room and honour the ones who passedduring our seven-week stay.Finally, we owe many, many, many thanks to our project liaisons, ChrisKrishna-Pillay and Carly Siebentritt for all of the help and support that they’ve givenus this term. Thank you to Carly for your helpful advice and feedback on our work,and for the effort you put in to find us bookings and travel arrangements for all of theprograms we needed to see. Without you, we never would have fulfilled our dream ofvisiting every train, tram, and bus station in the Melbourne metropolitan area. Thanksto Chris for sharing with us so much of your wisdom on program design, observationskills, presentation techniques, footy, Australia, and life. Your insights wereinvaluable to our project, and your comic relief was invaluable to our sanity. Ourchats with you were always the best parts of our days at work. Many thanks inparticular for saving the day when our initial project fell apart, helping us to put thepieces back together into a new project that was far more useful than the old oneii
could have been. Lastly, thanks to you both for taking time on a Saturday to see afooty game with us, specifically to Chris for reserving our seats and to Carly for goingalong (and even getting your face painted!) despite having no interest in the sportwhatsoever. We enjoyed the game and your company immensely!iii
Table of AuthorshipAll group members edited all .34.3.14.3.25.05.15.25.2.15.2.2Section TitlePrimaryAuthor(s)AbstractExecutive SummaryIntroductionBackgroundThe Commonwealth Scientific andIndustrial Research OrganisationCSIRO Science Education ProgramsMethodologyEducation Activity Research and CriteriaDevelopmentAnalysis of Education TechniquesDevelopment of an Activity EvaluationRubricFlexible solar cell activity designResearch into educational applications offlexible solar cell technologyDevelopment of flexible solar cell activityrecommendationsResults and DiscussionRubric DesignRubric CreationRubric WeightingAssessment of RubricValidity of RubricReproducibility of RubricUniversality of RubricActivity Design RecommendationsEnergy: Sources and UsesPolymers and Nano-ChemistryConclusions and RecommendationsProject SummaryRecommendations for RubricImplementationUse of RubricSuggested Rubric ImprovementsScottAllAllGregScottivGrace, ScottAllAllAllAllAllAllAllAllAllAllScott, GregAllGraceScottGreg, GraceAllGregGreg, GraceAllAllAllGraceGrace, Greg
Table of Contents1.0INTRODUCTION12.0BACKGROUND32.1 THE COMMONWEALTH SCIENTIFIC AND INDUSTRIAL RESEARCH ORGANISATION2.1.1 CSIRO SCIENCE EDUCATION PROGRAMS663METHODOLOGY133.1EDUCATIONAL ACTIVITY RESEARCH AND CRITERIA DEVELOPMENT3.1.1ANALYSIS OF EDUCATION TECHNIQUES3.1.2DEVELOPMENT OF ACTIVITY EVALUATION RUBRIC3.2FLEXIBLE SOLAR CELL ACTIVITY DESIGN3.2.1RESEARCH INTO EDUCATIONAL APPLICATIONS OF FLEXIBLE SOLAR CELLTECHNOLOGY3.2.2DEVELOPMENT OF FLEXIBLE SOLAR CELL ACTIVITY 1RUBRIC DESIGN4.1.1RUBRIC CREATION4.1.2RUBRIC WEIGHTING4.2ASSESSMENT OF RUBRIC4.2.1VALIDITY OF RUBRIC4.2.2REPRODUCIBILITY OF RUBRIC4.2.3UNIVERSALITY OF RUBRIC4.3ACTIVITY DESIGN RECOMMENDATIONS4.3.1ENERGY: SOURCES AND USES4.3.2VCE CHEMISTRY: POLYMERS AND NANO MMENDATIONS565.1PROJECT SUMMARY5.2RECOMMENDATIONS FOR RUBRIC IMPLEMENTATION5.2.1USE OF RUBRIC5.2.2SUGGESTED RUBRIC IX A: SUCCESSFUL ACTIVITY CRITERIA FROM LITERATURE REVIEWAPPENDIX B: ELEMENTS OF SUCCESSFUL ACTIVITIES MENTIONED IN EDUCATORINTERVIEWSAPPENDIX C: SUGGESTIONS FOR USES OF FLEXIBLE SOLAR CELLS IN EDUCATORINTERVIEWSAPPENDIX D: ACTIVITY OBSERVATION SHEETAPPENDIX E: PRESENTER-LED DISCUSSION QUESTIONSAPPENDIX F: GUIDELINES FOR TEACHER CONVERSATIONSAPPENDIX G: EDUCATOR CRITERIA IMPORTANCE SURVEYAPPENDIX H: FULL EVALUATION RUBRICAPPENDIX I: PAIRWISE COMPARISONS USED FOR CRITERIA WEIGHTINGAPPENDIX J: ACTIVITY SCORING SHEET59v2122616364656667697274
APPENDIX K: RUBRIC WEIGHTING SHEETAPPENDIX L: SUB-CRITERIA RUBRIC SCORES FOR OBSERVED ACTIVITIESCHEMISTRY: ACTIONS & REACTIONSENERGY: SOURCES AND USESFORCES, MOVEMENT AND SIMPLE MACHINESNATURAL DISASTERSVCE PHYSICS: MATERIALS & STRUCTURESAPPENDIX M: RUBRIC SUMMARY SHEETS FOR OBSERVED PROGRAMSCHEMISTRY: ACTIONS & REACTIONSENERGY: SOURCES AND USESFORCES, MOVEMENT & SIMPLE MACHINESNATURAL DISASTERSVCE PHYSICS: MATERIALS & STRUCTURESAPPENDIX N: SUB-CRITERIA RUBRIC SCORES FOR EDUCATOR PROGRAMEVALUATIONSAPPENDIX O: STANDARD DEVIATIONS IN EDUCATOR PROGRAM EVALUATIONSAPPENDIX P: SCORE RANGES IN EDUCATOR PROGRAM EVALUATIONSvi75767677787980818181828283848687
List of FiguresFigure 1: Science and Technology are important for society.Figure 2: I would like to become a scientist.Figure 3: Percentage of Australian Year 12 Students Taking Science Courses by SubjectGraph from (Ainley et al., 2008)Figure 4: Relative Importance Weighting of Overall CategoriesFigure 5: Two students written responses to the Energy ProgramFigure 6: Instructions/Expectations are Clear Rubric ExcerptFigure 7: Category Scores of Programs for Various Age Groupsvii345534414250
List of MainCategoriesTableC:WeightsofEducationalValueSub- b- ‐criteriaTableE:WeightsofSatisfactionSub- ‐criteriaTableF:WeightsofEaseofCompletionSub- MainCategoryviii333436373838404444474749
Executive SummaryThe Commonwealth Scientific and Industrial Research Organisation (CSIRO)is Australia’s national science agency, consisting of both research and educationbranches. The education branch offers a wide variety of informal science educationprograms for students of all ages in schools across Australia. In Victoria, CSIROEducation officers transport these programs into schools and present them to classesof 30 to 60 students. These informal science programs typically consist of three maincomponents: an introduction with a few demonstrations of the main concepts coveredby the program, a period of hands-on activities and experimentation, and then a wrapup discussion that may include another demonstration to conclude the program.These programs are designed to be interesting, engaging and educational.Over time, CSIRO Education will adjust the makeup of these programs,adding or removing activities to fill new needs or strengthen a particular aspect of aprogram. Currently, CSIRO Education holds a yearly staff meeting, in whicheducators discuss the strengths and weaknesses of these programs and proposemodifications. As this discussion includes no formal structure for analysing thestrengths and weaknesses of programs, decisions come down to subjectiveconversation based on years of experience.While these discussions are fairly effective, due to the staff’s years ofexperience with specific CSIRO education programs and education as a whole, theycan become difficult without some framework to guide the conversation. This projectset out to create an effective way for CSIRO education officers to standardise thiscommunication and allow them to quickly point out strengths and weaknesses of anactivity. To accomplish this goal, we developed an activity evaluation rubric thatenables educators to translate their experience with a program into quantified scoresacross several criteria that measure the effectiveness of the program’s hands-onactivities.Similarly, CSIRO lacks a systematic means of designing new activities andprograms to meet key objectives. This rubric will also give CSIRO educators astandardised starting point for the design of new activities and programs. To test theeffectiveness of the rubric as a design tool, we developed design recommendations foractivities involving new flexible solar cell technology designed by CSIRO’s MaterialsScience branch.ix
To develop the rubric, we studied current CSIRO education programs througha combination of background research, discussions with CSIRO educators, and firsthand experience observing CSIRO science education programs as they are normallyrun. The main purpose of this research was to generate an informed list of importantfeatures of hands-on activities, at the same time determining the effects of eachfeature and how important it was to the success of an activity. Through this research,we were able to create the following list of main criteria that successful activitiesshould exhibit, along with their measurable component sub-criteria: Educational Valueo Curriculum Relevanceo Potential for Teacher Extensiono Encourages Student Thought Logical Feasibilityo Health, Safety, and Environmento Costo Ease of Setup/Takedowno Ease of Transport/Handlingo Durabilityo Ease of Repair Initial Draw (No component sub-criteria) Student Satisfactiono Student Interactiono Novel Completion Processo Relevance to Students’ Liveso Surprising/Unexpected Results Ease of Completiono Clear Instructions/ Expectationso Intuitiveness to Completeo Appropriate LengthThese criteria measured the key elements of an effective program, covering howlikely a student was to begin an activity, how easy the activity was to perform, howmuch was learned, how much the student would enjoy and remember the activity, andhow many resources it required as part of the program. Each of the sub-criteria, whichx
made up these requirements, was represented on the rubric as a 0-5 scale, withdescriptions defining the traits that activities must exhibit to earn each possible score.These descriptions were made as clear as possible, to aid educators in giving similarscores based on their experiences, rather than subjective opinions on what thoseexperiences represented. This allowed educators to begin the discussion of a programfrom the common ground of their rubric scores, before deciding on the significance ofthese scores to any proposed program modifications.We ensured the rubric worked by evaluating a variety of programs and activities.Through these evaluations, we confirmed: The validity of the rubric in effectively and accurately measuring aspects ofthe main and sub-criteria The reproducibility of scores across educators to ensure that the rubric wouldgive them a common ground in their assessment of activities The universality with which the rubric can be used to evaluate the wide rangeof programs and activities that CSIRO offers.We determined that the large majority of criteria and sub-criteria were valid,reproducible and universally applicable. However, some sub-criteria are weaker thanothers in these aspects, and the project team has made recommendations on how therubric can be improved for continued use by CSIRO education staff. Theserecommendations are: Adding a “coolness” sub-criteria to the Satisfaction main criterion Separation of instructions/expectations are clear into two sub-criteria Adding text to the ease of repair scoring descriptions to includereplacement of broken parts Rewording the student interaction scoring descriptions to put lessemphasis on experimental variables Rewording relevance to students/familiarity of concepts or separatingit into two criteria Reducing weight of Logical Feasibility main criterion as the targetaudience of the program becomes older. Increasing weight of Educational Value main criterion as the targetaudience of the program becomes older.xi
The rubric we created will also be a useful design tool for future hands-onactivities and new programs. To validate this claim, we used the rubric as a guidelinefor providing design suggestions for two future flexible solar cell activities. Thesedesign recommendations are specifically geared towards the inclusion of theseflexible solar cell activities in the Energy: Sources and Uses program and Polymers &Nano-Chemistry program.xii
1.0 IntroductionAs the world becomes increasingly dependent on new advancements inscience to solve problems, improved tools and technologies are needed to keep upwith the demand for progress. In order for these improvements to be made, thereneeds to be a motivated group of potential scientists to help work on these problems,and a society that understands the effort and resources required to solve them. It istherefore important to stimulate student interest in science whenever possible,encouraging students to view science as a potential career path and giving them thescience awareness necessary to make informed policy decisions. If interest in sciencelessens on a societal level, crucial problems in that society could be left unsolved.Australia is one country where students’ interests in science are declining. Thepercentage of students taking higher level science courses has been steadilydecreasing since the 1980s (Ainley, Kos, & Nicholas, 2008). In contrast, the need fornew scientists has increased as many areas of Australia seek to improve their societyand environment through science-based endeavours such as carbon emissionreduction programs. As a smaller portion of the population studies physics, biology,and chemistry, the number of people capable of solving these problems decreases.This is especially worrisome for any projects requiring a gradual shift towards newtechnologies that will, over time, require new solutions and scientists to championthese innovations.There are many organisations already in Australia trying to reverse this trendof decreasing interest in science. The Commonwealth Scientific and IndustrialResearch Organisation (CSIRO) works to further the cause of science both throughresearch and development, and by raising public awareness. The CSIRO EducationVictoria branch designs and runs unique, non-formal, hands-on science programs thatare used by teachers to supplement their curriculum. Through these activities, CSIROendeavours to make science interesting for future generations by teaching lessons ininnovative and interactive ways. While CSIRO has a number of programs on a varietyof subjects, these programs are frequently added to and updated in order to improvehow effectively they stimulate interest in science.One such update is currently underway with CSIRO’s existing programs, inorder to include cutting edge flexible solar cell technology that CSIRO materials1
scientists have developed. While some programs have utilised solar technology in thepast, silicon based solar panels were used, which are both more difficult tomanipulate, and lack the unique characteristics of the polymers used in the flexiblesolar cells. CSIRO Education has not utilised flexible solar cell technology in itsprograms before and is interested in including novel activities that will allow for newdemonstrations of the technology.In addition, CSIRO has no standard by which to determine how programsshould be updated. CSIRO currently relies on the instincts and intuition of educatorsto determine which portions of programs are effective and which programs need tohave activities added, removed, or modified. A quantifiable method of judgingactivities that could be used by staff with a variety of backgrounds and experiencewill improve the evaluation process, requiring fewer abstract opinions and allowingeducators to more easily explain positive or negative features of an activity.Our project developed a rubric to judge activities based on importanceweighted design criteria, to be used by CSIRO educators to evaluate existing andproposed activities. Informed by these criteria, we also developed designrecommendations for two new hands-on educational activities involving flexible solarcell technology that CSIRO could add to its existing Energy: Sources and Usesprogram and to its Polymers and Nano-Chemistry program.2
2.0 BackgroundDeclining science interest is evident internationally and is a concerningworldwide problem. Studies conducted by the organisation known as Relevance ofScience Education (ROSE) surveyed secondary-school students’ opinions on sciencein 40 countries. The surveys consisted of ranking a number of given statements on ascale from 1 (disagree) to 4 (agree) (Sjoberg & Schreiner, 2005). Responses to thefollowing two statements show a concerning disparity between children’s opinions onscience: Science and technology are important for society (Figure 1) and I would liketo become a scientist (Figure 2). Students worldwide recognise that science andtechnology benefit society, but have little interest in pursuing science themselves; atrend that is especially prevalent in the highly developed countries.Figure 1: Science and Technology are important for society.Average scores for boys (filled symbols)
4.1 Rubric Design All 4.1.1 Rubric Creation All 4.1.2 Rubric Weighting Scott, Greg 4.2 Assessment of Rubric All 4.2.1 Validity of Rubric Grace 4.2.2 Reproducibility of Rubric Scott 4.2.3 Universality of Rubric Greg, Grace 4.3 Activity Design
