Transcription
Valerie Terry-EDTC6321-Spring 2012Visualization and Visual Aids in Persuasive Public Speaking:What's the Difference and Does it Matter?IU#5: The Formative EvaluationIntroduction and OverviewThis instructional unit reports the results of a formative evaluation of the instructional materials draftedin IU#4. Formative evaluation of the materials was gathered from three sources: one distance-learning subject matter expert (SME); one learner fitting the module's intended learner profile in a one-to-one face-to-face situation;and, a small group of eleven learners also fitting the module's intended learner profile but accessedin a classroom setting.Based on feedback from these sources, the rough draft materials have been revised, readied for postformative evaluation instructional use.Specific Instructional GoalUndergraduate students will differentiate between visual aids in public speaking and "visualization" inthe fourth step of the Monroe Motivated Sequence (MMS) persuasive public speech method.I. SME ReviewA. Introduction - A former colleague who is a distance learning expert reviewed my materials. She wasformerly the chair of the academic distance education department at Texas State Technical CollegeHarlingen. She also has chaired the distance education course review committee at TSTC-Harlingen, hasdesigned, developed and taught online courses for almost four years and has published scholarlyresearch on the subject of peer-to-peer mentoring in online course development among instructors inwhich I was an active participant as both mentee and mentor.B. Summary - The following table provides a summary of the results of the evaluation:INSTRUCTIONAL STRATEGYCOMPONENTSBefore evaluationPreinstructional activitiesContent presentationLEARNER COMMENTSMY RESPONSEThe SME agreed to review mymaterials via email. She hasexpressed strong interest in thisclass and my experiences in it."Love the podcast script." The SMEalso mentioned providing thelearners with a written version toaccompany the audio version. TheSME also liked the anticipationguide, noting that it "lets the learnerknow what the outcomes areand.need to [be] achieve[d] to besuccessful."The SME questioned whether all theI sent her a pdf file of theinstructional materials via email. Ialso sent her the analysis portion ofour IU#4 assignment.I will provide a written version ofthe podcast script to my small grouplearners for their formativeevaluation session.I will consider the SME's questions.1
Valerie Terry-EDTC6321-Spring 2012(Correlates to steps 1.0-6.0 ofinstructional goal.)instructional material would bereleased to the learner "at onetime"; she also wondered if the unitcould be completed in the orderpresented or could the learnercomplete the items "in any order."Learner participationAssessmentFollow-through activitiesNo concerns noted.No concerns noted.Aside from the comment about theanticipation guide alreadymentioned, no other concerns werenoted.The SME commented that thelearning module "could be'dropped' into any LMS." Shethought the assignment summarywas "very well-written andthorough."After evaluationMy original intention was that thematerials will be released to thelearner at one time, since themodule should be completed withina relatively short timeframe. And,yes, I designed the materials to becompleted in the precise ordergiven due especially to how eachassignment builds up on theconcepts engaged in the precedingones. I think this systematicapproach is prescribed in the IDdesign we are learning in thiscourse. The results of the one-toone and small group formativeevaluations should help to validateor invalidate this design approach.None required.None required.None required.None required.C. Synopsis - Short-term revisions: The SME I asked to review my instructional materials respondedpositively to their design and purpose. Aside from the implied suggestion that a written version of thepodcast script be provided to the learner, there were no other revisions or adjustments recommended.Long-term implications: The SME and I have worked together extensively and share a commonphilosophy regarding online learning, especially in its role and function in progressive higher education.Her comments were intentionally focused on what issues I might need to consider for possibleadjustment rather than necessarily on every aspect of the instructional package. Thus, what she did notcomment on speaks volumes in terms of approval, with particular emphasis on the unit's flexible LMScompatibility. An interesting side note is that this SME has been working on an ID project that involvestimed-release of instructional materials to learners, thus the source of her comment regarding mine.II. One-to-one EvaluationA. Introduction - The student learner fits the profile of the intended learner of this module. She isfemale, has the entry level skills required and is a non-traditional student because she is older than thetypical undergraduate. However, this module was designed with both traditional and non-traditionallearners in mind. Because the small group evaluation was to be done with a group of traditional learners2
Valerie Terry-EDTC6321-Spring 2012(male and female between the ages of 18-25), using a non-traditional learner for the one-to-oneevaluation yielded additional helpful feedback.The setting was at home. This was appropriate because many learners will be able to use thisinstructional unit at home, as long as they have Internet access. Also, this setting was convenient for thelearner.This evaluation was conducted on Sunday, April 22, 2012, start time, 2:15 p.m., and end time,2:43 p.m. (28 minutes).B. Summary - The following table provides a summary of the results of the evaluation:INSTRUCTIONAL STRATEGYCOMPONENTSBefore evaluationPreinstructional activitiesContent presentation(Correlates to steps 1.0-6.0 ofinstructional goal.)Learner participationAssessmentLEARNER COMMENTS/MYOBSERVATIONSMY RESPONSELearner appeared at ease; therewere no outside distractions; sheseemed to be able to concentratefully on the materials.The podcast script was too long;Learner started to appear restless asI read it to her. She requested awritten version of the script so shecould follow along with the audio.Learner had no problem with the"before instruction" part of theanticipation guide. She markedagreement with only the firststatement on the guide, indicatinglittle prior knowledge of the contentthe instructional unit presents.Learner had trouble finding keyterm definitions in textbook pagesprovided. She asked for clarificationof wording of the assignmentscorrelated to instructional steps 1.0(I) and 3.0 (I). Learner spent moretime on the reading portion of theassignments than expected.I gave a brief (1-2 minute) overviewof the purpose of the materials andthe evaluation. I gave some verybasic introductory directions.I will provide a written version ofthe podcast script to my small grouplearners for their formativeevaluation session.Learner completed the instructionalunit within the 30-minuteprescribed timeframe withoutinteraction with me. She appearedfully engaged with the instructionfrom beginning to end; there wereno pauses to stretch or seekrefreshments or go to thebathroom. Learner smiled severaltimes while completing the variousassignments, seeming to enjoy thevariety of tasks assigned.Learner scored 100% (A/Excellent)on the interactive objective testI will revise the wording of theassignments indicated. I will reducethe number of pages to bereferenced in the textbook duringthe instruction. The order in whichthe materials were presented to thelearner and their use as oneinstructional bundle were validatedin this evaluation.None required.None required.3
Valerie Terry-EDTC6321-Spring 2012Follow-through activitiesAfter evaluationitems. She also marked agreementwith all six of the statements on the"after instruction" part of theanticipation guide, indicatingachievement of the performanceobjectives of steps 1.0-6.0 of theunit's instructional goal.Learner looked closely at the colorillustration and post-learningchecklist provided as "take away"materials. She confirmed with methat these were print-outs to keep.Learner commented that the unitwas straightforward and easy tofollow. She liked how theassignments were structured. Shestated that she felt she had learnedthe material covered. This wasverified in her score on theinteractive objective test items andher responses on the anticipationguide after completing theinstruction. Learner noted that sheactually knew more about thecontent than she thought she didbefore beginning the instruction. Asa result, there were portions of thereading referenced in theassignments that she did not needto do before answering theinteractive objective test itemsassociated with those readings.Learner provided suggestedrevisions to the wording she foundconfusing in Assignments 1.0 (I) and3.0 (I).None required.I will note for the unit's learnersthat it might not be necessary toread all of the textbook pagesreferenced during the instruction,especially if read beforehand, asassigned in the course syllabus. I willincorporate the suggested wordingrevisions into Assignments 1.0 (I)and 3.0 (I).C. Synopsis - Short-term revisions: The one-to-one evaluation revealed that a script of the podcastneeds to be provided to the learner, so that he/she can follow along with the audio. Otherwise, thepodcast becomes overwhelming with the amount and type of information it gives which is actuallyintended to facilitate understanding and ease nerves. The wording in two of the unit's assignmentsneeds to be revised for clarity.This evaluation also indicated that some of the textbook reading referenced in severalassignments should be reduced and/or marked as optional for the learner during the instruction,especially if it has been read prior to the instructional unit. This will likely be the case since theinstructional unit is meant to address only one part of a larger course assignment (included in acomprehensive course syllabus). I will make these revisions accordingly and immediately, beforeproceeding with the small group evaluation.4
Valerie Terry-EDTC6321-Spring 2012Long-term implications: The most significant outcome of this evaluation was that it verified inquantifiable terms that the learner met the performance objectives for each of the six steps in the unit'sinstructional goal. This was confirmed by the learner's perfect score on the interactive objective testitems. Plus, the learner marked agreement to each of the six items on the anticipation guide correlatedto the unit's performance objectives after completing the instruction.III. Small Group EvaluationA. Introduction - Eleven students in a course on copyright and ethical issues in the Division of ComputerInformation Systems at Texas State Technical College-Harlingen conducted the small group formativeevaluation. These students fit the profile of the unit's intended traditional learner. These students weregiven the revised version of the unit's materials to evaluate, based on the feedback provided in the oneto-one evaluation. This included a copy of the podcast script.We met on Tuesday, April 24, 2012, in their regular classroom during their regularly scheduledclass time. All of the students are familiar with Moodle so could envision how the unit's instructionalmaterials would be presented as proposed on that learning management system. I was introduced bytheir instructor. I explained the purpose of my visit. The students appeared relaxed and eager to helpevaluate the materials. Their instructor had agreed to drop their lowest assignment grade in theircurrent course in return for their participation in the evaluation.The evaluation began at 10:25 a.m.: The first student finished working through the module at 10:51 a.m. (26 minutes); Six students finished on or before 11:00 a.m. (30-35 minutes); Three students finished by 11:10 a.m. (45 minutes); and, The last student did not complete the module until 11:16 a.m. (51 minutes).B. Summary - The following table provides a summary of the results of the evaluation:INSTRUCTIONAL STRATEGYCOMPONENTSBefore evaluationPreinstructional activitiesLEARNER COMMENTS/MYOBSERVATIONSMY RESPONSEOne learner had been in one of myF2F public speaking classes. Ichatted with her and several others,trying to make everyone feelcomfortable. All the learnersseemed interested in the purpose ofmy visit to their class; they wererewarded for participating my theirinstructor.Learners read the podcast scriptprovided (in response to the SMEand one-to-one learner's feedback)and seemed to move forward withthe materials with greater ease thanthe one-to-one learner had.Learners had no problem with the"before instruction" part of theanticipation guide. Seven learnersdid not mark agreement with anyNone required.I will adopt a written version of theaudio podcast script for the learnersto read from as the audio plays aspart of my final revised instructionalmaterials.5
Valerie Terry-EDTC6321-Spring 2012Content presentation(Correlates to steps 1.0-6.0 ofinstructional goal.)Learner participationAssessmentstatement on the guide, indicatingno prior knowledge among themajority of the learners with thecontent of the instructional unit.Two learners indicated that theassignments correlated toinstructional goal steps 4.0 and 6.0were confusing. Other generalcomments were: "too much readingand understanding the difference ofa lot of defining words"; "kind ofconfusing, but at the same timeinteresting"; ".simple and veryeasy to do.this would be a class Ican see myself doing again ifnecessary"; "Overall I learned thedifferent categories of publicspeaking. It was fairly easy for me."and "I thought this module was easyand understandable."Seven (a majority of the) learnerscompleted the instructional unitwithin the 30-minute prescribedtimeframe. The remaining fourrequired 10-15 more minutes tocomplete the unit. None of thelearners needed any interactionwith me. All of the learnersappeared fully engaged with theinstruction from beginning to end;there were no pauses to seekrefreshments or go to thebathroom. They all appeared to takethe unit seriously and give it theirbest effort.Only one learner did not pass theinstructional unit. Five learnersscored in the 90-100% (A/Excellent)range on the interactive objectivetest items. Three of these scored100%; one was my former F2Fpublic speaking student. Twolearners scored in the 80-89% ("B")range and three scored in the 7079% ("C") range. Those studentsI will re-word the assignmentsmentioned. Also, as a result of rewording the assignments correlatedto instructional goal steps 1.0 (I) and3.0 (I) per the suggestions from theone-to-one learner, the grouplearners did not encounter anyproblems with these revisedassignments.The unit still appears to be a littlelong in time required to complete it.I will make further adjustments tothe reading portion of theassignments, focusing on theinformation absolutely essential forthe learner to reference, ifnecessary, during the instruction.The order in which the materialswere presented to the learners andtheir use as one instructional bundlewere validated in this evaluation, aslong as the issue of timing isaddressed as indicated.I will make the adjustmentsmentioned above to shorten thetime required to complete theinstruction. The rest of the unitseems to flow efficiently whilesustaining learner interest.None required.6
Valerie Terry-EDTC6321-Spring 2012Follow-through activitiesAfter evaluationperforming in the "C" range all hadtrouble with Assignments 4.0 and6.0 already mentioned as needingrevision. All but one of the learnersmarked agreement with all six ofthe statements on the "afterinstruction" part of the anticipationguide, indicating achievement of theperformance objectives of steps 1.06.0 of the unit's instructional goal.The learner who did not marksimilar agreement was the learnerwho also did not pass the unit.All of the learners understood thatthe color illustration and postlearning checklist provided were"take away" materials for futurereference and use. They seemed tolike the color illustration inparticular.The comments from the studentswere generally positive, as alreadynoted. My former student wrote,"The information you teach in classactually sticks with you years later."I found this particularly gratifying.None required.None required.C. Synopsis - Short-term revisions: All but one of the revisions I incorporated into the unit based on theone-to-one learner's formative evaluation were successful. The exception has to do with the length ofthe instructional unit. Even though a majority of the learners completed the unit in the prescribetimeframe, the unit verges on being too lengthy. This still seems to be tied to the amount of readingincluded in the instruction, even though some is indicated as optional. Therefore, further adjustment isneeded so that the reading the learner might need to reference during the instruction is kept to anabsolute minimum.Long-term implications: As with the one-to-one learner, the most significant outcome of the smallgroup learning evaluation was that it verified, again, in quantifiable terms that the learners met the sixperformance objectives of the unit's instructional goal. This was confirmed by ten out of the elevenlearners scoring passing or better on the interactive objective test items. In addition, all but one learnermarked agreement to each of the six items on the post-instruction anticipation guide correlated to theunit's performance objectives. The same learner who did not pass the instruction did not agree with allof the learning statements in the "after" column on the anticipation guide.Summary and ConclusionIn Instructional Unit #1, an overarching instructional goal was identified along with a comprehensivedescription of the learning environment, the performance environment, and the intended learner for apotential thirty-minute stand-alone self-paced online instructional module on a relevant subject to bedesigned following the classic Dick and Carey (Dick, Carey, and Carey, 2009) instructional design model.7
Valerie Terry-EDTC6321-Spring 2012In Instructional Unit #2, a specific instructional goal was adapted from the overarching instructional goaland vetted with two levels of analysis. The specific goal pertains to how an intended learner mightdevelop an intellectual skill relevant to effective persuasive public speaking. Six major steps a learnerwould need to master toward accomplishing the specific instructional goal were identified and mappedin a hierarchical flow chart. Substeps were also included to add supporting instructional detail.Productive feedback from SME's and potential learners of the instruction was received and incorporatedinto the planned unit to validate and improve the instruction.Instructional Unit #3 detailed performance objectives for each of the six major instructional steps of theplanned learning module in terms of conditions, behavior, and criteria. Methods of assessment tomeasure learning progress and outcomes were also crafted. These include interactive object test items,a before- and after-instruction anticipation guide for student self-evaluation as well as instructormeasurement and a post-instruction rubric to assess learning and "real life" application of the learnedskills, respectively.Instructional unit #4 explained five learning components of the instructional strategy for the plannedlearning module, including preinstructional activities, content presentation, learner participation,assessment and follow-through activities. Instructional strategies in this unit also described andprovided a rationale for the planned instructional media and delivery system along with a rough draft ofthe proposed instructional materials.This fifth and final instructional unit reports the results of a formative evaluation of the instructionalmaterials drafted in IU#4. Formativ
formerly the chair of the academic distance education department at Texas State Technical College-Harlingen. She also has chaired the distance education course review committee at TSTC-Harlingen, has designed, developed and taught online
