Transcription
Training CourseEvaluation of Adult Educationand Training ActivitiesCourse Structure and ContentsCo-funded by the Erasmus Programme of the European Union
The European Commission support for the production of this publication does not constitutean endorsement of the contents which reflects the views only of the authors, and theCommission cannot be held responsible for any use which may be made of the informationcontained therein.Published by:Erasmus Strategic Partnership “Designing, monitoring and evaluating adult learning classes –Supporting quality in adult learning” (DEMAL)Agreement No. 2016-1-DE02-KA204-003346Bonn, 2018Author:Lars Alroe Olesen, Progress Consult, BudapestZsolt Vincze, Progress Consult, BudapestProject Partnership:German Institute for Adult Learning –Leibniz Centre for Lifelong learning (DIE)Projectcoordinationwww.die-bonn.deIDEC, Consultants, High TechnologyApplications, Training SAwww.idec.grInstitutul de Stiinte ale Educatiei Romaniawww.ise.roProgress Consult Kft.www.progress.huFondo Formacion Euskadi SLLwww.ffeuskadi.netSwiss Federation for Adult Learningwww.alice.chContact: Susanne Lattke, DIE, Bonn: lattke@die-bonn.deThis work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, .0/
Content1. Training Course Description . 42. Course Structure and Link to Competence Profile . 73. Overall Course Plan . 104. Description of the Thematic Units . 12Unit 1 – Introduction to the Course and Definition of Terminologies . 12Unit 2 – Evaluation in Education and Training . 15Unit 3 – Learner-Centered Evaluation . 17Unit 4 – Evaluation Methodology I. Qualitative Methods . 19Unit 5 – Evaluation methodology II. – Quantitative Methods and Use ofStatistics . 21Unit 6 – Design and Planning of Evaluation . 23Unit 7 – Self-evaluation of Teachers . 25Unit 8 – Final Assessment, Reflections on the Course . 28About DEMAL. 293
1. Training Course DescriptionTraining course descriptionTitleEvaluation of Adult Education and Training activitiesIntroductionEvaluation in adult educationQuality improvement and high impact of training and education foradults have over recent years become more and more importantas more resources are allocated for this field of education. Adulteducation is in many countries a rather new discipline in the fieldof teacher education and performance. Low impact and growingconcerns about the involvement is a second source for findingnew pathways to improve adult education. Recent research fromall over the world has proven that a precise and systematicevaluation of the education and training activities for adults play asignificant role in lifting the quality and in the improvement ofactual learning outcome. This course has been developed to findnew ways to improve the learning outcome for adults.The training course on Evaluation has been developed in theframework of the Erasmus project DEMAL – Designing,monitoring and evaluating adult learning classes – Supportingquality of adult learning.The main outputs of DEMAL are two training courses, associatedwith two specific key competences for the improvement of qualityand efficiency in adult education and training among professionalstaffs:-be able to design the learning process-be able to monitor and evaluate the learning process.The “Key Competences for Adult Learning Professionals”framework, published by Research voor Beleid (RvB) 1 in 2010,mapped the professional key competences that are relevant forthe field of adult learning in Europe. It identified seven generic andtwelve specific key competences which attempt to cover all typesof professional activity that is carried out within an adult learningprovider organisation, including not only teaching but alsomanagement and other types of supporting activities. The DEMALproject focuses on two of those twelve specific competences:Design of Adult Learning, andMonitoring and Evaluation of Adult Learning.Both training courses consist of an e-learning part which will offera theoretical preparation and a practical part with a transnationalworkshop which will consist mainly of case studies, role playgames and project work to help participants learn how to apply theacquired competence.1Research voor Beleid (2010). Key competences for adult learning professionals. Contribution to thedevelopment of a reference framework of key competences for adult learning professionals. Final report.Zoetermer. https://ec.europa.eu/epale/de/node/297064
At the end, the course will lead to a final assessment andcertification according to the “Key Competences for AdultLearning Professionals” framework on the basis of the learners’assessment.RationaleThe rationale of course is to present some of the key elements inwhat could announced as an evaluation-culture in adult education.Evaluation-culture is a prerequisite on how teacher develop“quality behavior”. By this we will understand the adult educatorcommitted to ongoing improvement of own training quality andcapacity. The rationale of course is to present these elements inway where it leads to transformation into own practice of theparticipants.Target groupAdult educators, and operational staff within institutions andorganizations who delivers various types of adult education intheir local settings.ObjectiveThe objective of the training course is toOverall methodology Develop a generic course format and content which canbe transferred towards adult educators in the participatingcountries. Create a course which in content and methodology makesit possible for national organizations to offer this coursefor improved quality development of adult education.The training course uses a blended learning methodology thatconsists of an e-learning part (8 weeks) and a face to faceworkshop (1 week).The e-learning phase has as objectives: to bring together theparticipants, to introduce them to the key themes and theoreticalprinciples related to Evaluation.The workshop will be aimed at providing the opportunity for theparticipants to learn how to put what they have learnt into theirown professional practice with the help of case studies, role playand reflection.Learning outcomes ofthe courseBy the end of this training course, the participants will be able to: Design and organize systematic evaluation of theireducation and training – of own practice and within theinstitution/organization they represent. Strengthen the ability to use evaluation as a measure toongoing quality development of own practice Transform own practice to become learner centeredThematic unitsSee overall plan belowLanguages availableEnglish, German, Greek, Spanish, Romanian, HungarianEvaluation of thecourseFor each part of the training course (e-learning and workshop)there is an evaluation questionnaire that the participants have tofill in upon the completion of the course. The evaluationquestionnaire of the e-learning is available online in the e-learning5
platform. The evaluation questionnaire of the workshop isdistributed to the participants at the last day of the workshop.CertificateCertificate for completion of the training courseEuropass Mobility6
2. Course Structure and Link to Competence ProfileElements from the DEMAL Competence profile „Designof Adult Learning Processes“2Corresponding CourseunitsKnowledge key terms and principles regarding evaluation,assessment, monitoring knowledge about how different types, approaches,principles etc. of evaluation/assessment relate todifferent contexts, and target groups; ability toanalyse and judge the appropriateness of a givenapproach/method etc for a specific context or areaof application different types and functions of evaluation andassessment (summative, formative) different types of criteria and indicators individual vs. group assessment characteristics of different tools for evaluation andassessment (e.g. tests, exercises, project work,reflective diaries etc.) basic principles of designing evaluation /assessment instruments (principles ofquestionnaire design, test design, .)Unit 1: Introduction to theCourse & DefinitionsUnit 2: Evaluation inEducation and Training,Unit 3: Learner-centeredevaluationUnit 3: Learner-centeredevaluation,Unit 4: EvaluationMethodology I. Qualitativemethods,Unit 5: EvaluationMethodology II.Quantitative methodsmethodological knowledge knowledge of various diagnostic methods and tools(test, self-assessment, portfolio, exercises, projectwork etc)observation principles and techniquescommunication in the course basic theoretical knowledge of communicationtheory, group dynamics; roles in the group phases of a course and their characteristics knowledge/awareness of various professional rolesas teacher, facilitator, counsellor feedback principles, methods and tools forcollecting feedbackUnit 4: EvaluationMethodology I. Qualitativemethods,Unit 5: EvaluationMethodology II.Quantitative methods,Unit 6: Planning & Designof Evaluation in Training,Unit 7: Self-evaluation ofthe teacher; assessment forcontinuous process of ownprofessional developmentlearner counselling principles of counselling learnersself-reflection professional development 2principles and techniques of self-reflectionUnit 7: Self-evaluation ofthe teacher; assessment forcontinuous process of ownprofessional l7
principles, methods and tools for professional selfdevelopment (e.g. reflective diaries, portfolios, peerfeedback, supervision, )Skills designing a consistent evaluation/assessmentframework for a given trainingUnit 2: Evaluation inEducation and Training,Unit 6: Planning & Designof Evaluation in Training designing evaluation and assessment tools andinstruments for different purposes practical application of different tools for evaluationand assessmentUnit 3: Learner-centeredevaluation, techniques for analysing evaluation andassessment results practical application of diagnostic tools includinganalysis of test resultsUnit 4: EvaluationMethodology I. Qualitativemethods, practical application of self-reflection exercises andtools techniques for recording and analysing selfreflection outcomes practical application of feedback collectingmethods and tools techniques of analysing and structuring differenttypes of feedback (oral/written; open/closedquestions) application of observation techniques practical communication skills; application of feedback techniques practical use of tools such as portfolios, reflectivediaries; designing a development plan for own professionaldevelopmentUnit 5: EvaluationMethodology II.Quantitative methodsUnit 7: Self-evaluation ofthe teacher; assessment forcontinuous process of ownprofessional developmentTransversal competences analytical and reflection skills intercultural awareness problem-solving and negotiation skills communication skills observation skills, empathy creativity flexibility, open-mindednessUnit 7: Self-evaluation ofthe teacher; assessment forcontinuous process of ownprofessional developmentUnit 3: Learner-centredevaluationCompetence in terms of autonomy and responsibilityProfessionals who are competent performers in the field ofEvaluation and Monitoring have a high degree ofUnit 3: Learner-centredevaluation8
responsibility and self-management skills. They are able toautonomously collect the necessary information on whichto base their work; they are able to autonomously drawappropriate conclusions from this information and take therelevant decisions. They are able to communicate,negotiate, and defend their decisions in direct interactionwith various stakeholders (learners, client); they are able toautonomously reflect on their own professional role andaction and critically review their own action and selfconcepts; they will consciously seek and develop strategiesfor improving their own professional performance on thebasis of the experience they have gathered and reflected.Unit 6: Planning & Designof Evaluation in TrainingUnit 7: Self-evaluation ofthe teacher; assessment forcontinuous process of ownprofessional developmentThe activity of Evaluation can be routinized only to a smallextent. In some cases, especially in the case of formaleducation, certain elements of this activity may havealready been established by external stakeholder, e.g. inthe case of standardized examinations at the end of acourse. In most cases however, adult educators canautonomously plan for the evaluation activities which theyintend to implement during a training. In doing so, adulteducators can draw on existing tools and models, but theywill in any case need to make well-founded decisionregarding the appropriateness of existing tools for a givenpurpose, taking into account a wide range of parameters.Moreover, much of the Evaluation activity cannot beplanned in advance but needs to be put in action during anongoing teaching-learning process. The adult educator hasto take instantaneous decisions, based on a quick analysisof the situation, and has to come up with creative solutionsto address individual challenges which arise during theprocess.The high proportion of action which cannot be planned inadvance makes this task of Evaluation a particularlydemanding one. A high level of competence in this fieldtherefore requires not only the accumulation of relevantknowledge and technical skills in designing and applyingrelevant evaluation and monitoring tools. It also essentiallyrequires the ability to make complex decisions in a very shorttime. This ability is closely linked to the level of professionalexperience which has already been accumulated by the adulteducator.9
3. Overall Course PlanE-learning course structureDuration8 weeksIndicative learning time2-3 hours per e-learning sessionUnit 1Unit 2Unit 3Unit 4Unit 5Unit 6Unit 7Unit 8Introduction tothe Course &DefinitionsEvaluation inEducation andTrainingLearner centeredevaluationEvaluationMethodology I.EvaluationMethodology II.QualitativemethodsQuantitativemethodsPlanning &Design ofEvaluation inTrainingSelf-evaluationof the teacher;assessment forcontinuousprocess of ownprofessionaldevelopmentFinalassessment &reflections on thecourseSee Unit 7 Below1. E-learningSee Unit 1 BelowSee Unit 2 BelowSee Unit 3 BelowSee Unit 4 BelowSee Unit 5 BelowSee Unit 6 BelowEvaluation of thecourse byparticipants2. Workshop3. Interface elearning andworkshop4. Transformation to ownpractice5. ImprovementsIndividualpresentation offuture evaluationin own practice10
Workshop structureDurationFive days / 30 hoursDay 1Day 2Day 3Day 4Day 5Concepts and definitionsEvaluation in education andtraining, learner-centeredevaluationEvaluation methodology, qualitativeand quantitative methodsPlanning and design ofevaluation in trainingSelf- evaluation of theteacher; final assessment,feedback of theparticipant, closureWorkshop assessmentReflection of the dayIndividual AssessmentportfolioIndividual reflection11
4. Description of the Thematic UnitsUnit 1 – Introduction to the Course and Definition of TerminologiesGeneral InformationFull NameUnit 1: Introduction to the Course and Definition ofTerminologiesCourse InformationE-learning course4-6 hours of trainingWork ShopIn the workshop this unit will consist of a 2x3 hour session. Thesession is composed by Introduction Feedback on E-learning sessions 1-7 Case elaboration Exercise and group work FeedbackThere will be one break of 15 minutes.Unit SummaryThe introduction of the course will give the participants a fulloverview about the aims, objectives and expected outcomes ofthe course. The introduction will also briefly introduce theindividual training units and how the training and learningprocess will lead them to expected knowledge, skills andcompetences to be acquired by the end of the course.Participants will also be introduced the actual expectationstowards their active participation in the training program. Thegroup of participants will be introduced shortly to each otherand in order to create an interest and view on how theparticipants internally can use each other’s experience andknowledge.The last part of the introduction will define how the participantscan address the training content into their own educationalpractice. Transformation of knowledge and competences toown practice with aim of giving the participants the opportunityto change and alternate own practice is a key element in thistraining program.The second part of the introduction will contain a sufficientintroduction to overall term of Evaluation. This will be adefinition part which will give the participants a clearunderstanding what is meant by evaluation and how theboundaries are towards similar wordings like assessment,monitoring, measurement – and how the interface is betweenthese different terms.12
Structure of the UnitLearning OutcomesLearning ContentAssessment MethodsKnowledgeE-learning courseE-Learning CourseBy the end of the coursethe participants have aclear overview on theterminology andmethodologies used inevaluation science andpracticeI. Course Introduction Part:Question and Answer session Aim and background ofthe DEMAL Program.Wrap up questionnaire aftereach e-learning session Target group of courseand perspectives of thewider use of the DEMALprogram andexperiences.SkillsThe introduction of thecourse will introduce shortlyhow the participant cananchor and transform thelessons learning into ownpracticeCompetencesThe unit will give theparticipants an overview onthe competences neededto 1) take part ahead in theprogram, and 2) how tobuild experience andcompetences in ownevaluation practice.Justification of Evaluation asan important activity ineducation and training. Latestresearch conclusion on thisissue and how evaluationaffects learning outcome andoptimization of learningresources in adult education.Transformation into ownpractice and how to utilize theknowledge and experience ofthe co-participants.Closing discussion about thedefinition of evaluation byinvolvement of the participantson how they define evaluation– pro and cons on thepresented definitions.WorkshopIntroduction to the workshop,by an assessment of theworking progress of theparticipants through the 7 elearning courses. Specialfocus on assessment oftransformation from course toown practice. This willintroduce the workshop.Practical information aboutthe course ahead. Timelineand expected input from theparticipants.II. Overall EvaluationDefinition and Terminology.What is evaluation?Overview of the mostimportant definitions ofevaluation – e.g. as“systematic assessment withthe aim of proving knowledgeon quantity and quality of acertain action or intervention”.Selection of 3 key definition. Short introduction to themain types of evaluation:Summativeevaluation formatsFormative evaluationformatsProcess evaluationformats13
-Qualitative evaluationformatsQuantitativeevaluation formatsWorkshopIntroduction of theparticipants face to faceReview of experience gainedin the E-learning ProgramCompetence Profiles onevaluation – findings andconclusionsThematic introduction ofEvaluation based on thecontent from the E-learning.In the workshop the focus willbe on clarification and thefocus on evaluation dilemmas(only Workshop)-learning vs. control-time vs outcome-involvement vs.assessment-output vs. outcomeGroup discussionsDidactic Format of theUnitIn the E-learning course the main focus will be to communicatethe knowledge base of the topics presented in the unit. Thepurpose is to provide the participants with a theoretical andmethodological overview, which will enable the participants toknow about the purpose of certain methods, their possiblestrengths and weaknesses in certain learning situation andalso be able to reflect on which type of methodology would fitthe best into own practice. The e-learning will have extendedthe knowledge base of its target group.The Workshop opens the possibility to enhance the dialogueand the critical involvement of the participants. The Didacticformat of the Workshop is to involve the participants indeveloping their skills and use their experience tosystematically assess how the topics and methods canfunction in practical situations, as a part of case-baseddiscussion and assessment on how to transform knowledgeinto a learning practice.References:Evaluation in education and training:-Ketlhoilwe, M. J. (2006). Course Evaluation in Adult Education Programmes. Insupport of environmental and sustainability education processes. SADC Regional14
Environmental Education Programme Course Developers’ Toolkit. Howick, ShareNet. 2006-G McNamara, P Joyce, and J O’Hara (2010). Evaluation of Adult Education andTraining Programs, Dublin City University, Dublin, Republic of Ireland, Elsevier Ltd.Evaluation in general:Web resources:-Program Operations Guidelines for STD Prevention-SHE Schools for Health in Europe: Evaluation types: Process, output & outcomeUnit 2 – Evaluation in Education and TrainingGeneral InformationFull NameUnit 2 – Evaluation in Education and TrainingCourse InformationE-learning course4-6 hours of trainingWork ShopIn the workshop this unit will consist of a 3-hour session. Thesession is composed by Introduction Case elaboration Exercise and group work Feed Back sessionThere will be one break of 15 minutesUnit SummaryThis unit shall elaborate the aim and the rationale of evaluationin teaching and learning activities. Evaluation is a timeconsuming activity, and in this respect, it needs clearjustification for the teacher to organize and use evaluation as adevelopment tool in own practice. Some evaluations arecompulsory, in order to assess the satisfaction of thelearners/participants in taking part in the training, and it willsatisfy the very direct output o of the training. If evaluation willgo deeper into the understanding of the learning process andthereby also the quality of the learning activity, it often needs adetailed preparation. This will require priorities by the teacherherself and very often also the institution or organization wherethe adult education program takes place.Structure of the UnitLearning OutcomesLearning ContentAssessment MethodsKnowledgeE-Learning CourseE-Learning CourseThe participant will know themain theoretical principles ofevaluation as an integratedThe E-learning program willpresent the following topics:Q&A15
part of educational planningand intervention. Justification ofEvaluation in educationand training (aim andobjectives) Rationale of Evaluation:SkillsThe Participant will be able toassess and measure theprecise impact(resources/time) a certainchoice of evaluation will haveon own and institutionalpractice and organization.Workshop-Improve teachercapability andperformance-Measure of learners’satisfaction-Quality improvement-Improved trainingefficiencyCompetencesThe participant will be able toanalyze the choice ofevaluation methods invarious types of educationprograms or initiativesWrap up questionnaire aftereach e-learning session Individual Assessment portfolio- Reflection roundtable(every morning in program)Overview of the mostused evaluation modelsand methodologies usedin the education andtraining sector.-Self-evaluationmethods-Learner centeredmethods-Link to Qualitativemethods-Link to QuantitativemethodsWorkshopIn the workshop this unit willbe used to raise a foundationdiscussion about thedilemmas of evaluation ineducation:Dilemmas in evaluation:Didactic Format in the Unit-learning vs. control-time vs outcome-involvement vs.assessmentIn the E-learning course, the main focus will be tocommunicate the knowledge base of the topics presented inthe unit. The purpose is to provide the participants with atheoretical and methodological overview, which will enable theparticipants to know about the purpose of certain methods,their possible strengths and weaknesses in certain learningsituation and also be able to reflect on which type of16
methodology would fit the best into own practice. The elearning will extend the knowledge base of its target group.The Workshop opens the possibility to enhance the dialogueand the critical involvement of the participants. The Didacticformat of the Workshop is to involve the participants indeveloping their skills and use their experience tosystematically assess how the topics and methods canfunction in practical situations, as a part of case baseddiscussion and assessment on how to transform knowledgeinto a learning practice.References:-Ketlhoilwe, M. J. (2006). Course Evaluation in Adult Education Programmes. In supportof environmental and sustainability education processes. SADC RegionalEnvironmental Education Programme Course Developers’ Toolkit. Howick, Share-Net.2006-G McNamara, P Joyce, and J O’Hara (2010). Evaluation of Adult Education andTraining Programs, Dublin City University, Dublin, Republic of Ireland, Elsevier Ltd.Web Resources-Katrina Chmolova (2016). Qualitative Vs. Quantitative Methods of Verification andEvaluation - Two methods of evaluating your students' knowledge and your courseeffectivenessUnit 3 – Learner-Centered EvaluationGeneral InformationFull NameUnit 3 – Learner-Centered EvaluationCourse InformationE-learning course4-6 hours of trainingWork ShopIn the workshop this unit will consist of a 3-hour session. Thesession is composed by Introduction Case elaboration Exercise and group work FeedbackThere will be one break of 15 minutesUnit SummaryThe aim of most education/training evaluation is focused onthe outcome of the learning activity. This means to what extendis the participating learner able to transform the education intonew skills, competences and knowledge and especially to whatextend is the learner able to utilize lessons learned from thetraining activities. Learner centered evaluation tries to explore17
which types of training intervention is the best for individuallearners or groups of learners, learning styles and ways oforganizing the training sequences. The aim is foremost to useevaluation to identify which teaching and learning interventionscreates the best outcome for the learner. The second part ofthis unit is dedicated to introduce how evaluation becomes apart of learners own activity and reflection on own learningprocess. How can the learner measure progress and how canthis be documented e.g. through portfolio systems or othertypes of written assessment.Structure of the UnitLearning OutcomesLearning ContentAssessment MethodsKnowledgeE-Learning CourseE-Learning CourseThe participant hasknowledge about thelearning styles of adultsand how this affect thechoice of learners’ selfevaluation and how thechoice will affect theirlearning abilitiesThe course will introduce thefollowing topics:Wrap up questionnaire aftereach e-learning session Theory of learnercentered evaluationWorkshop Examples (2-3) of learnercentered evaluationIndividual Assessmentportfolio Methodology ofassessment of the learnerReflection roundtable (everymorning in program) Transformation topracticeCritical friend assessmentSkillsThe participant is able toorganize a concreteevaluation process withand individual and/or agroup of adult learners in away where it positively canaffect learning outcomeWorkshopCompetencesCase: How to build a formatfor learner centeredevaluationThe participant can supportadult learners in varioussettings and conditions onhow to be able to evaluatetheir own learning processThe case will shop howdifferentiated the learnedcentered evaluation has to bein order to providecustomized supportExercise/Group work ; Workout 2 concrete evaluationprocess models for:1. An individual adultlearner2. Group of adultlearners3. Assessment ofresources4. Transformation topractice18
Presentation of conclusionsand recommendationsExercise: Create a model forlearner centered evaluation)Didactic Format in theUnitIn the E-learning course the main focus will be to communicatethe knowledge base of the topics presented in the unit. Thepurpose is to provide the participants with a theoretical andmethodological overview, which will enable the participants toknow about the purpose of certain methods, their possiblestrength and weaknesses in certain learning situation and alsobe able to reflect on which type of methodology would fit thebest into own practice. The e-learning will have extend theknowledge base of its target group.The Workshop opens the possibility to enhance the dialogueand the critical involvement of the participants. The Didacticformat of the Workshop is to involve the participants indeveloping their skills and use their experience tosystematically assess how the topics and methods canfunction in practical situations, as a part of case baseddiscussion and assessment on how to transform knowledgeinto a learning practice.ReferencesLearner-centred assessment/evaluation-James A. Athanasou (2005). Self-evaluations in adult education and training.Australian Journal of Adult Learning, Volume 45, Number 3, p. 290-303.-Aaron Kilner & Ryan Drew (2012). Learner Self-Assessment Toolkit: ELSA LevelsLiteracy to 7. Elsa Net. Canada-John D. Rich, Arabia N. Colon, Dominique Mines, & Kimberly L. Jivers (2014).Creating learner-centered assessment strate
Introduction Evaluation in adult education Quality improvement and high impact of training and education for adults have over recent years become more and more important as more resources are allocated for this field of education. Adult education is in many countries a rather new discipline in the field of teacher education and performance.
