Transcription
Doc 9995AN/497Manual ofEvidence-based TrainingApproved by the Secretary Generaland published under his authorityFirst Edition — 2013International Civil Aviation Organization
Doc 9995AN/497Manual ofEvidence-based TrainingApproved by the Secretary Generaland published under his authorityFirst Edition — 2013International Civil Aviation Organization
Published in separate English, Arabic, Chinese, French, Russianand Spanish editions by theINTERNATIONAL CIVIL AVIATION ORGANIZATION999 University Street, Montréal, Quebec, Canada H3C 5H7For ordering information and for a complete listing of sales agentsand booksellers, please go to the ICAO website at www.icao.intDoc 9995, Manual of Evidence-based TrainingOrder Number: 9995ISBN 978-92-9249-242-7 ICAO 2013All rights reserved. No part of this publication may be reproduced, stored in aretrieval system or transmitted in any form or by any means, without priorpermission in writing from the International Civil Aviation Organization.
AMENDMENTSAmendments are announced in the supplements to the Catalogue of ICAOPublications; the Catalogue and its supplements are available on the ICAOwebsite at www.icao.int. The space below is provided to keep a record ofsuch amendments.RECORD OF AMENDMENTS AND CORRIGENDAAMENDMENTSNo.DateCORRIGENDAEntered byNo.(iii)DateEntered by
FOREWORDThis manual is intended to provide guidance to Civil Aviation Authorities, operators and approved training organizationsin the recurrent assessment and training of pilots referred to in Annex 6 to the Convention on International Civil Aviation,Operation of Aircraft, Part I, International Commercial Air Transport — Aeroplanes, paragraphs 9.3, Flight crew membertraining programmes, and 9.4.4, Pilot proficiency checks. Implementation guidance is contained in the Evidence-basedTraining Implementation Guide (2012, a joint publication by ICAO, the International Air Transport Association (IATA) andthe International Federation of Airline Pilots Associations), available from the IATA website at www.iata.org.First EditionAt the inaugural meeting to develop constituent elements of the International Air Transport Association (IATA) Trainingand Qualification Initiative (ITQI) in 2007, representations were made by stakeholders to include a strategic review ofairline pilot training as part of the sponsored activity.The background to these representations is as follows: progress in the design and reliability of modern aircraft, a rapidlychanging operational environment and the realization that not enough has been done to address the human factorsissue required a strategic review of airline pilot training. In addition to the wealth of accident and incident reports, theprovision of flight data analysis offers the possibility to identify risks encountered in actual operations and to tailortraining programmes to mitigate those risks that flight crew members face in operations.In response, IATA facilitated the creation of an international working group to conduct the strategic review of airline pilottraining, commencing with the recurrent training element. The international working group comprised Civil AviationAuthorities, academic institutions, aircraft original equipment manufacturers, airlines, international organizations, pilotrepresentative bodies and training organizations.The international working group established a new methodology for the development and conduct of a recurrent trainingand assessment programme, titled Evidence-based Training (EBT), which is fully described in this manual.The aim of this programme is to identify, develop and evaluate the competencies required to operate safely, effectivelyand efficiently in a commercial air transport environment whilst addressing the most relevant threats according toevidence collected in accidents, incidents, flight operations and training. This document is intended to enable theimplementation of more effective training to improve operational safety. Additionally, and recognizing the criticality ofcompetent instructors in any training programme, the manual provides specific additional guidance on the requiredqualifications of instructors delivering EBT.Since the underpinning data analysis is what guides the content of Evidence-based Training programmes, theassociated data and related documents will be subject to a systematic review and update process to ensure theircontinued accuracy and the relevance of guidance material.Comments on this manual, particularly with respect to its application, usefulness and scope of coverage, would beappreciated. These will be taken into consideration in the preparation of subsequent editions. Comments concerning themanual should be addressed to:The Secretary GeneralInternational Civil Aviation Organization999 University Street,Montréal, Quebec H3C 5H7Canada(v)
(vi)Manual of Evidence-based TrainingThe following list contains the member organizations of the international working group, and other organizations thatcontributed towards the development of EBT:Air ArabiaFlight Safety InternationalAir FranceGeneral Civil Aviation Authority – United Arab EmiratesAir BerlinGriffith UniversityAirbus S.A.S.Gulf AirATRGulf Aviation AcademyThe Boeing CompanyInternational Air Transport AssociationBombardier Inc.International Civil Aviation OrganizationBritish Airways PlcInternational Federation of Airline Pilots Associations (IFALPA)CAE Inc.LOSA CollaborativeCathay Pacific Airways Ltd.LMQ LtdCivil Aviation Authority – United KingdomMechtronix Systems Inc.Civil Aviation Department – Hong Kong, ChinaNational Aerospace Laboratory (NLR) - NetherlandsCivil Aviation Safety Authority – AustraliaOxford Aviation AcademyDelta Airlines Inc.Qantas Airways LtdDeutsche Lufthansa AGQatar AirwaysDragonairResearch Integrations Inc.Direction générale de l'aviation civile – FranceRoyal Aeronautical Society (RAeS)EasyJetRoyal Holloway, University of LondonEmbraer S.A.Saudi Arabian AirlinesEmirates AirlineThomsonfly LtdETOPS S.A.S.Transport CanadaEtihad AirwaysVirgin AustraliaEuropean Aviation Safety Agency (EASA)Wizz AirEuropean Cockpit Association (ECA)Federal Aviation Administration – United States ofAmerica
TABLE OF CONTENTSPageGlossary .(xi)Abbreviations and acronyms .(xv)Publications .(xvii)PART I.EVIDENCE-BASED TRAINING DEVELOPMENT AND PROGRAMME OUTLINEChapter 1.Background .I-1-1Chapter 2.Applicability and aims .I-2-1Applicability .Aims .Benefits .I-2-1I-2-1I-2-2Principles and Programme Philosophy .I-3-1Background .Competencies .Training criticality survey .Data collection .Data analysis .Recurrent assessment and training — Programme outline .Guidance for regulatory oversight .EBT recurrent assessment and training programme lementation of the baseline EBT programme .I-4-1General principles .Staged implementation .Baseline EBT programme .I-4-1I-4-1I-4-2Implementation of the enhanced EBT programme .I-5-1Enhanced EBT programme .Collection and analysis of operations data .Collection and analysis of training data .Integration of analysis.Programme development .I-5-1I-5-1I-5-3I-5-4I-5-42.12.22.3Chapter 3.3.13.23.33.43.53.63.73.8Chapter 4.4.14.24.3Chapter 5.5.15.25.35.45.5(vii)
(viii)Manual of Evidence-based TrainingPageChapter 6.6.16.26.3Pilots and instructors.I-6-1Pilots.Instructors — General .Instructors — EBT .I-6-1I-6-1I-6-1Conduct of EBT .I-7-1General.Preparation .Briefing .Evaluation phase .Manoeuvres training phase .Scenario-based training phase .Assessment .Debriefing .I-7-1I-7-1I-7-1I-7-2I-7-2I-7-3I-7-3I-7-3Chapter 7.7.17.27.37.47.57.67.77.8PART II.EVIDENCE-BASED TRAINING PROGRAMMEChapter 1.Description of the process for developing an EBT recurrent training programme .II-1-1General.Guidance for using the EBT recurrent assessment and training matrix .Implementation of a baseline EBT programme .EBT modules .Evaluation phase .Manoeuvres training phase .Scenario-based training phase ent to Chapter 1. Summary process for end users wishingto implement the baseline EBT programme .II-1-Att-11.11.21.31.41.51.61.7Chapter 2.2.12.22.3Regulatory approval .II-2-1National Civil Aviation Authorities .Regulatory considerations .Obtaining regulatory approval .II-2-1II-2-1II-2-2Appendix 1.Core competencies and behavioural indicators . II-App 1-1Appendix 2.Training programme development guidance — Generation 4 (jet) . II-App 2-12.12.2General.Assessment and training matrix .Appendix 3.3.13.2II-App 2-1II-App 2-1Training programme development guidance — Generation 3 (jet) . II-App 3-1General.Assessment and training matrix .II-App 3-1II-App 3-1
Table of Contents(ix)PageAppendix 4.4.14.2General.Assessment and training matrix .Appendix 5.5.15.2II-App 5-1II-App 5-1Training programme development guidance — Generation 2 (turboprop) . II-App 6-1General.Assessment and training matrix .Appendix 7.II-App 4-1II-App 4-1Training programme development guidance — Generation 2 (jet) . II-App 5-1General.Assessment and training matrix .Appendix 6.6.16.2Training programme development guidance — Generation 3 (turboprop) . II-App 4-1II-App 6-1II-App 6-1Training programme development guidance — Generation 1 (jet) . II-App 7-1
GLOSSARYNote.— Much of the information in this section comes from Annex 1 — Personnel Licensing, Annex 6 —Operation of Aircraft, Part I — International Commercial Air Transport — Aeroplanes and the Procedures for AirNavigation Services — Training (Doc 9868).Assessment. The determination as to whether a candidate meets the requirements of the competency standard.Behaviour. The way a person responds, either overtly or covertly, to a specific set of conditions, and which is capable ofbeing measured.Behavioural indicator. An overt action performed or statement made by any flight crew member that indicates how thecrew is handling the event.Competency. A combination of knowledge, skills and attitudes required to perform a task to the prescribed standard.Competency-based training. Training and assessment that are characterized by a performance orientation, emphasison standards of performance and their measurement and the development of training to the specifiedperformance standards.Competency element. An action that constitutes a task that has a triggering event and a terminating event that clearlydefines its limits, and an observable outcome.Core competencies. A group of related behaviours, based on job requirements, which describe how to effectivelyperform a job and what proficient performance looks like. They include the name of the competency, adescription, and a list of behavioural indicators.Critical flight manoeuvres. Manoeuvres that place significant demand on a proficient crew.Critical system malfunctions. Aircraft system malfunctions that place significant demand on a proficient crew. Thesemalfunctions should be determined in isolation from any environmental or operational context.Error. An action or inaction by the flight crew that leads to deviations from organizational or flight crew intentions orexpectations.Error management. The process of detecting and responding to errors with countermeasures that reduce or eliminatethe consequences of errors, and mitigate the probability of further errors or undesired aircraft states.Evidence-based training (EBT). Training and assessment based on operational data that is characterized bydeveloping and assessing the overall capability of a trainee across a range of core competencies rather than bymeasuring the performance in individual events or manoeuvres.EBT instructor. A person who has undergone a screening and selection process, successfully completed an approvedcourse in delivering competency-based training, and is subsequently authorized to conduct recurrentassessment and training within an approved EBT programme.(xi)
(xii)Manual of Evidence-based TrainingEBT module. A session or combination of sessions in a qualified FSTD, as part of the three-year cycle of recurrentassessment and training.EBT session. A single defined period of training in a qualified FSTD that normally forms part of an EBT module.EBT scenario. Part of an EBT session encompassing one or more scenario elements, constructed to facilitate real-timeassessment or training.EBT scenario element. Part of an EBT session designed to address a specific training topic.Facilitation technique. An active training method, which uses effective questioning, listening and a non-judgementalapproach and is particularly effective in developing skills and attitudes, assisting trainees to develop insight andtheir own solutions and resulting in better understanding, retention and commitment.Factor. A reported condition affecting an accident or incident.Flight crew member. A licensed crew member charged with duties essential to the operation of an aircraft during aflight duty period.Instructor. A person authorised to provide practical training to a trainee or student for an aviation licence, rating orendorsement.Inter-rater reliability. The consistency or stability of scores between different raters.Line-oriented flight scenario. Training and assessment involving a realistic, “real time”, full mission simulation ofscenarios that are representative of line operations.Note.— Special emphasis should be given to scenarios which involve a broad set of core competencies. Itsimulates the total line operational environment for the purpose of training and assessment of flight crew members.Manoeuvres. A sequence of deliberate actions to achieve a desired flight path. Flight path control may be accomplishedby a variety of means including manual aircraft control and the use of auto flight systems.Performance criteria. Simple, evaluative statements on the required outcome of the competency element and adescription of the criteria used to measure whether the required level of performance has been achieved.Phase of flight. A defined period within a flight.Scenario. Part of a training module plan that consists of predetermined manoeuvres and training events.Threat. Events or errors that occur beyond the influence of the flight crew, increase operational complexity and must bemanaged to maintain the margin of safety.Threat management. The process of detecting and responding to threats with countermeasures that reduce oreliminate the consequences of threats and mitigate the probability of errors or undesired aircraft states.Training event. Part of a training scenario that enables a set of competencies to be exercised.
Glossary(xiii)Training objective. A clear statement that is comprised of three parts, i.e. the desired performance or what the traineeis expected to be able to do at the end of training (or at the end of particular stages of training), theperformance standard that must be attained to confirm the trainee’s level of competence, and the conditionsunder which the trainee will demonstrate competence.Unsafe situation. A situation, which has led to an unacceptable reduction in safety margin.
ABBREVIATIONS AND ACRONYMS(Used in this SOPAircraftAbove aerodrome levelAirborne collision avoidance systemActual navigation performanceApproachAdvanced qualification programmeAir traffic controlApproved training organizationAlternative training and qualification programmeCivil aviation authorityCommercial air safety teamClimbCrew resource managementCruiseDecision altitudeDescentEvidence-based trainingFederal Aviation Administration (of the United States of America)Fly by wireFlight data analysisFlight management systemFlight simulation training deviceGroundGlobal positioning systemInternational Air Transport AssociationInternational Federation of Airline Pilots AssociationsIn-seat instructionKnowledge, skills and attitudesLandingLine operations safety auditMinimum descent altitudeMaximum take-off massNavigationNew engine optionOriginal equipment manufacturer(s)operationsPilot flyingPilot monitoringPrecision runway monitorRisk assessment or ACAS resolution advisory (context dependent)Required navigation performanceRequired navigation performance — Approval requiredStandards and Recommended PracticesSafety management systemStandard operating procedure(xv)
(xvi)TATAWSTOV1V2Manual of Evidence-based TrainingTraffic advisoryTerrain awareness and warning systemTake-offTake-off decision speedTake-off safety speed
PUBLICATIONS(Referred to in this manual)Convention on International Civil Aviation (Doc 7300)Annex 1 — Personnel LicensingAnnex 6 — Operation of Aircraft,Part I — International Commercial Air Transport — Aeroplanes, andPart II — International General Aviation — AeroplanesProcedures for Air Navigation Services — Training (Doc 9868)Manual of Criteria for the Qualification of Flight Simulation Training Devices (Doc 9625), Volume I — AeroplanesManual on the Approval of Training Organizations (Doc 9841)Joint ICAO, IATA and IFALPA publication: Evidence-based Training Implementation GuideIATA publication: Data Report for Evidence-based Training(xvii)
Part IEVIDENCE-BASED TRAINING DEVELOPMENTANDPROGRAMME OUTLINE
Chapter 1BACKGROUND1.1The development of Evidence-based Training (EBT) arose from an industry-wide consensus that in orderto reduce the aircraft hull loss and fatal accident rates, a strategic review of recurrent and type-rating training for airlinepilots was necessary. The existing airline pilot training requirements in national regulations are largely based on theevidence of hull losses from early generation jets, and on a simple view that, in order to mitigate a risk, simply repeatingan event in a training programme was sufficient. Over time, many new events occurred and the subsequent addition ofthese events to the training requirements saturated recurrent training programmes and created an inventory or "tick box"approach to training.1.2At the same time, aircraft design and reliability improved substantially, leading to a situation where manyaccidents occurred in aircraft that were operating without malfunction. Controlled flight into terrain is a good example ofthis principle, resulting in a hull loss where inadequate situation awareness is almost always a contributing factor.1.3It is impossible to foresee all plausible accident scenarios, especially in today’s aviation system where itscomplexity and high reliability mean that the next accident may be something completely unexpected. EBT addressesthis by moving from pure scenario-based training, to prioritizing the development and assessment of key competencies,leading to a better training outcome. The scenarios recommended in EBT are simply a vehicle and a means to assessand develop competence. Mastering a finite number of competencies should allow a pilot to manage situations in flightthat are unforeseen by the aviation industry and for which the pilot has not been specifically trained.1.4The core competencies identified in EBT encompass what was previously known as both technical andnon-technical knowledge, skills and attitudes, aligning the training content with the actual competencies necessary in thecontext of contemporary aviation.1.5The availability of useful data covering both flight operations and training activity has improvedsubstantially over the last 20 years. Data sources such as flight data analysis, flight observation in normal operations(e.g. LOSA) and air safety reports give a detailed insight into the threats, errors and risks encountered in flightoperations and their relation to unwanted consequences. An enhanced analysis of training results demonstratesimportant differences of training needs between different manoeuvres and aircraft generations. Availability of such datahas both established the need for the EBT effort and supported the definition of the resulting training concept andcurriculum.Note 1.— The compendium of the data referred to in paragraph 1.5 is documented in the Data Report forEvidence-based Training, which can be obtained free-of-charge from the International Air Transport Association (IATA)web-site.Note 2.— Paragraph 3.1.2 provides a list of aeroplane types grouped in six generations.1.6The aim of this programme is to develop and evaluate the identified competencies required to operatesafely, effectively and efficiently in a commercial air transport environment whilst addressing the most relevant threatsI-1-1
I-1-2Manual of Evidence-based Trainingaccording to evidence collected in accidents, incidents, flight operations and training. This manual is intended to enablethe implementation of more effective training to improve operational safety, focusing on the recurrent training of airlinepilots. Additionally and recognizing the criticality of competent instructors in any training programme, the manualprovides specific additional guidance on the required qualifications of instructors delivering EBT.
Chapter 2APPLICABILITY AND AIMS2.1APPLICABILITY2.1.1This manual is intended to provide guidance to civil aviation authorities, operators and approved trainingorganizations in the recurrent assessment and training of pilots conducted in flight simulation training devices (FSTDs),referred to in Annex 6 to the Convention on International Civil Aviation, Operation of Aircraft, Part I, Internationalcommercial Air Transport — Aeroplanes, 9.3, Flight crew member training programmes, and 9.4.4, Pilot proficiencychecks in addition to Annex 1 to the Convention on International Civil Aviation, Personnel Licensing, 1.2.5, Validity oflicenses. It may also provide guidance for approved training organizations engaged in the recurrent assessment andtraining of flight crew operating large or turbojet aeroplanes in accordance with Annex 6, Part II — International GeneralAviation — Aeroplanes (Section 3 refers).2.1.2This manual includes guidance for the development of training programmes and the assessment of flightcrew performance, in addition to information for instructors conducting the training. The purpose of training andassessment within this programme is to establish competency and provide for continuous and measurable improvementin flight crew performance during line operations.2.1.3Aeroplanes considered for application of the guidance in this manual are those with a certified seatingcapacity of 50 or more passengers for turbo-jet aeroplanes referred to in this manual as jets, and of 30 or morepassengers for turbo-propeller aeroplanes referred to in this manual as turboprops.2.1.4This manual does not formally consider training media, but wil
training programmes, and 9.4.4, Pilot proficiency checks. Implementation guidance is contained in the Evidence-based Training Implementation Guide (2012, a joint publication by ICAO, the International Air Transport Association (IATA) and the International Federation of Airline Pilots Associations), available from the IATA website at www.iata.org.
