WIXLIB

Versant Aviation English Test:a job-specific test of languagethat complies with ICAOrequirementsleading tests of spoken language

Discussion Topics Overview of Ordinate Drivers for an ICAO-compliant test ofaviation English Versant Aviation English Test (VAET)– Test structure– Commercial options– Demo of test Test Development for VAET Validation and reliability research for VAET 2007 Ordinate CorporationFebruary 2008p2

About Ordinate The leading provider of technology and services fortesting spoken language. Founded in 1996 by two Stanford professors Created the patented Ordinate Testing System Wholly-owned subsidiary of Pearson– Purchased by Harcourt in 2004 (part of Reed Elsevier)– Ordinate and Harcourt Assessment spun out of Harcourtand purchased by Pearson 2007 Ordinate CorporationFebruary 2008p3

Ordinate Experience, ExpertiseCorporateNeeds: Recruitingselection Leadershipprograms TrainingplacementSchoolsSample customers: Dell P&G Accenture IBM BellSouth UK Network Rail Malaysia Airlines UAE civil aviationGovernmentNeeds: Employmentscreening Languagecertification Placement intotrainingNeeds: Teacher/TAcertification ELL placement Oral readingfluencySample customers: Dallas IndependentSchool District Univ. of Connecticut Tsinghua University Kobe University Waseda UniversityLanguage and Test SpecialistsSample customers: US Dept of Education Dutch ImmigrationMinistry US Dept of HomelandSecurity – Border Patrol US DOD – DefenseLanguage InstituteNeeds: Speechrecognition Languageprocessing Speaking tests 2007 Ordinate CorporationSample customers: ACT WorkKeys ALC Japan Pearson Yoons Berlitz ETS CTB/McGraw-Hill Thomson HeinleFebruary 2008p4

Versant spoken language tests A recognized leader fortesting spoken language Taken over the phone oron a computer Automatically scored Proven to be valid andreliable Tests of spoken:– English– Dutch– Spanish– Arabic (in dev) 2007 Ordinate CorporationFebruary 2008p5

Versant AviationEnglish Test (VAET)Drivers for an ICAOcompliant test of aviationEnglish 2007 Ordinate CorporationFebruary 2008p6

What is ICAO?International Civil Aviation Organization A United Nations agency Purpose is to codify international air navigation standards,requirements, and recommended practices Individual country civil aviation authorities (member states)agree to ICAO standards and requirements:– so their licensed professionals, planes, safety routines, etc. will beaccepted by other CAAs when they fly internationally– to better harmonize and smooth international aviation– to support safer international aviation In addition, ICAO defines the protocols for air accidentinvestigation followed by transport safety authorities inmember states 2007 Ordinate CorporationFebruary 2008p7

Why test aviation English? ICAO analyzed accidents in the 1980s and 90s Noted that poor communication due to languagedifferences between a pilot and air traffic controllercontributed to some incidents– i.e. a SAFETY ISSUE Introduced Annex 1 with a new requirement forEnglish proficiency– Defined language levels (ICAO levels 1 to 6)– All pilots and ATCs involved in international travel mustspeak sufficient English (level 4 or above)– Outlined requirement to test, with guidelines foracceptable testing– Defined timetables for initial compliance and subsequentretesting 2007 Ordinate CorporationFebruary 2008p8

ICAO requirements for testing The test must evaluate English speaking and listening skills The test must address aviation work-related topics used inradiotelephonic communications The test must cover ICAO phraseology and plain or generalEnglish The test must be able to assess across all 6 six ICAO-specifiedlanguage skills The test’s development team must be qualified The test development process must be validated – All tasks and items reviewed by independent experts– Test instructions reviewed by stakeholdersThe test must be secureThe test must be reliable, valid and practicalAll sessions must be recorded and stored securelySample and practice tests need to be available 2007 Ordinate CorporationFebruary 2008p9

Versant AviationEnglish Test (VAET)Overview of the Test 2007 Ordinate CorporationFebruary 2008p10

VAET Overview Developed under a Cooperative Researchand Development Program with The FAA Validation Report published documentingValidity and Reliability sufficient to satisfyICAO requirements Can be taken by phone or computer 24x7 Scores available within minutes Designed to support both training andcertification programs 2007 Ordinate CorporationFebruary 2008p11

VAET Meets ICAO RequirementsICAO Checklist for Aviation English Testing9 99 9 99 9 99 9Test evaluates speaking and listeningDevelopment team is qualifiedDevelopment process is validated All tasks and items reviewed by independent expertsTest instructions reviewed by stakeholdersTest addresses aviation work-related topicsTest covers ICAO phraseology and plain or generalEnglishTest assesses all 6 six ICAO-specified language skillsTest is secureTest is reliable, valid and practicalAll sessions are recorded and stored securelySample and practice tests are available 2007 Ordinate CorporationFebruary 2008p12

Test Structure and ContentaviationEnglishSectiongeneralEnglishItem TypePart AReading (Aviation Phraseology)Part BReading (Common English)Part CRepeatPart DShort Answer QuestionsPart EReadbacksPart FCorrections and ConfirmationsPart GStory RetellingsPart HOpen Questions 2007 Ordinate CorporationFebruary 2008p13

Test Security Secure – resistant to cheating and coachingForm 1Form 2Form 3Form 4Form 5Form randomlygeneratedformsITEM POOL 2007 Ordinate CorporationFebruary 2008p14

Anytime, Anywhere Delivery Tests delivered in a secured,proctored location bytelephone or computer Test administration is easy– Instructions and test papers arecreated for administrator whentests are ordered– Administrator running theproctored location providesboth to the test-taker whenready to begin– No complicated scheduling isrequired – the test administratorcan have the test-takercomplete the test at any time,from any authorized location 2007 Ordinate CorporationFebruary 2008p15

Efficient, Simple Test-Taker Process123456781. Test-taker receives test paper fromproctor and reviews before starting2. Candidate accesses the secureOrdinate Testing System (telephoneor computer)3. Candidate enters unique TestIdentification Number (TIN)4. Candidate responds to the testquestions or prompts to completethe test 2007 Ordinate CorporationFebruary 2008p16

Efficient, Automated Scoring Once the test-takersubmits all responses tocomplete the test, it issent to the patentedOrdinate ScoringSystem The test is automaticallyscored and usuallyavailable within minutesafter the completed testis received 2007 Ordinate CorporationFebruary 2008p17

Two Versions of VAET Available NowCertificationPractice PlacerThe high stakes test used to assess the ICAOEnglish language proficiency language level forofficial CERTIFICATION purposes. The test mustbe proctored, with appropriate test-takeridentification and test security.A low stakes version of the test that can be used:1. to assess ICAO level and place individuals intothe right level for training.2. to allow test-takers to practice for theCertification test.3. to use as an instructional activity within training. 25 to 30 minutes78 questions 15-20 minutes54 questionsAviation Reading, Common English Reading,Repeats, Short Answer Questions, Readback,Corrections and Confirmations, Story Retelling,Open QuestionsAviation Reading, Common English Reading,Repeats, Short Answer Questions, Readback,Corrections and Confirmations, Story RetellingScoringMost precise score, with reliability statistics basedon field studies. Will produce the final ICAOlanguage level, with a precise point scoreon a 10-to-70 point scale.Estimate of what final score may be within aconfidence range. Will produce an estimate ofthe current ICAO language level only. No pointscore is provided.SecurityAnonymous test IDs, secure test responsecapture, encrypted storage, secure reporting.,Randomized test question generationAnonymous test IDs, secure test response capture,encrypted storage, secure reporting. Uses a fixedform without random questions.PurposeLengthContent 2007 Ordinate CorporationFebruary 2008p18

VAET – part of Jeppesen Solutions1. Full ICAO Language Certification Solutions––for Airline, Corporate and General Aviationtailored for pilots and air traffic icationand IDverificationcertificationtest deliveryto testtakertest proctoringtest certificateand scoreacceptance bylocal authoritiescertificationrecordkeeping, auditsupportJeppesenOrdinate 2007 Ordinate CorporationFebruary 2008p19

VAET – part of training programs2. Preparation for ICAO Language CertificationOption A:placement test to seewhere people areOption B:dry-run practicetest before certOption C:trainingtraining ifrequiredcertificationRe-testingtrainingif Training provider 2007 Ordinate CorporationOrdinateFebruary 2008p20

Versant AviationEnglish Test (VAET)Demonstration 2007 Ordinate CorporationFebruary 2008p21

VAET DemoPART A: AVIATION READINGYou will be asked to read three aviation specific messages in a randomorder from among those printed in Part A. Read the messages out loudsmoothly and naturally.Sample:1. World Air 891, request descent.2. World Air 891, maintain flight level 280expect descent shortly.3. Maintaining flight level 280, World Air891 2007 Ordinate CorporationLISTENFebruary 2008p22

VAET DemoPART B: COMMON ENGLISH READINGYou will be asked to read three sentences in a random order from amongthose printed in Part B. Read the messages out loud smoothly andnaturally.Sample:1. The Lee sisters are flying to Seoul thissummer for a family reunion.2. Our flight didn’t start very well.3. Then all outgoing flights were delayedbecause of the weather conditions. 2007 Ordinate CorporationLISTENFebruary 2008p23

VAET DemoPART C: REPEATYou will hear 16 sentences in this section, one at a time. Repeat eachsentence you hear exactly as you hear it. If you can’t repeat thewhole sentence, repeat as much of the sentence as you can.For example:When you hear:You say:Sample:“My next flight is on Saturday.”“My next flight is on Saturday.”LISTEN 2007 Ordinate CorporationFebruary 2008p24

VAET DemoPART D: SHORT ANSWER QUESTIONSIn this section, you will hear 24 short questions. Answer each questionwith a single word or a short phrase of two or three words.For example:When you hear:You say:“Where in the airplane do the pilots control the aircraft?”“cockpit” or “in the cockpit”Sample:LISTEN 2007 Ordinate CorporationFebruary 2008p25

VAET DemoPART E: READBACKYou will hear several spoken radiotelephony messages. Say an appropriate readbackafter each one. For each question, a call sign is printed on the test sheet for yourinformation. Use this call sign to say an appropriate readback.For example:The test sheet shows a call sign:When you hear:one possible answer is:or another possible answer is:Sample:Global Air 295Cessna 29"Cessna two niner, exit taxiway Hotel.","Exit taxiway Hotel, Cessna two niner.""Exiting taxiway Hotel, Cessna two niner."LISTEN 2007 Ordinate CorporationFebruary 2008p26