WIXLIB

CETE @ NACTEITesting & Credentials in CTE:Alignment is ImperativeJames T. AustinCETENACTEI, May 21-22 20081CONTENT AND PRESENTERCredentialing within CTE covered two waysBasic concepts: secondary, post-secondaryImplementation: steps, align, evaluateJim Austin (CETE(CETE, 1997-Present)1997 Present)Industrial-Organizational PsychologistVirginia Tech PhD (1987)Testing to evaluate persons, programsCredentialing presentations: 2000, 04, 0522008, Center on Education and Training for Employment, The Ohio State University1

CETE @ NACTEITRENDS: EDUCATIONAL TESTINGTests sample domains, test-taker responds,items scored, scores “used” for purposesTechnical skills definition: top tier Q atData Quality Institutes (DQI; 2005-2007)Classroom AND standardized testingDrivers: teachers or districts, states, nationsAlignment of two testing levels lackingNeed for “assessment literacy “3CTE: NATIONAL INITIATIVESInitiatives: States Career Cluster (NASDCTEc) &College and Career Transitions (CCTI)Perkins IV legislation: accountability ramp-upProgram of study: national/SEA/LEA sourcesp gEnd-of-programOR end-of-course ((EOP-EOC))Industry-based credentials a way to measuretechnical skill in tune with businessTransitions emphasis in CCTI & P-O-S42008, Center on Education and Training for Employment, The Ohio State University2

CETE @ NACTEICTE: STATE OF STATESWhat are other states doing? Other districts?Ohio building seamless adult systemOhio Skills Bank, Stackable Certificates (H.O.)g crucial for testinggBenchmarkingassessment leadership (look around)Need to scrutinize what others are doingWho are your comparable states? DistrictsNeighbors, thought leaders, NACTEI5TESTING PROJECT SAMPLERSampling of CETE projects: experienceStatewide testing systems & Webxamtesting/data collection portalOCTCA hosts WFL tests, also customized systemOccupational analyses with verificationAligning standards or test items (CTE, ABE)Certification testing and systemsWork Readiness: State & national levelsTest Dev Workshop: capacity building62008, Center on Education and Training for Employment, The Ohio State University3

CETE @ NACTEICTE Standards‐based Educational Reform:“What is what”Covered AND PrescribedAcademic, SkillStandardsCoveredANDTested ANDPrescribedTestedANDPrescribedTeaching &InstructionState Testing,CredentialsCovered AND Tested7TECHNICAL SKILL & ATTAINMENTFramework (3WH)WHOWHATWHENHOWWHY test: PerkinsIV, but mostlylearnersMeasurementsLocal (LEA)State (SEA)Third-partyNational/federalCredentials fitwith third-party,national model82008, Center on Education and Training for Employment, The Ohio State University4

CETE @ NACTEIWHO IS TESTED?Student definitionsConcentratorNumerator / DenominatorHow are special needs CTE students testedand reported?Effects of POS on who is tested?9WHAT IS TESTED?Definition: Knowledge / Competency portionsRelationship of technical skill attainment toother indicators: academic, completionDistinguishingfoundational/genericDi tii hifd til/i anddoccupational skillsClarifying clusters, pathways, P-O-STesting aligned content that is instructed!!102008, Center on Education and Training for Employment, The Ohio State University5

CETE @ NACTEIWHEN TO TEST?Temporal: Most testing is EOP, but EOCprovides better diagnostic capability (drilldown)Importance of showing gain (baseline – final)or even maintenance (multiple post-tests)Example: Career-BasedCareer Based Intervention in OHFunding from Fed-Ed to develop noncognitive measures usable by practitioners atlevels: local (LEA) and state (SEA)11HOW TO TEST?Development or selection: LEA vs SEA locusEvaluating suitability & value to studentDefining & establishing validity / reliabilityAligned to skill standards (academics?)Development / administrative costsPooling / sharing resourcesNational Item Bank or clearinghouse122008, Center on Education and Training for Employment, The Ohio State University6

CETE @ NACTEIHOW TO TEST: Alignment onGlobal Skills Xchange; www.gskillsxchange.com/13CREDENTIALS: BASIC CONCEPTSWhat is a credential?Who earns a credential? How do theyearn it?What career outcomes are associatedwith credentials (tipping points)?Are there standards for certifications?142008, Center on Education and Training for Employment, The Ohio State University7

CETE @ NACTEICREDENTIALS: BASICSWhat is a credential?License or certification granted by a body[government, association, corporation, etc]Government does licenses, much wider for certsNational Organization for Competency Assuranceis professional group for certifications (voluntary)Council on Licensure, Enforcement, andRegulation is similar group for licensingProbably 10,000 certifications15CREDENTIALS: BASICSType of BodyExample BodyExample CredentialState regulatoryboardState Board: Welldriller, CosmetologyLicensed ational Board forRespiratory CareCPFT, CRT, NPS, RPFT,RRTTrade assocNational AutomotiveTechnicalTh i l EducationEdtiFoundationAutomotive ServiceEExcellence;llE-O-P TestsBusiness firmMicrosoft, CiscoMCSE / MCP; CCNAFederationECDL FoundationECDL, ICDL basicdigital ITAC162008, Center on Education and Training for Employment, The Ohio State University8

CETE @ NACTEICREDENTIALS: BASICSWho earns credentials? How are they earned?Credentials awarded by decision rules (3-E)Education (HS, AA, BS, etc.)Experience (time, quality)Examination score (knowledge, performance)Usage implies validation for all “E”, butexams get most scrutinyGranting body stipulates attempts,recertification, etc.17CREDENTIALS: BASICSWhat labor market outcomes are associatedwith earning credentials (tipping point)?Do credentials increase compensation,occupational level, & job security?How to know? Marketing versus evidence!!Perceived Value of Certification Tool (PVCT) is an18-item measure with intrinsic & extrinsic partsWashington state research indicates “oneyear credential” is positive tip point182008, Center on Education and Training for Employment, The Ohio State University9

CETE @ NACTEICREDENTIALS: BASICSAre there quality standards for credentials?You bet!! Accredited credentials offer bestvalue, but still just one piece for CTEgyStandards for certifications ppromulgatedbyorganizations for audit purposesNOCA (2005) has 21 standards [10-18 on test]ISO/ANSI (2003) has 6 standardsAsk potential vendors: members? accredited?19CREDENTIALS: CTE PROCESSHow could CTE stakeholders usecredentials?Hypothetical example of credential useHow should CTE stakeholders usecredentials?A process for using credentials202008, Center on Education and Training for Employment, The Ohio State University10

CETE @ NACTEICREDENTIALS: CTE PROCESSHow could CTE stakeholders use credentials?What are models-in-use?Exploratory in which SEA allocates to earlyadopters, who report back to SEA on lessonslearned(Ohio allocated , no datald (Ohillt d d t backb k yet)t)Implemented, for example Virginia, with processoutlined & applied (web-based look-ups)Aspirational/future to synthesize what is knownand make improvements21CREDENTIALS: CTE PROCESSHypothetical example of use of credentialsCross-cutting example: CTE TEACHERSState licensing involves 3E but mostly exams(Praxis II-III; NES) at basic levelNational Board for Professional TeachingStandards (NBPTS) certifies advancedknowledge using portfolio & assessment centerCarry logic backwards to CTE teacher prep(CPTP pathway in Education-Training Career22Field is great for transitions!!)2008, Center on Education and Training for Employment, The Ohio State University11

CETE @ NACTEICREDENTIALS: CTE PROCESSHow should CTE stakeholders use credentials?Aspirational/future model meets Perkins &moves beyondSynthesize what is known into defensible stepsPrescriptive approachDescribe, use, then improveSampling states: CETE doing telephone interviewsto ID themes & needed services, addressing needsin quarterly Centergram & other formats23CREDENTIALS: CTE PROCESSHow should CTE stakeholders use credentials?Overall, an evolved process from MahlmanAustin (2003) nine-step model, linkable top[[CCTI]]P-S transitions emphasisStandards for certifications [NOCA, ANSI]Levels of career clusters initiative [SCCI]242008, Center on Education and Training for Employment, The Ohio State University12

CETE @ NACTEICREDENTIALS: CTE PROCESSProcess for using credentials1. Database state / local CTE programs (levels)2. Identify / import licenses & certifications3.3 Conduct preliminary screen for usabilityA. Virginia process one modelB. Generic model of usability (need rubric)4. Survivors receive enhanced screeningA. Options for rigor25CREDENTIALS: CTE PROCESSProcess for using credentials5. Investigate perceived / actual valueA. Stakeholders include business, students6.6 Disseminate project results in multiple waysA. Searchable web interface through portalB. Database download for savvy tech userC. PDF to print remotely for late adoptersD. Word/WP/HTML for manipulation262008, Center on Education and Training for Employment, The Ohio State University13

CETE @ NACTEICREDENTIALS: CTE PROCESSProcess for using credentials?What is enhanced screening? Varies, butcould include .Alignment of practice analysis to skill standards infull GSX model, and statewide CT2 articulationStudy perceived/actual value (surveys, focusgroups, salary data)Field test w/ CTE providers (districts, CTPD)27SUMMARY: CREDENTIALSCredentials complement standardized tests, tooffer access alternatives as “parallel universe”Basic & use/process features addressedApplication experience awaits a project withcareer pathways in OhioNot with CTE, but through WIA!!Have to start somewhere! Why not now?282008, Center on Education and Training for Employment, The Ohio State University14

CETE @ NACTEILEVEL SLIDERRCLUSTERDigital (ICDL, IC3),first aid/CPR, foodsafety, OSHA 10hr tools,hr,tools plantsPATHWAY?OCCUPATION /SPECIALIZATIONASE, licenses,NCCER, STNA,A , CDA29DEPARTING THOUGHTS .Hope that your goals were metFour themes for credentials – definition,technical quality, CTE usability, & fairness/qEmail with additional feedback/questionstoaustin.38@osu.edu302008, Center on Education and Training for Employment, The Ohio State University15

CETE @ NACTEIDEVELOPING TESTS: STEPS1 D1.Determinet i TestT tPPurpose2. Define Content Domain (Practice / Job Analysis, Skill Standards)3. Create/Evaluate Test Blueprint4. Write, Manage, & Evaluate Items (and item writers)5. Set Cutoff Scores6 D6.Develop,l Evaluate,E l t RefineR fi TestT t FormsF7. Use Reliability, Validity, Fairness as Quality Yardsticks8. Interpret & Report Scores to Stakeholders9. Maintain Testing System10. Evaluate Tests: Own & OthersPost-SecondaryEducation: Academic31Post-SecondaryEducation: TechnicalAcademic ContentStandards (NCLB)Career-Technical Industry-RecognizedSecondarySkill Standards (Basic PreparationAcademic ContentStandardsCareer AwarenessN/AMiddle School322008, Center on Education and Training for Employment, The Ohio State University16

CETE @ NACTEITable Four: Selected Usability-Technical Quality Indicators with CriteriaSample Evaluative StandardExample Levels (finalized by panel)Alignment with Ohio Career FieldTechnical Content Standards (CFTCS)and/or Program of Study (Usability)Cost to Student, to LEA or to SEA(Usability)Data Availability for Student / CTESystem (Usability)Credential Time-span (Usability)Reliability Evidence (Technical Quality)At least 60% alignment (at pathwaylevel)Validityl d Evidenced(TechnicalT hl Quality)lCutoff Evidence (Technical Quality)Price points with comfort levels(tradeoffs)Maximum for maximum availabilityAt least 3 to 5 yearsTechnical report; level of reliabilityestimateT hTechnicall report; qualityl off evidencedTechnical report section; quality ofprocedures & controls for bias332008, Center on Education and Training for Employment, The Ohio State University17

2008, Center on Education and Training for Employment, The Ohio State University 2 TRENDS: EDUCATIONAL TESTING Tests sample domains, test-taker responds, items scored, scores "used" for purposes Technical skills definition: top tier Q at Data Quality Institutes (DQI; 2005-2007) Classroom AND standardized testing