Transcription
Crow ds ourcin g Vis ua l In s p e ct ion13th International Symposiumon NDT in Aerospace 2021Adoption Challenges andOpportunities of NDE 4.0J u d i E . S e e , M . A . , M E N G , P h D, C P ESandia National LaboratoriesP.O. Box 5800, MS 0151Albuquerque, NM 87185-0151jesee@sandia.gov; 505-844-4567NDE Nondestructive Evaluation; NDT Nondestructive TestingOctober 5, 2021Sandia National Laboratories is amultimission laboratory managedand operated by National Technology& Engineering Solutions of Sandia,LLC, a wholly owned subsidiary ofHoneywell International Inc., for theU.S. Department of Energy’s NationalNuclear Security Administration undercontract DE-NA0003525.SAND2021-11446 C
2Pa n e lis t : Dr . Ju d i Se eEmployment Current: Systems analyst and human factors engineer at Sandia NationalLaboratories, Albuquerque, New Mexico, since 2010 Previous: Department of Defense contractor for 16 yearsEducation M.A. (1992), Human Factors, MENG (2021), Systems Engineering Ph.D. (1994), Experimental Psychology (Human Factors) BCPE professional certification in human factors and ergonomicsPublications Vigilance, target detection, signal detection theory, visual inspection, mental workload, andhuman readiness levels
3Multiple challenges must be managedto realize the benefits of NDE 4.0 forvisual inspection in aerospace
4Tr a d it io n a l Vis u a l In s p e ct io n Ap p r o a ch1. InitiateOne Inspector5. RespondOne Component2. AccessVisualInspectionSteps4. Decide3. Search
5Tr a d it io n a l Vis u a l In s p e ct io n Ch a lle n g e s Human inspectors are imperfect Missed defects typically range from 20% to 30% (Drury & Fox, 1975; See, 2015) Minimum error rate of 1 in 1000 limited to very simple inspection tasks (Swain & Guttmann, 1983) Inconsistencies occur within and between inspectors Nearly one-fourth of inspectors reverse decisions during reinspection (McCornack, 1961) Multiple inspections can minimize differences between inspectors but only if implemented correctly
6Tr a d it io n a l Vis u a l In s p e ct io n Ch a lle n g e s Two inspectors are typically better than one (Drury, Karwan, & Vanderwarker, 1986) Five different ways of combining two inspectors were investigated Optimal detectability occurs if “two rejects” approach is used Exceptions do occur Two inspectors are not better than one if inspection occurs in parallel (Drury, Karwan, & Vanderwarker, 1986) Two inspectors are not better than one if reinspection is based on first inspector’s confidenceratings (See, 2015)
7Tr a d it io n a l Vis u a l In s p e ct io n Ch a lle n g e s Different inspectors do not necessarily find the same defects Inspection accuracy can be increased with up to six independent inspections (Harris, 1969)
8NDE 4 .0 Op p o r t u n it ie s fo r Vis u a l In s p e ct io n NDE 4.0 presents opportunities to leverage multiple inspections for even greaterperformance improvements Issues stemming from a scarcity of inspection experts can be managed more effectively Collect second opinions from multiple inspection experts in real time (Drury, 2021) Engage “worldwide NDE network” of experts (Meyendorf, Schallert, Pal, & Bond, 2017) Enhance decision makingRejectRejectAcceptRejectRejectRejectReject
NDE 4 .0 Op p o r t u n it ie s fo r Vis u a l In s p e ct io n Inputs from multiple dispersed experts can help aid decision making The good news is that crowdsourcing worksAsk the Audience80%Percentage of Audience970%60%50%40%30%20%10%0%ABCDResponse OptionWho Wants to Be a MillionaireCrowdsourcing consists of outsourcing tasks to a crowd of individuals (Howe, 2006).
NDE 4 .0 Ch a lle n g e s fo r Vis u a l In s p e ct io n The bad news is that crowdsourcing does not always work Ask the Audience80%Percentage of Audience1070%60%50%40%30%20%10%0%ABCDResponse OptionWho Wants to Be a MillionaireWhat can be done to realize the immense benefits of NDE 4.0 for visual inspection in aerospace?
11NDE 4 .0 Ch a lle n g e s fo r Vis u a l In s p e ct io n Managing logistics challenges1Accessing Expertise2Coping with Time Differences3Factoring in Other Work Demands4Meeting DeadlinesHow do you identify and access the right experts?Is expertise in aerospace inspection required?How do you access people in different time zones?How do you involve experts when they have theirown inspection tasks to complete?How do you ensure the time-critical nature ofvisual inspection in aerospace is met?
12NDE 4 .0 Ch a lle n g e s fo r Vis u a l In s p e ct io n Handling input from human inspectors and from artificial intelligence (AI) Should the source be identified or remain anonymous? Biases may be introduced if it is known which human inspector provides input (See, 2012) Trust issues may deter decision making if it is known which inputs are provided by AIInspection Should confidence ratings be provided? AI confidence ratings can facilitate human decision making (See, Davis, & Kuperman, 1997) Human inspectors are not inclined to use low confidence ratings (Ainsworth, 1982; See, 2015)
13NDE 4 .0 Ch a lle n g e s fo r Vis u a l In s p e ct io n Combining inputs frommultiple sources Should a simple tally of inputsbe used? Should inputs be weightedbased on factors known toimpact visual inspection (Drury,2021)Both approaches have been used tocombine crowdsourcing inputs inmedical image analysis (Ørting, et al., 2020) Majority voting: 23% of applications Weighted voting: 16% of applicationsTOMES Task, Operator, Machine, Environment, and Social [factors]
14NDE 4 .0 Ch a lle n g e s fo r Vis u a l In s p e ct io n Involving multiple dispersed experts may introduce new problems not seen before intraditional “one inspector, one component” visual inspectionDiffusion of Responsibility Sense of responsibility decreases as thenumber of people present increases Obligation to help is divided among thepeople presentSocial Loafing People tend to contribute less when part ofa group Group may be less effective than any oneindividual alone
15Pa t h Fo r w a r d Leverage a vast body of visual inspection research dating back to the 1950s (See, 2012) Leverage existing crowdsourcing applications in other fields such as medical image analysis(Ørting, et al., 2020) Review of 57 papers demonstrated validity of crowdsourcing for medicalimage analysis Crowdsourcing can reduce both costs and time Guidelines should be tailored to the specific crowdsourcing task Conduct targeted research for unique challenges presented by NDE 4.0 Support evidence-based decision making regarding novel aspects of visual inspection afforded byNDE 4.0 Identify how to tailor NDE 4.0 methods for aerospace visual inspection tasksAdoption of NDE 4.0 is a journey.Benefits can be realized only through openness and transparency in sharing crowdsourcingexperiences, both successes and failures.
16Re fe r e n ce sAinsworth, L. (1982). An RSM investigation of defect rate and other variables which influence inspection. In Proceedings of the HumanFactors Society 26th Annual Meeting (pp. 868-872). Human Factors and Ergonomics Drury, C. G. (2021, October 5-7). Human-systems integration in aerospace NDT [Keynote address]. 13th International Symposium on NDTin Aerospace 2021, Williamsburg, VA, United States.Drury, C. G., & Fox, J. G. (1975). The imperfect inspector. In C. G. Drury & J. G. Fox (Eds.), Human reliability in quality control (pp. 11-16).Taylor & Francis.Drury, C. G., Karwan, M. H., & Vanderwarker, D. R. (1986). The two-inspector problem. IIE Transactions, 18 (2), arris, D. H. (1969). The nature of industrial inspection. Human Factors, 11(2), 139-148. https://doi.org/10.1177/001872086901100207Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1-4. https://www.wired.com/2006/06/crowds/McCornack, R. L. (1961). Inspector accuracy: A study of the literature. SCTM 53-61 (14).Meyendorf, N. G., Schallert, R., Pal, S., and Bond, L. J. (2017, September 4-8). Using remote NDE, including external experts in the inspectionprocess, to enhance reliability and address today’s NDE challenges [Conference Paper]. Readability 2017, Potsdam, �rting, S. N., Doyle, A., Van Hilten, A., Hirth, M., Inel, O. Madan, C. R., Mavridis, P., Spiers, H., & Cheplygina, V. (2020). A survey ofcrowdsourcing in medical image analysis. Human Computation, 7(1), 1-26. https://doi.org/10.15346/hc.v7i1.1See, J. E. (2012). Visual inspection: A review of the literature (Report SAND2012-8590). Sandia National Laboratories.See, J. E. (2015). Visual inspection reliability for precision manufactured parts. Human Factors, 57(8), See, J. E., Davis, I., & Kuperman, G. G. (1997). Automatic target cueing and operator performance with enhanced APG-70 syntheticaperture radar imagery (Report AL/CF-TR-1997-0171). Air Force Materiel Command. https://apps.dtic.mil/sti/pdfs/ADA347534.pdfSwain, A. D., & Guttmann, H. E. (1983). Handbook of human reliability analysis with emphasis on nuclear power plant application (TechnicalReport NUREG/CR-1278-F SAND80-0200). Sandia Corporation. https://www.nrc.gov/docs/ML0712/ML071210299.pdf
17Ab s t r a ctVisual inspection is a technique commonly used in aerospace NDE to detect defects and mitigate theirconsequences. The traditional “one inspector, one component” approach to visual inspection has facedmany challenges, with missed defects typically ranging from 20% to 30%. NDE 4.0 methods offer thepossibility of overcoming these challenges by leveraging inputs from a worldwide network of inspectionexperts. However, the potential benefits of NDE 4.0 can be realized only if multiple practicalimplementation challenges are effectively managed. Toward that end, a vast body of visual inspectionresearch dating back to the 1950s can be supplemented with additional targeted research to address theunique challenges presented by NDE 4.0 for visual inspection.
Leverage a vast body of visual inspection research dating back to the 1950s (See, 2012) Leverage existing crowdsourcing applications in other fields such as medical image analysis (Ørting, et al., 2020) Review of 57 papers demonstrated validity of crowdsourcing for medical image analysis Crowdsourcing can reduce both costs and time
