Transcription
Design Verification – The Case for Verification, Not ValidationOverview:The FDA requires medical device companies to verify that all the design outputs meet the design inputs.The FDA also requires that the final medical device must be validated to the user needs. When there areso many more design inputs and outputs than specific user needs, why do companies spend so littletime verifying the device and so much time and money on validation? And what is the role of riskmanagement in determining the amount of testing required? This presentation will demonstrate that ifdevelopers conduct more complete verifications of design outputs and risk mitigations, validations canbe completed in a shorter time, more reliably, and more successfully.Introduction:In our experience working with medical device manufacturers to improve their Quality ManagementSystems and to gain regulatory clearance of new devices, we have found that Design Verification is anoften underutilized tool for ensuring success during the latter stages of device development efforts. Inmany cases, limited verification efforts represent a lost opportunity to significantly reduce the scope ofvalidation efforts – and thereby reduce time to market and development costs.This paper explores the use and misuse of Design Verification and how device manufactures can get themost out of their verification efforts. But first, some background . . .Background:Medical device manufacturers have been working to align their design and development systems withthe FDA’s design control regulations (21 CFR 820.30) since their release in 1996. The regulations arestructured to ensure that device manufacturers maintain control over their designs throughout thedevelopment process and that the marketed device is safe and effective for its intended use. At theirmost basic level, the regulations require that manufacturers: Clearly state what they intend to produce (Planning and Design Inputs); Develop a design that meets those needs (Design Outputs and Design Review); Confirm that the design meets the original intent (Verification); Confirm that manufactured product can be produced reliably and achieve the desired result(Transfer and Validation); andPage 1 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not Validation Maintain records of key development activities and decisions (Design Changes and DesignHistory File)It would be hard to argue that any of these steps should be excluded from any development effort.Whether you are building a bridge, a cell phone, or a surgical tool, this combination of steps has a logicalflow that most engineers would simply consider to be “good practice”.The challenge arises when you move from the conceptual world, where product development can beseen as single stream of water flowing down a hill: steadily moving from one point to the next withoutinteruption; to the real world, full of rapids, eddies, branching streams, and (occasionally) deep pools ofwater that don’t seem to be moving at all. In this turbulent environment, it’s can be difficult to maintaina disciplined design approach – particularly when business schedules demand rapid progress and anydelays in moving on to the next development phase risk affecting project milestones and developmentstaff deployment plans. These challenges are multiplied when the device has multiple components andintegration points that must be managed throughout the development process.Much has been written about the importance of early stage planning and problem solving to speedtime‐to‐market and reduce development costs – and the value of clear Design and Development Plansand well‐defined Design Inputs cannot be overstated. An equal amount of attention has been focusedon Design Transfer and Validation activities. At this late stage of a development effort, the project scopewill have expanded to include more active involvement of Clinical, Operations and Marketing staff. Inaddition, the cost of validation studies and the “make or break” aspect of these studies require a greatdeal of attention and company resources. Problems at the validation stage will have serious implicationsfor the success of the project and ‐ especially for small companies ‐ could determine the fate of thewhole company.Compared to these early‐ and late‐stage activities, the importance of Design Verification tends to beoverlooked. There are several reasons why: Definitions: Many practitioners (and established Quality Systems) continue to have troubledefining exactly what “verification” is, and how it fits into the design control process. The term isoften considered to be synonymous with “validation” ‐ and the development of combined“V&V” plans that do not establish a clear distinction between the objectives of the two effortsdon’t help. [Note: when discussing verification with the FDA, it’s best not to talk about your“V&V plan” – increasingly, inspectors’ preferences are to address the two activities separately.] Timeframe: Verification is often an ongoing effort conducted throughout the development ofoutputs. So, a great deal of time may pass between when the first and last output is verified.Page 2 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationWith this extended timeframe, verification activities can get lost within all of the “churning”associated with developing final outputs. Approaches: There are a variety of ways to complete the verification for an output, so it can bedifficult to communicate that all of these activities, as a group, represent the design verification.The problem with this lack of attention is that it weakens a critical link in the design control “chain”,affecting the strength of the overall design control effort. Poor design verification can lead to problemsduring design transfer and validation, and can reduce your ability to track down and correct problems if(and when) they occur. Just as importantly, it may represent a lost opportunity to optimize validationactivities, reduce time to market, and increase your overall confidence in the safety and efficacy of yourproducts.Key Concepts:To make sure that we’re aligned on the proper use of the term “verification”, here are a few key pointsfrom the FDA’s Design Control Guidance For Medical Device Manufacturers: Definition: Verification means confirmation by examination and provision of objective evidencethat specified requirements have been fulfilled [§820.3(aa)]. Types of Verification Activities: Verification activities are conducted at all stages and levels ofdevice design. The basis of verification is a three‐pronged approach involving: tests, inspections,and analyses.Any approach which establishes conformance with a design input requirement is an acceptablemeans of verifying the design with respect to that requirement. In many cases, a variety ofapproaches are possible . . . the manufacturer should select and apply appropriate verificationtechniques based on the generally accepted practices for the technologies employed in theirproducts.Table 1 provides some examples of the types of verification activities that device manufacturers oftenemploy.Page 3 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationTable 1: Types of Verification ActivitiesooooTestsMaterial performance /fatigue testsPackage integrity tests*Biocompatibility testing ofmaterials*Bioburden testing of productsto be sterilized*ooInspectionsFirst article inspectionComparison of a design to aprevious product having anestablished history ofsuccessful use*oooooAnalysesWorst case analysis of anassembly to verify thatcomponents are deratedproperly and not subject tooverstress during handlingand use*Thermal analysis of anassembly to assure thatinternal or surfacetemperatures do not exceedspecified limits*Fault tree analysis of aprocess or design*Failure modes and effectsanalysis*Engineering analyses:o Finite Element Analysis(FEA)o Computational FluidDynamics (CFD)o Tolerance stack-up* Source: Design Control Guidance For Medical Device Manufacturers (1997)A key distinction between design verification and design validation activities is that verification onlyrequires that a single unit be assessed. What constitutes that single unit will vary depending on theintent of the verification. It might be one batch of raw material (for material performance tests), onemachined part (for first article inspection), on one package sample (for integrity tests). The intent ofverification is to confirm that the design outputs (i.e., the materials or components specified in designdocuments) meet the design input requirements.Validations, on the other hand, require that studies be conducted to ensure that the device can bemanufactured to meet design specifications on a consistent basis (i.e., process validation), and to ensurethat the finished device is safe and effective for its intended purpose (i.e., design validation). For theprocess validation, multiple devices must be manufactured and evaluated to confirm that theproduction process is capable of producing devices within specifications on a consistent basis. For thedesign validation, multiple devices must be used to treat multiple patients to confirm that the treatmentis safe and effective.Page 4 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationThe number of devices that need to be produced and the number of patients that need to be treated isa function of the risk associated with the particular aspect of the device and the need to establishconfidence in the study results. As we will see, the effective use of risk analysis tools and solidverification results can help to reduce the size of the validation studies without negatively affecting theconfidence in the study results.What Goes Wrong?As described above, too often design verification does not receive the amount of attention needed toensure success in the latter stages of a device development program. While not a perfect representationof what goes on in all device companies, it is interesting to look at the FDA’s design control auditfindings to see what problems they have found with companies’ design verification programs. Figure 1,shows the number of warning letters in the FDA’s database that include findings related to specificsections of the Design Control regulation (21 CFR 820.30).Figure 1: Design Control Warning LettersPage 5 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationInterestingly, the most observations were found with regard to the Design Change, Design Validation,and the General design control sub‐headings (the “General” findings suggest an overall failure of thecompany’s design control process). We argue that failures in these three areas are largely the result ofpoor performance in the earlier stages of the program (e.g., validation failures due to poorly structuredinput requirements, outputs and incomplete verification; and design change failures due to anineffective system to update verifications and validations when changes are made).Of the remaining categories, most warning letters cite 21CFR 820.30(f) – Design Verification. In these 64warning letters, the FDA identified that the manufacturer did not complete all required verifications innearly 45% of the cases. A lack of (or significant gaps in) verification procedures were identified in about25% of the warning letters. In addition, out‐of‐specification results, a lack of records in the DHF, and alack of acceptance criteria were found in 16%, 14%, and 14% of these companies, respectively. Figure 2provides a breakdown of all of the types of violations identified in these warning letters.Figure 2: Types of Design Verification ViolationsPage 6 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationNote: 92 separate violations were identified in the 64 warning letters.Why are these failures occurring? One possibility is that by viewing verification as a “regulation”, devicedevelopers are losing sight of the fact that conducting verifications is simply “good engineering”. Forexample, it makes no sense to move forward with the development of a component before you are surethat the material properties meet the performance and safety requirements established in the designinputs.This “meeting the regulations” mindset can lead engineers to view design verification simply as a taskthat must be checked off by QA instead of valuable tool for building knowledge about their device.Failures occur because sometimes the “checkoffs” get missed.Even when all the boxes are checked, the “regulation” mindset can drive engineers to focus only theminimum number of verification activities needed to satisfy the design control requirements. While thisapproach may help satisfy short‐term budget or schedule constraints, the development tea m will losethe opportunity to learn potentially important information about the performance and behavior of itsdevice and its components. If that learning is put off to the validation stage, the cost of failure can growdramatically.What To Do?There are a variety of tools that developers can use to improve the effectiveness of their designverification process. In this paper we discuss three tools that can help ensure that verification activitiesare appropriate and complete and, if well documented, can provide regulators with sufficientconfidence in the design that excessive validation can be avoided. The three tools are Traceabilty Matrix Failure Modes and Effects Analysis (FMEA) “Fault Tree” of Design InputsTraceability Matrix: The traceability (trace) matrix is a basic design control tool that all devicedevelopers should use throughout the design control process to establish clear linkages between DesignInputs, Outputs, Verifications, Validations and Risk Analyses. Too often the trace matrix is left for QA tocomplete just in time for the final design review. However, if it is begun as soon as Design Inputs areapproved and managed throughout the development effort, the trace matrix provided an excellentroadmap – guiding developers through key steps of the design control process and ensuring thatrequired documentation is created and controlled.Page 7 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationTo illustrate how the trace matrix can be used, Table 2 shows what one row in a trace matrix might looklike when the matrix is first developed and approved. At this stage of the development effort, the matrixis solely a list of the design inputs (the reference number from the Product Requirements Document,and a description). At this stage there is no information about how that design input will be satisfied, butit provides a roadmap for what questions need to be asked and what documents need to be developed.As the project proceeds through the develop stages, subsequent cells will be filled in, identifying howthe design input is being met.Table 2: Traceability Matrix at Design Inputs StageTraceability MatrixDesign Design InputDesignRiskVerification ProcessDesignReq.(Requirement) OutputAnalysesValidation ValidationNo.(Specification)1.1.1Can be ETO‐‐‐‐‐sterilizedComments‐Table 3 illustrates how a row of a trace matrix might look when being reviewed at the Final DesignReview. At this point the design outputs have been approved and the requirement that the material inquestion is appropriate for ETO sterilization is established in Drawing No. 1.2. The matrix also identifiesthat identified risks associated with the device have been mitigated (in part) through the use of thismaterial. The references to design FMEA 1.3 and process FMEA 5.4 identify two places where thismaterial is addressed. Verification was achieved by confirming that the specified material is included inthe purchasing bill‐of‐material (BOM 2.3). The Process Validation column identifies that production ofthe material was assessed in an operation qualification (OQ Study 3.5) and a performance qualification(PQ Study 4.3). Finally, it shows that a design validation (Sterilization Study 2.1) was conducted toconfirm that the final manufactured device could in fact be effectively sterilized using ETO.Table 3: Traceability Matrix at Final Design ReviewTraceability MatrixDesign Design Input(Requirement)Req.No.1.1.1Can be ETOsterilizedDesignOutput(Specification)Material A(Dwg 1.2)RiskVerification ProcessDesignAnalysesValidation ValidationdFMEA1.3pFMEA5.4BOM 2.3OQ Study3.5PQ Study4.3CommentsSterilization NoneStudy 2.1Page 8 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationThe trace matrix now serves as a clear record of the chain of analyses and studies used to ensure thatthe specific design input is met. All of the documents referenced in the trace matrix are included in yourdesign history file (DHF), so if a question arises regarding any stage in the process, the trace matrixpoints to the key documentation developed at that stage.While the trace matrix is a necessary and valuable tool, it largely serves just a recordkeeping role. It doesnothing to inform the team about how to prioritize verification and validation efforts. For that input, riskanalyses are needed.FMEA: Like the trace matrix, risk analyses need to be started early in the design and developmentprocess and filled‐in and enhanced as project proceeds. While this paper will not discuss the details ofthe application of risk analyses, and FMEAs in particular, we will address the link between FMEAs andverification.Too often, risk analyses are conducted in isolation from the other design control activities. As discussedearlier, the “regulation” mindset can lead to the perspective that risk analyses are just a required task tobe completed and not as tools to support decision making – which they are.FMEAs are conducted to provide developers with a shared understanding of the risks associated withtheir device – usually from three perspectives: design, use/application, and processing. The result of thistool, as illustrated in Figure 1, is the classification of risks to people, property, and the environment intothree categories: Broadly Acceptable, As Low as Reasonably Practicle (ALARP), and Intolerable. Typically,application of this tool focuses on the Intolerable risks and the development of mitigation strategies toreduce the probability of occurrence for those risks, bringing them into the ALARP region.Page 9 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not ValidationFigure 2: Risk ClassificationWhile risk identification and mitigation is clearly the primary objective of the risk management activity,these analyses can also be used to focus verification and validation efforts. For example, consider adevice that includes two components, one affects how the device is held by the user and the othercomes into contact with the patient. Both components have a similar number of physical dimensions toverify, but risks associated with the user‐facing component are “Broadly Acceptable”, while the patient‐facing component risks are “ALARP”. While a First Article inspection of the user‐facing component maybe appropriate, a more thorough analysis of the patient‐facing component may be warranted. Thisanalysis could include a tolerance analysis or the inspection of multiple pieces to provide a betterindication of the potential capability to produce the component within specification.While the additional efforts described above may not be required at the verification stage (conducting aFirst Article would be sufficient to claim that the verification is complete), the ALARP risk classification isan indication that validation of this component may be challenging. The more you learn about thiscomponent during verification, the better prepared you will be once you get to the validation. If fact, ifthe verification is thorough, it may eliminate the need for some aspects of validation – saving time andmoney later in the development effort.The key point is that incomplete or minimal verifications may not provide a sufficient understanding ofthe likely performance of a device during validation, and excessive verification can be a waste ofresources. To effective focus your verification efforts, risk analyses, and FMEAs in particular, can providePage 10 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not Validationa clear rationale for how to focus your time and resources – potentially reducing validationrequirements.Fault Tree Analysis: While FMEAs are a well accepted risk analysis tool, one weakness of the approach isthat each risk is considered in isolation. There is no ability to assess the risk of two independent failures(i.e., dimension A is too short and the user applies too much force). As illustrated in Figure 3, Fault treeanalysis (FTA) allows you to consider the risk of two independent failures occurring in parallel (the“AND”), and the risk of any one of a series of failures (the “OR”). For example, in this “Bad Coffee”example, adding “Old Cream” requires the cream to be past its expiration date AND for the coffeedrinker not to read the date. Unsatisfactory serving temperature could be due to the coffee being eithertoo hot OR too cold.Figure 3: Fault Tree AnalysisWhile more challenging to structure and assess than an FMEA, the FTA provides a better understandingof the risks associated with the “system”. Since the performance of the system is the objective ofvalidation activities, FTA is an ideal tool to prepare for validations. With the top of the fault treerepresenting the intended use of the device, this tool allows device developers to fill out the tree –Page 11 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
Design Verification – The Case for Verification, Not Validationidentifying that range of design or usage faults that could lead to a problem. Just as the FMEA helped tofocus attention on independent risks, the FTA helps to focus on system risks.Once the branches of a tree associated with a significant risk are identified, additional verificationresources can be focused on verifying the design of components linked to that risk. Again, the objectiveof verification and validation activities is to build confidence within your organization and withregulators that your device is going to be safe and effective. A well‐structure FTA combined withthorough verifications of key components can be instrumental in the identification and resolution ofproblems early in the development process, allowing your team to enter into the validation phase of theprocess with confidence in the performance of the device and to focus its validation on those systemelements that most directly affect user needs.What are the Benefits?While it can sometimes get lost in the churning of engineering process, verification is a critical elementof the design control process. While meeting the threshold requirement of documenting a verificationfor each design input may help to move the design through the development stages, such an approachdoes not provide the value that a strong verification effort could provide. By developing a trace matrixto ensure that all design inputs are properly addressed and leveraging risk analysis tools, medical devicedevelopers would be better able to focus scarce resources on those verification activities that willprovide the greatest benefit. Well‐structured verification activities provide the foundation forvalidations. If the verifications are sound, validations can be better focused, helping to reduce the scopeof these activities – reducing validation costs and accelerating time to market.Page 12 of 12MEDIcept, Inc.200 Homer AvenueMay not be reprinted or copied without expressed permission from MEDIceptAshland, MA 0172111/2010
The FDA requires medical device companies to verify that all the design outputs meet the design inputs. The FDA also requires that the final medical device must be validated to the user needs. . A key distinction between design verification and design validation activities is that verification only .
