Transcription
What regulators expect frommedical device manufacturersof software with artificialintelligence (AI) & machinelearning (ML)Q&A taken fromour live webinar1. What is the difference between software standardizationbetween CE marking and FDA?CE marking and conformity assessment in the EU isaccording to the requirements of Medical DevicesDirective(s) /and EU MDR 2017/745.The European Commission provides a range of guidancedocuments (MDCG endorsed documents) to assist withimplementation of the regulations.The US conformity assessment is based on 510(k)premarket submission or PMA/PMA supplementsubmissions made to FDA to demonstrate that the deviceto be marketed is safe and effective. The 510(k) routerequires proving substantial equivalence (SE) to a legallymarketed device (predicate device) that is not subject toPremarket Approval (PMA). The FDA provides publicationson guidance for industry and FDA staff with respect toexpectations for software submissions.The International Medical Device Regulators Forum(IMDRF) is a voluntary group of medical device regulatorsfrom around the world which develops internationallyagreed upon documents for topics affecting medicaldevices and has a working group to develop guidancerelevant for software.2. What is the regulatory strategy for global Software as aMedical Device (SaMD) prospective at development level?The manufacturer should determine their regulatorystrategy according to the countries that the device isintended to be marketed and the regulations and guidancerequired to be complied with (See above for 1). A mappingor gap assessment may help to assist determination ofthe most appropriate strategy or course to follow.3. What are the regulatory requirements for SoftwareRequirement Specification (SRS) for Mobile MedicalApplication (MMA)?Regulatory requirements are present in EU MDR2017/745; including Annex I, and General Safety andPerformance Requirement 17.3 which refers to mobilecomputing platforms.
Applicable standards such as EN/IEC 62304 and EN/IEC 82304-1 also include requirements for softwaredevelopment including software requirements. Furtherguidance for example MDCG 2019-16 guidance oncybersecurity of medical devices may also provide inputto a software requirements specification4. Why do you identify EN/IEC 62304 as relevant, Ithought for SaMD EN/IEC 82304-1 is relevant?The scope of the EN/IEC 62304 is for the lifecyclerequirements for Medical Device Software, processesand activities and tasks. The scope of EN/IEC 82304-1 isfor application of safety and security of health softwareproducts and does not apply to health software intendedto become part of specific hardware, medical electricalsystems covered by IEC 60601/80601, In vitro diagnosticequipment covered by IEC 61010 series or implantabledevices covered by ISO 14708 series.SaMD is not defined in relation to 2017/745 and isreplaced by software driving or influencing the use of adevice and Medical Device Software (MDSW) which alsodefined in EN/IEC 62304 Amendment 1. Note that EN/IEC82304-1 requires the use of software lifecycle processesdefined in EN/IEC 62304 clauses 4.2, 4.3, 5, 6, 7, 8 and 9.5. You mentioned IEC 14764. Is compliance with EN/IEC62304 and ISO 14971 for the EU not sufficient?The state-of-the-art changes over time and standards arenot always sufficient. EN/IEC 62304, EN/IEC 82304-1 andEN ISO 14971 are applicable to medical devices togetherwith other standards, guidance etc. which would representstate of the art. ISO/IEC 14764:2006 is applicable forsoftware engineering and provides a framework formaintenance of software which may support or lead toimproved development process.IEC 14764 can support and lead to a deeper understandingand improved process for SaMD development.6. Shall we follow the edition 2019 of ISO 14971 for CEmark of SaMD?There are no harmonised standards currently to theregulation 2017/745 MDR. The EN ISO 14971:2019 isthe latest version of the standard and can be appliedfor medical device software conformity assessment asrelevant for state of the art.7. Which document should we refer to for Softwarechanges in Europe?The presentation references IMDRF/SaMD WG/N10FINAL:2013 5.3 SaMD Changes which includesinformation on software changes and may be found atimdrf.org/documents/documents.aspRelevant standards for software lifecycle processes, forexample EN/IEC 62304 and EN/IEC 82304-1 also coversoftware changes and maintenance process.Change management is also relevant in relation tomedical devices quality management system processand ISO 13485:2016 (e.g. Clauses 7.3.9, 7.3.10, 7.4.2, 7.4.3,7.5.6, 7.6, 4.1.6), and for conformity to Medical DevicesDirective(s) /Regulation (EU) 2017/745 on medical devicesand conformity assessment to Annex IX, X, XI.MDCG 2020-3 includes guidance on significant changesregarding transitional provision under article 120 ofthe MDR with regard to devices covered by certificatesaccording to MDD or AIMDD.NBOG BPG 2014-3 contains guidance regarding softwarechanges that would be considered ‘substantial’ for devicescovered by MDD or AIMDD certificates.8. Should we refer to the US guidance for changes insoftware for CE Marking?No, the US guidance is directed for the US althoughthe artificial intelligence/ machine learning (AI/ML)discussion paper is the proposal of the US FDA and theEU could incorporate in future guidance documents oradjustments of the legislations.9. Is IMDRF guidance applicable to US legislation only oralso applicable to EU legislation?The IMDRF guidance is not legally binding for the USor EU. Only the US or EU legislation is legally binding.However, the IMDRF guidance is a global agreement fromthe regulators, and the regional or national legislationsreflect the IMDRF guidance.10. Is there a difference in the approach for machinelearning (ML) software that is regulated under MDR vsunder IVDR?No, the requirements of the regulations apply to both atthe moment, although it is under review.
11. When can we expect the AI/ML-based approach of FDAto be implemented? Any companies do you know of whichgot FDA clearance on this new approach?There is no timeline given by the US FDA for the AI/MLbased approach. The status is a proposal and in discussionwith other stakeholders, e.g. industry.12. Should we refer to the US guidance for changes inartificial intelligence for CE Marking?No, the AI/ML discussion paper from the US FDA is aproposal. For CE marking refer to the EU 2017/745(MDR) and the MDCG and IMDRF guidance documentsapplicable.There may be further regulations and guidance in thefuture.13. How detailed should an AI/ML software be documented?Should the network architecture be documented, andthe effect of different layers? Or only the purpose of thealgorithm?An AI/ML-software is a medical device itself as medicaldevice software (MDSW) or as software that drives orinfluences a medical device. State of the art standards(EN/IEC 62304, EN/IEC 82304-1) provide a frameworkfor software lifecycle development including devicearchitecture and detailed design. Note that EN/IEC62304 has increasing requirements around documentingarchitecture and detailed design based on software safetyclassification (i.e. Class ‘A’ has less stringent requirementswhile Class ‘C’ has more stringent requirements). Theregulation 2017/745 also has requirements relatingto general safety and performance requirements, ITenvironment including disclosure in instructions for use.MDCG 2019-16 Guidance on cybersecurity for medicaldevices was made available in December 2019; https://ec.europa.eu/health/sites/health/files/md sector/docs/md cybersecurity en.pdf14. Is there a guideline for the verification strategy ofSaMD containing artificial intelligence/ machine learning(AI/ML)?There is no guidance finalised at this time. The designprocedures as required by EN ISO 13485 determinethe design and development process and requirementsfor verification and validation including plans anddocumented output. The current regulations including(EU) 2017/745 and other guidance documents (MDCG,IMDRF, FDA) are developed relating state of the art.Further guidance may become available at a future date.15. Is this new regulatory pathway (with pre-specification/Algorithm Change Protocol) already in place, or what is itsstatus?The new regulatory approach is a proposed regulatoryframework to enable the FDA and manufacturersto evaluate and monitor software products withAI technology. The regulators are in the process ofdetermination of guidance and regulatory approach withsuch devices.16. If I understand correctly, there is no way under MDRthat a real-time updating/ changing algorithm can have aCE-mark at all. Is that correct?The requirements of the regulation 2017/745 (MDR)applies in the EU and conformity assessment apply toall devices applied for where notified body assessmentis needed according to general performance andsafety requirements to the state of the art with riskmanagement framework. An application for such a devicecould be applied for, however the outcome of conformityassessment would determine whether such a devicecould achieve CE mark. The topic of artificial intelligence(AI) technology is under discussion amongst regulatorsand notified bodies and may result in further guidance tosuch a device being able to achieve a CE mark.In the USA the 510K premarket submission applies tosubstantially equivalent devices to a legally marketeddevice; the proposed discussion paper from the FDA is aframework relating to continuously updating devices andwould allow manufacturers to plan modifications duringpre-market review and an approach for modificationsand does include scenarios which may not be appropriate(Proposed Regulatory Framework for Modifications toArtificial Intelligence/Machine Learning (AI/ML) – basedsoftware as a Medical device)
17. Under the current MDR rules, can a software collectdata, e.g. vital parameter, to be used offline to developthe algorithm and then get clearance from a notified body(NB) before implementing changes?The design and development process may include use ofand/or collection of data to aid in the device developmentand relevant regulations would apply, for example theData Protection Act GDPR and Clinical Trial Regulation(EU) 536/2014. These data may be used for the conformityassessment and evidence of performance and safety witha notified body.18. Let‘s say a parameter is introduced in the SaMD,which does not alter the intended use but if that featureis clinically relevant, is new clearance or submissionrequired? Example: radiological image post-processingsoftware including a new feature for lesion measurement.Yes, a new parameter, which is outside the intended useneeds an additional conformity assessment accordingto EU MDR 2017/745 Annex IX, X, XI. Each change ofintended use is a significant change.19. How do you concretely define the borders what in thealgorithm is allowed to change? How do you justify theborders chosen for your algorithm?The manufacturer defines the architecture of thedevice design including software units and algorithmfunctionality within each part of the device. Sucharchitecture and design may assist with definition ofborders/boundaries in conjunction with requirements ofapplicable standards and regulations, for example EN/IEC62304, EN/IEC 82304-1 and EN ISO 14971 (application ofrisk management to medical devices), 2017/745 etc. forsafety and performance taking into account the state ofthe art and clinical data. For software changes refer toquestions 1 and 2 above.The design and development process may include use ofcollection of data to aid in the device development andrelevant regulations would apply, for example the DataProtection Act GDPR and Clinical Trial Regulation (EU)536/2014. These data may be used for the conformityassessment and evidence of performance and safety witha notified body.An example is endoscopy software, which supports thesurgeon to identify cancer cells in an early stage. Thissoftware could have live data collection and conform withrelevant regulations with anonymous data process andcode improvement in the software lab of the endoscopymanufacturer.21. The software UDI guidance talks about the significantchanges (which needs to change the DI) a little bit, couldwe use that as the definition of the significant changes ofthe software?The manufacturer quality management processes inrelation to EN ISO 13485 should have procedures relevantfor determining significant changes. MDCG 2020-3guidance may be used as an input to determination ofa significant change together with any other relevantfactors for example change to intended use or technologychange.22. Would you consider a change in Software of UnknownProvenance (SOUP) a possible ‚significant change‘ in theEU or the US? How about a change in the validationprocess of SOUPs?Yes, a change in a SOUP or validation process of SOUP,may be a significant change. There are requirements instandards such as EN ISO 13485, EN/IEC 62304, EN/IEC82304-1 and legislation of the MDR relating to designand development, software maintenance, changes andrisk management.20. Can you show an example for the algorithm changeprotocol? (ACP)?23. Which points are important to consider validating asoftware based on AI or ML clinically?For now, no SaMD product is in the market relating toACP.The design procedures as required by EN ISO 13485determine the design and development processand requirements for validation including plans anddocumented output.The MDCG guidance document MDCG 2020-1details guidance related to clinical evaluation (MDR)/ performance evaluation (IVDR) of medical devicesoftware. Conformity assessment to the requirementsof the regulation 2017/745 (MDR) is also required for CEmarking.
ector/docs/md mdcg 2020 1 guidance clinic evamd software en.pdf24. Do you have any information on clinical evidence datato be provided on SaMD with AI/ML class I or III?The design procedures as required by EN ISO 13485determine the design and development processand requirements for validation including plans anddocumented output.The MDCG guidance document MDCG 2020-1details guidance related to clinical evaluation (MDR)/ performance evaluation (IVDR) of medical devicesoftware. Conformity assessment to the requirementsof the regulation 2017/745 (MDR) is also required for CEmarking.Classification to the requirements of the regulation2017/745 Annex VIII including Rule 11 apply for softwarefor all device risk classes. State of the art standards, forexample EN/IEC 62366 may also be used.25. How to deal with the software used in proof ofconcept stage in a medical device development? Shouldwe validate all the software used in this stage when wetest different aspects of the product?The design procedures as required by EN ISO 13485determine the design and development processand requirements for validation including plans anddocumented output.The plan may include validation of software at proofof concept stage but should also be appropriate to thecompleted device with changes revalidated. At minimum,manufacturers should ensure that any software code thatis retained from the proof of concept stage and makesit into the final product software should be subject tosoftware development and testing requirements per EN/IEC 62304 and any other applicable standards.26. What is the difference between computer systemvalidation and software validation? How do you determineif you need validation or verification? Can you pleaseelaborate?A computer system validation is a validation testing(establishment of objective evidence) of a complete system(software installed on hardware in intended environmentwith an operating system and any recommendedsoftware patches that the device specifications conformto the user requirements or needs and that the devicesystem works as intended. Computer system validationis often applied to non-product software that can affectfinal medical device quality. Examples include softwareused in the manufacturing process (see ISO13485 clause7.5.6, 7.6) and software used in the quality managementsystem (see ISO13485 clause 4.1.6), such as documentcontrol system software or requirements managementsoftware.The design procedures as required by EN ISO 134857.3 determine the design and development process andrequirements for verification and validation includingplans and documented output. The plan for each devicemakes a determination of verification and validationapplicable to that device to provide objective evidencethat specifications and applicable requirements or userneeds are met and that the device works as intended withappropriate performance and safety.27. Can you please give an example of software validationand verification? E.g. what kind of testing is done invalidation and verification? Is black box testing and whitebox testing part of validation and verification? Is thesoftware developer responsible for any kind of testingrelated to software and to document them?The design procedures as required by EN ISO 13485determine the design and development process andrequirements for verification and validation includingplans and documented output. Medical device softwarevalidation is the validation testing (establishment ofobjective evidence) of the medical device software todemonstrate that the software (either alone or whenused as part of a system) is capable of meeting userrequirements or needs and works as intended. Note thatEN/IEC 62304 does not cover medical device softwarevalidation and covers only up to SOFTWARE SYSTEMVERIFICATION.Medical device software validation is the validationtesting (establishment of objective evidence) of themedical device software (either alone or when used aspart of a system) is capable of meeting user requirementsor needs and works as intended. Note that EN/IEC 62304does not cover medical device software validation andcovers only up to SOFTWARE SYSTEM VERIFICATION.
Medical device software verification is the verificationtesting (establishment of objective evidence) thatthe product requirements (software requirementsspecification) are met confirming that the device outputmeets design input. White box and black box testing maybe used within the plans for the device. White box testingusually requires intimate knowledge of the code and maybest be conducted by a software developer. White boxtesting would typically be considered part of softwareverification (e.g. software unit testing and softwareintegration testing per EN/IEC 62304). Black box testingcould represent either verification or validation testing.Software developers may be responsible for someaspects of the verification testing; however, verificationand validation testing should be independent of design.28. How do you validate an off-the-shelf (OTS) softwarelike Amazon Web Services (AWS) cloud services if it is anintegral part of SaMD Product?A computer system validation is a validation testing(establishment of objective evidence) of a completesystem in intended environment with an operatingsystem and any recommended software patches or offthe shelf software (OTS) that the device specificationsconform to the user requirements or needs and that thedevice system works as intended. The design proceduresas required by EN ISO 13485 determine the design anddevelopment process and requirements for verificationand validation including plans and documented output.The plan for the device should include any requirementsfor verification and validation of software or services.There is MDCG 2019-16 guidance also available relatingto cybersecurity of medical devices.29. Where can I get sample size requirements for softwaretest cases?A performance test model or clinical study needs to besufficiently statistically powered to provide performance/ clinical significance, and statistical output to the stateof the art and requirements of appropriate regulations.MDCG 2020-1 Guidance on clinical evaluation (MDR)/Performance evaluation (IVDR) of medical device softwarewas made available in March 2020, and the FDA also haspublished guidance relating to clinical trials, for exampleE9 Statistical Principles for Clinical Trials, Statisticalconsiderations for clinical trials during the COVID-19public Health Emergency Guidance for Industry. Note thatat the level of software verification, a sample size of oneexecution may be considered acceptable if the softwarebehaviour under test can be considered deterministic innature. Where deterministic software/system behaviourcannot be assured, a statistically based sample sizeselection approach should be used. The manufacturerwill need to provide sample size justifications basedon risk and on whether the testing can be considereddeterministic in les/mdsector/docs/md cybersecurity en.pdfNote that OTS software incorporated into a medicaldevice would be considered as SOUP per the EN/IEC62304 definition and would be subject to EN/IEC 62304clauses required for SOUP.Do you have any questions?Please contact us.Tel.: 1800 862 4977E-Mail: us.medicaldevices@bsigroup.comWebsite: www.bsigroup.de/medical-devices
An AI/ML-software is a medical device itself as medical device software (MDSW) or as software that drives or influences a medical device. State of the art standards (EN/IEC 62304, EN/IEC 82304-1) provide a framework for software lifecycle development including device architecture and detailed design. Note that EN/IEC
