Transcription
The Verification CompanySoftware Development and Verificationcompliance to DO-178C/ED-12C
DO-178C/ED-12C in Context Verocel 2017
Airworthiness Requirements Federal Aviation Regulation (FAR) 25 —Airworthiness Standards: Transport CategoryAirplanes Certification Specification CS-25 is theEuropean equivalent Others exist for gliders (CS-22), light aircraft(FAR 23/CS-23), helicopters (FAR 27/CS-27 &FAR 29/CS-29) and hot air balloons (FAR31/CS-31HB) Verocel 2017
CAST Certification Authorities Software Team International group of certification authorityrepresentatives Harmonization of certification positions onsoftware & electronic hardware CAST position papers http://www.faa.gov/aircraft/air cert/design approvals/air software/cast/cast papers/ Verocel 2017
Document Overview Verocel 2017
Software Level Software levels determined by system safetyassessment process (usually done inaccordance with SAE ARP4754) Based on potential failure conditions 5 levels from Level A (the most rigorous) toLevel E (the least rigorous) Objectives & independence varied by softwarelevel We’ll outline these objectives in thispresentation Verocel 2017
Failure Condition Software criticality levelsFailure ConditionSoftware LevelCatastrophicLevel AHazardous/Sever - MajorLevel BMajorLevel CMinorLevel DNo EffectLevel E Verocel 2017
SOFTWARE LIFE-CYCLE Verocel 2017
Software Life Cycle Processes Software planning process (DO-178C/ED-12C §4) Software development processes (DO-178C/ED-12C §5) Integral processes– Software verification process (DO-178C/ED-12C §6)– Software configuration management process (DO178C/ED-12C §7)– Software quality assurance process (DO-178C/ED-12C §8)– Certification liaison process (DO-178C/ED-12C §9) Verocel 2017
Conventional Waterfall Model Verocel 2017
Example From DO-178C/ED-12C Verocel 2017
DO-178C PROCESSES ANDACTIVITIES Verocel 2017
Planning process Purpose– Defines the means of producing software which satisfy thesystem requirements and provide the level of confidencewhich is consistent with the airworthiness requirements Output:– Plan for Software Aspect of Certification (PSAC)– Software Development Plan (SDP)– Software Verification plan (SVP)– Software Quality Assurance Plan (SQPP and SQAP)– Software Configuration Management Plan (SCMP)– Design standards (SDS) Verocel 2017
Planning process – Table A-1 Verocel 2017
Development process Purpose:– Develop the system requirements in one or more level ofsoftware requirements– Develop the software architecture– Produce the source code– Integrate the software components to produce executable Outputs– Software Requirement Specification (SRS)– Software Design Description (SDD)– Source Code– Executable object code Verocel 2017
Development process – Table A-2 Verocel 2017
High-Level Requirements Compliance with system requirementsAccuracy and consistencyCompatibility with the target computerVerifiabilityConformance to standardsTraceabilityAlgorithm aspects Verocel 2017
Verification of S/W requirements – Table A-3 Verocel 2017
Verification of S/W Design Verocel 2017
Low-Level Requirements Compliance with high-level requirementsAccuracy and consistencyCompatibility with the target computerVerifiabilityConformance to standardsTraceabilityAlgorithm aspects Verocel 2017
Software Architecture Compatibility with the high-level requirementsConsistency, esp. data flow and control flowCompatibility with the target computerVerifiabilityConformance to standardsPartitioning integrity Verocel 2017
Software Coding Process Compliance with LL requirements andarchitecture Accuracy and consistency Verifiability Conformance to standards Traceability Parameter Data Items Integration Process is correct Verocel 2017
Parameter Data Items Parameter Data Items can be developed andverified separately if certain conditions are met– Can be used to configure run-time environment The high-level requirements describe how thesoftware uses the parameter data items The low-level requirements define the structure,attributes and allowable values of the parameterdata items Verification should show that every data elementhas the correct value Verocel 2017
Coding and Integration Process – Table A-5 Verocel 2017
Verification processes Purpose:– Verification of the software requirement process– Verification of software design process– Verification of the SW coding and integration Challenges:– The cost may represent up to 50% of the totaleffort. Verocel 2017
Reviews and Analyses Reviews provide a qualitative assessment ofcorrectness, e.g. an inspection of an output ofa process guided by a checklist or similar aid(DO-178C/ED-12C §6.3) Analyses provide repeatable evidence ofcorrectness (DO-178C/ED-12C §6.3) Verocel 2017
Reviews and Analyses High-Level Requirements (DO-178C/ED-12C §6.3.1)Low-Level Requirements (DO-178C/ED-12C §6.3.2)Software Architecture (DO-178C/ED-12C §6.3.3)Source Code (DO-178C/ED-12C §6.3.4)Outputs of the Integration Process (DO-178C/ED-12C §6.3.5)Test Cases, Procedures and Results (DO-178C/ED-12C §6.4.5) Verocel 2017
Outputs of the Integration Process Detailed examination of the linking andloading data and memory map Topics include:– Incorrect hardware addresses– Memory overlaps– Missing software components Verocel 2017
SOFTWARE TESTING ANDVERIFICATION Verocel 2017
Test Environment Preferred test environment includes the softwareloaded into the target computer and tested in ahigh fidelity simulation of the target computerenvironment Some testing may need to be performed on asmall software component that is functionallyisolated from other software components Selected tests should always be performed in theintegrated target computer environment Emulators and simulators Tool qualification Verocel 2017
Normal Range Test Cases Real and integer input variablesTime-related functionsState transitionsSoftware requirements expressed by logicequations Verocel 2017
Equivalence Classes Exhaustive testing is impractical for non-trivialprograms Equivalence class: “The partition of the inputdomain of a program such that a test of arepresentative value of the class is equivalentto a test of other values of the class” (DO178C/ED-12C Glossary) Verocel 2017
Robustness Testing Real and integer variables System initialization during abnormalconditions Possible failure modes of the incoming data Loops Protection mechanisms for exceeding frametimes Time-related functions State transitions Verocel 2017
Testing of Integration Process – Table A-6 Verocel 2017
Verification of Verification Process – Table A-7 Verocel 2017
Test Coverage Analysis Requirements-based test coverage analysis Structural coverage analysis Verocel 2017
Requirements Coverage Analysis Test cases exist for each software requirement Test cases satisfy the criteria of normal androbustness testing Test coverage of high-level requirementsrequired at Levels A, B, C and D (withindependence at Level A) Test coverage of low-level requirements notrequired at Level D Verocel 2017
Structural Coverage Analysis MC/DCDecision CoverageStatement CoverageData Coupling and Control CouplingAll test cases used to achieve structuralcoverage should be traceable to requirements Verocel 2017
Structural coverage TerminologyConditionsif A B and (C or D 3) thenDecisionBoolean VariableBoolean Operators Verocel 2017
Decision coverage Boolean expressions tested in control structures(such as the if-statement and while-statement)must be evaluated to both true and false.Additionally, this measure includes coverage ofswitch-statement cases, exception handlers, andinterrupt handlers. For the decision (A or B), test cases (TF) and (FF)will toggle the decision outcome between trueand false. However, the effect of B is not tested;that is, those test cases cannot distinguishbetween the decision (A or B) and the decision A. Verocel 2017
Condition coverage Requires that each condition in each decisionevaluate to both TRUE and FALSE at least once For the decision (A or B) test cases (TF) and(FT) meet the coverage criterion, but do notcause the decision to take on all possibleoutcomes. As with decision coverage, a minimum of twotests cases is required for each decision. Verocel 2017
Condition Decision coverage Combines the requirements for decision coverage withthose for condition coverage. That is, there must besufficient test cases to toggle the decision outcomebetween true and false and to toggle each conditionvalue between true and false. Hence, a minimum oftwo test cases are necessary for each decision. Consider the following C/C code fragment:if ( A 0 or B 0 ) /* supposed to be a and */C sqrt (A) sqrt (B);– Tested OK with ( 1 , 1 ) and ( -1, -1). Will fail with (1,-1) and(-1,1). Verocel 2017
MC/DC The MC/DC criterion enhances the condition/decisioncoverage criterion by requiring that each condition beshown to independently affect the outcome of thedecision. The independence requirement ensures thatthe effect of each condition is tested relative to theother conditions. In general, a minimum of N 1 test cases for a decisionwith N inputs. For the example (A or B), test cases (TF),(FT), and (FF) provide MC/DC. For decisions with alarge number of inputs, MC/DC requires considerablymore test cases than any of the coverage measuresdiscussed above. Verocel 2017
Structural coverage Must account for “hidden” decision:A (C and D);if (A)/* something */A decision is not synonymous with a branch point. MC/DC appliesto all decisions, not just those within a branch point. And also :A B or C; (statement 1)E A and D; (statement 2)These two statements are logically equivalent to:E (B or C) and D; (statement 3) A test set that provides MC/DC for statements 1 and 2 individually will notnecessarily provide MC/DC for statement 3. For this example, tests (TFT),(FTF), and (FFT) for (B,C,D) provide MC/DC for statements 1 and 2individually, but do not provide MC/DC for statement 3. Verocel 2017
Coverage at Level Aif (A 0 && B 2 && C 5) { }if A 0 thenif B 2 thenif C 5 thenP;end if;end if;end if;MCDC not required for this code At the object code level, MCDC is equivalent todecision coverage. Verocel 2017
Data Coupling and Control Coupling Data coupling – The dependence of a softwarecomponent on data not exclusively under thecontrol of that component (DO-178C/ED-12CGlossary) Control coupling – The manner or degree bywhich one software component influences theexecution of another software component(DO-178C/ED-12C Glossary) Verocel 2017
Verification of Data & Control Coupling Reviews and analysis of Software Architecture(DO-178C/ED-12C §6.3.3.b) Reviews and analysis of Source Code (DO178C/ED-12C §6.3.4.b) Requirements-based testing, confirmed bystructural coverage analysis (DO-178C/ED-12C§6.4.4.d) Verocel 2017
Analysis of Data & Control Coupling “Test coverage of software structure, both data coupling andcontrol coupling, is achieved” (DO-178C/ED-12C §6.4.4.d) “Analysis to confirm that the requirements-based testing hasexercised the data and control coupling between codecomponents” (DO-178C/ED-12C §6.4.4.2.c) The intent behind this objective is to ensure that applicantsdo a sufficient amount of hardware/software integrationtesting and/or software integration testing (DO-248C/ED-94CFAQ #67) Verocel 2017
Structural Coverage Analysis Resolution Shortcomings in requirements-based testcases or procedures Inadequacies in software requirements Dead code Deactivated code Verocel 2017
SOFTWARE CONFIGURATIONMANAGEMENT Verocel 2017
CM process Purpose– Provide defined and controlled configuration of thesoftware– Provide the ability to consistently replicate the excutableobject code (or re-generate it if needed)– Provide consistency and repeatability in the processactivities– Provide baselines and know points for reviews– Provide controls to ensure problems receive attention andchanges are recorded, approved and implemented Verocel 2017
CM process Verocel 2017
SOFTWARE QUALITY ASSURANCE Verocel 2017
Quality Assurance Process Purpose– Provide assurance that SW development andintegral process comply with the approved plansand standards– Provide assurance that transition criteria forprocesses are satisfied– Provide assurance that a conformity review of thesoftware product is conducted. Verocel 2017
QA process Verocel 2017
Certification Liaison Purpose :– Establish communication and understandingbetween the applicant and the certificationauthority Verocel 2017
Certification Evidence (Life cycle data) Plan for software aspects ofcertification (PSAC) Software quality assuranceplan Software configurationmanagement plan Software development plan Software design document Version descriptiondocument Traceability matrix Software development folder Design reviews Code reviews Software requirements Test reviewsstandards Functional tests Software design standards Software coding standards Software verification plan Software requirementsspecification Coverage results Tool qualificationdocumentation Software accomplishmentsummary (SAS) Verocel 2017
Software Verification Results Detailed and overall pass/fail results Configuration item or software versionverified Results of tests, reviews and analyses Verocel 2017
TOOLING CONSIDERATIONS Verocel 2017
How to Prove Traceability?ReviewTest ResultsReviewReviewReviewReviewTest ProceduresSource CodeDesignRequirementsLinkage Verocel 2017
Verocel VeroTrace VeroTrace– Verification Life-Cycle Management Tool– Manages Requirements, Design, Tests,Coverage, Problem Reports, and more.– Provides full Traceability between all of theArtifacts Eases showing completeness of traceability– Enforces Software Development Processes– Impact Analysis for Changes– Generates Browseable Certification Evidence (onDVD)– Qualified to DO-330, TQL-5Verocel Proprietary61
Verocel Tools – Verification Tools VerOCode– Level A Object Code Coverage tool– Test on target without instrumenting the code– Addresses MCDC coverage– Qualified to DO-330, TQL-5VeroSource– Level A Source-based coverage tool– Qualified to DO-330, TQL-5VeroLink– Satisfies Control Coupling criteria– Qualified to DO-330, TQL-5VeroStack– Measures and calculates Worst Case stack use– Qualified DO-330, TQL-5PICSim– Instruction level simulator, Coverage Analyzer, Test Manger– QualifiedVerocel Proprietary62
Tool Qualification Tool qualification is necessary when DO178C/ED-12C processes are eliminated,reduced or automated by use of a softwaretool without its output being verified (DO178C/ED-12C §12.2.1) Tool qualification is handled quite differentlyin DO-178C/ED-12C compared to DO178B/ED-12B Verocel 2017
Tool QualificationCriteria 1: A tool whose output is partof the airborne software and thuscould introduce an errorCriteria 2: A tool that is used to justifyeliminating a development process ora verification process other than theone automated by the toolCriteria 3: Any other tool that could failto detect an error SoftwareLevelCriteria 1Criteria 2Criteria TQL-4TQL-5TQL-5 Verocel 2017
The Verification CompanyThe End
control coupling, is achieved" (DO-178C/ED-12C §6.4.4.d) "Analysis to confirm that the requirements-based testing has exercised the data and control coupling between code components" (DO-178C/ED-12C §6.4.4.2.c) The intent behind this objective is to ensure that applicants
