WIXLIB

8December, 2009ISSN 1866-5705www.testingexperience.comfree digital versionprint version 8,00 printed in GermanyThe Magazine for ProfessionalProfessiional TestersStandards –What about it? iStockphoto.com/belknap

iStockphoto.com/NeustockimagesIEEE829:2008 A major change of focusby Bernard HomèsThe test plan is the basis for all future test activities on a software application. It is a contract between the test and development teamsand the management.Note: This article was originally presented atCONQUEST09, but has since been improvedIEEE 829:1998 was focused on documentationand this documentation detailed the strategy tobe used for the testing activities for:1-Management to understand the addedvalue of the tests and the remaining risks,the costs and requirements as well as thetime frame,1.Synthesis of the IEEE829:1998 andIEEE829:2008 standards2.Relationship between the documents-Development team to focus on the areasthat are most critical or where a requiredlevel of quality has been set,3.How to be compliant to the 2008 versionand how to migrate from 1998 to 2008version-Test team to develop and execute the testcampaign on time and within budget,4.Advantages and drawbacks of theIEEE829:2008 version-Test project manager to have a referencedocument to keep the project on courseand on budget.1.1In 2008, the IEEE Standards Organization published a revised version of IEEE829, expanding it from software to also cover systems.This article describes the different types ofdocument used in the IEEE829:2008 standardand their relation to one another. Building onthis information, it details the different stepsand aspects of the test plan as described in thestandard and the relations between the different chapters, how they complement each otherin order to provide a complete solutionIEEE829:2008 introduces a number of changes, one of which is the change of focus from adocument-centric standard to a process-centricstandard.This article maps changes from the old version with the new version, and suggests waysto implement the new version of the standardin your context.the IEEE 829 standard. Other suggested audiences include business, line and purchase managers, in order to understand what is required in thecontext of testing their systems, so thatthey are able to evaluate proposed testingstrategies.2NoveltyExposeKey pointsThe following key points will be highlightedin this article:Intended audienceThe intended audience includes: Test and development managers as wellas project managers and any testers, analysts or consultants intending to deployProjectDocIEEE 829 is a well-known standard for software test documentation, that is not specifically new. The standard has been widely commented, sometimes attacked, by the differentschools of testing, and is a bone of contentionbetween the Agile testing school – which consider documentation to have very limited added value – and the systematic school, wherethe strict adherence to established processesand standards is considered mandatory.Since 2003, a group of international expertsfrom the different schools of testing have gathered under the auspices of the IEEE, in order toimprove the standard and make it ready to evelDesignPrepareITRTest executionTLIRLevelExecute&TrackReportDirectly from one of the members of theIEEE829:2008 team, learn how to change yourfocus from documentation to testing anceTest ceSystemIntegrationComponentMTRFigure 1: IEEE829:1998 vs IEEE829:2008 structure6The Magazine for Professional Testerswww.testingexperience.com

the new challenges, adapting it to the evolvedand more complex world facing us.This endeavour reached its climax in July2008 with the official publication of theIEEE829:2008 version of the standard, whichnow addresses both software and system testdocumentation. This expansion of scope from“software” to “systems” make this standard aleading item when one takes into account theincreased complexity of systems and the emergence of systems-of-systems.This paper describes the highlights of the newstandard, its possible implementation in youror-ganization, and the tasks needed for you tobecome compliant with the new version of thestandard.2.1OrganizationThe organization of the documentation suiteof the new version of the standard, comparedto the 1998 version is as described in the diagram (Figure 1). Shown on the left side is theIEEE829:1998 structure of documents, and onthe right side the new 2008 version.(See Figure 1)Legend for IEEE829:1998 is as follows: TP:Test Plan; TD: Test Design; TC: Test Case;TPr: Test Procedure; ITR: Item TransmittalReport; TL: Test Log; TSR: Test SummaryReport.Legend for IEEE829:2008 is as follows: ATP:Acceptance Test Plan; STP: System Test Plan;CITP: Component Integration Test Plan; CTP:Component Test Plan; ATD: Acceptance TestDesign; ATC: Acceptance Test Case; ATPr:Acceptance Test Procedure; ATL: AcceptanceTest Log; AITR: Acceptance Interim TestStatus Report; AR: Anomaly Report; ATR:Acceptance Test Report; STR: System TestReport; CITR: Component Integration TestReport; CTR: Component Test Report; MTR:Master Test Report.3BenefitsCompliance with a standard means that theperson / organization intends to ensure thatits processes and documentation fit a specificframework. This usually involves complying with mandatory (or regulatory) requirements and is based on the principle that anorganization can improve its deliverables ifthe development processes are repeatable andmeasurable. In IEEE829:1998, the only itemsthat were mandatory were the title of the documents and their chapters. This was sometimesconsidered as too burdensome and at othertimes too light-weight to fit the type of development undertaken.With the advent of different SDLCs1 and theincreased complexity of some systems, it became clear that a new, updated and upgradedstandard was necessary. This standard wouldbe aligned to the ISO/IEC 12207 standard andadaptable to the criticality of the software andsystems being developed.1SDLC : Software Design Life Cycle, described in ISO/IEC 12207www.testingexperience.comThe inclusion of persons like James Bach andCem Kaner from the Agile / context-drivenschool of testing ensured that the standardwould focus on tasks that added value to thesystem and that it is adaptable to different contexts.3.1ApplicabilitySoftware becomes more and more pervasivenowadays. It is no longer limited to applications running on mainframes, minis or micros,linked or not by networks. It is now presentin almost any commercial appliance, fromfridges to cars and from telephones to personalassistants. The hardware verification aspect isnow as important as the software-specific aspect. The new version of the IEEE829:2008standard addresses this.The new applicability of the standard is reflected in the decision to use the terms “software” and “system” interchangeably to reflectan “identified component or collection of components that include but are not restricted tosoftware [ ] that comprise a software-basedsystem”.This generalization to include hardware components implies that the future test managersand test analysts in charge of defining teststrategies, test plans (Master Test Plans andLevel Test Plans), will be required to have anunderstanding of the environment the softwarewill be running on.In terms of contexts, this standard can be applied in multiple contexts and environments,from aerospace to automotive industry, butalso to other industries such as banking, insurance, and de-velopment of any software.3.2Tasks vs. documentsIn IEEE829:1998, only a limited set of documents was suggested (see figure 1, left side),and testing tasks were defined based on thedeliverables. In the new 2008 version of theIEEE829 standard, a number of tasks areprovided per process, and each task has associated inputs and outputs, and can be associated with specific levels defined in ISO/IEC12207:2008.This enables authors of test documentation todefine the tasks required, and to use the standard as a checklist from which to pick andchoose the different tasks according to theirown context, ensuring that all tasks selectedprovide added value.It is important to note the linkage betweenIEEE829:2008 and ISO/IEC12207. This allows mapping of testing tasks to other tasksnot directly linked to systems development,such as acquisition.As can be understood, this shift of focus froma document-oriented standard to an activityfocused standard enables the users to maptheir processes to the standard and check forany deficiencies. It is thus easier for users toimplement this new version of the standardthan the previous version.Another aspect that should not be forgotten isthe possibility to use this standard even whenusing agile methods and exploratory testingtechniques that de-emphasize the use of documentation. As these methods and techniquesare task-focused instead of being documentfocused, compliance to the standard can beattained.3.3Integrity levelsFailure of some software and/or systems maylead to serious or even catastrophic consequences that cannot be solved by just rebooting the system.Some examples include the Osprey V22 systems and the multiple developmental problemsit suffered, killing in the process a number oftest pilots. Other similar issues abound, so itwas deemed important that the IEEE829:2008standard addressed this in detail, showingwhat tasks were recommended at each level.These levels were called “integrity levels” andimplement in IEEE829:2008 the ideas thatwere defined in DO178B/ED12B, ECSS andother standards.The standard proposes a four-level integrityscheme, numbered 1 to 4 (4 is the highest integrity) described as follows:- Negligible : “Software must execute correctly or intended function will not be realizedcausing negligible consequences. Mitigationnot required.”- Marginal : “Software must execute correctlyor an intended function will not be realizedcausing minor consequences. Complete mitigation possible.”- Critical : “Software must execute correctly orthe intended use (mission) of system/softwarewill not be realized causing serious consequences (permanent injury, major system degradation, environmental damage, economic orsocial impact). Partial-to-complete mitigationis possible.”- Catastrophic : “Software must execute correctly or grave consequences (loss of life, lossof system, environmental damage, economicor social loss) will occur. No mitigation is possible.”This concept of integrity level trickles downto other aspects of testing, such as the levelof mandatory documentation according to theintegrity level, the recommended minimumtasks per level and their intensity, and levelof tester independence (see section 3.3.3). Integrity levels can also be defined based on theresults of a risks analysis.Even though the IEEE829:2008 standard doesnot mandate the use of integrity levels, it isnoted that the use of an integrity level is “arecommended best practice that facilitates theselection of the most appropriate activities andtasks”.Incidentally, the implementation of integritylevels is also an incentive for defensive programming and other similar reliability enhancement techniques for software development.The Magazine for Professional Testers7