Transcription
Int'l Conf. Software Eng. Research and Practice SERP'18 89Integrating Software Testing Standard ISO/IEC/IEEE29119 to Agile DevelopmentNing Chen1, Ethan W. Chen1 and Ian S. Chen2Department of Computer Science, California State University, Fullerton, California, USA2Raytheon Company, Tucson, Arizona, USA1Abstract - The IEEE standard 29119 on Software and SystemsEngineering - Software Testing which replaces an olderstandard of IEEE Std 829 and others is designed with the needof agile process in mind. It provides an explanation on Agileprojects and some suggestions on integrating the standard toAgile process. Nevertheless, integrating the standard to Agile,still is not that straightforward and may need furtherelaboration. This paper addresses the following issues: theneeds for a testing standard, the mechanism that integrates thetesting standard to Agile, and on how to tailor theconformance to a proper level that involves the maturity levelsof the Test Maturity Model Integration (TMMi). The paperconcludes with a suggested tailored conformance plan that issuitable for a typical Agile Development.Keyword: IEEE Std 29119, Software Testing, conformance,Agile, TMMi1IntroductionTesting is an integral part of the software developmentprocess. ISO/IEC/IEEE 29119 is a relatively new standard forsoftware testing with the most recent part published in 2016[1]. As a new international standard, IEEE 29119 becomes thebenchmark upon which testing processes are measuredagainst. Organizations are then influenced to adhere to thestandard to some degree and implement IEEE 29119conformance within their existing software developmentprocess.Agile methodology is one of the most popular softwaredevelopment philosophies and widely used. Per a recent 2017survey, 94% of respondents claimed that their organizationspracticed Agile [2]. As a result, many companies will face orare currently facing the task of implementing IEEE 29119within their Agile process. One of the most notable standardsthat IEEE 29119 is replacing is IEEE 829-2008, whichdescribes test documentation standards [3]. The old standard,IEEE 829, is deeply rooted in the traditional Waterfalldevelopment lifecycle, but is not entirely at odds withimplementation into Agile [4]. Though IEEE 29119 waswritten with more consideration for Agile in comparison toIEEE 829, it does not clearly describe how it can beimplemented into Agile. The question then arises of whetherIEEE 29119 can be easily integrated into Agile and how.Though IEEE 29119 does not necessarily conflict withAgile, it is possible that full conformance to the standard, asdefined in IEEE 29119, may be too cumbersome for properAgile methodology. This produces a conundrum whereextensive adherence to testing standards is required to be fullycompliant with IEEE 29119, but in doing so would violateAgile principles. At the same time, it would not be sensible tocompletely forgo any testing standards in Agile. Someindustry surveys estimate that testing comprises up to 30 to 50percent of the total cost of development [5]. As a result, sometesting standard is desired in order to properly exhibit theresults of and to justify the expense invested in testing.Though IEEE 29119 provides some leeway in thisproblem with the option of tailored conformance, theimplementation of tailored conformance is arguably too loose.Tailored conformance in IEEE 29119 is defined to becompletely dependent on a specific organization and/or projectneeds, and thus loses its utility as a standard. Since Agileholds significant market-share in the community, a standardadoption of IEEE 29119 within Agile is needed to maintainparity between different organizations. If no Agile standard ofIEEE 29119 is available, implementations of IEEE 29119tailored conformance across multiple organizations may be tooradically different to be properly compared.This paper starts with an analysis of IEEE 29119 andAgile methodology. We then explore the need for a testingstandard, propose an integration of IEEE 29119 within Agile,and describe IEEE 29119 conformance maturity levels relatedto the Test Maturity Model Integration (TMMi).2What is IEEE Std 29119?IEEE 29119 is an internationally agreed set of standardswith the purpose of supporting software testing. IEEE 29119consists of five different parts:1.2.3.4.5.IEEE 29119-1: Concepts & DefinitionsIEEE 29119-2: Test ProcessesIEEE 29119-3: Test DocumentationIEEE 29119-4: Test TechniquesIEEE 29119-5: Keyword Driven TestingIEEE 29119 is intended to replace the following existingstandards for software testing:ISBN: 1-60132-489-8, CSREA Press
90Int'l Conf. Software Eng. Research and Practice SERP'18 ooooIEEE 829 Test DocumentationIEEE 1008 Unit TestingBS 7925-1 Vocabulary of Terms in Software TestingBS 7925-2 Software Component Testing StandardIEEE 29119-1: Concepts & Definitions – This documentcorrelates to BS 7925-1 and serves as a glossary of definitionsof testing terms and concepts used in the overall IEEE 29119standard.their situation. Lastly, IEEE 29119-5 defines a modularapproach to describing test cases through the Keyword-DrivenTesting framework which lends itself to test automation and isa specific implementation of the test process model describedin IEEE 2119-2. Similarly, to the rest of IEEE 29119, thetesting framework itself is described in detail, but the specificintegration of the framework into different types of softwaredevelopment lifecycle models is not.IEEE 29119-2: Test Processes – This document describessoftware testing processes at multiple levels. The processes aremeant to be generic as to be able to be implemented in anyorganization for any kind of software testing and for any typeof software development life cycle model. Most notably, IEEE29119-2 focuses on a risk-based approach to testing with theprimary goal of risk mitigation. The standard test processmodel described is multi-layered with a top levelorganizational test process, a middle level test managementprocess, and a bottom level of dynamic test processes [Figure1].IEEE 29119-3: Test Documentation – This part of IEEE29119 provides templates and examples of test documentation.These work products are aligned with the test processesdescribed in IEEE 29119-2 and cover the organizational level,test management level, and dynamic test level. Examples andsuggestions are provided for both Agile and traditionalsoftware development life cycle models.IEEE 29119-4: Test Techniques – A description of softwaretest design techniques that align with the test processesdescribed in IEEE 29119-2 is given. The document describeshow to derive test conditions, test coverage items, and testcases.Figure 1. IEEE 29119 Multi-layer Test Process Model3What is Agile Development?Agile development is a broad term encompassing a set ofpractices that promote the values espoused in the AgileManifesto [6]. From the Agile Manifesto [Figure 2] arederived the Twelve Principles of Agile that make up the Agilephilosophy. In total, Agile is meant to enhance the ability toeffectively respond to the constant change present in thesoftware development process. Agile advocates continuous,small incremental releases of working software. Mostimportantly Agile describes the methods, or the “how”, insoftware development (e.g. Scrum, Kanban).IEEE 29119-5: Keyword Driven Testing – The most recentlypublished portion of IEEE 29119 describes the usage ofkeyword-driven testing as a strategy of modular testing forusage of test automation frameworks.IEEE Std 29119 lays out a set of requirements to beconsidered as adherent to the standard. Though IEEE 29119provides examples of how test documentation might look likein Agile or traditional software development lifecycle models(found in IEEE 29119-3), it does not provide specificinstruction on how IEEE 29119 should be implemented.Regarding test process, IEEE 29119-2 sets standards of whatforms of test processes should be defined at multiple levels[Figure 1] but no specifics on integration into various softwaredevelopment lifecycle models. Similarly, IEEE 29119-3describes what work products in the form on documentationshould be produced throughout the testing process, butprovides minimal guidance on how it should be implemented.IEEE 29119-4 describes various test techniques that can beused, but does not specify any particular requirements as towhich are universally needed. Rather the user is left to pickand choose which techniques are appropriate and suitable forFigure 2. The Agile Manifesto3.1Software Testing in Agile DevelopmentIn Agile Development, software testing is performeddifferently in comparison to traditional testing methodologylike Waterfall Model. Instead of testing performed as aseparate and distinct phase after coding, Agile software testingis performed in concurrence with coding. Furthermore, testingis integrated into the development team as well as relevantstakeholders and the distinction between developers and testersis blurred.In Agile methods, there is no true definitive or standardtesting strategy. Though Test Driven Development (TDD) isISBN: 1-60132-489-8, CSREA Press
Int'l Conf. Software Eng. Research and Practice SERP'18 91an extremely popular method of software testing in Agile, it isnot required, strictly speaking, to adhere to the AgileManifesto. The first line of the Agile Manifesto describes thissuccinctly: "Individuals and interactions over processes andtools". Agile does not technically prescribe any specific testingprocesses only that the software development process remainsresponsive to change. Literature has expanded on this topicand produced recommendations and trends on best practices ofimplementing testing into Agile [7] [8].development lifecycle. What Agile does not specify is whatprocesses should be carried out or what work-products shouldbe produced throughout the testing process. In that sense,testing standards do not conflict with Agile development andinstead can be complementary to Agile. The question thenbecomes how should testing standards, in particular IEEE29119, be implemented in Agile environments.3.2Testing in Agile development aims at different goals than thatof traditional iteration life process models (e.g., Water Fall) [7].There are no firm requirements at the beginning and typically nosolid, fixed architecture plan either. Furthermore, one of theimportant goals in each iteration (e.g., Sprint) is to have a “workingsoftware”. The term “working software” implies the existing ofrequirements and the process of verifying those requirements eventhey are very informal, not in the form of formal documentation.This opens the idea that if the development activity can be iterativeand incremental, then the forming of test plans, test documentation,and test tool policy can also be iterative and incremental [4] [9]. Wepresent an effort in forming of plans, documentation, and policy in aScrum setting as shown below.The Role and Need of Testing Standard inAgile DevelopmentAgile, in a strict interpretation, does not prescribe orrequire any sort of testing standard. This is not to say that atesting standard would not be valuable in an Agileenvironment. A standard provides common ground uponwhere the community can agree upon what practices arenecessary to say that the overall testing process is sound.Individuals and interactions over processes and tools Though the Agile Manifesto does prefer individuals andinteractions over processes, that does not mean that testingprocesses are not needed. In the end, having standardizedtesting processes and organization provides structure to theoverall testing effort. Utilizing standard processes also doesnot run contrary to Agile as long as it remains flexible anddoes not become rigid to the point that responsiveness tochange is affected, thus violating Agile philosophy.Working software over comprehensive documentation Though Agile does explicitly value "Working software overcomprehensive documentation", the total absence of anytesting documentation is not desirable. Some documentation isdesired to have records that can demonstrate testing wasadequately performed. At the same time, excessivedocumentation would run contrary to Agile. Agileorganizations then must individually determine what level ofdocumentation is needed to satisfy their own internaldocumentation requirements while remaining Agile. With sucha broad specification, a testing standard would be useful here.Customer collaboration over contract negotiation - Thoughthis particular part of the Agile Manifesto does not explicitlyinvolve testing, customer collaboration can be an importantpart of the Agile testing process. For example, AcceptanceTest Driven Development, where customer engagement is keyto testing, can be implemented into Agile though it is notnecessarily a standard.Responding to change over following a plan - Similarly,while more value is placed on the capability to respond tochange, this does not mean that a plan has no value. A test plandoes provide value in an Agile environment as long as it doesnot hinder responsiveness to change.Agile philosophy describes how testing should beimplemented. It should be continuous and integrated into the44.1How to Integrate?Sprints in ScrumWe briefly describe the Scrum process first. The Scrumdevelopment is done in time-boxed efforts called Scrumsprints. As the business requirements evolve incrementally,the Product Backlog (e.g., user stories) forms. Each sprintaims for reducing some project backlog in one to four weeksof time. Although the business requirements typically focus onfeature and functionality of the software product, it is possibleto treat the need to have processes, documentation, techniquesand automation related keyword driven testing framework aspart of the product backlog and allocate special sprints, forexample, sprint zero or other sprints, for this purpose.4.2Processes, Documentation, Techniques andKeyword Driven FrameworkPart 1 of 29119 describes the concepts and definit
IEEE 29119 is an internationally agreed set of standards with the purpose of supporting software testing. IEEE 29119 consists of five different parts: 1. IEEE 29119-1: Concepts & Definitions 2. IEEE 29119-2: Test Processes 3. IEEE 29119-3: Test Documentation 4. IEEE 29119-4: Test Techniques 5. IEEE 29119-5: Keyword Driven TestingFile Size: 231KBPage Count: 7
