Best Practices For Improving The Quality And Speed Of Your .

Transcription

A Conformiq White PaperBest Practices for Improving the Qualityand Speed of Your Agile TestingAbstractWith today’s continually evolving digital business landscape, enterprises are increasingly turning to Agile approachesto speed up development and address growing consumer demands for innovation. Agile approaches aim to buildquality assurance into product development from the ground up, by having developers heavily involved with testingfrom the outset. The idea is that if problems can be found and corrected earlier in the design process, companiescan get defect-free products to market more quickly. Unfortunately, Agile is often unable to deliver on its promiseof early, aggressive, and continuous testing, because many of the testing approaches being used today areinsufficient to get the job done.Below we explore some of the popular alternatives to traditional manual testing, including those being used toimprove the speed of functional test design. However, it is important to note that speeding up test design is not thesame as improved test design productivity. Even in Agile programs, the focus must be on improving the overallproject testing process, not just faster test design. Conformiq 360 Test Automation solves this issue by automatingthe entire systems development life cycle (SDLC) process, in addition to automatically generating test designs andexecutable scripts at the speed of development. This paper is intended to provide more insight into these olderdeveloper test design methodologies, what they do and don’t deliver, and then to compare and contrast them withthe newer Conformiq 360 Test Automation process.BackgroundOne of the processes promoted for speeding testdesign is Test-Driven Development (TDD). It is a corepart of Extreme Programming (XP) and other “lightweight” development practices and, though not acore part of Agile development, is a common partnerto Agile.Their next objective was to write the minimal codethat would pass that test case. Once the test passes,they refactor the code making sure that the test stillpasses. More broadly, TDD is used to describe anyprocess where tests for a feature are written beforethe feature is implemented.As originally described by Kent Beck, TDD meant thatbefore a developer could add a feature in thesoftware, he/she first must write a failure test case.In practice, TDD has been considered toounstructured, so Behavior-Driven Development(BDD), as an enhancement over TDD, is currentlyPage 1

receiving significant interest as being the “next” greattest improvement process for Agile developmentprojects.BDD is a software development process that aims tocombine the techniques and principles of TDD andObject-Oriented Design by leveraging ideas fromdomain-specific designs. It has excellent philosophicalgoals and ambitions, as BDD fundamentally aims toengage all stakeholders in the software developmentprocess by enabling Subject Matter Experts (SMEs)and non-technical stakeholders, such as businessanalysts and customers, to contribute and collaboratein the process by writing user stories.In principle, BDD is founded on the use of a simpleand informal notation, which is very close to commonlanguage and based on the main concepts of featuresand scenarios. Scenarios detail the “desired behavior”for each feature; they are essentially acceptance testsin the form of user stories. Probably the most widelyknown and used notation is Gherkin, which is used bytools like Cucumber, FitNesse, and JBehave.Here is a Gherkin example: Feature: The online shop keeps track of goods ina shopping basket.Scenario: Put an item into an empty shoppingbasket.Given the shopping basket is empty.When user adds one item to the shopping basket.Then the shopping basket should contain oneitem.It is easy to see what is happening here. But if youtake another look at this example, Gherkin is reallynothing more than a requirement and an informaltest description. This informality is both a strengthand a weakness when it comes to test automation.Automation frameworks revolving around Gherkincan only generate simple code stubs, and still requirea significant amount of implementation by softwaredevelopers to get these codes ready for execution.Each Gherkin scenario clause is “just text,” or textcreated with the preferred wording by the Gherkinauthor. Since automation codes need to be writtenmanually and mapped to that description, dealingwith change management is a significant issue.Scenarios make it very difficult to assess the qualityand completeness of your software testing.For example, how much have you actually tested yourapplication? The use of Gherkin does not guaranteesystematic coverage of functionality. Have you reallytested all the possible data combinations? Which datacombinations actually make sense? Do you havescenarios that fully cover all decision points withinyour functionality to be tested? Have you consideredboundary values?Overall, BDD and TDD are extremely limited insupporting Agile development because they areunable to deliver a complete testing process andactually reduce software developer efficiency. Also,because they were developed prior to automatedtesting tools, they don’t account for improvedefficiency from advancing technology.ChallengesWhile TDD and BDD testing can be used in an Agileprocess, the results show that with these methods,improved speed comes at the cost of quality andknowledge, especially the understanding of testcoverage. Other issues include:1. TDD was created to match the need of asoftware development process with shortdevelopment cycles. Unfortunately, constanttime to market pressure makes it difficult tomaintain and update the (regression) testsuites.2. Test cases written by developers were createdto cover their own code. They did not fullycover the whole system operation, or take intoaccount how the multiple code parts written bydifferent developers would run together. Thismeant that additional system end-to-end testcases needed be written later in the process,Page 2

often causing critical system-level defects to befound much later.3. Because the developers wrote their own tests,this took away time from writing code andsubsequently reduced their design efficiency.4. Finally, the process of having the developertest his/her own code goes fully against thetenets of the Independent Verification andValidation (IV&V) process. Systemic defects canslip through the developer’s “blind spots”.The TDD/BDD testing process is shown below.Figure 1: The TDD and BDD testing processA More Advanced Method is NeededThis is where Conformiq 360 Test Automation comesin, as a transformational approach which includesModel Based Testing (MBT) technology and more.This next-generation solution enables testers andsoftware developers to complement the work doneby business analysts, system engineers, andcustomers by generating the tests they created asboth Gherkin scenarios and test automation scriptsfor the entire test logic as code that includes testdata. Tests are automatically derived and generatedby Conformiq tools, which provide systematic andrepeatable coverage of the functionality to be tested.As requirements change, the model can be quicklychanged to match them, and all generated scenariosand test automation codes are automaticallyupdated, eliminating the issue of maintenance.Requirements can be directly downloaded fromrequirement management tools linked to theConformiq models; requirements traceability isautomatically established.The Conformiq 360 Test Automation approach tomodel-based black-box testing starts with simple,high-level formal models of the system under test(SUT) that is being designed, and then automaticallygenerates test cases. The model of the system can becontinuously modified in parallel with development ofthe system itself. With TDD, you write a test case for afeature before you write the feature. WhenConformiq 360 Test Automation is applied, youaugment your SUT model to express your feature,then Conformiq automatically regenerates the tests(which will include one or more tests relating to yournew feature), before you write the feature.In our experience with industry use cases forConformiq 360 Test Automation, we have found twokey transformations for testing advantage. First, theproductivity improvement for actual test casegeneration versus manual creation of tests issignificant, on the level of an order of magnitude.Second, this productivity improvement is even higherin the context of incremental development:Conformiq tools automatically regenerate the full testsuite when the model is changed, includingdetermining which prior test cases are no longerapplicable. It also automatically generates the testoracle, i.e., the expected correct test execution result.Page 3

prototyping is a good and efficient way of findinginconsistencies in the requirements.Varying requirements.Figure 2: Conformiq 360 Test Automation methodWhat Are the TDD/BDD Issues thatConformiq 360 Test AutomationSolves?These real-world experiences argue that Conformiq360 Test Automation is particularly well suited forAgile, especially as an improvement to TDD methods.In addition, it can help in improving some of theinherent issues associated with TDD and BDD,including the following challenges.Poorly understood requirements.With TDD, the requirements are not well understoodearly in the process, and they typically containambiguities, omissions and contradictions. Onebenefit of Conformiq 360 Test Automation is thatjust the act of modeling the system behavior oftenimproves the quality of the requirements specified.This means many defects can be spotted early, inmodeling the specifications and requirements, beforea single line of code is even written. As you create amodel of the system behavior, you often raise manyquestions regarding the requirements, so already themodeling process can expose a lot of issues with therequirements. This should not come as much of asurprise. After all, system modeling involves thedevelopment of a compact high-level prototype of thereal system, and it has been long known thatThis is especially important within Agile developmentprojects where the requirements are updated duringthe project. In Conformiq 360 Test Automation, asimple formal model of the SUT explicitly embodiesthe requirements. The refactoring of thoserequirements with a model change, and automation,is dramatically less effort than refactoring a set ofmanual tests. With the Conformiq approach, theeffort corresponds only to the number ofrequirements that change. In contrast, with a manualprocess, all test cases need to be checked for allrequirements that have changed.Poor/Inadequate Tests.With TDD, the developers have not implemented thesolution yet, so the tests are typically not “goodenough” to explain the solution. With Conformiq 360 Test Automation the idea is that the model representsthe actual, desired behavior of the system itself – notthe test cases nor how it should be tested. It improvesthe quality of the test cases because this automatedapproach to test design lowers the risk of havingincorrect, missed and redundant tests. An engineercan, for example, accidentally miss a test case that isdictated by the requirements, such as for an errorhandling case, a limit value of a data parameter, or anexpiration of a rarely activated timer, but not so withthe Conformiq algorithmic approach.Unit tests instead of system tests.As mentioned previously, TDD test cases are writtenby developers and cover only their own code. Theseunit tests do not cover the system operation, becausethey do not cover the operation of multiple codeparts written by different developers all runningtogether. This means that, with the TDD approach,additional, end-to-end system test cases must bewritten at a later time, often leading to a delay inPage 4

finding the system level defects. With Conformiq 360 Test Automation, these system tests areautomatically created as the model grows or asmodels are combined into the full system.Over-fitting tests to the code.A common concern is that if a developer writes thetests first, he may over fit the actual implementationto the tests. With Conformiq 360 Test Automation,the developer doesn’t design the test cases, so thereis no risk of fitting the implementation to the tests. Akey point of model-based black-box testing is that thesystem is judged against an independent reference.Without this approach there is naturally a possibilitythat the developer reflects the same fault both in thetest and in the implementation code.Reluctance to do early testing.Adherents believe that tests should be guided bycode as well as expert knowledge of theimplementation on where the problems might be.Therefore, with TDD, implementing tests too early isviewed as unnecessarily expensive, because you donot have the details of the code available. This is avery common white-box view to the problem, wherewe expect to have access to the implementationdetails for devising test cases. However, with modelbased black-box testing, we approach the problemfrom a different angle. We assume noimplementation details of the system, and validatewhether the given system conforms to its design andfunctional specification. The test cases thatConformiq CreatorTM generates (using MBTtechnology) are black-box by nature, which meansthat they depend on the model and the interfaces ofthe SUT, but not on the internal structure of theimplementation. You do not have to understand howthe system has been architected internally to create amodel, thus lowering the cost of test creation andallowing tests to be generated prior to incrementalcode drop.Developer’s lack of interest and/or ability to test.Testing requires a different mindset fromdevelopment, and some developers are poor in doingtest design. This issue can be resolved by pairingdevelopers with people who know how to test. Whenthe Conformiq 360 Test Automation approach isapplied, modeling can be accomplished by a nondeveloper (an SME, a modeler, etc.) who is an integralpart of the development team. This means thatdevelopers do not need to also be testers. So ratherthan spending their time writing test cases, insteaddevelopers can spend their time writing code. This is akey benefit in an Agile delivery process with shortsprint times. In addition, this approach supports thetenets of the IV & V process, and helps to reduce theerrors that naturally result when the same personwrites both the testing assets and the code.Conformiq 360 Test AutomationApproachConformiq has developed an approach to modelbased black-box testing that starts with simple, highlevel formal models of the SUT, then automaticallygenerates test cases without further user involvementor direction. The model of the system can thencontinuously be fleshed out and easily updated inparallel with development of the system itself.Figure 3: With Conformiq 360 Test Automation, the modelis manually created from the requirements and test scriptsare automatically generated for executionBecause Conformiq automatically generates testcases, they are consistent in how they are written,whereas manually created BDD textual test cases varyPage 5

by author. Beyond test cases, Conformiq ensures thattesting is done well, with high coverage, and that allstakeholders have the knowledge they need.Can these Processes be UsedTogether?Instead of seeing BDD and MBT as competingapproaches, we can view them as beingcomplementary. It can be beneficial to integrate thetwo approaches to get the best of both worlds.Instead of requiring and relying on softwaredevelopers to “connect the dots” through manualimplementation of code stubs and maintenance ofevery clause in every scenario, the Conformiq solutionautomatically updates a powerful MBT system model.The model is then processed by Conformiq’s engine,which generates an optimal collection of tests thatcan be exported as Gherkin scenarios for businessanalysts, system engineers, and customers for modelreview. These can also be executed automaticallyusing frameworks such as JUnit, Cucumber, FitNesse,and output for execution with HP/UFT, Selenium, andother frameworks.SummaryThe promised speed of Agile development is a key toits appeal and broadening use. However, testingmethods developed over a decade ago are insufficientfor delivering quality at the necessary speed. Theinnovative Conformiq 360 Test Automationapproach for test design eliminates the inherentproblems with those previous developer-centrictesting methods, and instead introduces a nextgeneration methodology that delivers productivity,quality and documentation at the speed ofdevelopment.Author Kimmo Nupponen is the Chief Scientist atConformiq. He has been developing automated testdesign software for over ten years. He understandswhat is needed for real world use and the differencesbetween tool engines.Page 6

Conformiq is transforming software testing with Conformiq 360 Test Automation , providing the mostsophisticated and comprehensive automated test design solution in the industry. The unique Conformiq360 Test Automation technology enables the next generation of testing: transforming, streamliningand automating even the most complex system-level testing environments. Conformiq 360 TestAutomation improves efficiency with a 40% faster test case development cycle; enables delivery ofhigher quality code with 50% more defects found; increases manageability with 50% bettercollaboration: and reduces costs with a 400% return on investment. Conformiq serves enterprise IT,communications and embedded software markets worldwide. Privately-held Conformiq isheadquartered in San Jose, California, with a worldwide delivery and support organization includingoffices in Finland, Germany, Sweden, and India.www.conformiq.comUSA4030 Moorpark AveSan Jose, CA 95117Tel: 1 408 898 2140Fax: 1 408 725 8405FINLANDWestendintie 102160 EspooTel: 358 10 286 6300Fax: 358 10 286 6309sales@conformiq.comSWEDENStureplan 4CSE-11435 StockholmTel: 46 852 500 222Fax: 358 10 286 6309GERMANYMaximilianstrasse 3580539 MunichTel: 49 89 89 659 275Fax: 358 10 286 6309 Conformiq 2015. All company names, trademarks and copyrights are the property of their respective owners.INDIA29 M.G. Road Ste 504Bangalore 560 001Tel: 91 80 4155 0994V1115Page 7

TDD and BDD testing can be used in an Agile process, the results show that with these methods, improved speed comes at the cost of quality and knowledge, especially the understanding of test coverage. Other issues include: 1. TDD was created to match