WIXLIB

A Conformiq White PaperHow to Succeed at Multi-Platform TestingIt should come as no surprise that the numbers of platforms and device types are more varied now than everbefore. Customers continue to demand the latest devices, features and functionality, as well as increasedmobility and accessibility. With the proliferation of mobile and portable device platforms and the Internet ofThings, the workload of developers, and especially testers, has greatly increased. As traditional testingpractices prove unable to keep up with the demand, businesses of all types are experiencing significant productdelivery delays and, in some cases, costly product defects. Naturally, there is a growing demand for moreefficient and cost-effective testing across all platforms.With the Internet of Things and more devices of alltypes, it is critical for development teams to buildproducts that interoperate and functionconsistently across multiple platforms and devices.Regardless of the industry, multi-platform testingcapability is now mandatory.Due to fragmentation and differentiation ofdevices, displays, and inputs, the traditional manualapproach to testing suffers many drawbacks andlimitations. Among other things, it is extremelytime-consuming and prone to missed coverage.This not only promotes lower product quality, butalso leads to cost escalation. Testing is even moredifficult because features do not function in thesame way across all platforms. This requiresdevelopers to implement custom code for differentplatforms, which, in turn, requires more testing.Adding to the testing complexity is the vast numberof APIs (application programming interfaces) thatconnect all these devices.Introducing Cross-Platform Test AutomationIn order to improve product quality across allplatforms, it is necessary to have known testcoverage. In parallel, it is important to ensuregreater efficiency throughout the testing processand reduce testing turnaround time in order tospeed time-to-market. The industry agrees thatimplementing cross-platform test automation isreally the only effective way to scale. This requiresbusinesses to make an investment in implementingand deploying a test execution infrastructure thatworks across platforms such as Appium, Calabash,Perfecto, Selendroid, and Telerik. At the sametime, for other platforms and legacy applications,automating execution utilizing existing tools like HPQTP/UFT and Selenium might also be needed.When combined, these infrastructures allow teststo be executed on a variety of devices without thecore structure having to be re-engineered, butsupporting them all can make automated testingvery complex. Thus it becomes critical to use atesting process that can easily support all neededexecution frameworks. Conformiq 2015. All company names, trademarks and registered trademarks are the property of their respective owners.

However, while the industry has identified theimportance of automating test with cross-platformtesting, most traditional test automation focusesonly on regression testing. When testing newfunctionality, traditional manual test design andmanual test execution are still the prevailingapproaches, even though “state-of-the-art” crossplatform test automation is now available.Manually created test cases typically focus on“happy paths”. They target the most commonstates and activities within the application, notcorner or negative cases where defects often occur.This focus on happy-path testing carries over evenwhen regression tests are automated (scripted).There are numerous reasons for this, but the mostcommon one is because traditional test scripting isa complex activity that requires a lot of time andeffort. Manually, it is often too labor intensive todesign tests and their expected results for thisremaining functionality.Unfortunately, because of this focus on happy-pathregression testing, state-of-the-art test automationsuffers from numerous shortcomings associatedwith the manual test design approach, includingincorrect, missed and redundant tests, ad-hoccoverage, and manual traceability. As oftenhappens when only the testing of happy paths isautomated, all the remaining functionality (i.e., theless-used functionality and usage scenarios wherebugs typically lie) are still tested using completelymanual ad-hoc approaches, which are neitheraccurate nor repeatable. Even worse, these bugsmay only be found by product users.Since the core purpose of testing is to find all flawsas efficiently and cost-effectively as possible,automating test design is the obvious solution. Inthis way, the effort of testing across multipleplatforms can be significantly reduced, whilesimultaneously increasing the quality of the testingperformed.Manual Test Design LimitationsAs described above, one of the most fundamentalproblems with current state-of-the-art multiplatform testing is that test design remains amanual activity. Test design involves makingdecisions on: What to test and what not to testHow to stimulate the system and with whatdata valuesHow the system should react and respondto the stimuli (test oracle)Test design is a separate task from test execution,and is done before executing the tests against thesystem.So what is the problem with doing test designmanually?First of all, manual test design cannot guaranteesystematic, known, and repeatable coverage of thesystem behavior. This already presents a huge risk.With manual test design, it is difficult to assess thequality of your testing efforts, which often leadsone to evaluate the quality and progress of themanual test design process using spurious metricssuch as the number of test cases or the number ofhours spent testing. Neither of these metricsdirectly correlates with application test coverageand test quality.Additionally, manual test design is a very expensiveand time consuming process, especially when thereare changes in the requirements. In practice, testengineers are forced to manually analyze each ofthe test cases individually in order to see which testcases need to be updated and reworked; whichneed to be removed: and which need to be addedto fill the coverage gap when there are changes inthe requirements. The manually written test scriptsneed to be updated both when the requirementschange and also when some implementationdetails change. All this erodes productivity and is Conformiq 2015. All company names, trademarks and registered trademarks are the property of their respective owners.

extremely error-prone, especially when releasedates dictate very short testing times.By automating test design, testing efforts can besignificantly reduced, while at the same time, thequality of the testing can be increased.Introducing Conformiq 360 Test AutomationThe most effective way to automate test design isfrom the top down. Conformiq provides the mostsophisticated and comprehensive automated testdesign solution in the industry. Its uniqueConformiq 360 Test Automation technologyenables the next generation of testing for complex,system-level testing environments.It includes a model-based testing (MBT) approach,driven by system models. System-model driven,next-generation MBT solutions, such as ConformiqDesigner and Conformiq Creator , do notcompete with existing test automation solutions,instead they are complementary. Conformiq toolscan directly leverage existing investments in testautomation and directly integrate with crossplatform test execution infrastructures, meaningthat test cases generated by Conformiq can bedirectly executed against multiple different devicesand platforms without additional manual efforts.Conformiq 360 Test Automation also makes itpossible to perform testing either on the actualtarget device and/or in a simulated environment.Automating test design using Conformiq tools hasnumerous benefits, some of which are listed below.Improved quality and fault detection – Anautomated approach to test design lowers the riskof having incorrect, missed, and redundant tests.An engineer can, for example, accidentally miss atest case that is dictated by the requirements foran error handling case, a limit value of a dataparameter, or an expiration of a rarely activatedtimer. An algorithmic approach to test designeliminates randomly incorrect tests. This yieldsfewer missing tests because the algorithm does notaccidentally miss corner cases. There are fewerredundant test cases because the resulting test setsare rigorously optimized by a computer andchecked for relevance. Tests are always related tothe requirements, the quality of the generated testsuite is always measurable, and the entire processis systematic and repeatable.Reduced time and cost – Creating a system modelfor Conformiq software is straightforward and lesserror prone than describing the tests themselves.One model can be used to generate multiple testsuites for different purposes, including theexpected results of execution (test oracle), savingboth time and money.Improved traceability – Using Conformiq softwarehelps ensure that none of the functionalrequirements have been ignored in test casedesign, gives a rationale on why all tests weregenerated, helps in understanding tests, and assistsin post-execution analysis of tests to pinpointwhich feature(s) were actually malfunctioning.Improved maintenance – The maintenance effortfor Conformiq models are significantly reducedcompared to a traditional test case approach. Forchanges, once the tests are regenerated, theConformiq tools generate an impact analysis withincremental traceability. The tools directly reportwhich of the test cases were removed, which wereadded, and which became redundant. In this way,the regression test suite is optimized because it nolonger includes useless test cases. This caneliminate up to 40% of regression test cases that,over time, have been added but never eliminatedwhen design changes made them no longerrelevant. With automated test design, testers knowwhy they have each test case and what each onecovers. Conformiq 2015. All company names, trademarks and registered trademarks are the property of their respective owners.

Conformiq 360 Test Automation SetupWith Conformiq tools, instead of manuallydesigning test cases, a test engineer or testarchitect creates an abstract model of the systemunder test (SUT). He or she essentially takes thespecification or requirement document andencodes it into a computer readable graphicalmodel. A Visio model is an example of a graphicalrepresentation of a specification. Before generatingtest cases, the test engineer states his or her goalswith regard to what the test generation tool shouldproduce for the test suite. Next, he or she definesand configures the test selection heuristics such asrequirements, paths, pair-wise, boundary value,and others. Then these test cases can beautomatically generated without humaninvolvement.The output of the test generation is a collection oftest cases, a coverage report, a message sequencechart of the test steps, graphical coverage of eachtest case for the model, and a traceability matrix,plus executable scripts for automation. Thecoverage report gives valuable information on howwell the generated test cases cover the model withrespect to coverage criteria, and gives informationon the quality of the test suite. The traceabilitymatrix gives the linkage between the model andthe requirements. The test cases list all steps anddata in the user’s format and language.Conformiq tools directly integrate with testexecution infrastructures, meaning that Conformiqgenerated test cases can be readily executedagainst multiple devices and platforms via thecross-platform test execution automationinfrastructure of your choice. Conformiq 2015. All company names, trademarks and registered trademarks are the property of their respective owners.