WIXLIB

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack Upmainframe because of back-end application modernization investments, enterprises often requirecontinuous delivery pipelines and testing to be on mainframe too. Players like Compuware withTotal Test are leading testing solutions for DevOps automation on the mainframe.›› AI and ML to better test and automate. AI and machine learning (ML) bring new technologies tobear, like natural language understanding, text understanding, voice, video and image recognition,super-efficient pattern matching, and the ability to make predictions. But all require big data.Testers in services organizations like Accenture, Brillio, Cognizant, HCL Technologies, Infosys, IBMGlobal Services, Infostretch, TCS, Tech Mahindra, and Wipro are leveraging AI and ML to makebetter decisions and optimize automation. And new startups like Applitools, accelQ, Diffblue,Functionize, and Sofy.AI are making testing more autonomous.OFTA Evaluation OverviewTo assess the state of the OFTA market and see how the vendors stack up against each other,Forrester evaluated the strengths and weaknesses of top OFTA vendors. After examining pastresearch, user need assessments, and vendor and expert interviews, we developed a comprehensiveset of evaluation criteria. We evaluated vendors against 26 criteria, which we grouped into three highlevel buckets:›› Current offering. We evaluated each tool’s operating environment and technology breadth,ranging from mobile to desktop, software-as-a-service (SaaS), device clouds, browsers, packagedapps, and even mainframe (the latter two to a smaller degree but still relevant for enterpriseDevOps). We also evaluated front- and back-end automation and increased our focus on design,TDD, and behavior-driven development (BDD) testing; UI- and API-led execution automation withcontinuous integration/continuous delivery (CI/CD); and various other technology combinations.We then looked at tool qualities like reuse, maintenance, and target personas addressed; thirdparty integrations with nonfunctional testing tools; and real-time monitoring for connecting userexperience with testing.›› Strategy. We reviewed each vendor’s product road map to assess how it plans to differentiate itstools in the continuous testing market, support Agile and DevOps operating models, and preparefor the future of AI. Other evaluation criteria included pricing strategies, technology partnerships,and number of system integrator (SI) partners with relevant OFTA deployments in field.›› Market presence. To determine each vendor’s market presence, we evaluated the growth of itsinstalled base of OFTA clients and product revenue in the past 12 months as well as its geographicsales reach and distribution. 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-73786

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpEvaluated Vendors And Inclusion CriteriaForrester included 15 vendors in the assessment: Conformiq, Eggplant, Experitest, IBM, Micro FocusSilk, Micro Focus UFT, LogiGear, Microsoft, Parasoft, Perfecto, Ranorex, Sauce Labs, SmartBearSoftware, Tricentis, and Worksoft. Each of these vendors has (see Figure 4):›› Omnichannel testing capabilities. All of the tools enable functional automation testing acrossat least three of the four most popular browsers: Chrome, Firefox, Internet Explorer, and Safari.They can also execute automation on mobile phones running iOS and Android as well as mobilebrowsers on these.›› Design, UI, and API omnichannel functional testautomation. Vendors had to deliver at least one ofthree capabilities: UI automated testing, the abilityto automate design, and the ability to create andexecute API and web services tests. The tool alsohad to address at least one of the three testingpersonas identified: business, technical, anddeveloper.›› Customer references. Vendors provided contactinformation for at least three customers that agreedto be surveyed by Forrester about their use of theOFTA tools.Functional designautomation, breadth oftechnologies, testingpersonas, and shiftingleft and right testing incontinuous delivery arekey differentiators in OFTA.›› Presence in client inquiries and/or a tool that put it on Forrester’s radar. Forrester clients oftendiscuss these vendors and platforms through inquiries. Alternatively, the vendor may, in Forrester’sjudgment, warrant inclusion or exclusion in this evaluation because of technology trends or itsmarket presence. 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-73787

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpFIGURE 4 Evaluated Vendors: Product Information And Selection CriteriaVendorProduct evaluatedConformiqConformiq 360 Test Automation, consisting of-Creator (3.1.1) andTransformer (1.4)EggplantEggplant AI v2.1, Eggplant Functional 18.1, Eggplant Performance v19,Eggplant Network v3, Eggplant Automation Cloud v6, Eggplant Manager v6,Eggplant RUM (SaaS), Eggplant Monitoring (SaaS), Eggplant PerformanceAnalyser (SaaS)ExperitestSeeTest continuous testing platform for web and mobile apps v.11.7IBMRational Test Workbench 9.2LogiGearTestArchitect 8.3 Update 4Micro FocusSilk 18.5Unified Functional Testing 14.03MicrosoftVisual Studio Team Services (VSTS), Team Foundation Server (TFS), VisualStudio App Center (VSAC)ParasoftSOAtest 9.10.4Test Automation ToolPerfectoContinuous Quality LabRanorexRanorex Studio 8.1.1Sauce LabsContinuous Testing CloudSmartBear SoftwareTestComplete 12.5, ReadyAPI 2.3, CrossBrowserTesting, QAComplete,TestLeftTricentisTricentis Tosca 11.2, Tricentis FloodWorksoftCertify 10, Certify Execution Manager 10, Certify Business ProcessProcedure (BPP) 10, Certify Impact for SAP 10 and Analyze 10 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-73788

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpFIGURE 4 Evaluated Vendors: Product Information And Selection Criteria (Cont.)Vendor inclusion criteriaOmnichannel testing capabilities. All of the tools enable functional automation testing across at leastthree of the four most popular browsers: Chrome, Firefox, Internet Explorer, and Safari. They can alsoexecute automation on mobile phones running iOS and Android as well as mobile browsers on these.Design, UI, and API omnichannel functional test automation. Vendors had to deliver at least one ofthree capabilities: UI automated testing, the ability to automate design, and the ability to create andexecute API and web services tests. The tool also had to address at least one of the three testingpersonas identified: business, technical, and developer.Customer references. All of the participating modern application delivery functional test automationvendors provided contact information for at least three customers that agreed to be surveyed by Forresterabout their use of the omnichannel functional test automation (OFTA) tools.Presence in client inquiries and/or a tool that put it on Forrester’s radar. Forrester clients oftendiscuss these vendors and platforms through inquiries. Alternatively, the vendor may, in Forrester’sjudgment, warrant inclusion or exclusion in this evaluation because of technology trends or its marketpresence.Vendor ProfilesThis evaluation of the OFTA market is intended to be a starting point only. We encourage clients toview detailed product evaluations and adapt criteria weightings to fit their individual needs through theForrester Wave Excel-based vendor comparison tool (see Figure 5 and see Figure 6). Click the link atthe beginning of this report on Forrester.com to download the tool. 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-73789

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpFIGURE 5 Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018Omnichannel Functional Test Automation ToolsQ3 rongercurrentofferingEggplantTricentisMicro Focus UFTExperitestParasoftMicro Focus SilkSmartBear SoftwareIBMConformiqPerfectoLogiGearMicrosoftSauce LabsWorksoftRanorexWeakercurrentofferingWeaker strategyStronger strategyMarket presence 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-737810

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpCurrent OfferingSilcu ksroUsoFTftcusFoMicroFoMicrroeaiciGMLogForw resei tegh r’stingConfoEg rmigp qlaEx ntperitesIBtMFIGURE 6 Forrester Wave : Omnichannel Functional Test Automation Tools Scorecard, Q3 201850%2.65 4.11 3.33 3.38 2.40 2.98 3.74 2.285%1.00 4.00 4.00 3.00 3.00 3.00 4.00 4.00GUI design and automation15%1.50 2.50 4.50 3.50 3.00 4.00 4.00 2.50API testing and automation15%3.00 5.00 2.00 5.00 2.00 3.00 3.00 2.00Test and automation design15%5.00 4.00 2.00 2.00 3.00 1.00 2.00 2.005%3.00 5.00 1.00 3.00 3.00 1.00 3.00 1.00Automation execution/continuous testing15%3.00 5.00 5.00 3.00 3.00 3.00 5.00 3.00Qualities and integration15%1.80 4.40 4.00 3.00 2.00 4.00 3.60 3.00Mainframe and device testing15%2.00 3.50 3.00 4.00 1.00 3.50 5.00 1.00Strategy50%4.00 4.00 3.00 3.50 1.00 3.00 3.00 3.00Product road map50%5.00 5.00 3.00 3.00 1.00 3.00 3.00 1.00Pricing strategy25%3.00 3.00 3.00 3.00 1.00 3.00 3.00 5.00Partner strategy25%3.00 3.00 3.00 5.00 1.00 3.00 3.00 5.000%4.20 2.60 4.60 2.60 1.80 2.60 2.60 2.60Installed base40%5.00 3.00 5.00 3.00 1.00 3.00 3.00 3.00Product growth40%5.00 3.00 5.00 1.00 3.00 1.00 1.00 1.00Sales reach20%1.00 1.00 3.00 5.00 1.00 5.00 5.00 5.00Operating environmentsPackaged application testingMarket PresenceAll scores are based on a scale of 0 (weak) to 5 (strong). 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-737811

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpForw resei tegh r’stingParasoPe ftrfecRa tonoreSa xuceSm LabsartBeaTricen r SotftwW isaroreksoftFIGURE 6 Forrester Wave : Omnichannel Functional Test Automation Tools Scorecard, Q3 2018 (Cont.)Current Offering50%3.30 2.54 1.65 2.49 3.29 3.53 2.285%4.00 3.00 2.00 4.00 5.00 5.00 3.00GUI design and automation15%3.00 4.50 1.50 3.00 4.00 3.00 2.50API testing and automation15%5.00 1.00 3.00 1.00 4.00 5.00 1.00Test and automation design15%3.00 2.00 1.00 1.00 3.00 3.00 1.005%1.00 1.00 1.00 1.00 1.00 5.00 5.00Automation execution/continuous testing15%3.00 3.00 1.00 5.00 3.00 3.00 3.00Qualities and integration15%3.80 2.60 2.00 2.40 3.40 3.20 2.00Mainframe and device testing15%2.50 2.50 1.50 2.50 2.50 3.00 3.00Strategy50%4.00 3.00 1.50 3.50 3.50 3.50 3.00Product road map50%5.00 3.00 1.00 3.00 3.00 3.00 3.00Pricing strategy25%3.00 3.00 1.00 5.00 5.00 5.00 3.00Partner strategy25%3.00 3.00 3.00 3.00 3.00 3.00 3.000%3.80 3.40 3.40 3.40 2.60 4.60 2.60Installed base40%3.00 5.00 5.00 3.00 3.00 5.00 3.00Product growth40%5.00 3.00 3.00 5.00 3.00 5.00 3.00Sales reach20%3.00 1.00 1.00 1.00 1.00 3.00 1.00Operating environmentsPackaged application testingMarket PresenceAll scores are based on a scale of 0 (weak) to 5 (strong). 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-737812

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack UpLeaders›› Eggplant leverages AI and automation to test UX and business outcomes. In the past 18months, Eggplant evolved and improved in three important aspects: It developed a scriptless AImodel-based language, extended functional testing to include early steps of customer experiencetesting, and addressed its API testing weakness. Eggplant is connecting its recently acquiredperformance management and user experience (UX) monitoring technology to functional testing,giving testers the opportunity to model business outcomes with Eggplant AI and validate thembefore releasing the apps into production. Customers using Eggplant are very positive about theextended testing persona options offered, the collaboration possible among them, the level ofreuse the tool provides, and the levels of automation they achieve (60% to 70%).While the AI technology and algorithms acquired and developed internally are part of its road map,Eggplant has yet to determine how to price these features simply, which is a concern we heardfrom customers. Eggplant also has relatively weak global sales reach and needs to improve its SIpartner strategy to expand its customer footprint.›› Parasoft strengthens its developer focus and makes API testing easier. Parasoft hastraditionally been a go-to testing platform for developers. In the past 12 to 18 months, it has addedfeatures to appeal to less technical folks too. Its recent addition of a specific UI enables testerswith no coding experience to easily generate and manage API tests — all facilitated by underlyingML. Regarding AI, Parasoft has an impressive and concrete road map to increase test automationfrom design to execution, pushing autonomous testing. Parasoft shined in our evaluationspecifically around effective test maintenance, strong CI/CD and application lifecycle management(ALM) platform integration, as well as reporting through its analytics system PIE. Clients like therecent changes, and all reference customers reported achieving test automation of more than 50%in the past 12 months.The recent tool updates are important because customers complain that the tool is relatively hardto master. Customers often need Parasoft specialists’ support to leverage its full potential. Pricingwould benefit from flexibility for gradual enterprise adoption. Parasoft also has some work to do inaddressing front-end mobile testing, which it only offers through partners adding little value-add.›› Tricentis extends its end-to-end testing capabilities and gives a nod to developers. Tricentishas a multipronged approach to continuous testing, addressing both the packaged and the customdevelopment worlds. Its tool, Tosca, offers a strong scriptless environment, which is well designedwith nicely engineered wizards that help design and automate tests quickly and effectively ina technology-independent way. Tricentis also created an easy-to-use UI for nontechnical folksto test APIs and strengthened its integration with Jira to support CI/CD. Although not part ofthis evaluation, the very recent acquisitions of QASymphony for test management automationand the test data management company Q-Up will make Tosca a more complete end-to-endtesting platform. Its mobile testing strategy extends through cloud device farm partners for 2018 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law.Citations@forrester.com or 1 866-367-737813

For Application Development & Delivery ProfessionalsJuly 26, 2018The Forrester Wave : Omnichannel Functional Test Automation Tools, Q3 2018The 15 Providers That Matter Most And How They Stack Updistributed mobile testing on multiple types of devices and OSes. Clients appreciate the improvedtransparency and simplicity of Tosca pricing. Tosca also experienced strong market growth —thanks to its acquisitions especially.Tricentis needs to strengthen the mobile device side of its omnichannel offer and make sure itsclients get more added value directly from Tosca rather than just a loose third-party integration.Despite its strong scriptless approach, Tosca does not yet offer a competitive TDD or BDDsolution. Improvements are on the road map. As testing shifts right, and not just left, Tosca lacksin integration and user monitoring tools in testing. Although it’s a conscious decision, Tosca doesnot support full-stack developers who want tight integration with their integrated developmentenvironment (IDE) and full code scripting.Strong Performers›› IBM keeps its focus on the heterogeneous enterprise for DevOps at scale. For broad, complexenterprise-at-scale testing, IBM is front and center. Its mobile strategy focuses on assuring mobilecloud labs’ independence, offering freedom of choice to its customers in addition to its partnershipwith Perfecto. By including mainframe in its DevOps and cloud strategies, IBM’s test automationincorporates both terminal emulation and back-end API testing. API testing and test assets reuseshined in this year’s evaluation. Product evolution for IBM revolves around four main themes:integration, ease of use, openness, and enterprise-grade testing. Long-term clients highlightedgreat and consistent customer support.Mobile independence unfortunately comes with weak front-end mobile testing (e.g., testingfeatures based on sensors). IBM’s road map focuses mostly on tactical improvements: extendingtechnologies and protocols and expanding on SAP — with some exciting innovation but very longterm. The average automation level across referenced clients was less than 50%.›› SmartBear Software extends its testing personas beyond developers. SmartBear Softwarecontinues to expand its breadth of testing personas beyond technical and developer testers andhas had great success with its SoapUI open source offering. SmartBear Software offers superiorcapabilities in API testing and coding-based scripting. It also provi

Conformiq, Experitest, Sauce Labs, Micro Focus Silk, and Perfecto offer strong-performing options. Microsoft and Worksoft are competitive for some features, and LogiGear and Ranorex lag behind. App Dev Pros Are Looking For Omnichannel Functional Test Automation OFTA adoption is increasing because more AD&D