Transcription
Test Automation inDevOpsMOVING TOWARDS CONTINUOUS TESTING1COPYRIGHT UTOPIA SOLUTIONS, INC.
About Me CTO of Utopia Solutions – QA/testing focused service provider Involved in software quality and testing for over 25 years Most of that time focused on test automation Passionate about learning and helping organizations benefit from doingautomation the right lutions.com@USI com/blog12COPYRIGHT UTOPIA SOLUTIONS, INC.
Agenda Intro What is DevOpsTesting in DevOpsContinuous TestingTest Automation in DevOps Effective Test Automation Scope and Approach Test Environment and Data Management Testing in Production3COPYRIGHT UTOPIA SOLUTIONS, INC.
What is DevOps?4COPYRIGHT UTOPIA SOLUTIONS, INC.
DevOpsSoftware engineering culture and practice aimed at unifyingsoftware development (Dev) and software operation (Ops)Shorter development cyclesIncreased deployment frequencyHigher quality releasesClose alignment with business objectivesWikipedia https://en.wikipedia.org/wiki/DevOps5COPYRIGHT UTOPIA SOLUTIONS, INC.
The DevOps Cycle- Dan testing-in-devops/6COPYRIGHT UTOPIA SOLUTIONS, INC.
A Delivery onmentDeployProductionPerformance /LoadEnvironment7COPYRIGHT UTOPIA SOLUTIONS, INC.
DevOps TermsContinuous IntegrationFrequent integration of developers’ code into a shared repositoryContinuous DeliveryDevelop, build, test and release software in short cyclesContinuous Deployment Automated deployment of qualified application features toproductionContinuous TestingStay tuned!Continuous Processes DevOps8COPYRIGHT UTOPIA SOLUTIONS, INC.
State of DevOpsAspect of Delivery PerformanceEliteHighMediumLowDeployment FrequencyOn-demand(multipledeploys per day)Once per hourto once per dayOnce per weekto once permonthOnce per weekto once permonthLead Time for Changes (commit toproduction)Less than onehourOne day to oneweekOne week toone monthOne month tosix monthsTime to Restore Service (outage orimpairment)Less than onehourLess than onedayLess than onedayOne week toone monthChange Failure Rate (% of changesrequiring rollback, hot fix, patch, etc.)0 to 15%0 to 15%0 to 15%46 to 60%2018 Accelerate State of DevOps Report9COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing in DevOps10COPYRIGHT UTOPIA SOLUTIONS, INC.
Traditional Testing in DevOps?PlanBuildTestReleaseTesting is PART of the dev cycle11COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing in DevOpsWhere does testing fit?12COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing in DevOpsTesting spreadsTHROUGHOUT thedevelopment cycle and becomeseveryone’sresponsibility- Dan testing-in-devops/13COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing Across the Delivery PipelineEnd to end testsExploratory testsDeveloperWorkstationCode analysisUnit testsIntegration roduction testsPerformance /LoadEnvironmentPerformance testsSecurity tests14COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing in DevOps FastDevOps testing activities must provide Accurate feedback Actionable15COPYRIGHT UTOPIA SOLUTIONS, INC.
Continuous Testing16COPYRIGHT UTOPIA SOLUTIONS, INC.
Continuous TestingRunning the right tests at the right time to assess the risk of promoting a build throughthe delivery pipelineRisk ?DeveloperWorkstationRisk ?DeployIntegrationEnvironmentRisk ?DeployProductionRisk ?Performance /LoadEnvironment17COPYRIGHT UTOPIA SOLUTIONS, INC.
Successful Continuous Testing Provides fast, accurate and actionable feedback Avoids creating a bottleneck in the delivery pipeline Is embedded throughout the development process (shift Left AND shift Right) Includes effective test automation practices Promotes effective manual testing activities Is executed in stable test environments with valid test data Continually reviews and refines test suites to ensure relevance and efficiency18COPYRIGHT UTOPIA SOLUTIONS, INC.
Continuous Testing vs. Test AutomationTest AutomationPass/fail results associatedwith user stories andbusiness requirementsContinuous TestingBusiness risk associated withreleasing the softwarePrimary DifferencesRiskTests built with businessrisk in mindBreadthTests that detect issues thatimpact user experienceTimeTests that provide immediatefeedback and are executed asclose as possible to appelement they are testing- Adapted from Continuous Testing vs Test Automation by Wayne Ariola19COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Automation in DevOps20COPYRIGHT UTOPIA SOLUTIONS, INC.
Importance of Effective AutomationContinuous Integration requires three capabilities: A comprehensive and reliable set of automated tests that validate we are in adeployable state A culture that “stops the entire production line” when our validation tests fail Developers working in small batches on trunk rather than long-lived featurebranchesThe DevOps Handbook (Kim, Humble, Debois & Willis)21COPYRIGHT UTOPIA SOLUTIONS, INC.
Automation Impact on DevOps PerformanceType of Manual ActivityEliteHighMediumLowConfiguration s5%10%30%30%Change Approvals10%30%75%40%- 2018 Accelerate State of DevOps Report22COPYRIGHT UTOPIA SOLUTIONS, INC.
Current State of Test AutomationSome alarming statisticsThe level of test automation is below 20%2017-2018 WorldQuality ReportThe value generated by test automation is largelyunchangedAlmost 80% of test automation implementationsfail232017-2018 WorldQuality ReportUnknownCOPYRIGHT UTOPIA SOLUTIONS, INC.
Obstacles to Test Automation Success Too much / not enough automation Automating the wrong testsCommon AutomationTraps Focusing on UI tests Unstable test environments and data Not treating automation as code24COPYRIGHT UTOPIA SOLUTIONS, INC.
Effective Automation25COPYRIGHT UTOPIA SOLUTIONS, INC.
Pillars of Effective AutomationScopeApproach / MethodEnvironment & Data26COPYRIGHT UTOPIA SOLUTIONS, INC.
SCOPE AND APPROACH27COPYRIGHT UTOPIA SOLUTIONS, INC.
Pipeline TradeoffsIncreasing Confidence in Build’s Production ReadinessEnvironments Become More onEnvironmentDeployProductionPerformance /LoadEnvironmentFaster Feedback- Continuous Delivery by Jez Humble and David Farley28COPYRIGHT UTOPIA SOLUTIONS, INC.
Which Test Suites Do We Need?Test Suite Canvas for evaluating test suitesPurposeDependenciesConstraintsPipeline / ExecutionEnvironment / DataWhat business riskdoes this test suitemitigate?What systems ortools must be inplace for this testsuite to runsuccessfully?What can prevent usfrom implementingthis test suite in anideal way?Will this test suite bepart of a pipeline?What environmentwill the test suite runin?When will it betriggered?How often will itrun?How will test data bemanaged?Ownership and ResponseMaintainabilityEffectivenessWho will create the test suite?What will the process be to reviewcode?How will we know the test suite iseffective?Who should own it?Who will address test failures andhow?What documentation needs toexist?- Ashley Hunsberger (adapted from original concept by Katrina Clokie)29COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Suites Mapped to the PipelineSuitePurposeUnit TestsIs the change beingpushed ready for a codereview?IntegrationTestsAre the integrated systemcomponents ready forfurther testing?End-to-EndTestsIs the system functionallyready for deployment?PerformanceDoes the system meetperformance SLAs? Commit ChangesPerf.TestsUnitTestsDeploy toPerformance Env.Deploy toIntegration Env.Int.TestsE2ETestsDeploy toProduction30COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Scope: Risk vs. SpeedCoveragevs.SpeedFeedback Loop31COPYRIGHT UTOPIA SOLUTIONS, INC.
Which Tests Do We Need?Using FITR to assess your automated tests for inclusionFocusedTest is tied as close to the functionality being tested as possible UI,API, unit, etc.InformativeTest clearly and concisely communicates its intent and resultTrustworthy Test executes reliably and doesn’t provide false negatives / positivesRepeatableTest can be executed on demand environment and data dependencies- Bas files/volume-issue/pdf/V20I3.pdf32COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Pyramid- Mike Cohn, Mountain Goat Software“All models are wrong, but some are useful”George Box, Statistician33COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Pyramid Re-imagined- Noah Sussman34- Katrina ClokieCOPYRIGHT UTOPIA SOLUTIONS, INC.
Or Not a Pyramid at All Gives equal importance to eachlayer Removes the focus on solelyautomation Highlights that tools and skills mustbe used appropriately throughout- John Stevenson35COPYRIGHT UTOPIA SOLUTIONS, INC.
Continuous Testing Improvement If an automated acceptance test fails - investigate writing a unit test to catch theproblem If a defect is discovered in exploratory testing - write an automated unit oracceptance test as appropriate If an automated test generates a false positive error - fix the test or remove itIf tests are unreliable, they will lose their privilege of being in thebuild pipeline and eventually decay into uselessness
ENVIRONMENT AND DATA37COPYRIGHT UTOPIA SOLUTIONS, INC.
Biggest Challenge to Automation?Test environment and data availability continue to be topchallenges to achieving desired levels of automation- 2018-19 World Quality ReportWell it worked on my machine!38COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Environment and Data SpectrumStaticSharedStaleEnvironment / Data Issues Data staleness and instability Scheduling conflicts Not production-likeDynamicDedicatedFresh Automation AverseUnreliable executionLong/complex setupFalse positivesProlonged analysis39 Automation FriendlyCreate clean environmentson demandDedicate environments tosingle purposeProduction-like (if needed)Load with clean/fresh dataCOPYRIGHT UTOPIA SOLUTIONS, INC.
DevOps Environment TechnologiesApp 1Bins/LibEvolvedPlatformsGuestOSApp 2Bins/LibGuestOSApp 3App 1App 2App 3Bins/LibBins/LibBins/LibBins/LibGuestOSContainer EngineHypervisorOperating SystemInfrastructureInfrastructureVirtual MachinesContainersCloudInfrastructureas Code40COPYRIGHT UTOPIA SOLUTIONS, INC.
Benefits to DevOps Testing / AutomationCloud / Virtual Machines / Containers Infrastructure as codeEnvironment configurations can be version controlledEnvironment state can be included with test resultsManual process becomes automated repeatable, scalable and reliableAbility to develop and test in more production-like environmentsEnvironment creation can become part of the delivery pipeline if desiredInfrastructure management processes/code can be tested via automationDestructive testing41COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Data ChallengesThree of the top test data challenges according to the World Quality Report 2017-18 Keeping test data in sync with tests Lack of test data for complex integration testing Maintaining consistent test data across systems under testInability to reliably execute automated tests on demand42COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Data Management StrategyKey Strategy Input QuestionsWhat data do your tests require?TypeValueWhat are the dimensions of therequired data?is data structured or unstructured?what specific values or class of data arerequired?Timedoes data change over time?Reuse Can data be used more than once?Volume how much do data you need?What are the data sources?43COPYRIGHT UTOPIA SOLUTIONS, INC.
Test Data Management StrategyCommon Data Management TechniquesSystem / applicationtechniquesData refreshData partitioningDedicated automation environmentsTest suite techniquesPre-test / test suite data creationPost-test / test suite data cleanupOn-demand environment creation / destructionDynamic data handlingExternal techniquesCommercial test data management solutions44COPYRIGHT UTOPIA SOLUTIONS, INC.
Dynamic Data Handling TechniquesHandle application/systemdata that changesperiodicallyLogin with Login with Login with Login with Login with Static metadata tags used in test casesLogin with ADMIN CREDENTIALS Centralized test datarepositoryMetadata tags mapped to current data valuesADMIN CREDENTIALS Administrator / Yyfht% asTest suite replaces tags with current data at runtime45COPYRIGHT UTOPIA SOLUTIONS, INC.
Dynamic Data Handling TechniquesHandle test input data thatmust be dynamic or uniqueParameterValueFirstName AUTO FirstName LastName AUTO LastName DateOfBirth AUTO DOB 35 45 Email AUTO Email mailinator.com Test cases contain data generation tagsFramework DataGeneration UtilitiesTest suite replaces parameter tags at COPYRIGHT UTOPIA SOLUTIONS, INC.
Service VirtualizationSupports stable execution for tests that require access toapplication dependencies including: Dependencies that have not yet been developed Dependencies that don’t have stable/appropriate data 3rd party dependencies that don’t allow testing Any other situation where the dependency can’t beinteracted with in a stable and predictable manner47COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing in Production48COPYRIGHT UTOPIA SOLUTIONS, INC.
Testing vs. Monitoring in ProductionKnown KnownsKnown UnknownsUnknown UnknownsTestingMonitoringProduction usersAnalysisProduction SystemQualityAssessmentTelemetry dataTest cases and monitors49COPYRIGHT UTOPIA SOLUTIONS, INC.
Key PracticesProduction Testing Not a replacement for upstream testing Tests should evolve from heartbeats to userimpacting scenarios Systems should “know” they are being testedProduction Monitoring Development needs to be aware of monitoring info Monitoring and alerting should be tested Production testing and monitoring strategy requires broad IT input Feedback (analytics/logging) should be clear and concise Testability of the system should be continuously reviewed50COPYRIGHT UTOPIA SOLUTIONS, INC.
Key Takeaways DevOps is a culture, not just processes and tools Effective test automation is required to achieve Continuous Testing in DevOps Focus on the three pillars to ensure fast, reliable and accurate feedback Scope Approach Environment and Data Management51COPYRIGHT UTOPIA SOLUTIONS, INC.
Questions ? . and Answers!Direct future questions to:Lee BarnesCTOUtopia Solutions, olutions.com@USI com/blog152COPYRIGHT UTOPIA SOLUTIONS, INC.
References Ariola, W. (2018). Continuous Testing vs Test Automation: What Testers Need to /wp-content/uploads/2018/06/Continuous Testing vs Test Automation0618.pdf Ashby, D. (2016). Continuous Testing in -testing-in-devops/ Clokie, K. (2017). Test Automation st-automation-canvas.html Clokie, K. (2017). A Practical Guide to Testing in DevOps Dijkstra, B. (2018). Support Continuous Testing with Automation. Better Software, Summer s/volume-issue/pdf/V20I3.pdf Eliot, S. The Future of Software Testing Part the-future-of-software-testing-part-two Humble, J., Farley, D. (2010) Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation Hunsberger, A. (2016). Transform Culture Using DevOps Principles. SauceCon Conference 2018.https://www.youtube.com/watch?v RBrAj9jKgX0 Kim, G., Humble, J., Debois, P., & Willis, J. (2016) The DevOps Handbook Stevenson, J. (2015). Blog Post based on MEWT4 Session titled Sigh, It’s That Pyramid Again by Richard t4-post-1-sigh-its-that-pyramid.html Wikipedia – DevOpshttps://en.wikipedia.org/wiki/DevOps World Quality Report 2017-18 World Quality Report 2018-1953COPYRIGHT UTOPIA SOLUTIONS, INC.
Continuous Integration Frequent integration of developers' code into a shared repository Continuous Delivery Develop, build, test and release software in short cycles Automated deployment of qualified application features to production Continuous Deployment Continuous Testing Stay tuned! Continuous Processes DevOps
