Transcription
EECS814: Software Quality AssuranceSix Sigma for quality assurance andimprovementJacob Schleith,Poorna Chandra Gopal,Sophie Su,Supriya Medipalli
Agenda IntroductionSix SigmaMethodologyCase study: six sigma in software engineeringLevel measurementsTeam developmentTools and technologyCase study: blending six sigma to meet business goalsComparisonCriticismsConclusion2
References C. Redzic and Jongmoon Baik, "Six Sigma Approach in Software QualityImprovement," Fourth International Conference on Software EngineeringResearch, Management and Applications (SERA'06), 2006, pp. 396-406,M. Murugappan and G. Keeni, "Blending CMM and Six Sigma to meet businessgoals," in IEEE Software, vol. 20, no. 2, pp. 42-48, March-April 2003Goncalves, F.M.G.S, et al. “A Strategy for Identifying, Classifying and PrioritizingImprovement and Innovation Actions: A CMMI Level 5 and Six SigmaApproach.” 19th Australian Conference on Software Engineering (Aswec 2008),2008, pp. 104–111."Process area (CMMI)" (2021). Wikipedia. Available athttps://en.wikipedia.org/wiki/Process area (CMMI) (Accessed: 15 July 2021)"ISACA CMMI Performance Solutions" (2021). Wikipedia. Availableat https://cmmiinstitute.com/ (Accessed: 15 July 2021)3
Introduction – quality [1/2] Quality– “Conformance to requirements”– “The losses a product imparts to the society from the time the product isshipped”– “Quality should be aimed at the needs of the customer, present and future”– “Quality is what makes it possible for a customer to have a love affair withyour product or service”– “The degree to which a set of inherent characteristics fulfills the requirements,i.e., needs or expectations that are stated, generally implied or obligatory”– “The quality of a product is its ability to satisfy, and preferably exceed, theneeds and expectations of the customers”4
Introduction – quality [2/2] Quality concept approaches– Transcendent-based– Product-based– User-based– Manufacturing-based– Valuebased
Introduction - quality improvement [1/2] Why quality improvement?“Quality is free. It is not a gift, but it is free. What costs money are in-qualitythings - all the actions that involve not doing jobs right the first time.”– 80% of the cost in a software engineering projectis commonly related to after-delivery corrections– Up to 90% of the unsatisfied customers will not dobusiness with you again, and they will not tell you– 95% of the unsatisfied customers will remain loyalif their complaints are handled fast and well6
Introduction - quality Improvement [2/2] Success factors– More satisfied and loyal customers– Reduced costs due to waste and rework– Higher productivity Improved quality connects with company prosperityImprove qualityIncrease profitsCosts decrease because of lessrework, fewer mistakes, fewerdelays, snags, better use of machinetime and materialsStay in business7ProductivityimprovesCapture themarket withbetter quality
Introduction - software quality [1/3] Software quality– Quality factors developing in the right waymatching the requirement specificationgood performance meeting customer’s expectationsfitness for use8
Introduction - software quality [2/3] Software quality in line with five approach ofquality– Transcendental view: ideal, but may neverimplement completely– User view: High quality software shall meet theuser’s needs, and have a good reliability,performance and usability9
Introduction - software quality [3/3] Software quality in line with five approach ofquality– Manufacturing view: focuses on product qualityduring production and after delivery to avoidrework– Product view: assesses quality by measuringinternal product properties– Value-based view: high quality product alwaysmeans a high cost.10
Introduction - quality improvement [1/2] Software processImprovement– Aims to have a bettercontrol in softwaredevelopment– Project divided intosmaller phases, such asrequirement analysis,planning, coding, testing,releasing, etc.11
Introduction - quality improvement [2/2] Improved by process improvement because there iscorrelation between processes and outcomes Process: a sequence of steps performed for a given purpose– provides regular method of using the same way to do the same work Process improvement focuses on defining and continuallyimproving process There are many models and techniques for processimprovement, such––––CMMI,ISO9000 series,SPICE,Six Sigma, etc.12
Six Sigma [1/4] Six Sigma– Bob Galvin the CEO of Motorola, 1980’s– Structured quantitative method– Invented for reducing defects in manufacturing– Uses statistical analytic techniques in enhancingorganization’s performances, and to improvingquality13
Six Sigma [2/4] Six Sigma: a quality improvement philosophy that focuseson eliminating defects through reduction of variation in aprocess Defect: a measurable outcome that is not withinacceptable (specification) limits Identify defects and eliminate inefficiency within theproduct lifecycle Ensures cost savings and customer satisfaction14
Six Sigma [3/4] Definition– As a metric: (“σ”) used to symbolize howmuch deviation exists in aset of data Sigma level is to showhow well the product isperforming– Software companyDefects Per Million Opportunities (DPMO) 106* D/ (N*O)D - number of defects,N - number of units producedO - number of opportunities per unit 200,000 lines of code(loc) Each loc, companyDPMO 106*191 / (200,000*1) 955performs 1 check to testquality 191 defects detected15
Six Sigma [4/4]– As a Methodology: Managing the customer requirements Aligning the processes to achieve those requirements Analyzing the data to minimize the variations in thoseprocesses Rapid and sustainable improvement to those processes Process improvement models: DMAIC, DMADV,Breakthrough strategy, Roadmap, New Six Sigma16
Six Sigma methodology DMAIC– Define, Measure, Analyze, Improve, Control– DMAIC also known as Lean Six Sigma focused onquality improvement over existing processes DMADV– Define, Measure, Analyze, Design, Verify– DMADV acts as data-driven qualitystrategy for new product development17
DMAIC pre-requisiteProcess - a series of step and procedureto deliver a service or product to customer Is there an existing process? Is there a problem affecting performances? Is the problem measurable? Does the problem affect customersatisfaction?18
DMAIC - Define Identify and define existing problemstatement through research Documenting problems discovered Benchmarking against industrial standards Gather input from various sources as regardsto the problem Continuous improvement19
DMAIC - Measure Gather measurable information Visualize the data with process mapping– Map out the cause-and-effect diagram forpattern identification Assign process variable to structuralize data Classify process variable for– Discreteness– Variation– Scale20
DMAIC - Analyze [1/2] Popular strategy to analyze the data– Probability– Statistics– Inferential Statistics Analyze distribution– Linear correlation methods– Linear Regression– Normal distribution21
DMAIC - Analyze [2/2] Reproducing the problem cause– Obtain challenges faced for the problem Investigating measureable baseline Retrieve list of loot causes for identifiedproblems22
DMAIC - Improve Brainstorm ideas for possible solution Develop improvement testing basedon discovered patterns Implement the idea and collect data toconfirm the measureable difference23
DMAIC - Control Establish control plan Ensure implemantations are maintained Standalizing the process with documentedprocedures Review and follow up New rounds of iteration for project ideas24
DMADV - Define Identify purpose of the projectResearch for customer's voiceEstablish measureable goalsCreate project management flowAssess potential risks25
DMADV - Measure Measure factors critical to qualityDefine requirement and market segmentIdentify design parametersRe-assessing risks and capability26
DMADV - Analyze Develop design alternatives Optimize requirement selection Provide comprehensive estimation of theproject lifecycle27
DMADV - Design Finalize selection alternatives Establish architecture with possible phototype Prepare detailed model to identify potentialerrors for modification28
DMADV - Verify Design is validated against stakeholderrequirements Pilot production run to ensure quality Deployment of the system29
Six Sigma in software engineering [1/7] Case study: “Six Sigma Approach in SoftwareQuality Improvement”– Six Sigma approach used for software qualityimprovement– Process and tools used in each phase of themodel(DMAIC)– Mainly targeted towards quality improvement of new Product Design, training, processes, tools, andtracking30
Six Sigma in software engineering [2/7] Define– Understand the problem, identify input andoutput variables, form the team, define goals andmilestones of the project, and understand theproject’s merit– Outputs: project charter – business case, opportunity statement,goal statement, project scope, project plan, teamselection31
Six Sigma in software engineering [3/7] Measure– Document the existing process in order to determine whatwe would measure to diagnose the problem inquantitative terms– Standard operational definitions were used for phasecontainment, software reliability– Sigma level was determined for high severity faults, fromphase containment data set, for high severity reliabilitydefects from system test, and for CRUD (CustomerReported Unique Defect)32
Six Sigma in software engineering [4/7] Analyze– investigated potential sources of variation byexamining the relationship between input andoutput variables in order to uncover potentialareas for improvement by conducting Pareto analysis correlation between release software size and defectinjected software reliability33
Six Sigma in software engineering [5/7] Improve– Gathered and statistically analyzed ideas forimprovement– Relation between injected and detected defectsand release size were identified– Solution effectiveness was verified– Cost/benefit was documented34
Six Sigma in software engineering [6/7] Improve– To identify and select potential solutions forprocess improvement phase-based defect removal model: summarized therelationship among three metrics: defects escaped fromprevious phase, defect injection, and defect removaleffectiveness tactical changes plan– Pilot plans35
Six Sigma in software engineering [7/7] Control– Team is working to ensure that the gains are sustained for thelong term– New processes are documented, and trainings are provided– Efforts include responsibilities - dedicated to key team members performance reviews - success evaluation and tracking scheduledon a monthly basis metrics for software quality tracking - Fault prediction model36
Break TimeBreak time arranged for 10 minutes37
Sigma levels Statistical calculation regards to defectper opportunities– Sigma level provides value score todetermine the quality capability of theprocess– The goal is to minimize thedeviations because deviation is theenemy of quality process37
Sigma distribution38
Sigma level measurements Sigma represents variation level onnormal distribution chart The higher the sigma level, the lower thedefect rate39
Sigma level measurements - tableSigma LevelDefect 2%620950.023%23260.00034%3.440
Sigma at level 4 99% of the time the process isoperating with acceptable results– 50 minutes unsafe drinking water a day– 5,000 Incorrect surgical operations perweek– 200,000 thousands of wrong medicalperspectives each year.– 7 hours software malfunction per month41
Benefit of Six Sigma level Provides understanding of qualitycapability in statistical way Demonstrate loss capacity Verify improvements archived42
Roles and responsibilities Yellow belt– Basic understanding of Six Sigma Green belt– Lead to support intermediate Six Sigma processimprovement projects Black belt– Manage and lead advanced Six Sigma project Master black belt– The master of the domain capable to instruct andeducate the followers43
Development team type Types of the Six Sigma team– Process improvement team– Celluar team– Self-directed team– Parallel team44
Stages of team development Development stages– Forming– Storming– Norming– Performing– Adjourning45
Tools and technology Value stream mapping JIT & KanBan Poka Yoke46
Case study: blending Six Sigma to meetbusiness goals [1/16] Many companies face challenges in aligning– Customer requirements– Process improvement goals– Estimating and monitoring capability with time effort quality48
Case study: blending Six Sigma to meetbusiness goals [2/16] For example, TCS (Tata Consultancy Services)– Developed their own QMS (Quality ManagementSystem)– TCS-QMS was structured based on SW-CMM Level 2 and 3 requirements enhanced further with concepts of Six-Sigma– Offers an organized framework with quality prepares the approach to CMMI Level 549
Case study: blending Six Sigma to meetbusiness goals [3/16] TCS - QMS framework historyYearImprovement description1993Documented software development practices, maintenance, andconversion to conform to ISO 90001995Reinforced all improvement initiatives under Tata Business ExcellenceModel umbrella1996Adopted SW-CMM to setup, define and consistently implementedmature software processes throughout the organization1998Applied Six Sigma methods to analyze problems quantitatively withproducts and processes to ensure customer satisfaction2000Adopted the People-CMM to integrate people practices to sustainSW-CMM practices and ensure employee satisfaction50
Case study: blending Six Sigma to meetbusiness goals [4/16] TCS - QMS ensures customer satisfaction by– Designing processes with product and quality ofservice– Integrating quality control activities into softwaredevelopment– Stressing quality assurance– Using metrics to manage processes and quality– Introducing new technologies51
Case study: blending Six Sigma to meetbusiness goals [5/16] Key features of TCS – QMS framework– First, provides a basis for effective projectmanagement by monitoring triple constraints functionality– Second, provides architecture for organization– Third, emphasizes data – driven management– Lastly, facilitates continuous process improvement52
Case study: blending Six Sigma to meetbusiness goals [6/16] Blending CMM and Six Sigma– Six Sigma is customer-centric & data- drivenapproach– Focuses on reducing process variations– Six Sigma & CMM together help organizations improves marketplace competitiveness achieve business goals53
Case study: blending Six Sigma to meetbusiness goals [7/16] Process management – Six Sigma way– TCS designed their metrics program by translatingcustomer needs into operational measurements– Conducted extensive analysis of process trends– Eliminated ambiguities in calculating processcapability using the sigma capability54
Case study: blending Six Sigma to meetbusiness goals [8/16] Customer - centric metrics program– Team identified the software process and productmetrics– Tracking based on QFD (Quality FunctionalDevelopment)– From the result of QFD team selects in-process metrics indicates progress based on the customer objectives55
Case study: blending Six Sigma to meetbusiness goals [9/16] Customer - centric metrics programQuality Function DeploymentFirst house ofQuality ductrequirementsuse casesTraceabilitycoding tability79333Testability59991258186246146Total of rating xrequirement56
Case study: blending Six Sigma to meetbusiness goals [10/16] Metrics analysis and management– Process variability is a key natural process variation special cause variation– For better output, reduce both the variations57
Case study: blending Six Sigma to meetbusiness goals [11/16] Metrics analysis and managementControl chart patterns for a sample process58
Case study: blending Six Sigma to meetbusiness goals [12/16] Process capability calculations– Six Sigma monitors the process using controlcharts– In terms of sigma capability, process capability is no. of standard deviations fits between mean andspecification limit59
Case study: blending Six Sigma to meetbusiness goals [13/16]Process capability in terms of Z capabilityUse of Z capability for predicting process capability60
Case study: blending Six Sigma to meetbusiness goals [14/16] Continuous improvement– Six Sigma ensures meeting goals of continuous improvements basis to DMAIC– Process mapping ensures ordering of process elements interfaces interdependencies among elements61
Case study: blending Six Sigma to meetbusiness goals [15/16] Continuous improvementProcess mapping. Implemented as a CMM Level 3 process62
Case study: blending Six Sigma to meetbusiness goals [16/16] Benefits to the organization– TCS development center has various process andtechnology enhancements– For example, it implemented a statement of work for capturing all projectrequirements estimation guidelines standard libraries error-proofing tools63
How does Six Sigma compare? How does Six Sigma compare to CMMI(Capability Maturity Model Integration)? What are the differences? When would each be appropriate?64
What is CMMI? "ISACA’s CMMI model is a proven set of bestpractices organized by critical businesscapabilities which improve businessperformance. It is designed to beunderstandable, accessible, flexible, andintegrate with other methodologies such asagile." - https://cmmiinstitute.com/65
Understanding CMMI capability levels66
How to achieve CMMI compliance Each Level of CMMI compliance is obtained bymeeting certain "Process Areas" for each level A "Process Area" is define as "A cluster ofrelated practices in an area that, whenimplemented collectively, satisfies a set ofgoals considered important for makingimprovement in that area." https://en.wikipedia.org/wiki/Process area (CMMI)67
CMMI compliance example: To achieve level 2 the following PA's must bemet:–––––––CM - Configuration ManagementMA - Measurement and AnalysisPMC - Project Monitoring and ControlPP - Project PlanningPPQA - Process and Product Quality AssuranceREQM - Requirements ManagementSAM - Supplier Agreement Management68
Six Sigma compliance No such compliance exists for Six Sigma thesame way there does for CMMI Methodology Individual certification exists on belt levels:yellow, green, black. That indicate a personexpertise in administration the Six Sigmaworkflow69
How does CMMI compare to Six Sigma? The CMMI are a set of standards that definesthe process structure. Six Sigma is a way toimprove an existing process throughquantitative and statistical analysis Six Sigma is focused on eliminating specificdefects defined by the organization toimprove customer satisfaction70
CMMI and Six Sigma together? Six Sigma would work best when incorporatedinto an organization that has reached a CMMIlevel 3 or higher. Utilization of the Six Sigma methodology ofstatistical analysis to eliminate defectcan help an organization achieve levels 4and 5.71
Case study: a CMMI level 5 and SixSigma approach Identify performance and customer/employeeneeds Analyze and align business goals andorganizational processes Establish an organizational processes structure Identify opportunities and establishperformance goals72
Criteria for classification Classification todetermine classificationeither as DMAIC,DMADV, PDCA, orSimple Action73
Criteria for prioritization Due to resourcelimitation prioritizationmust occur based onimprovementopportunities and ROI74
Mapping different opportunities75
DAR action classification76
DAR priority classification77
Criticisms Create additional overhead and unintendedred tape Create large amounts of data that need tobe analyzed and understood to be beneficial Can inhibit creativity and out-side-the-boxcritical thinking to solve problems78
Conclusion Can be very beneficial to any organizationlooking to improve the quality and eliminatedefects Originally created to find manufacturingdefects, Six Sigma has evolved to be appliedacross multiple industries and disaplines79
Questions?80
Six Sigma in software engineering [6/7] Improve –To identify and select potential solutions for process improvement phase-based defect removal model: summarized the relationship among three metrics: defects escaped
