Transcription
Test Re-prioritization in Continuous Testing EnvironmentsYuecai ZhuDepartment of Computer Scienceand Software EngineeringConcordia UniversityMontreal, QC, CanadaEmail: zhuyuecai@gmail.comEmad ShihabData-driven Analysis of Software Lab (DAS)Department of Computer Scienceand Software EngineeringConcordia UniversityMontreal, QC, CanadaEmail: emad.shihab@concordia.caAbstract—New changes are constantly and concurrentlybeing made to large software systems. In modern continuous integration and deployment environments, each changerequires a set of tests to be run. This volume of tests leadsto multiple test requests being made simultaneously, whichwarrant prioritization of such requests. Previous work ontest prioritization schedules queued tests at set time intervals.However, after a test has been scheduled it will never bereprioritized even if new higher risk tests arrive. Furthermore,as each test finishes, new information is available whichcould be used to reprioritize tests. In this work, we use theconditional failure probability among tests to reprioritize testsafter each test run. This means that tests can be reprioritizedhundreds of times as they wait to be run. Our approachis scalable because we do not depend on static analysis orcoverage measures and simply prioritize tests based on theirco-failure probability distributions. We named this approachC O DYNAQ and in particular, we propose three prioritizationvariants called C O DYNAQS INGLE, C O DYNAQD OUBLE andC O DYNAQF LEXI. We evaluate our approach on two data sets,C HROME and Google testing data. We find that our co-failuredynamic re-prioritization approach, C O DYNAQ, outperformsthe default order, F IFO BASELINE, finding the first failure andall failures for a change request by 31% and 62% faster,respectively. C O DYNAQ also outperforms G OOGLE TCP byfinding the first failure 27% faster and all failures 62% faster.Keywords-Regression Testing; Test Minimization; DynamicTest Prioritization; Test Dependency; Continuous Testing; Continuous IntegrationI. I NTRODUCTIONThe practice of releasing often and releasing early - alsoknown as rapid release - has seen wide adoption in thesoftware engineering field [1]. A key technology to enablerapid release is the use of a continuous integration (CI)process, where feedback on changes to the software aregiven quickly [5]. CI requires continuous testing of everychange which can be very expensive.For large projects, the burden of continuous testing canbe significant and lead to many thousands of tests that aretriggered daily. Prior studies have shown that for industrialprojects with as little as 100 changes, such as the videoconference system analyzed in Marijan et al.’s work [14], thetesting time may surpass 2 days. This long testing cycle canPeter C. RigbyDepartment of Computer Scienceand Software EngineeringConcordia UniversityMontreal, QC, CanadaEmail: peter.rigby@concordia.cabecome a bottleneck delaying the development and releaseof code. To address this issue, prior work, which we refer toas G OOGLE TCP, has proposed the reordering of tests aftera certain time period has passed [7].Although prior work has been able to improve the speedof testing, most studies are based on the assumption that testsare independent. In practice with G OOGLE TCP this means,that tests are prioritized once when the request is made. Weargue that the tests (or their failures) are not independentand that prior test outcomes should be used to prioritizethe continuous testing efforts. In particular, we propose thetest re-prioritization approach, C O DYNAQ, which uses theco-failure distributions of tests to dynamically reprioritizequeued tests. For example, if test A and test B co-fail in thepast 75% of the time, then if we observe a failure of test Ain the current run, we may be able to speed up the executionof test B since there would be a high probability that it willfail as well. In fact, as we show later, more than 58% of thetests in C HROME co-fail with at least another test.Our approach introduces two novel concepts: first, prioritization based on the co-failure distribution of tests, andsecond, re-prioritization after each test run. We implementthis approach as the single queue re-prioritization, C O DYNAQS INGLE . Since we continuously reprioritize tests, wefind that tests with low failure probabilities can be pushedback in the queue (i.e. “starved”) and delay final test resultfor a change. To deal with this problem, we introduce adouble re-prioritization queue, C O DYNAQD OUBLE, and aflexible re-prioritization queue, C O DYNAQF LEXI.To evaluate C O DYNAQ and compare it with the state-ofthe-art, G OOGLE TCP, we use two data sets. The first is theinternal Google testing data that Elbaum et al. [7] releasedwhen developing G OOGLE TCP. The second is scraped datafrom the C HROME project which runs as many as 149 testsper minute. The C HROME data set includes over 4.4 milliontest runs.Our goal is to speed-up the detection of test failures. Weuse the actual test request order as a baseline and compare thespeedup/slowdown we attain in finding the first failure andthe time to find all failures for each request. On both data sets
Table I: The median and maximum number of test requestsper minuteGoogleC HROMEMedian2726Maximum2,461149C O DYNAQ outperforms the state-of-the-art, G OOGLE TCP.C O DYNAQ is most effective when there are many co-failuresand the speedup compared to G OOGLE TCP can be as muchas 26.65% and 61.8% faster in finding F IRST FAIL andA LL FAIL respectively. In general, we are able to find the firstfailure and all failures approximately 11.33% and 61.84%faster than the default order (F IFO BASELINE) in C HROMEand find the first failure and all failures 31.01% and 39.60%faster in the Google data set.This paper is organized as follows. We provide backgroundon continuous testing in Section II. Section III summarizesthe related work. Section IV sets up our case study andintroduces our proposed re-prioritization algorithms andevaluation criteria. Section V, presents our preliminary results.Section VI presents the results of our case study. Section VIIdiscusses how data set characteristics can impact which reprioritization algorithms should be used. Sections VIII andIX discuss the threats to validity, future work, and concludethe paper.II. BACKGROUNDThe continuous integration process and the sheer volumeof changes at large companies, such as Google and Microsoft,derive a need for efficient test prioritization techniques. Inmost cases, the status quo is to prioritize tests based ontheir request order. We suggest that co-failure conditionalprobabilities based on the past failure distributions may leadto finding failures earlier.Multi-Request Environment: Both, Marijan et al. [14]and Elbaum et al. [7] reported heavy work loads in theproduct testing queues. We find that C HROME has a similarlyheavy load. In the C HROME project, when a new change issubmitted for review, the developer or reviewer requests aset of tests to determine the quality of a change. Usually theset of tests requested are different from those requested for adifferent change. We will refer to the set of tests requested fora new code change as a test request. In some cases, a largenumber of tests can be requested for each change leadingto slow test results and delaying the merging of changes.Long wait times for test results can be a serious issue whendevelopers need to pass quality checks before moving ontoother work.To solve this problem, multi-request test environmentsfacilitate the handling of multiple test requests at the sametime. Figure 1 shows the distribution of test requests andTable I shows the median and maximum number of testrequests per minute. As the table shows, in C HROME, aFigure 1: Number of changes submitted to the test queueper minute for Google and C HROMEmedian of 26 tests are requested per minute, and in extremecases, that number can be as high as 149 test requests. Inthe Google data, the median is 27 test requests per minute,and in extreme cases, it can be as high as 2,508 test requestsper minute.Concurrent Test Request Execution: In a multi-requestenvironment, multiple test machines are used and testsare run in parallel. On both C HROME and at Google thedefault approach is to schedule tests based on their requesttime. This common strategy is used as our F IFO BASELINEapproach. Elbaum et al. proposed to schedule tests onlybased on their assigned priority in the dispatch queue [7].Since tests are scheduled solely based on their prioritywithout considering when they are requested, test requestsare executed concurrently.We illustrate this idea in figure 2, where test requestsR1, R2, R3 are submitted in order, but the priority of theirtests T 1, T 2, T 3 is not based on a FIFO scheme. Rather, thetest execution order is based on some other priority scheme.We adopt this idea in C O DYNAQ, i.e., we also reprioritizetests based on their failure likelihood and not their arrivalorder.III. R ELATED W ORKProgram analysis based test selection and prioritizationtechniques have been extensively researched in the regression testing literature [13][18][10][21][17][15][16]. Kim andPorter [11] were the first to use historical test failuredistributions. Their approach does not consider multiple testrequests or re-prioritization. Elbaum’s et al. work was the firstto acknowledge the problems of multiple test requests andto employ historical failures. Elbaum et al. [7] proposed theG OOGLE TCP algorithm and evaluated it by the time requiredto provide feedback on failing tests. In their approach, they
Figure 2: Example of concurrent test executionprioritize tests in the scope of the whole waiting queueinstead of per request. When the prioritize window checkingcondition is met, a certain number of tests in the head ofthe waiting queue will be pulled into a prioritized queue andare scheduled to be run. This prioritized queue is referredto as the dispatch queue. They prioritize test suites in thedispatch queue based on how often a test has failed in the pastwithout considering when the test is requested. Hence, testrequests are executed concurrently and developers obtain thetest results faster than in the traditional approaches. Elbaumet al.’s approach is limited because once a test is prioritizedin the dispatch queue, it will not be reprioritized before it isrun missing co-failure information.A further problem commonly seen in real time tasksscheduling is starvation of low priority tasks. To solve thisproblem, Elbaum et al. [7] introduced an algorithm calledprioritize window checking. We further develop this approachwith double queues and flexible double queues prioritizationmodels.Marijian et al. [14] also improved test case prioritization incontinuous regression testing and used test execution time asone of the metrics to evaluate their algorithm. Their approachassumes a limit on time allotted for test execution and doesnot make use of runtime results. Furthermore they onlyprioritize tests in the scope of each change request. Comparedto our approach, they did not reprioritize tests and thusprioritization becomes non-optimal after a few tests are run.Saff et al. [20] presented a form of continuous testingwhere regression tests are run continuously as developerswrite code. This work, however, focuses on the protocol ofcontinuous testing in CI instead of the regression testingoptimization techniques. Jiang et al. [9] considered the useof test case prioritization following code commits to helporganizations reveal failures faster in continuous integrationenvironments. However their work focus on improving faultlocalization with test case prioritization techniques and hencesolves a different problem.Our work is novel in the use of co-failure distributions andre-prioritization after each test run. These simple advancesallows our approach to outperform the state-of-the-art.IV. C ASE S TUDY S ETUPOur paper presents an approach to prioritize testing ina multi-request environment. In this section, we detailthe methodology, highlighting the data, the prioritizationalgorithms, and the performance evaluation criteria used tocompare the proposed algorithms.A. Case Study DataTo perform our case study, we require test request andexecution data. We obtained data from two different projects- namely, Google’s internal test data provided by Elbaumet al. [7] and C HROME data, which we mined. We detaileach data set below.Internal Google data: Elbaum et al. [7] made the testdata used in their study publicly available. The data setprovides tests from two phases, the pre-submission and postsubmission (of commits) testing. Like Elbaum et al. we usethe post-submission testing data. The data set contains 11,457change requests that result in 847,057 test executions, whichare executed over 17 days. We also obtain the time cost andthe result of each test execution. We separate the data setinto two periods since we need two folds, a training data setto obtain the co-failure distributions and a testing data set toevaluate the re-prioritization algorithms.C HROME: C HROME has two sub-projects: the Chromiumweb browser and the operation system Chromium OS. Everysource code change in C HROME is reviewed and tested aftersubmission. Once a change is submitted, reviewers can selectthe tests that they think need to run against the change.1 All1 In most cases, these tests can be considered as test suites since they arecomposed of multiple tests
of the test execution information is recorded in their codereview tool. We mine this data to obtain 1) the test requests,2) the submission time of a test request and 3) the outcome ofeach test execution. C HROME does not store the running timefor each test. As a result, we randomly assigned a runningtime between 1 and 60 minutes. While this affects the validityof our approach for use on by C HROME developers, it has noimpact on our work from a theoretical research perspective ofevaluating our test prioritization strategies. Our C HROME dataset contains 235,917 change requests that result in 4,487,008test executions. For experimental purposes, we divided thedata into 5 six month periods, which also corresponds toC HROME’s six month release cycle.B. Prioritization AlgorithmsTo cope with the large number of test requests, prioritization algorithms are often used to facilitate effectivetest executions. In this section, we first discuss the currentstate-of-the-art algorithm used to prioritize test execution,F IFO BASELINE [7], [19], [11] and then we present threenovel algorithms that we propose to effectively prioritize testexecutions.F IFO BASELINE: The most basic algorithm is to provide noprioritization of tests at all. In such a case, tests are prioritizedbased on a First-In-First-Out criteria. This approach is verycommon and in our discussions with companies is used byEricsson, Microsoft and Google by default. When multipletests are requested at the same time, they are put into adispatch queue. In our experiments, we set the dispatchqueue to a size of 60 per machine. The order of the testsin the dispatch queue is arbitrarily determined, and in mostcases is based on the request time. We refer to this as theF IFO BASELINE. Previous work showed that, prioritizing testsrandomly is as cost-effective as advanced program analysisbased techniques [11].However, prior work by Kim et al. suggested that historicaldata from test cases can be leveraged to prioritize tests sothat they fail earlier [11]. Hence, we are motivated by theaforementioned work and the idea that test failures are notcompletely independent [22], and devise three algorithmsthat leverage this insight. Specifically, our algorithms takeinto consideration the past test co-failure distributions todynamically prioritize test execution. We adopt a dynamicscheduling protocol in real time task scheduling to performthe test re-prioritization. Full details of the original dynamicscheduling protocol can be found in Chetto et al.’s taskscheduling work [2]. Essentially, the algorithm updates thetest order in the dispatch queue based on the result (i.e.,pass/fail) of the recently completed tests in the same request.The computation of the modified priority is calculated basedon the following scoring function.Given a pair of tests t1 , t2 in the same request, the scoringfunction is given as follows:new sc previous sc (P (t2 f ail t1 f ail) 0.5)(1)If t1 is passed, then:new sc previous sc (P (t2 f ail t1 pass) 0.5)(2)If t2 is a newly added test, then we can either set itsconditional failure rate to 1 to give it the maximum priority,or 0.5 to maintain its original priority. However, since weare only interested in the performance of re-prioritizationbased on past co-failure, we did not reprioritize new testsin our simulation. Equation 1 predicts the failure rate of atest by its co-failure history, while Equation 2 is anotherway to estimate its failure rate. Algorithm 1 explains theprocedure on how to reprioritize a test based on the twoaforementioned equations. In Algorithm 1, line 2 checkswhether the input test t is a recently added test, namely noco-fail history. If t is not a new test then reprioritize t byEquation 1 at line 4 if tf inish fails or by Equation 2 at line6 otherwise. Algorithm 1 takes no action if t is a new test.The time complexity of reP rioritize() is O(1).Algorithm 1: reP rioritize(t, tf inish )input : tf inish : the test that just finished its executiont : the test that needs to be reprioritizedResult: reprioritize t based on the result of tf inish1 initialization;2 if t is not a new added test then3/* Apply Equation 1 to update thepriority of t if tf inish fails. */4if tf inish failed then t.setPriority(t.getPriority (P (t f ails tf inish f ails) 0.5)) ;5/* Apply Equation 2 to update thepriority of t if tf inish passes.*/6else t.setPriority(t.getPriority (P (t f ails tf inish passes) 0.5)) ;7/* Only reprioritize tests withpast co-fails. No priorityupdate to new added tests*/8 endC O DYNAQS INGLE: Our first enhancement is the singlequeue test re-prioritization scheme, called C O DYNAQS IN GLE , which executes tests concurrently and reprioritizestests based on their past co-failure distribution. Initially,tests that are added to the queue are simply based on theirrequest time. Then the tests are reprioritized according to thenewly generated information and their co-failure history. Thepseudo code of C O DYNAQS INGLE is shown in Algorithm 2.C O DYNAQS INGLE operates on the dispatchQueue which isa priority queue of tests that are waiting to run. We assume
Table II: Summary of Re-prioritization Algorithm FeaturesTechniquePrioritizationSchemeF IFO BASELINEArrival time FIFOG OOGLE TCPrecent failed testdistributionC O DYNAQS INGLEC O DYNAQD OUBLEC O DYNAQF LEXIPast co-failuredistributionPast co-failuredistributionPast Numberof QueuesRe-prioritizationStarvation Control1NoYes2NoNoYes, when the elapsedtime matches theprioritize windowYes1Yes, after each test runYes2Yes, after each test runYes2Yes, after each test runthat the dispatchQueue is smaller than the waitingQueue,which is the most likely practical scenario. We also assumethat two important functions are already implemented inthe test infrastructure: getF inishT ests() listen on the testexecutors and returns an array of recently finished tests;getOtherT ests(tf inish , dispatchQueue): a function takesa finished test as its argument and returns the other teststhat are in the same test request and waiting for execution.Given m the total of test executors and n the largestnumber of tests that would be included in a test request,getF inishT ests() is O(m) and getOtherT ests() is O(n).Hence C O DYNAQS INGLE is O(mn).Algorithm 2: C O DYNAQS INGLE ()Result: reprioritize the relative tests in the dispatchqueue1 initialization;2 while TRUE do3f inishT ests getF inishT ests();4for tf inish f inishT est do5otherT ests getOtherT ests(tf inish , dispatchQueue);6for t otherT ests do7reP rioritize(t, tf inish );8end9end10 endDuring the simulation on the C HROME and Google data,we found that in some cases tests face the starvation problemin C O DYNAQS INGLE. Some tests are highly unlikely tofail, these tests are assigned a low priority and since newtests request constantly arrive, these low failure probabilitytests will never be run. In essence, tests that have a higherpriority are always jumping the line and being executed, whiletests that have a low priority are being ‘starved’ and beingconstantly pushed to the back of the queue. This would notbe a problem if these low priority tests never fail. However,in cases where they do actually fail, then this behaviour isNoYes, after the thedispatch queue is emptyYes, after the size of thedispatch queue isbelow a thresholdundesirable since the developer requesting the test would nothave any feedback for a long time. Therefore we developour second algorithm, the C O DYNAQD OUBLE.C O DYNAQD OUBLE: To address the aforementioned starvation problem, we propose another prioritization algorithmcalled C O DYNAQD OUBLE. In C O DYNAQD OUBLE, we limitthe capacity of the dispatch queue and add another FIFOwaiting queue to temporally store the newly arriving testrequests.When new tests are requested, they are stored in the waitingqueue. After the dispatch queue is cleared, we fill it withtests in the head of the waiting queue. Then we execute testrequest concurrently and reprioritize tests in the dispatchqueue the same way as in C O DYNAQS INGLE. The pseudocode of C O DYNAQD OUBLE is shown in Algorithm 3. Incontrast with C O DYNAQS INGLE, the dispatchQueue is apriority queue with predefined capacity of tests that areprioritized and waiting to run. C O DYNAQD OUBLE usesthe waitingQueue, a FIFO queue of tests, to hold therequested tests before they are added to the dispatchQueuefor prioritization. In line 3 to 6, when C O DYNAQD OUBLEfinds that the dispatchQueue is cleared, it pulls tests fromthe waitingQueue and adds them to the dispatchQueue.Once a test is finished, the reprioritize algorithm is the sameas C O DYNAQS INGLE. Hence C O DYNAQD OUBLE is O(mn)as well.The approach that controls the size of the prioritizeddispatch queue is very similar to the prioritization windowchecking approach in Elbaum et al.’s G OOGLE TCP [7]. Themain difference however, is that in our approach when thedispatch queue is empty, only a few waiting tests will beadded to the dispatch queue based on the dispatch queuecapacity, while in G OOGLE TCP, all waiting tests will beadded to the dispatch queue and prioritized. Although ourproposed algorithm helps reduced the starved tests in oursimulation, the improvement is not substantial. Therefore wedevelop our third model that we describe next.C O DYNAQF LEXI: While C O DYNAQD OUBLE reduces teststarvation, we found that many tests still experience substantial delays. In order to minimize the number of delayed test ex-
Algorithm 3: C O DYNAQD OUBLE ()Result: reprioritize the relative tests in thedispatchQueue; Fill the dispatchQueue withtests from the waitingQueue when thedispatchQueue is cleared1 initialization;2 while TRUE do3if dispatchQueue is empty then4while dispatchQueue is not full waitingQueue is not empty nd8f inishT ests getF inishT ests();9for tf inish f inishT est do10otherT ests getOtherT ests();11for t otherT ests do12reP rioritize(t, tf inish );13end14end15 endAlgorithm 4: C O DYNAQF LEXI ()input : p: the prioritize window ranges between 0 to 1Result: reprioritize the relative tests in the dispatchqueue; Fill the dispatchQueue with tests fromthe waitingQueue when the size ofdispatchQueue is smaller than p c1 initialization;2 while TRUE do3if dispatchQueue.size p dispatchQueue.capacity then4while dispatchQueue is not full waitingQueue is not empty nd8f inishT ests getF inishT ests();9for tf inish f inishT est do10otherT ests getOtherT ests();11for t otherT ests do12reP rioritize(t, tf inish );13end14end15 endecutions, we add a prioritization window to C O DYNAQD OU BLE and develop C O DYNAQF LEXI . In C O DYNAQF LEXI ,if the size of the dispatch queue is less than the prioritizewindow which is the product of a predefined threshold timesthe dispatch queue capacity in our implementation, the testsin the head of the waiting queue will be prioritized randomlyand added to the dispatch queue until the dispatch queueis full. The prioritize window is the size at which we addwaiting tests to the dispatch queue. This allows us to addtests to the dispatch queue and achieve a tradeoff betweenstarving (low priority) tests and getting higher gain for highlyrisky tests, with high priority. Once a test is finished, allother tests that are in the same request and waiting in thewaiting queue or dispatch queue, are reprioritized in the sameway as in the previous two algorithms. Algorithm 4 displaysthe steps of C O DYNAQF LEXI. The only difference betweenC O DYNAQF LEXI and C O DYNAQD OUBLE is the conditionon which to trigger the filling of the dispatchQueue. In line3 of Algorithm 4, when the number of prioritized tests in thedispatch queue is less or equal to the product of the prioritizewindow p and dispatch queue capacity c, C O DYNAQF LEXIfills the dispatchQueue with tests from the waitingQueue.The time complexity of C O DYNAQF LEXI is the same asthat of C O DYNAQD OUBLE.C. Performance Evaluation MeasuresFigures 3, 4, and 5 illustrate the main differences betweenthe three algorithms. Table II summarizes the differentfeatures of the prioritization algorithms.To measure the performance of our prioritization algorithms, we use three measures 1) time to the first failure(F IRST FAIL), 2) time to detect all failures in a test suite(A LL FAIL) and the percentage of delayed failure tests. Wedetail each measure below:Speedup in the First Failure Detection (F IRST FAIL).When developers submit a change to be tested the firstfeedback they can respond to is a failing test or, if there areno failures, the time to pass all the tests. We refer to thismeasure as the First Failure Detection Time (F IRST FAIL).We measure F IRST FAIL in testing time saved. This measureis very similar to the measure Response Time in systemprogramming which is used to evaluate how fast a sharedsystem reacts to the user [12].For a given test request, Gain in F IRST FAIL is calculated astaking the difference between the F IRST FAIL of the baselinealgorithm and that of the algorithm under evaluation. ThenSpeedup in F IRST FAIL is calculated by Equation 3:speedup median gainmedian waiting time(3)If speedup in F IRST FAIL is positive, then the evaluatedalgorithm provides result for a single change earlier than theF IFO BASELINE algorithm and vice versa.Speedup in All Failure Detection (A LL FAIL). For a givenfailing suite (which includes many tests), A LL FAIL is thewaiting time for a test failure to be reported for the entire
Figure 3: C O DYNAQS INGLEtest suite. In our case, we use the period as an indicator ofa test suite. Using A LL FAIL gives us a different perspectivefrom F IRST FAIL, since F IRST FAIL is about how quickly weget results for a single change, whereas A LL FAIL gives usan indication of how fast we get feedback for all test failures.In many ways, these two measures complement each other.F IRST FAIL tells us how quickly the prioritization is able toprovide feedback for a change, whereas A LL FAIL tells ushow well the prioritization can handle the starvation problem,since it considers all test failures.The gain in A LL FAIL is calculated as the differencebetween the time it takes for all failures to be found inthe tests suite using the baseline algorithm and the evaluatedalgorithm. And the speedup in A LL FAIL is given by Equation3. If speedup in A LL FAIL is positive, then the evaluatedalgorithm fails the given tests earlier than the F IFO BASELINEalgorithm and vice versa.Percentage of Delayed Failure Detection. Another view ofhow well an algorithm copes with starvation is to measurethe difference in delayed test failures compared to theF IFO BASELINE. The main difference between percentage ofdelayed failure detection and A LL FAIL lies in the fact thatA LL FAIL measures the relative amount of time gain, whilepercentage of delayed failure detection does not consider theamount of delay but just the percentage of delayed failures.For example, if many failures were delayed by a negligibleamount A LL FAIL would still report good results, whereaspercentage of delayed failure detection would not. Since thismeasure is related to delayed failure detection, the smallerthe value the better.V. C O - FAILING B EHAVIOUR OF C HROME T ESTSPrior to delving into our results, we wanted to examinethe co-failing behaviour of tests in C HROME. Generally,test prioritization work assumes that tests are independentfrom each other [6]. However, we believe that tests are notindependent, in particular for system level testing [22][3].Some tests may cover different functionality located in thesame file collection, while other tests may cover differentfiles where a high degree of file dependency is found. TheseFigure 4: C O DYNAQD OUBLEFigure 5: C O DYNAQF LEXItests tend to fail together and result in co-failing behavior.In both cases, the basis of the co-failing behavior comesfrom the inherent interrelation inside the file system, whichis not likely to change. Hence, we believe that the co-failingbehavior is a long term phenomenon.We used the machine learning library, Apache Spark, tocapture the frequently co-failing test suites in the C HROME’stest data and generate Table III. The column Test Pair is thepair of tests under investigation, the column Fail Together isthe number of times they all failed in the same request, thecolumn Run Together is the number of times t
double re-prioritization queue, CODYNAQDOUBLE, and a flexible re-prioritization queue, CODYNAQFLEXI. To evaluate CODYNAQ and compare it with the state-of-the-art, GOOGLETCP, we use two data sets. The first is the internal Google testing data that Elbaum et al. [7] released when developing GOOGLETCP. The second is scraped data
