WIXLIB

1) AndroidProjectsCollection2) Snapshotcreation3) CorruptionCheckGitHub AndroidProject Metadata(8,216)GitLab Instance WithAndroid Repositories(8,216)Corrupt Projects(244)AndroidTimeMachineNeo4j DBFig. 1: Datasetcollection processproducerconsumerStepGitLab Instance WithNon CorruptAndroid Repositories(7,972)DataContainerArtifactclear distinction between regular Android applications andapplications containing Kotlin; (ii) source code and projecthistory must be fully accessible for all entries. Hence, to builda valid dataset, we adopt the process summarized below andshown in Figure 1.Applications collection - As a starting point for the collectionof our dataset, we use AndroidTimeMachine, an independentlybuilt dataset of open-source Android applications [11]. Wechoose this dataset as our starting point as (i) it provides alarge number of open-source applications, thus increasing thelikelihood of having a representative sample of applicationseven after subsequent filtering steps, and (ii) it includespointers to the GitHub repository of each entry, thus meetingthe requirement of having full access to the source codeand project history. Although AndroidTimeMachine is madeavailable in ready-to-use components, it is not usable in anout-of-the-box fashion for the purpose of this study as projecthistories included in it have been collected in 2018, thus usingthem would have meant excluding more than a year’s worth ofapplication versions. Therefore, we create an updated snapshotof source code and project histories for applications includedin AndroidTimeMachine.we found that the repository was empty. These corruptionswere most likely introduced in the time between our study andthe AndroidTimeMachine study. Thus, our dataset of Androidapplications consists of a snapshot of 7,972 git repositories,hosted in a local GitLab instance for easy access.Identifying Kotlin Applications - For finalizing the dataset,identification of projects that contain Kotlin code is required.This process is achieved by following the algorithm describedin Listing 1. First, we clone each repository in our local GitLabinstance (line 5 in Listing 1). Cloning the repository willautomatically check out the main branch from the originalGitHub repository. The process is then continued by listing allcommits in the main branch by using the git log command(line 5). All commits are then iterated on, and files changedin each are checked for the presence of a Kotlin file extension(lines 6-7). If a Kotlin file extension is found, we tag theAndroid repository as a Kotlin application (line 8). The fullprocess results in a filtered dataset of 451 applications thatever contained any Kotlin code, and thus classified as Kotlinapplications. Our filtering process, does not check for thepresence of Kotlin code in applications’ dependencies, due tothe increased complexity it introduces and due to the fact thatchoice of libraries is not always fully controllable by Androiddevelopers themselves.Listing 2: Kotlin projects identification procedurefunction countSlocForRepositories(repos: List){results map();for repository in repos {repository repository.cloneRepository();for commit in repository.commits {commit commit.checkout();sloc Cloc.countSloc(commit);results.put(repository, sloc.java, sloc.kotlin);}}return results;}Listing 1: Kotlin projects identification procedure1234567891011121314function filterRepositoriesOnKotlin(repos: List){kotlinRepos set();for repository in repos {for commit in repository.commits {for file in commit {if y);}}}}return kotlinRepos;}The snapshot creation starts by extracting from AndroidTimeMachine the list of all entries with an attached GitHubrepository. This query results in 8,216 GitHub repositories. Wecontinue by importing these repositories into a GitLab Dockerimage2 instance. For 244 repositories, insertion resulted in acorrupt repository containing zero commits. After inspection,we found that the GitHub repository no longer exists for 243of these projects while for the sole project that was leftover2 https://docs.gitlab.com/omnibus/docker/Measuring Java and Kotlin SLOC - Having access to theentire git history for each application in our dataset enablesus to measure the lines of code for each language directly onthe source code. We do so by using the tool CLOC3 . CLOCis a tool that counts the source lines of code of an application,grouped for each of the multiple programming languages itrecognizes. It is able to detect blank lines and comments so,in our study, we only measure lines of actual code. The processof counting the SLOC is described in Listing 2. It starts withcloning the repository (line 5 in Listing 2) and iterating onevery commit (lines 6). For each, CLOC is run on the checkedout source code (lines 7-9). It results in the count of Kotlinand Java SLOC for each version of the application for all 451Kotlin applications in our dataset.C. Data analysisIn the following, we describe the steps undertaken toanalyze the collected data towards answering our researchquestions.3 https://github.com/AlDanial/cloc12345678910111213

Measuring the degree of Kotlin adoption - For answeringRQ1, we categorize the 7,972 Android applications in ourdataset using the three categories defined by Mateus et al. [5]:(i) Entirely written in Java, (ii) Entirely written in Kotlin,(iii) Written in both Java and Kotlin. Every application thatis part of our Android applications dataset but not part ofthe Kotlin applications are assigned to the first category.Kotlin applications that never contained any lines of Java codeare assigned to the second category. All remaining Kotlinapplications are assigned to the third category. We presentthese results visually in the bar chart of Figure 3.Measuring the proportion of Kotlin code - Furthermore, toprovide a more in-depth answer for RQ1, we take the counts ofSLOC for the most recent version of every Kotlin applicationand calculate the proportion of Kotlin code compared to Javacode, disregarding any other programming language present inthe codebase. We plot these in the histogram visualization ofFigure 4, which shows the Kotlin proportion distribution forKotlin applications in our dataset.Measuring the run-time efficiency impact of a Kotlinmigration – We answer RQ2 quantitatively. In the following, we describe the experiment design by first covering theselection of the subjects and then defining the independent anddependent variables, together with how the latter are measured.We continue with a formulation of hypotheses that ultimatelyanswer RQ2 and the design of our experimental setup. Finally,we describe the statistical methods for analyzing the data andaccepting or rejecting the hypotheses.Subjects selection – As subjects for the experiment, we needa sample of applications for which both a version entirelywritten Java and a version entirely written in Kotlin exists.An important aspect to consider in the selection of theseapplications is that the transition from Java to Kotlin must bea pure migration. A migration from Java to Kotlin is pureif it does not introduce any new functionality in the app.By considering pure migrations, we are reasonably confidentabout the functional equivalence of the Java and Kotlin versions of an app. We start our sampling by identifying in ourdataset those projects for which the Kotlin code replaces allJava code between two consecutive commits in the projecthistory. To do so, first, we identify all logically equivalentcode chunks in Kotlin and Java for all projects in our dataset,by applying the cross-language clone (CLC) detection methodproposed by Cheng et al. [12]. Employing this methodology,all commits in revision histories are analyzed to identifyKotlin migration commits. Whenever a Java deletion and aKotlin addition resulting in a CLC is detected, the commit isclassified as a Kotlin migration, since it meets the definitionof containing deleted Java code that is replaced by logicallyequivalent Kotlin code. This led to the identification of 3,674Kotlin migration commits. Among these, we found 62 projectsfor which Kotlin code replaces all Java code between twoconsecutive versions.Adopting these projects as the base for our subject selectionincreases the likelihood that the migration is a pure migration,TABLE I: Experiment Entertainmentwhitakers-words-androidBooks & ReferenceslounikBooks & aries & egoryJava SLOC Kotlin 02,0126,053372505421728168as it was completely performed in a single commit. From theinitial 62 projects, we pick a sample of 10 applications usingstratified random sampling [13]. We use two characteristics forthe stratified random sampling: (i) the application’s categoryas listed in the Google Play store, (ii) the total Kotlin and JavaSLOC being either lower than a threshold t or greater or equalto it. We stratify by app category in order to have a balancedset of apps with respect to their provided functionalities. Wechose t 5000 SLOC since we observed that apps with lowerSLOCs tend to be either single-purpose or extremely basic.This sampling procedure increases the likelihood that a variedset of categories, as well as both small and large projects,are well represented in our sample. To be certain that thesample consists of only pure migrations, we manually inspectthe sampled migration commit via three heuristics: (i) verifythat the paths of the deleted Java files and added Kotlin files arematching; (ii) verify that the deleted Java code and the addedKotlin code are functionally similar; (iii) we install both theJava and Kotlin versions of the app and manually check thatthey provide exactly the same functionalities, i.e., all buttonsand screens are functionally equal in the two versions. If themigration passes all of these manual checks, we consider itpure. Our initial sample of 10 migrations successfully passedall three heuristics, and therefore, we did not have to introducemeasures to deal with impure migrations. The 10 applicationsare presented in Table I.Independent and dependent variables – As an independentvariable, we use the programming language used in the application. It has two treatments: a 100% usage of Java beforemigration to Kotlin and a 100% usage of Kotlin after themigration to Kotlin.The dependent variables of this experiment consist of wellknown metrics for both performance and energy efficiency ofAndroid apps: CPU usage (cpu): optimizing CPU usage provides a fasterand smoother experience to the user while also preservingbattery life [14]. We define CPU usage as the percentageof the device’s total CPU capacity used by an applicationat given points in time during its lifetime. It is measuredusing the Android Debugging Bridge (ADB) dumpsyscpuinfo command throughout the entire duration of theexperiment at a sampling frequency of one second.Memory usage (mem): physical memory is constrainedon mobile devices due to clear limitations in space, andtherefore, memory is a valuable resource in Android.

Excessive memory consumption can degrade app performance and can cause application crashes [15]. We definememory usage as the amount of RAM in KB used by anapplication at given points in time during its lifetime. It ismeasured using the dumpsys meminfo ADB commandthroughout the entire duration of the experiment at asampling frequency of one second.Number of calls to the garbage collector (gc): The system’s memory is freed up automatically by the garbagecollector. Poor memory management, such as the introduction of memory leaks, not only causes more garbagecollector calls but also intensifies the work done by eachcall, thus degrading performances [16]. The number ofcalls is counted by reading device logs through the ADBlogcat utility. Additionally, we checked the source codeof every application on potential explicit invocations ofthe garbage collector. None of the 10 apps were makingexplicit calls to the garbage collector.Frame times (ns) and the number of delayed frames(df): When Android renders a frame, it takes a certainamount of time to do so. This frame time is, therefore,an essential factor in perceived performances when usingan Android application. The ideal frame rate is 60 framesper second (FPS). To achieve this rate, frames must berendered in under 16ms; otherwise, the system is forcedto skip frames. As the human eye is very keen on noticingdrops in FPS, the user will perceive such events asstuttering in the app [15]. Therefore, the amount of suchdelayed frames directly affects the user’s experienced performance as well. We measure these metrics by runningthe dumpsys gfxinfo framestats ADB commandthroughout the entire duration of the experiment. Wecount frames that took more than 16ms to be rendered asdelayed frames.App size (as): App size is defined as the size of theapplication when packaged into an APK. App size caninfluence how users perceive an app since devices havelimited storage available, and in some circumstances,costs may be involved when downloading large files fromthe Internet. It is measured by taking the size of the APKbinary file of each app in bytes.Energy consumption (en): energy consumption standsfor the number of Joules consumed by the device ina period of time. Low power consumption is a criticalnon-functional requirement when building an Androidapp, as mobile devices have limited battery and, whenneglected, it seriously impacts the users’ perceived appquality [17]. In our experimentation, energy consumption is represented in Joules and it is measured bymeans of a software-based technique based on the ADBbatterystats tool. In the literature, the accuracy ofsoftware based approaches has been reported to be reasonably close to hardware-based ones [6], [18].Hypotheses – To answer RQ2, we formulate a null hypothesisfor each dependent variable, specifically: cpu: being µcpu the mean CPU usage per run for a givenapplication version, we define the null and alternativehypotheses as:cpucpucpuH0cpu : µcpu: µcpujava µkotlin H1java 6 µkotlin mem: being µmem the mean of memory usage per runfor a given application version, we define the null andalternative hypotheses as:memmemmemH0mem : µmem: µmemjava µkotlin H1java 6 µkotlin gc: being µgcv the mean of the number of GC calls perrun for a given application version, we define the nulland alternative hypotheses as:gcgcgcgcH0gc : µgcjava µkotlin H1 : µjava 6 µkotlin ft: being µfv t the mean of frame time values per runfor a given application version, we define the null andalternative hypotheses as:ttttH0f t : µfjava µfkotlinH1f t : µfjava6 µfkotlin df : being µdfv the mean of the number of delayed framesper run for a given application version, we define the nulland alternative hypotheses as:dfdfdfdfH0df : µdfjava µkotlin H1 : µjava 6 µkotlin as: being µasv the size of a given application version, wedefine the null and alternative hypotheses as:asasasasH0as : µasjava µkotlin H1 : µjava 6 µkotlin en: being µecv the mean energy consumption per runfor a given application version, we define the null andalternative hypotheses as:ecenececH0en : µecjava µkotlin H1 : µjava 6 µkotlinExperiment design and execution – We automate the execution of the tests for each subject employing an ad-hoc scriptthat automatically clicks through the application and coversall of its features with the necessary waiting operations inbetween steps. The scripts are available in the online replication package1 . Each script is designed to execute one userscenario for each of the features that the app contains (e.g.,editing settings, visiting screens, scrolling through content,and creating entries). We create these scenarios based on amanual exploration of the experiment subjects, during whichwe explored the running app, its Google Play Store page,and its source code. While running the scripts, all metricsare measured in parallel. Before and after the experiment, athorough setup and reset phase is included. The scripts and thecode for the measurements are implemented using AndroidRunner, an open-source Python framework for automatingexperiments on Android devices [6].We perform our experiment on two different devices. Thetwo devices consist of an older generation device (GoogleNexus 5) and a more recent device (Google Pixel 3a). We

consider the type of Android device as a blocking factor sincethe two devices have different hardware specifications.Before running the experiment, we manually prepare ourfactory reset device by fully charging the battery, removing theSIM and SD card, setting brightness and sound to a minimum,enabling the stay awake developer option, and finally, disablingnetwork data, WiFi, Bluetooth, and notifications of other apps.Continuing, before starting a single test run, we implementeda setup step that clears the device log files, and performs afresh install of the app. Finally, after every single test run, weinclude a timeout step that waits 2 minutes in order to preventtail energy consumption from influencing our measures.Figure 2 gives an overview of the experiment executionprocess. Android Runner hosts the plugins that measureour defined metrics as well as the scripts that execute theexperiment. Our created test scripts that interact with thesubject application are ran using MonkeyRunner4 , a utility forautomated testing of Android apps. In parallel, all plugins arerunning and collecting the necessary data for each metric. Foreach device, the experiment is run 20 times per subject inrandomized order.Analysis of experimental data – We start our data analysisby verifying our assumption that the data gathered for eachhypothesis is not normally distributed. It consists of visualanalysis of each metric’s Q-Q plots [19] and the applicationof the Shapiro-Wilk test [20]. We make use of the ShapiroWilks test because it is proven to be one of the most powerfultests for testing normal distribution [21]. The result of thisverification indicates which statistical test we can use fortesting our hypotheses.For testing our hypotheses, we need to prove that a difference between the Java and Kotlin versions of the sameapp exists. In order to find out whether the two independentdistributions measured for each hypothesis are different, weperform statistical tests. Since we obtained evidence thatverifies our assumption on the non-normal distribution of ourdata (further elaborated upon in the results Section IV), wemake use of the non-parametric Mann-Whitney U test [22]. Itassesses whether the Kotlin and Java measurements come froma different distribution and thus rejects the null hypothesis.For interpreting the resulting p-values, we use the BenjaminiHochberg [23] p value corr

The study aims to support Android developers, maintainers of the Android platform and its Kotlin runtime, and re-searchers. The former are provided with evidence on the level of Kotlin adoption in Android development and on the run-time impact of Kotlin, which forms an objective basis for de-ciding about