WIXLIB

Michael N. Huhns, Wei Li, and Wei-Tek Tsai33.139Overview of TalksLarge-Scale Performance Testing by Cloud and CrowdXiaoying Bai (Tsinghua University – Beijing, CN)Creative Commons BY 3.0 Unported license Xiaoying BaiJoint work of Bai, Xiaoying; Tsai, Wei-Tek; Wu, WenjunLicenseSoftware scale and complexity increase tremendously in recent years. Performance testing ofInternet scale software systems is usually expensive and difficult. The challenges include:(1) to simulate diversified usage scenarios; (2) to generate various workload distributions;and (3) to measure and evaluate performance from different aspects. Cloud computing andCrowdsourcing are two emerging techniques in recent years. They promote new testingarchitectures that are promising to address the challenges. Cloud-based testing, such as TaaS(Testing-as-a- Service), aims to support on-demand testing resources and services in/on/overclouds for testers at any time and all time. Testing by crowdsourcing, such as uTest andMob4hire, follows the Web 2.0 principle of harnessing collective intelligence and varioustesting tasks can be crowdsourced including test case design, script development, scriptdebugging, test execution, and test result evaluation.This paper first investigates the framework to facilitate large-scale performance testingby integrating cloud infrastructure and crowd wisdom. A Cloud-based testing platform,Vee@Cloud was built to provide a cross- cloud virtual test lab to support on-demand testscripts provisioning and resource allocation, scalable workload simulation, and continuousperformance monitoring. Following the crowdsourcing approach, participants can join theplatform for different purposes, such as biding testing tasks, contributing test cases andscripts, renting test resources, and carrying on test executions. The process of establishingtest collaboration can be formulated as a multi-criteria decision making problem, usingqualitative evaluation models for different factors like task allocation and scheduling, qualitycontrol, and cost control.3.2Finding ExpertsXavier Blanc (Univ. Bordeaux, LaBRI – Talence, FR)LicenseCreative Commons BY 3.0 Unported license Xavier BlancHeavy usage of third-party libraries is almost mandatory in modern software systems. Theknowledge of these libraries is generally scattered across the development team. When adevelopment or a maintenance task involving specific libraries arises, finding the relevantexperts would simplify its completion. However there is no automatic approach to identifythese experts. In this talk we propose LIBTIC, a search engine of library experts automaticallypopulated by mining software repositories. We show that LIBTIC finds relevant experts ofcommon Java libraries among the GitHub developers. We also illustrate its usefulness througha case study on the Apache HBase project where several maintenance and developmentuse-cases are carried out.13362

4013362 – Cloud-based Software Crowdsourcing3.3Crowdsourcing Cloud Infrastructure using Social NetworksKyle Chard (University of Chicago, US))LicenseCreative Commons BY 3.0 Unported license Kyle ChardThe increasing pervasiveness of online social networks is not only changing the way that peoplecommunicate and interact but it is also allowing us to represent, document and explore interpersonal relationships digitally. Social networking platforms have gone beyond a platformfor communication and are now a viable platform in their own right on which to implementunique socially oriented services with access to an increasingly complex social graph modelingevery aspect of an individual life. Building upon this social fabric services can leveragereal world relationships, inferring a level of trust between users, exploiting intrinsic socialmotivations, and developing socially aware algorithms. This talk describes the experiencesand lessons learned developing a Social Cloud, a platform that enables the construction ofa crowdsourced cloud infrastructure to facilitate resource and capability sharing within asocial network. Social Clouds are motivated by the need of individuals or groups for specificresources or capabilities that can be made available by connected peers. Such resourcesare not necessarily only computational or software resources, but can be any electronicallyconsumable service, including human skills and capabilities. Social Clouds leverage lessonslearned through volunteer computing and crowdsourcing such as the willingness of individualsto make their resources available and offer their expertise altruistically for good causes. Thistalk explores aspects such as inter-personal trust, platform implementation, social incentivesand use cases, by leveraging methodologies from computer science, economics and sociology.It looks specifically at these aspects in the context of social network based crowdsourcingand attempts to generalize the approaches used in Social Clouds to other forms of softwarecrowdsourcing.3.4Cloud based Crowdsourcing Software Development – KeynoteSchahram Dustdar (TU Wien, AT)LicenseCreative Commons BY 3.0 Unported license Schahram DustdarIn this talk I begin with analyzing the historical roots of large-scale collective problems solvingresearch in Computer Science. this is followed by a detailed analysis of the mechanisms,algorithms, models, and software architectures and deployment models of software solutionshelping to enable crowdsourcing software development on a large scale. This is discussed inrelationship with an analysis of currently missing aspects of solutions aiming at supportingcloud based crowdsourcing for software development.

Michael N. Huhns, Wei Li, and Wei-Tek Tsai3.541Hyperscale Development of SoftwareMichael N. Huhns (University of South Carolina, USA)LicenseCreative Commons BY 3.0 Unported license Michael N. HuhnsThe benefits of the open-source (Bazaar? approach to software development have not beenfully realized, because the number of software developers is still relatively small and orders ofmagnitude smaller than the number of users. Developers typically are experts in computing,whereas users typically have domain expertise: this produces a disparity in viewpoints, causinga mismatch between the developed software and its desired use. Moreover, the proposition,given enough eyeballs, all bugs are shallow,would take on much greater significance, if alarger fraction of the users could also be developers. A solution to this problem has so farbeen impractical: end-users often do not have sufficient expertise to contribute software, northe time to learn how to do so, and there was no way to meld it with existing software untilit was proven correct or from a trusted developer.In this paper I describe an approach for broadening dramatically the number of peoplewho contribute to the development of software. If successful, the approach will result ina more effective software-development process, greatly improved software reliability, andincreased end-user satisfaction. The approach is ambitious – in that it affects all stages ofsoftware development and several levels of the software execution process and transformative– in that the software will be developed and executed in new ways.The approach makes use of take advantage of-all contributed code, components, anddesigns until they have proven to be of no value. The talk will describe how agent-basedwrappers can manage the necessary collaboration and competition, allowing the contributionsto be used alongside their existing counterparts until their behavior and features can beassessed. The solution exploits concepts from N-version programming, multicore processors,model- driven architectures, test-driven development, autonomic computing, negotiation inmultiagent systems, group decision-making, and consensus.The two main threads of the proposed research are broader participation in developingsoftware and multiagent systems for the use and execution of the software. For each weare developing prototypes and realistic evaluations demonstrating greater robustness andusefulness. The research can help solve the problem of incorrect software by improvingrobustness, while enabling a wider cross-section of society to develop and personalize software.This can lead to greater understanding, satisfaction, and utility of software that behaves aspeople generally want it to.3.6“Microtask” vs. freelancer platforms – how crowdsourcing cancomplement software developmentRobert Kern (IBM Deutschland – Böblingen, DE)LicenseCreative Commons BY 3.0 Unported license Robert KernIn the era of cloud computing, mobile computing, collaboration and big data, softwaredevelopment requirements are significantly changing. Users and organizations are asking forshorter development cycles, improved ease of use, better integration and lower administrationand operation overhead. This results in a need for flexible software development and operation13362

4213362 – Cloud-based Software Crowdsourcingprocesses combined with a new assignment of roles in order to hide manual efforts from theuser.Crowdsourcing has the potential to address these challenges in several ways. On the onehand, platforms like TopCoder or oDesk enable for flexible outsourcing of design and implementation efforts to freelancers. On the other hand, “microtask” platforms like Crowdfloweror Amazon MTurk provide human workforce as a scalable service in order to fulfill formalizedtasks which are difficult to automate. Such services can either be integrated into SW servicesto deliver complex services like search engine optimization, data cleansing and social mediaanalysis, or they can complement SW operation and development, e.g. by providing testingservices.After contrasting the two types of platforms, this presentation focuses on the latter type.By elaborating on the analogy of cloud computing and crowdsourcing, a “cloud labor” stack isintroduced that seamlessly integrates with the concepts of cloud computing. The capabilitiesprovided by the different layers of the stack are identified and implementation options areoutlined. The considerations are then validated by discussing a crowd-based medical codingservice offered by IBM. Finally, a bow is drawn back to the freelancer platforms and initialideas are provided on how the two complementary concepts could be merged in the future.3.7Software Development Crowdsourcing Issues: Organization Designand Incentive DesignDonghui Lin (Kyoto University, JP)LicenseCreative Commons BY 3.0 Unported license Donghui LinIn the talk, we discuss two important issues in Software Development Crowdsourcing:Organization Design and Incentive Design. We share the experience of problem-basedlearning course of crowdsourcing in university.3.8Crowdsourcing with ExpertiseGreg Little (ODesk Corp. – Redwood City, US)LicenseCreative Commons BY 3.0 Unported license Greg LittleCrowdsourcing often focusses on small tasks that require no special skills other than beinghuman. I’m interested in tasks and workflows that do require special skills, like programmingor art skills. For instance, one could imagine a crowdsourced logo that involves several phases.The first phase might pay 20 sketch artists to sketch logo ideas, and then pay a traditionalcrowd to vote for the best 5. The next phase might then pay 5 designers to flesh out thebest 5 logos from the previous phase, followed by another round of voting. The final phasemight involve hiring an expert to finalize the top design.Crowdsourcing with expertise is difficult today because it is difficult to programmaticallyhire many experts – hiring experts typically involves looking at resumes and portfolios. Myresearch has focussed on finding scalable ways to reliably identify experts. Two promisingideas include: (1) having people play simple subjective games based on their expertise, andrating each others results; (2) having public work histories which include both the task and

Michael N. Huhns, Wei Li, and Wei-Tek Tsai43the results, as opposed to portfolios today which typically only display the results, but itis hard to know what this worker contributed, or whether their result met their client’srequirements.3.9Multi-Agent System Models and Approach of CrowdsourcingSoftware DevelopmentXinjun Mao (National University of Defense Technology – Changsha, China)LicenseCreative Commons BY 3.0 Unported license Xinjun MaoThe advent and successful practices of software crowdsourcing needs to investigate its indepth essence and seek effective technologies to support its activities and satisfy increasingrequirements. We highlight crowdsourcing participants consist of a multi-agent system andsoftware crowdsourcing is a multi-agent problem-solving process. This paper discusses thecharacteristics and potential challenges of software crowdsourcing in contrast to traditionalsoftware development, and present a general analysis framework based on multi-agent systemto examine the organization and behaviors of software crowdsourcing. Several softwarecrowdsourcing models performed on typical platforms like Topcode, uTest are established andtheir organization and coordination are discussed. We have developed a service-based multiagent system platform called AutoService that provides some fundamental capabilities likeautonomy, monitoring, flexible interaction and organization, and can serve as an infrastructureto support software crowdsourcing models and tackle its challenges. A software crowdsourcingprototype is developed and some scenarios are exemplified to illustrate our approach.3.10Crowds, Clouds, Agents and CoordinationDave Murray-Rust (University of Edinburgh, GB)Creative Commons BY 3.0 Unported license Dave Murray-RustJoint work of Murray-Rust, Dave; Robertson, DaveLicenseThe term “social machines” describes a class of systems where humans and machinesinteract, and the mechanical infrastructure supports human creativity. As well as softwarecrowdsourcing projects such as TopCoder and oDesk, this includes distributed developmentplatforms such at GitHub and Bitbucket. In this paper, we describe a formalism for socialmachines, consisting of i) a community of humans and their “social software” interactingwith ii) a collection of computational resources and their associated state, protocols andability to analyse data and make inferences. These social machines are increasingly thetarget of software development, and as such, they represent an interesting problem, as thecommunity must be “programmed” as well as the machines. This leads to evolving andunknown requirements, and having to deal with much softer concepts than formal systemsdesigners usually work with. Our model hence uses two coupled social machines. Thereis the target machine, and the machine which is used to create it—much as GitHub andassociated resources might be used to form a social machine to create “the next Facebook”.We draw on the ideas of ’desire lines’ and ’play-in’ to argue that top down design of socialmachines is impossible, that we hence need to leverage computational support in creatingcomplex systems in an iterative, dynamic and emergent manner, and that our formalismprovides a possible blueprint for how to do this.13362

4413362 – Cloud-based Software Crowdsourcing3.11Collaborative Majority Vote: Improving Result Quality inCrowdsourcing MarketplacesKhrystyna Nordheimer (Universität Mannheim, DE)Creative Commons BY 3.0 Unported license Khrystyna NordheimerJoint work of Nordheimer, Dennis; Nordheimer, Khrystyna; Schader, Martin; Korthaus, AxelLicenseCrowdsourcing markets, such as Amazon Mechanical Turk, are designed for easy distributionof micro-tasks to an on-demand scalable workforce. Improving the quality of the submittedresults is still one of the main challenges for quality control management in these markets.Although beneficial effects of synchronous collaboration on the quality of work are wellestablished in other domains, interaction and collaboration mechanisms are not yet supportedby most crowdsourcing platforms, and thus, not considered as a means of ensuring high-qualityprocessing of tasks. In this paper, we address this challenge and present a new method thatextends majority vote, one of the most widely used quality assurance mechanisms, enablingworkers to interact and communicate during task execution. We illustrate how to apply thismethod to the basic scenarios of task execution and present the enabling technology for theproposed real-time collaborative extension. We summarize its positive impacts on the qualityof results and discuss its limitations.3.12The Open Source Volunteering ProcessDirk Riehle (Universität Erlangen-Nürnberg, DE)LicenseCreative Commons BY 3.0 Unported license Dirk RiehleToday’s software systems build on open source software. Thus, we need to understand howto successfully create, nurture, and mature the software development communities of theseopen source projects. In this article, we review and discuss best practices of the open sourcevolunteering and recruitment process that successful project leaders are using to lead theirprojects to success. We combine the perspective of the volunteer, looking at a project, withthe perspective of a project leader, looking to find additional volunteers for the project. Weidentify a five-stage process consisting of a connecting, understanding, engaging, performing,and leading stage. The underlying best practices, when applied, significantly increase thechances for a successful open source project.3.13Artifact-centric Incentive Mechanisms for Socio-technicalSystemsOgnjen Scekic (TU Wien, AT)Creative Commons BY 3.0 Unported license Ognjen ScekicJoint work of Scekic, Ognjen; Truong, Hong-Linh; Dustdar, SchahramMain reference O. Scekic, H.-L. Truong, S. Dustdar, “Incentives and rewarding in social computing,”Communications of the ACM, 56(6):72–82, 2013.URL owdsourcing systems of the future (e.g., social compute Units – SCUs, collective adaptivesystems) promise to support processing of richer and more complex tasks, such as softwaredevelopment. This presupposes deploying ad-hoc assembled teams of human and machine

Michael N. Huhns, Wei Li, and Wei-Tek Tsai45services that actively collaborate and communicate among each other, exchanging differentartifacts and jointly processing them. Major challenges in such environments include teamformation and adaptation, task splitting and aggregation, and runtime management ofdata flow and dependencies, collaboration patterns and coordination mechanisms. Thesechallenges can be somewhat alleviated by delegating the responsibility and the know-howneeded for these duties to the participating crowd members, while indirectly controlling andstimulating them through appropriate incentive mechanisms.In this paper we present a novel, artifact-centric approach for modeling and deployingincentives in rich crowdsourcing environments. Artifact’s lifecycle model is augmented withincentive mechanisms to create encapsulated units that can be offered to the cr

Crowdsourcing software development or software crowdsourcing is an emerging software engineering approach. Software development has been outsourced for a long time, but the use of Internet with a cloud to outsource software development to the crowd is new. Most if not all software development tasks can be crowdsourced including requirements,