Transcription
Hohl et al. Journal of Software Engineering Research and(2018) 0059-zR ES EA R CHOpen AccessBack to the future: origins and directionsof the “Agile Manifesto” – views of theoriginatorsPhilipp Hohl1* , Jil Klünder2 , Arie van Bennekum3 , Ryan Lockard4 , James Gifford5 , Jürgen Münch6 ,Michael Stupperich1 and Kurt 1Daimler AG,Research and Development,Wilhelm-Runge-Str. 11, 89081 Ulm,GermanyFull list of author information isavailable at the end of the articleAbstractIn 2001, seventeen professionals set up the manifesto for agile software development.They wanted to define values and basic principles for better software development. Ontop of being brought into focus, the manifesto has been widely adopted by developers,in software-developing organizations and outside the world of IT. Agile principles andtheir implementation in practice have paved the way for radical new and innovativeways of software and product development. In parallel, the understanding of themanifesto’s underlying principles evolved over time. This, in turn, may affect current andfuture applications of agile principles. This article presents results from a survey and aninterview study in collaboration with the original contributors of the manifesto for agilesoftware development. Furthermore, it comprises the results from a workshop withone of the original authors. This publication focuses on the origins of the manifesto, thecontributors’ views from today’s perspective, and their outlook on future directions. Weevaluated 11 responses from the survey and 14 interviews to understand the viewpointof the contributors. They emphasize that agile methods need to be carefully selectedand agile should not be seen as a silver bullet. They underline the importance ofconsidering the variety of different practices and methods that had an influence on themanifesto. Furthermore, they mention that people should question their currentunderstanding of “agile” and recommend reconsidering the core ideas of the manifesto.Keywords: Manifesto for agile software development, Agile manifesto, Agile methods,Interview review1 IntroductionNowadays, companies are confronted with high-frequent changes due to innovations andnew technology (Broy 2006). Innovations such as cloud based services, big data and digitalization are spreading into all markets and all kinds of products. A lot of innovationis achieved in the combination of software and new types of devices. This combinationoffers new opportunities in a fast pace, which asks one by one a response from the market.Consequently, the amount of software and connected solutions, e.g. in cars (Broy 2006),increased exponentially during the last decades. Therefore, it is a competitive advantageto develop and distribute high-quality software and connected solutions at a high pace.Agile software development methods are a promising solution to keep pace with thisprogress. The rise of agile methods and practices started to get significant attention 17 The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to theCreative Commons license, and indicate if changes were made.
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15years ago with the manifesto for agile software development, often referred to as “agilemanifesto” although not meant to be agile itself (Thomas 2017).A lot of publications (Highsmith 2002; Dingsøyr et al. 2012; Abrahamsson et al. 2002)which were published during the last 17 years interpret the statements from the manifestoand introduce agile practices, principles, and methods. The manifesto is usually referredto whenever developers aim at conducting agile development (Beck et al. 2001):We are uncovering better ways of developing software by doing it and helpingothers do it. Through this work we have come to value:Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a planThat is, while there is value in the items on the right, we value the items on theleft more.The manifesto was written by 17 software practitioners in 2001 in Snowbird, near SaltLake County, Utah. On being asked about the motivation for setting up the manifesto,Kent Beck mentions that they wanted to “uncover better ways of developing software”(Beck et al. 2001). Ron Jeffries adds that the original idea “Kent Beck had in mind withExtreme Programming, [was] to make the world safe for programmers” (Lockard et al.2017). A “safe” development environment is not only the motivation of Extreme Programming (XP). It is common for most agile methods, to improve the working conditions ofdevelopers. This can be seen as a main factor for the success of the manifesto for agilesoftware development.The rise of agile methods was welcomed with enthusiasm by many developers but alsoled to criticism. At least two mindsets of approaching agile development with potentiallydifferent outcomes could be observed: There are those developers who apply agile practices because they believe in the values and principles of the manifesto, and those whodo it because it is seen as best practice. The manifesto for agile software developmentseemed to promise a more successful way of developing software by “just” following theoriginal values and principles. This, in turn, makes the manifesto special. Some developers believed and still believe in the manifesto as the “Holy Grail” for successful softwaredevelopment, while others denote it as a marketing gimmick to sell intuitive developmentbehavior within a new livery.Nevertheless, the ongoing success of agile had several implications. Among others, newtrends like scaling agile with Scrum of Scrums (SoS) or the Scaled Agile Framework(SAFe) are common talk. The original ideas of the manifesto have become more andmore commercialized. Many developers and managers that are now adopting agile arenot aware of the initial diversity of agile methods and the underlying principles. Scrum isoften seen as the only agile practice (Klünder et al. 2017).The so-called agile transformation of organizations is a frequently discussedtopic because agile ways of working promise to make companies future-proof(Lockard and Cleff 2016). However, the developers often declare themselves “agile”when just using Scrum by the book. In this case, the meaning of agile isPage 2 of 27
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15wrongly interpreted, not understood, and commercialized (Klünder et al. 2017).This is one of the reasons why we conducted this survey and the interviewstudy: In our daily work, we have often seen the trend of doing agile forthe sake of doing agile and we wanted to know what the originators thinkabout it.A variety of agile methodologies influenced the manifesto, its values and principles.The aim of this article is to recall the past and to reflect on the present agile world.Therefore, we scrutinized the manifesto from various perspectives. We collected firsthand data to draw an undistorted picture. We combine the results of a survey, inwhich 11 of the 17 original contributors participated, with the outcome of an interview study with 14 of the 17 contributors. Furthermore, we gained insights from aworkshop with Arie van Bennekum who is among the authors of the agile manifesto and also a co-author of this article. We explicitly indicate throughout the articlewhen his individual perspective or opinion is presented. As secondary data, we alsoincluded insights from previously held talks at conferences of given interviews of theoriginal authors.The main contributions of this work consist in:- Collect background information to identify the techniques and methods thatinfluenced the manifesto.- Identify the evolution of interpretations of the statements in the manifesto and futuredirections from the original contributors’ point of view.- Present selected challenges, prerequisites and success factors for an agiletransformation.The remainder of the article is structured as follows. Section 2 discusses related work.Section 3 defines the study approach and the research design, while Section 4 presents anddiscusses the findings of our research. In Section 5, we discuss the findings and concludethe publication in Section 6.2 Background and related workThere are interesting contributions from the time before the rise of the manifesto (Larmanand Basili 2003) and afterwards (Dingsøyr et al. 2012).2.1 Before the manifestoLarman and Basili (2003) describe the history of Iterative and Incremental Software development (IID), starting in the pre-70s and ending up with the manifesto in 2001. Theydescribe the relationship between IID and agile methods as follows: “In February 2001, agroup of 17 process experts [.] interested in promoting modern, simple IID methods andprinciples met in Utah to discuss common ground” ((Larman and Basili 2003), p. 54). Inthe publication by Larman and Basili (2003) they structured the agile history into decades.According to the research of Larman and Basili (2003), the agile mindset started in the1930s with the idea of “plan-do-study-act” cycles. They mention several projects, suchas NASA’s project Mercury (the first human spaceflight program of the United States) orthe software development for the US Navy’s helicopter-to-ship weapon system, which allapplied IID practices. They point out that practices such as short iterations and test-firstdevelopment were already used in the Mercury project. These practices remained presentin the agile methods such as Scrum or XP (Larman and Basili 2003).Page 3 of 27
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15In the seventies, Royce (1970) published an article which is considered as the basis forthe waterfall-model. In his article Royce describes his personal views of managing largesoftware developments and what is“necessary to transform a risky development processinto one that will provide the desired product” ((Royce 1970), p. 335). His approach aimsat reaching the condition of working software delivered on-time and within costs (Royce1970). Royce proposes to use the waterfall model’s phases with an iterative relationshipbetween successive phases. The process benefits by scoping down the development process to manageable limits (Royce 1970). He furthermore suggests using prototypes to getan early simulation of the final product. In addition, he presents first reflections aboutiterative development, feedback and adaption (Larman and Basili 2003; Royce 1970).In the 1980s, many approaches of incremental software development were presented(Wallis 1984; Swartout and Balzer 1982), such as Boehm’s spiral model (Boehm 1986). Thespiral model is a risk-driven approach to software development. According to Boehm, thespiral model resulted from a “risk-management variant of the waterfall model” ((Boehm1986), p. 23). The development process is divided into several circles which comprisesteps like risk analysis, circle planning and reviews. As he points out, “the model holds thateach cycle involves a progression through the same sequence of steps” ((Boehm 1986),p. 24). He concludes that the risk-driven approach is more adaptable than documentdriven (waterfall) and code-driven (evolutionary development) approaches (Boehm1986).The work of Boehm and others present approaches at that time which led to theoccurrence of Rapid Iterative Production Prototyping (RIPP) in the mid-to-late eighties (Buragga and Zaman 2013). This project management methodology was a propertyof Du Pont Information Engineering Association, the consulting business unit of DuPont Co. (Margolis 1988). Scott Shultz, who was a manager at Du Pont at that time,states: “What users tell you they want isn’t always what they really want” ((Margolis1988), p. 110). Direct customer collaboration led to a definite plus with clear benefits(Margolis 1988).James Martin learned about the RIPP methodology from Scott Shultz at Du Pont(Larman 2004). Based on the success of RIPP, Martin promoted the iterative and timeboxed development himself, which led to the creation of the Rapid Application Development (RAD) (Larman 2004). The RAD approach included a quick and iterative deliverywith a small team of highly skilled developers (Abbas et al. 2008). In addition, RAD setthe basis for the Dynamic Systems Development Method (DSDM), which had a wide useraudience in north-west Europe such as the United Kingdom, the Netherlands, Sweden,and Denmark.Several decades of software development and best practices from developmentapproaches and applications in industry ended up in the meeting in Utah, where the mostsuitable techniques were combined within the manifesto for agile software development.2.2 After the manifestoA variety of publications (Dingsøyr et al. 2012; Theocharis et al. 2015) consider theevolution of agile software development after 2001. Diverse theoretical approaches andnumerous points of view on the manifesto, the practices used, the implementation of theprinciples, or human involvement have led to a wide range of publications within the last16 years.Page 4 of 27
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15Dingsøyr et al. (2012) present a literature review of the evolution in the agile community.They present an overview of 1551 papers on agile development until 2010. This reflectsthe still increasing interest in adapting agile methods whose applications are consideredin many different contexts and combinations.Theocharis et al. (2015) conducted a literature study in 2015 resulting in a variety ofdifferent development approaches. The identified approaches often come from a combination of traditional approaches and agile methods. They called those combinationshybrid approaches and identify the method Water-Scrum-Fall (Theocharis et al. 2015) asthe most widespread one. Water-Scrum-Fall is the combination of Scrum and the waterfall model. This model is seen as one approach to the adaption of agile elements intotraditional development approaches.Kuhrmann et al. (2017) set up the HELENA study, an international study on the useof Hybrid dEveLopmENt Approaches in software systems development1 . The HELENAstudy started in 2017 to figure out how hybrid approaches have spread into the development of software worldwide. The HELENA team consisting of more than 80 researchersfrom all over the world investigates the distribution of hybrid software and system development in practice. The HELENA study is ongoing research and final results have notbeen presented yet.In 2002, Abrahamsson et al. (2002) published a literature review aiming at the analysisof agile methods. They analyzed ten methods such as Extreme Programming and Scrumwhich are characterized as “being agile” with respect to the authors’ definition. Furthermore, they compared the similarities and differences of these methods. In contrast to ourapproach, Abrahamsson et al. (2002) based their research on publications instead of asking the original contributors whether the agile teams are “agile” according to the originalcontributors’ understanding.Pathak and Saha (2013) reviewed different agile methods and compared them withconventional process methods. Furthermore, they analyzed pros and cons of applyingthese agile processes to research projects. They also pointed out difficulties in adapting agile methods. The contribution focused on existing literature in that field instead ofpresenting the thoughts of those who started “thinking agile”.In 2002, Highsmith (2002) described the most important principles of agile development from his point of view, such as delivering value to the customer and focusing onindividual developers and their skills. He also presented all major agile methods such asScrum or Extreme Programming. Highsmith (2002) proposed a method of transforminga company into a truly agile organization.3 Research methodThis section presents the data collection consisting of a survey, an interview study, aworkshop with one original contributor, and data provided by the survey participants.3.1 Research questionsThe aim of this study is to preserve the initial thoughts of the original contributorsof the manifesto and to elicit their viewpoint of the future of the manifesto and agiledevelopment in general. We are interested in answering the following research questions:RQ 01: What influences and intentions led to the manifesto for agile softwaredevelopment?Page 5 of 27
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15Page 6 of 27This question focuses on the agile practices and methods that had an influence on themanifesto. Furthermore, it will clarify the original contribution to the manifesto.RQ 02: What do the original authors think about the manifesto today?This question examines if there are parts of the manifesto the original authors wouldskip or change today. It helps to clarify whether the interpretations of the manifesto arestill valid or represent the original ideas.RQ 03: What do the original authors think about the manifesto’s future?This question deals with the necessity of new concepts and adjustments of the manifesto. According to the initial idea of being “agile”, it is necessary to learn and adaptaccording to the given context. Therefore, it is important to rethink the original ideas andto examine their influence on today’s software development in practice.RQ 04: What do the original authors think about the future of agile software development?This question focus on the future directions of agile software development. Furthermore, it presents necessary steps to preserve the origin of agile in the future.3.2 Data collectionThe data collection and analysis bases on four different data sources, which are presentedin this section. An overview can be found in Table 1. The survey and the interview studywere set up at the same time with the same goal to preserve the original agile mindset2 . The workshop resulted from the survey and aims at providing deeper insights intothe agile movement. In order to gain a consistent picture of the ideas, we combined thestudies.3.2.1 SurveyResearch design. According to Yin (2009), the type of the research questions defines theusage of different research methods. The defined research questions are mainly “what”questions which are used in exploratory studies. This class of research question areknowledge questions resulting in a clearer understanding of the phenomena (Easterbrooket al. 2008). According to Forza (2002), surveys can be divided into three different groups:exploratory surveys, descriptive surveys, and explanatory surveys.Our study is founded on a qualitative and a descriptive survey design to obtain knowledge about the manifesto. The survey aims at describing and explaining the phenomenonof the manifesto by using a questionnaire for data collection. A characteristic of survey research is the selection of a representative sample from a well-defined population(Easterbrook et al. 2008). The sample selection is explicitly limited to the 17 originalauthors of the manifesto because we wanted to obtain first-hand information. With thisselection, the sample size is sufficient to represent the population of interest.Table 1 Overview of data sourcesData SourceInitials of executingauthorsInitials of analyzingauthorsNumber ofparticipantsFurtherinformationSurveyP.H., J.K.P.H., J.K.11Section 3.2.1Interview studyR.L., J.G.P.H., J.K.14Section 3.2.2WorkshopP.H., J.K., A.vB.P.H., J.K.3Section 3.2.3Secondary data-P.H., J.K.-Section 3.2.4
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15The survey questions are classified as open questions according to Dreschet al. (2015). The number of questions is limited to five in order to shorten timeto answer the questionnaire. This increases the respondent involvement and thusreduces the amount of missing data (Forza 2002). Furthermore, the participants areasked for permission to use the given answers and information anonymously in thispublication.All survey questions are directly mapped to the research questions.1. Why did you participate in the manifesto for agile software development 16 yearsago? (aims at answering RQ1)2. What was your original contribution to the manifesto for agile softwaredevelopment? (aims at answering RQ1)3. Which parts of the manifesto would you skip or change today? (aims at answeringRQ2)4. Are there necessary new concepts and adjustments for the manifesto? (aims atanswering RQ3)5. What trends will shape the future of agile development from your perspective? (aimsat answering RQ4)Forza (2002) recommends conducting a pilot test of the questionnaire with colleagues,industry experts and target respondents. A pilot test was conducted with anotherresearcher and an industry expert. The feedback they provided led to some modificationsof the questionnaire to make the questions more comprehensible for the participants.Execution. The survey was conducted in January and February 2017. We sent an invitation letter and the questionnaire to all contributors of the manifesto. In total, we received11 answers.In order to collect the data, we chose two different ways to get in touch with the originalcontributors and to send them the questionnaire. First, we searched for email addresses onthe private blogs and homepages. We directly contacted Martin Fowler and Ron Jeffriesvia personal email. Both provided answers to the questionnaire. To get in contact with theother 15 participants, Researcher 1 signed up for a recruiter membership on LinkedIn3 .With the recruiter profile, it was possible to get in contact with the original contributorsand to send them the questionnaire via LinkedIn. The recruiter membership contract wascanceled after one month. Within that month, we received answers from 9 participants.In most of the responses, the items of the questionnaire were directly answered. Thelength of the provided answers varied from one sentence per question to multiple pagesto answer the complete questionnaire.We were invited by one participant to a Skype call to discuss the questions face-toface. The following interview was a one-hour Skype conference in which two researchersand one contributor participated. The interview was recorded by both researchers, andthe meeting minutes were subsequently combined.Two other participants from the study referred to the ongoing project “Agile Uprising Agile Manifesto Review” conducted by Ryan Lockard and James Gifford (2016)(cf. Section 3.2.2).Data analysis. The collected qualitative data was analyzed using categorization anddata coding. According to the classification of Stol et al. (2016), the coding concepts ofStraussian Grounded Theory were used. We used the presented three coding phases: openPage 7 of 27
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15coding, axial coding, and selective coding (Stol et al. 2016; Corbin and Strauss 1990).The interpretive process of open coding generates categories and concepts by breakingdown the data analytically. The written answers were chunked into small, logically relatedphrases and tagged with a code. Each code represents a certain theme, whereas the codesform a hierarchy of codes and sub-codes in the axial coding. In the selective coding,the central categories were defined. This leads to the final result, a formalized body ofknowledge (Runeson and Höst 2009). To analyze, categorize and sort the collected data,we used the tagging and coding functionality in the reference management and citationtool Citavi.3.2.2 Interview studyThe interview study was mainly conducted by Ryan Lockard and James Gifford as partof the “manifesto authors review” in a podcast (Lockard 2016) for the Agile Uprising network. The network focuses on the advancement of the agile mindset and globalprofessional networking between leading agilists. The network purely focuses on theadvancements of the agile craft removing external influences from sponsors, partners andother organizations (Lockard 2016).Research design. Since October 2016, Ryan Lockard and James Gifford have performedan interview study (Lockard 2016), asking the original contributors about the reasons forparticipating in the meeting that ended up with the manifesto.The study has been designed as an exploratory semi-structured interview. For the interviews, an interview guide has been designed. This guide has been developed in an iterativeway. It was pilot-tested and adjusted to the problems which have arisen. The structure ofthe guide is divided into three parts: (1) before the meeting, (2) at the meeting and (3)afterwards (cf. Appendix A)Execution. The interview study took place from October 2016 till March 2017. Gettingin touch with the contributors was an iterative process. Ryan Lockard and James Giffordfirst contacted three of them, namely John Kern, Alistair Cockburn and Bob Martin. In theinterview, they asked them to share the idea of the podcast with other contributors. Thisway, another batch of three contributors to continue was created. In total, 14 contributorswere interviewed. Two of them have been interviewed in person (John Kern and AlistairCockburn), the others have been interviewed via Skype . Skype sessions and face-toface interviews were recorded and afterwards published in the uprising podcast channel.The average duration for an interview is 50 min, whereas the shortest interview, amashup of interviews published by Scrum.org, is 17 min and the longest 1 h and 18 min. Acomplete overview of the conducted interviews with their lengths can be found in Table 2.Data analysis. The data analysis was conducted by Researcher 1 and Researcher 2by coding the meeting minutes. The coding process followed the coding concept presented in Section 3.2.1. The results have been analyzed according to the researchquestions.3.2.3 WorkshopAfter having answered the questionnaire, Arie van Bennekum invited Researcher 1 andResearcher 2 for a 1-day workshop to discuss the origins and future directions of the manifesto. He furthermore suggested, to discuss the future of the agile movement in general.He organized a workshop to discuss these topics in detail.Page 8 of 27
Hohl et al. Journal of Software Engineering Research and Development(2018) 6:15Page 9 of 27Table 2 Overview of the interviews conducted as part of the agile uprising podcastParticipantInterviewerPublication DateLengthArie van BennekumRyan Lockard, Andy Cleff2017-02-1050 minRef.Lockard and Cleff (2016)John KernRyan Lockard2017-02-1157 minLockard (2016a)Martin FowlerRyan Lockard2017-02-1155 minLockard (2017)Stephen MellorRyan Lockard, James Gifford2017-02-1948 minLockard and Gifford (2017)Ken SchwaberMashup of interviewspublished by Scrum.org2017-03-0617 minScrum.org (2017)Jim HighsmithRyan Lockard, James Gifford2017-03-1043 minLockard and Gifford (2017c)Ron JeffriesRyan Lockard, Jason Cusack,Troy Lightfoot2017-03-1052 minLockard et al. (2017)Mike BeedleRyan Lockard, James Gifford2017-03-1049 minLockard and Gifford (2016a)Brian MarrickRyan Lockard, James Gifford2017-03-1078 minLockard and Gifford (2016c)Jeff SutherlandRyan Lockard, Andy Cleff2017-03-1045 minLockard and Gifford (2017a)James GrenningRyan Lockard, James Gifford2017-03-1053 minLockard and Gifford (2017b)Andy HuntRyan Lockard, James Gifford2017-03-1066 minLockard and Gifford (2016b)Alistair CockburnRyan Lockard2017-03-1150 minLockard (2016b)Robert MartinRyan Lockard, James Gifford2017-03-3148 minLockard and Gifford (2016)The workshop took place in Oktober 2017 in Rotterdam, Netherlands. The workshop was separated in two sessions of 2, 5 h each. In the first session, Researcher 1and Researcher 2 presented the results from the survey and discussed the findingswith Arie van Bennekum. In the second session, Arie van Bennekum presented hisview on the future of the agile movement and the importance of agile transformation. Researcher 1 and Researcher 2 wrote meeting minutes and merged them after theworkshop.The data analysis was done by Researcher 1 and Researcher 2, according to thepreviously identified codes from the data analysis of the survey.3.2.4 Secondary dataSome participants of the survey referred to previously held presentations at conferences,and interviews about the manifesto in previous publications. We used this informationas a secondary data source. Therefore, we also transcribed the talk “Agile is Dead” fromthe conference “GOTO Amsterdam” given by Dave Thomas (2017) in 2015. Furthermore, we included an interview with Kent Beck (2011) from 2011. To analyze, categorizeand sort the collected data, we used the tagging and coding functionality in the reference management and citation tool Citavi. The secondary data sets are limited to therecommendations given by the survey participants.3.3 Threats to validityIn accordance with Wohlin et al. (2012), we consider four commonly used criteria forvalidity.Construct validity. The construction of the questionnaire aims at preventing a wronginterpretation of the questions. To minimize the risk of misunderstandings, a pilot testwas conducted and a short explanation was given to the participants. The semi-structuredSkype interview was based on the questionnaire.All conducted interviews were guided by pre-defined open questions (cf. Appendix A).They all started with a description of the interviewee’s background and the reason for attending the meeting in Snowbird in 2001. The focus of the interview
The so-called agile transformation of organizations is a frequently discussed topic because agile ways of working promise to make companies future-proof (Lockard and Cleff 2016). However, the developers often declare themselves "agile" when just using Scrum by the book. In this case, the meaning of agile is
