Transcription
Software Solutions Symposium 2017Agile Project Success andFailure(The Story of the FBISentinel Program)Thomas FriendSoftware Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 152131
Software Solutions Symposium 2017Tom Friend - BioTom FriendIntroductionsCorporate ITMilitary / Aviation Agile Scrum Coach 12 Years Agile Scrum 25 Year IT Application Dev Airline Transport Pilot US Naval & Air ForcePilot Air War College Squadron Commander B.S. Aeronautics2
Software Solutions Symposium 20171.2.3.4.Why is this case study is of value?Stress Systematic FailureTell the FBI StoryFederal Agile Tools and programs3 3
Software Solutions Symposium 2017Why? The Old Way is Not Working!1. Traditional programs are train wrecks2. Long Cycle can’t react to change3. Base assumptions are not validCost!!! Time-Effort-Opportunity-Money4 4
Software Solutions Symposium 2017Overview of the Cost of Federal Failures1.2.3.4.Federal IT is 78 Billion90% of large IT projects failOver half were miss expectations41% failed completely - 32 Billion Waste!5 5
Software Solutions Symposium 2017How Much is a Billion Dollars in 100 Dollar Bills?6 6
Software Solutions Symposium 2017Original Source of FBI StoryBrian Wernham1. Director at The Association for Project Management2. Delivery of 100m IT-enabled programs3. Business Case development at Director Level 7 7
Software Solutions Symposium 2017Brian’s Research Focus Area1. Brian’s research focused on critical, main-streamFederal projects.2. Projects that had used various methods.3. Based on the FBI case study, here is the questionthat we will address together today: What evidence it there that Agile is suitable for largescale projects?8 8
Software Solutions Symposium 2017Viewpoints on Project Management Approaches1. Normal argument is Waterfall or Agile.2. The Federal Waterfall approach will tend towardswhat Kent Beck called ‘Big Design Up Front’(BDUF). Due to BDUF being the starting point for waterfall3. Brian stresses that we should aim for EnoughDesign Up-Front (EDUF), not BDUF.9 9
Software Solutions Symposium 2017What is Big Design Up Front (BDUF)1. ‘The Simpsons’ Homer the average American, isthe perfect person to design a new car.2. The car turns out to be totally unusable and tooexpensive to produce!1010
Software Solutions Symposium 2017Remembering Failed DOD BDUF ProgramsWe Repeat What We Forget1111
Software Solutions Symposium 2017Airborne Laser (ABL) 5,000,000,00012
Software Solutions Symposium 2017XM2001 Crusader 2,000,000,0001313
Software Solutions Symposium 2017Expeditionary Fighting Vehicle (EFV) 3,300,000,0001414
Software Solutions Symposium 2017RAH66 Comanche 6,900,000,0001515
Software Solutions Symposium 2017Brian’s Case Study:A Story of Agile Success at the FBIThe following FBI projects detail a fascinating case study ofproject methodologies and how Agile leadership deliveredwhere traditional approaches failed.Three reasons this case study is relevant:1. Size: This was a large project - hundreds of millions of dollars:Agile Scales.2. Prior Failure: This Agile project succeeded where twoprevious waterfall projects using the same technology hadfailed.3. Visibility: This was a high-profile and mission-critical project in government. Precisely the environment and scale at whichAgile is often misconstrued as being inappropriate.1616
Software Solutions Symposium 2017Case Study: History (Part 1)1. 2001 FBI Agent Robert Hanssen arrested2. 1.4 million in payments3. 20 separate occasions1717
Software Solutions Symposium 2017Case Study: History (Part 2)1. Three months later, inMay 2001, the OklahomaCity bomber was about tobe executed.2. 700 documents had notbeen disclosed to thedefense.3. The FBI had forgot tosend and lost evidence.4. Stay of execution wasgranted the FBI cameunder severe criticism.1818
Software Solutions Symposium 2017FBI Looks for a Solution1. Root Cause: old computer system and outdated manualprocesses2. A secure and reliable set of systems and processes wererequired3. The FBI sets up a project to build a new Virtual Case Filesystem (VCF)a) One 400 contractb) Brand new system as no market alternatives were deemed sufficient4. (SAIC) won the bid and created a classic waterfall projecta) 200 person team spent 6 months creating the requirementsb) A grand design upfrontc) Go live at once - a classic big-bang1919
Software Solutions Symposium 2017Scope Change 9/11 - 20011. The 9/11 attacks increased political pressure for betterhomeland security and data sharing between agencies.2. Responding to this pressure, the FBI made promises theycould not keep.3. The FBI received an additional 78M of funding forcompliance and also promised to chop off another 6 monthsfrom the schedule.4. Summarya)b)c)d)Waterfall-style Big Bang rollout 400 Million 78 Million Additional Funding300 person team for Requirements 6 months 600 pages oflisted requirements2020
Software Solutions Symposium 2017Results of First Attempt Over 4 YearsFour CIO’s – one per year1. 2002 – no delivery2. 2003 – no delivery3. 2004 – no delivery4. 2005 – no deliveryFour years of no results!2121
Software Solutions Symposium 2017Results of First Attempt 1. The 600 pages of requirements2. 400 documented ‘change requests’,3. 700,000 lines of program code had been being writtenand re-written time and time again.4. 1,200 pages requirements documents5. Classic Big Design Up Front.2222
Software Solutions Symposium 2017Final Disposition FailureProgram Cancelled1. A 318 page post-mortem report concluded that thecontractor had: “badly bungled the project – it shouldbe abandoned the software is incomplete,inadequate and (incompetently) designed -essentiallyunusable under real world conditions.”2. Randolph Hite, GAO, said: "When you do a programlike this, you need to apply a level of rigor anddiscipline that's very high.”2323
Software Solutions Symposium 2017The Need is Still There .1. That year, yet another FBI agent, LeandroAragoncillo, is arrested.2. The ancient FBI system had failed to spot hissuspicious behavior as he fished through theexisting case-management systems for over 9months.3. Obviously the need for a new system remained.2424
Software Solutions Symposium 2017When you Fail, Blame the Vendor andTry Again with More Money!1. Lockheed Martin wins the new ‘Sentinel’ system project 3years 2006 to 2009. The total project budget was 425M.a) 305M was budgeted for Lockheed Martin.b) 120M was allocated for the FBI to run a massive program officeto carry out detailed and prescriptive oversight of the work.2. That's one quarter of the budget being spent on planningand control of the contractor!2525
Software Solutions Symposium 2017The Sentinel Project Second TryThe new project run as Waterfall1. 60M spent just creating a web-based front-end to thealready broken system in-place.2. Some agents called it “lipstick on a pig”.3. Due to its incompleteness, they stopped using it.2626
Software Solutions Symposium 2017Took Their Eye Off the Ball1. Overly optimistic reporting.2. In 2010, the stakeholdersrejected the system eventhough it was compliant withthe original specifications.3. The dream of implementingelectronic information sharingsystem was shattered.2727
Software Solutions Symposium 2017Time to Get Serious - Remove Impediments1.2.3.4.Congressional inquiryProject internalizedThe FBI CIO takes ownershipAgile is adopted as the project frameworka)b)c)d)e)f)Design is broken into 670 user storiesSelf-organizing teams45 staff (not 300 as previous)Product Owner prioritized the workTwo week sprintsDemo every sprint2828
Software Solutions Symposium 2017Outcome, Rubber Meets Road1. After a few sprints, it became possible to forecastthe rough timescales and start to plan the datesfor incremental business change and adoption ofreleases of the new software.2. System delivered using only half of the budget.3. Agents used the system on real cases. In the firstquarter of its use, over 13,000 agents progressedover 600 cases, meeting or exceeding allexpected targets.4. The old mainframe system was turned off.2929
Software Solutions Symposium 2017Outcome in Dollars and Cents1. The three-year Agile project delivered therequested system and improvements.2. A success after 10 years of failure and 600million wasted on the two previous aborted'Waterfall' attempts.3. Total cost of only 99 million.3030
Software Solutions Symposium 2017“Lessons Learned” From This?1. Patient Protection andAffordable Care Act (PPACA).2. Healthcare.gov changed theconversation.3. HealthCare.gov went live in2013.3131
Software Solutions Symposium 2017US Federal Agile Agency Leaders WereCalled Upon for their Lessons Learned1.2.3.4.FBI Federal Bureau of InvestigationUSCIS US Customs and Immigration Service.Veterans AdministrationUSPTO Patent and Trademark Office3232
Software Solutions Symposium 2017Directives1. U.S. Digital Services Playbook (Website)2. The TechFAR Handbook (GitHub Collaboration)3. Leveraging Best Practices to Help EnsureSuccessful Major Acquisitions (GAO, Feb 2014)3333
Software Solutions Symposium 2017Digital Services Playbook3434
Software Solutions Symposium 2017Purpose of the Playbook1. Establishing standards to bring the government’sdigital services in line with the best private sectorservices.2. Identifying common technology patterns that willhelp us scale services effectively.3. Collaborating with agencies to identify and addressgaps in their capacity to design, develop, deploy andoperate excellent citizen-facing services.4. Providing accountability to ensure agencies seeresults.3535
Software Solutions Symposium 2017Tech-Far Handbook GitHub3636
Software Solutions Symposium 20173737
Software Solutions Symposium 2017In the End, What We Need is Agile Leadership1. We need proof that Agile leadership will bringsuccess.2. We need people to lead projects, not manage them.3. Agile Leadership Behaviorsa.b.c.d.e.f.g.h.Satisfy the customerHarness ChangeBe very incrementalCreate trust through ‘high touch’ leadershipEncourage face to face conversationsSet targets and reword progressPursue simplicity, not complexityGive team space they need to excel3838
Software Solutions Symposium 2017Questions?www.TomFriend.com39
Software Solutions Symposium 2017Sources1. The Agile approach saves the FBI Sentinel ProjectBrian Wernham FBCS FAPM2. SOFTWARE DEVELOPMENT; Effective Practicesand Federal Challenges in Applying Agile MethodsUnited States Government Accountability Officea) David A. Powner, (202) 512-9286 or pownerd@gao.govb) Dr. Nabajyoti Barkakati, (202) 512-4499 orbarkakatin@gao.gov4040
Software Solutions Symposium 2017Title of the Presentation Goes Here41
Agile Scales. 2. Prior Failure: This Agile project succeeded where two previous waterfall projects using the same technology had failed. 3. Visibility: This was a high-profile and mission-critical project - in government. Precisely the environment and scale at which Agile is often mi
