Transcription
Business IntelligenceVol. 4, No. 12Agile Data Warehousing:Incorporating Agile Principlesby Dr. Ken Collier, Senior Consultant, CutterConsortium; with Jim Highsmith, Director,Cutter Agile Project Management PracticeThis Executive Report targets organizations that are considering adata warehousing project, organizations that have struggled toimplement a data warehouse, data warehouse developers, and ITexecutives who are responsible for overseeing such projects. Thereport recommends a leaner approach than that of traditionalmethods; it advocates a working system that meets users’ businessneeds rather than heavily process-centric development approachesthat emphasize documentation and contractual agreements. Armedwith the agile data warehousing approach, organizations can increasethe likelihood of a successful data warehouse implementation on timeand within budget.
Cutter Business Technology CouncilRob AustinTom DeMarcoChristine DavisAccessto theExpertsLynne EllynJim HighsmithTim ListerKen OrrLou MazzucchelliEd YourdonAbout Cutter ConsortiumCutter Consortium is a truly unique IT advisory firm, comprising a group of more than100 internationally recognized experts who have come together to offer content,consulting, and training to our clients. These experts are committed to delivering toplevel, critical, and objective advice. They have done, and are doing, groundbreakingwork in organizations worldwide, helping companies deal with issues in the coreareas of software development and agile project management, enterprisearchitecture, business technology trends and strategies, enterprise risk management,metrics, and sourcing.Cutter offers a different value proposition than other IT research firms: We give youAccess to the Experts. You get practitioners’ points of view, derived from hands-onexperience with the same critical issues you are facing, not the perspective of adesk-bound analyst who can only make predictions and observations on what’shappening in the marketplace. With Cutter Consortium, you get the best practicesand lessons learned from the world’s leading experts; experts who are implementingthese techniques at companies like yours right now.Cutter’s clients are able to tap into its expertise in a variety of formats includingcontent via online advisory services and journals, mentoring, workshops, training,and consulting. And by customizing our information products and training/consulting services, you get the solutions you need, while staying withinyour budget.Cutter Consortium’s philosophy is that there is no single right solution for allenterprises, or all departments within one enterprise, or even all projects within adepartment. Cutter believes that the complexity of the business-technology issuesconfronting corporations today demands multiple detailed perspectives from which acompany can view its opportunities and risks in order to make the right strategic andtactical decisions. The simplistic pronouncements other analyst firms make do nottake into account the unique situation of each organization. This is another reason topresent the several sides to each issue: to enable clients to determine the course ofaction that best fits their unique situation.For more information, contact Cutter Consortium at 1 781 648 8700 orsales@cutter.com.
Agile Data Warehousing:Incorporating Agile PrinciplesBUSINESS INTELLIGENCEADVISORY SERVICEExecutive Report, Vol. 4, No. 12by Dr. Ken Collier, Senior Consultant, Cutter Consortium; with Jim Highsmith, Director,Cutter Agile Project Management PracticeINTRODUCTIONThis Executive Report targetsorganizations considering a datawarehousing project, organizations that have struggled to get adata warehouse implemented,data warehouse developers, andIT executives whose job it is tooversee such projects. By combining experience in data warehousing (Dr. Ken Collier) and in agiledevelopment practices (JimHighsmith), the authors havedeveloped a new approach todata warehousing projects calledagile data warehousing (ADW).While the ADW method incorporates the tried-and-tested fundamentals of data warehousing, itadapts many of the successfulagile software developmentpractices and principles to datawarehousing techniques. Thereport doesn’t suggest a revolutionary change in data architectures, modeling methods, orwarehousing technologies;instead it recommends a leanapproach that emphasizes aworking system that meets users’business needs rather than heavilyprocess-centric developmentapproaches that emphasizedocumentation and contractualagreements.The goal of this report is to changethe ways that data warehousingprojects are implemented. Datawarehousing is difficult, and thereare many accounts of failed projects. Our experience suggeststhat an ADW approach willgreatly increase the likelihoodof successful implementation ofa data warehouse on time andwithin budget.This approach has strong roots inthe work of AgileAlliance members. We have adapted publishedtechniques such as Test-DrivenDevelopment (TDD) from CutterConsortium Senior ConsultantKent Beck, the Framework forIntegrated Test (FIT) from WardCunningham, Agile Modelingfrom Cutter Consortium SeniorConsultant Scott Ambler, andothers to the unique characteristics of data warehouse development. We have also incorporatedexperience and lessons learnedfrom real ADW projects and thework of other experts.
2WHEN DATA WAREHOUSINGGOES BADTraditional data warehouse projects follow a typical waterfalldevelopment model in which rigorous efforts are made to collectcomplete requirements, designcomprehensive architectures anddata models, develop and populate repositories, and, ultimately,develop the analytical reports andartifacts that users want. Theseprojects are complex affairs,involving a project manager leading a team of specialists includingbusiness analysts, data architects,and so forth. Depending on theirmagnitude, these projects generally run at least six months andcan easily exceed US 1 million.I have worked with some talentedand experienced data warehousedevelopers. I’ve also had the benefit of working with savvy clientswho have reasonable expectationsand a pretty stable set of requirements. Sounds like a recipe fortotal success, right? But often,business users are less than ecstatic about the first data warehouseproduction rollout. User reactionscommonly range from “This isn’twhat I was told I would get” to “Ican see where this might be usefulwith some refinement.”Most data warehouse developershave participated in projectsthat were less than successful.BUSINESS INTELLIGENCE ADVISORY SERVICEI recently worked with a midsizecompany seeking to replace itsexisting homegrown reportingapplication with a properly architected data warehouse. At theoutset, the project seemed poisedfor success and user satisfaction.Despite the best efforts of developers, project managers, andstakeholders, however, the project ran over budget and pastdeadline, and users were lessthan thrilled with the outcome.Because this project in particularmotivated the development ofADW, the following offers a briefretrospective to highlight the principles and practices presentedlater in this report.reports. No advanced analyticalcapabilities were provided.Project AttributesExternal drivers. A sales teamfrom a leading worldwide vendorof data warehousing and businessintelligence (BI) software initiallyenvisioned the data warehousingproject. Providing guidance andpre-sales support, the sales teamhelped project sponsors understand the value of eliciting thehelp of experienced BI consultants with knowledge of industrybest practices. But as with manysales efforts, initial estimates ofproject scope, cost, and schedulewere too ambitious.Existing application. Internally,the company’s existing reportingapplication was referred to as a“data warehouse.” In reality,though, the data model was areplication of parts of the legacyoperational database. This replicated database did not includedata scrubbing and was wrappedin a significant amount of customJava code to produce the specified reports required. At various times, users had requestednew custom reports, thus overburdening the application with highlyspecialized and seldom-usedreporting features. All the reportscould be classified as cannedProject motivation. Because theexisting “data warehouse” wasnot architected according to datawarehousing best practices, it hadreached the practical limits ofmaintainability and scalabilityneeded to continue meeting userrequirements. Additionally, withthe new billing system comingonline, it was evident that theexisting system could not easilybe adapted to accommodate thenew data. Therefore, at the executive level, there was strong support for a properly designed datawarehouse.Development team. The development team comprised exclusively external data warehousingconsultants. Because theThe Business Intelligence Advisory Service Executive Report is published by Cutter Consortium, 37 Broadway, Suite 1, Arlington, MA02474-5552, USA. Client Services: Tel: 1 781 641 9876 or, within North America, 1 800 492 1650; Fax: 1 781 648 1950 or, within North America, 1 800 888 1816; E-mail: service@cutter.com; Web site: www.cutter.com. Group Publisher: Kara Letourneau, E-mail: kletourneau@cutter.com.Production Editor: Linda M. Dias, E-mail: ldias@cutter.com. ISSN: 1540-7403. 2004 by Cutter Consortium. All rights reserved. Unauthorized reproduction in any form, including photocopying, faxing, and image scanning, is against the law. Reprints make an excellent training tool. For informationabout reprints and/or back issues, call 1 781 648 8700 or e-mail service@cutter.com.VOL. 4, NO. 12www.cutter.com
3EXECUTIVE REPORTcompany’s existing IT staff hadother high-priority responsibilities,the development team did nothave extensive knowledge of thebusiness or existing operationalsystems. The development teamdid, however, have open accessto business and technical expertswithin the company as well astechnology experts from thecompany’s software vendor.While initial discovery effortswere challenging, all stakeholdersparticipated extensively.Customer role. The company’sfinance division filled the roleof the primary “customer” forthe new data warehouse, andthe CFO sponsored the project.Together, the customer team andthe CFO had a relatively focusedbusiness goal of gaining more reliable access to revenue and profitability information. They also hada substantial amount of existingreports used in business analysison a routine basis, offering a reasonable basis for requirementsanalysis.Project management. CorporateIT handled project managerresponsibilities using traditionalProject Management Institute(PMI) practices. The IT group wassimultaneously involved in twoother large development projects,both of which had direct or indirect impact on the scope of thedata warehouse project.Hosted environment. Due to limited resources and infrastructure,the company’s IT leadership had 2004 CUTTER CONSORTIUMrecently decided to partner withan ASP to provide hosting servicesfor newly developed productionsystems. The data warehouse wasexpected to reside at the hostingfacility located on the West Coastof the US, while company headquarters were located on the EastCoast. Because legacy systemsremained on the corporate infrastructure, separate geographiclocations for the warehouse andheadquarters created complications — though not insurmountable ones — for the movement oflarge volumes of data.Project OutcomeThe original project plan called foran initial data warehouse launchwithin 90 days. As any experienced data warehouse developerknows, such a deadline imposesan ambitious, but not impossible,schedule (assuming appropriatescope definition). But the datawarehouse launch for this projectcame a full eight months afterproject start. User acceptancetesting did not go well. Userswere already annoyed with project delays, and when they finallysaw the promised features, therewas a significant gap betweenuser expectations and the endproduct. As is common with lateprojects, staff members wereadded to the development teammidstream to try and get the project back on track. So project costsfar exceeded the planned budget,and the project was placed onhold until further planning couldbe done to justify continueddevelopment.RetrospectiveSo who was to blame? Usersthought that developers hadmissed the mark and failed toimplement all their requirements.Developers believed that userexpectations had not been properly managed and thus projectscope had grown out of control.Project sponsors thought that thevendor and the consulting firmhad overpromised and underdelivered. The vendor and consultingfirm believed that internal politicsand organizational issues were toblame. Finally, many members ofthe company’s IT staff felt threatened by their own lack of ownership on the project and secretlycelebrated the failure.The project degenerated into aseries of meetings to review contracts and project documents todetermine who should be heldresponsible. And guess what?Everyone involved was partiallyto blame. In addition to the common technical challenges of dataextraction, integration, and cleansing, the following were identifiedas root causes of project failure:n The project contract did notsufficiently define the projectscope.n Requirements were incomplete, vague, and open-ended.n There were conflicting interpretations of the previouslyVOL. 4, NO. 12
4approved requirements anddesign documents. Developers put in long nightsand weekends in a chaoticattempt to respond to userchanges and new demands. Developers did not fully understand user requirements orexpectations and did notmanage requirementschanges well. Users had significant misconceptions about the purpose ofa data warehouse since theprevailing understanding of awarehouse was based on theprevious reporting application(which was not a good model). Vendors made ambitiouspromises that developers couldnot fulfill in the time available. T
data warehouse implemented, data warehouse developers, and IT executives whose job it is to oversee such projects. By combin-ing experience in data warehous-ing (Dr. Ken Collier) and in agile development practices (Jim Highsmith), the authors have developed a new approach to data warehousing projects called agile data warehousing (ADW).
