Transcription
Using Analytics To IncreaseCustomer Engagement and Revenue Tom Coghlan, CMA, VillanovaUniversity Center for BusinessAnalytics Thom Rossi, PMP, ProjectManagementInstitute (PMI)#IMA16ACE
AgendaFramework for Benefits of AnalyticsIllustrate Use of Analytics via ProjectManagement Institute Case Study#IMA16ACE2
Learning Objectives1. Describe analytical techniques that can beused to collaborate with operations andinformation technology.2. Recognize how a successful organizationdeveloped and deployed an analyticsstrategy to achieve worldwide benefits.3. Identify analytics tips that can beimplemented in the workplace, regardlessof size or technical sophistication.#IMA16ACE3
The Story Starts With The CMAExam Johnson & Johnson.Imperial Chemical Industries (ICI)IT consulting / Business Intelligence.Villanova University.Managerial Accounting in MBA program.Special Interest Group in Analytics.#IMA16ACE4
What is Analytics?“The extensive use of data, statisticaland quantitative analysis, explanatoryand predictive models, and fact-basedmanagement to drive decisions andactions”.Thomas Davenport and Jeanne Harris“Competing on Analytics”#IMA16ACE5
Analytics Is A Journey“Competing onAnalytics”By Davenport and yCompetitor#IMA16ACE6
Getting Started With Analytics“Analytics at Work”By Davenport, Harris andMorrisonDELTA MODEL Data Enterprise Leadership Targets Analysts#IMA16ACE7
Stages of AnalyticsPRESCRIPTIVE Holistic optimization What's the best we can do?PREDICTIVE Statistical Modeling What will happen next?DIAGNOSTIC Ad-Hoc Analysis Why did it happen?DESCRIPTIVE Data is historical Decisions areabout the future Leverageknowledge andtechnology tobridge the gap Automated reporting What happened?#IMA16ACE8
What, Why, What, What?#IMA16ACE9
Where To Start?Balanced Scorecard Perspectives1. Customer2. Internal Processes3. Learning and Innovation4. Financial#IMA16ACE10
2016 Top Issues for CFO’s“The Digital Finance Imperative”, Oraclesponsored research study conducted by theAICPA and CIMA. The survey asked respondents to rank themost important determinants of value. The top 3 are:1. Customer satisfaction (76%).2. Quality of business processes (64%).3. Customer relationships (63%).#IMA16ACE11
Importance of Intangible AssetsIntangible assets comprised 84% of the S&P 500corporate valuations in Jan 2015, up from 20% in1975. 2015 study by Ocean Tomo, Mgt. AdvisoryFirmUber – Worlds largest car service; owns no cars.Apple – Outsources almost its entire value chain;yet has over 175 billion of capital. iCloud is run on outsourced hardware.#IMA16ACE12
Villanova Center for BusinessAnalyticsThe mission of the Center for BusinessAnalytics is to spearhead the development ofthe analytical capabilities of business students,conduct applied research, and disseminatebest practices to improve organizationalperformance.#IMA16ACE13
Strategic PillarsProvide focus for the development, executionand tracking of strategic objectives.1. Curriculum2. Research3. Practice#IMA16ACE14
About PMI World’s leading not-for-profit professionalmembership association for the managementof: Projects Programs Portfolios Over 2 million project managementprofessionals in nearly every country#IMA16ACE15
IMA and PMI Similarities1.2.3.4.5.6.7.International.Members pay annual dues.Certifications.Continuing education (CPE and PDU).Local chapters.Thought leadership.Annual meetings.#IMA16ACE16
PMI Member EngagementGlobally mic ties#IMA16ACETools17
Keeping Track of MembersCertificationsElectronicDownloadsData WarehouseChapters#IMA16ACE18
Understanding MembersData Warehouse#IMA16ACE19
Origin of PMI/Villanova Collaboration PMI internal resources constrained Interns/Co-ops 6 month duration Villanova Center for Business AnalyticsConsulting Phased project approach Scope of work with deliverables Timeline with milestones and fixed price#IMA16ACE20
How You Start Is How You Finish1. Start with end in mindScope, deliverables and timeline clearly defined2. Integrated project teamClient: Sponsor, managers, subject matter expertsCenter for Business Analytics: Student consultants,technical advisors, project manager/mentor3. Communications are keyInternal team, PMI/Villanova weekly calls; on-sitecheckpoint reviews#IMA16ACE21
Need Data To Prove “Beliefs”PMI “Believed”1. There was a drop-off in member renewalsafter the initial membership year.2. After members renew the first time(renew for their second year) they werelikely keep renewing in subsequent years.#IMA16ACE22
First Two Deliverables#1. Determine if there are differentiatingfactors between:Members who renewed theirmemberships versusMembers who do not renew theirmemberships.#2. Based on the findings of #1, develop amodel to predict the likelihood of memberrenewal.#IMA16ACE23
Some Technical DetailsCloud Development Environment Fully secure. View of PMI data warehouse. No member personal identity.information; member # and country. Over 100 tables, updated monthly. Access via remote desktop connection.#IMA16ACE24
Scope of Member Data AnalyzedPMI members who joined between 01/01/2003and 01/01/2013. Exclude members who joined after1/1/2011 Retention for three years after joining PMI. Includes approximately 850,000 members.“Join Year” People who joined in the same calendaryear.#IMA16ACE25
Key Attributes Person ID (identity is anonymized)Join dateRegionCertified (PMP, others)Chapter memberElectronic downloads#IMA16ACE26
Early Finding: Need for Visualization Number of members who join each year isincreasing Increasing from 2006-Present Overall retention rates are decreasing Decreasing from 2005-Present#IMA16ACE27
Start By Visualizing the DataNumber of times renewed3 2Retention# of times amemberrenewed theirmembership10#IMA16ACE28
Retention By Join Year#IMA16ACE29
Present Insight, Not DataFor insights to be actionable, you need to : Develop an understanding of what leversare available to the business. Conceptualize how those recommendationscan impact the business.#IMA16ACE30
Impact of Certification on RenewalsCertification increases membership length 55% of members are certified Nearly twice as likely to renew 1st year ifcertifiedFour times as likely to remain a member for 4years or greater 37% of certified members were membersfor over 4 years.#IMA16ACE31
Impact of Certification on Renewals#IMA16ACE32
Impact of Joining a ChapterJoining a chapter increases membershiplengthTwice as likely to renew 1st year if in achapterOver three times as likely to remain amember for 4 years or greater#IMA16ACE33
Impact of Joining a Chapter#IMA16ACE34
Developing a Prediction Model Data in the data warehouse In-depth profile of all members. Data about who did or did notrenew. Data became knowledge Which attributes contribute mostto member renewal. Predict whether a specific memberwill renew or not.#IMA16ACE35
Best Practice in Developing Models Building the model TrainingTestingValidation Choose rules to bestpredict the training setAdjust rules based onhow they perform fora validation set Testing the model easure the accuracyby how well rulesperform on a hold outtesting set36
Decision Tree Analysis Determine most important predictors of1st year membership renewal Categorize into 2 groups:1. Members who renewed2. Member who did not renew Establish set of rules to predict whether ornot someone will renew#IMA16ACE37
Decision TreeAll RowsLevel1Level2Level3Level41:PMPCertified3:All OtherRegions2:Not PMPCertified4:AsiaPacific7:ChapterSize 1155:ChapterSize 608:ChapterSize MA16ACE10:DegreeMissing6:ChapterSize 6011:EMEA &NorthAmerica12:All OtherRegions15:AnnualDownloads 116:AnnualDownloads 138
Measuring Model Accuracy:“Confusion Matrix”#IMA16ACE39
Key Learning: Avoid Being TooTechnicalThe term “confusion matrix” is widely used indata mining. True positive False positive True negative False negativeLabeling the analysis of the accuracy as a“confusion matrix” confused most of theattendees at a meeting.#IMA16ACE40
Initial Model AccuracyInitial model had 70% prediction accuracy Correctly predicts renewal in 69.8% of cases. Correctly predicts non-renewal in 70% of cases.#IMA16ACE41
About the First Model “Member Lifetime Experience” Model Analyzes engagement from date memberjoins PMI. Determined most important lifetimeattributes that predict membershiprenewal. E.g. “Was member ever certified?” “Was member ever a chapter member?#IMA16ACE42
Phased Approach Start small.Deliver.Show success.Develop credibility.Look to extend initial success with anothersmall project.#IMA16ACE43
Phase Two1. Test existing “Member LifetimeExperience” model.2. Develop new “Member Renewal Decision”model3. Analyze new factors such as chapter size,regional differences and gender.#IMA16ACE44
Member Renewal Decision Model Second model developed. Analyzes current level of memberengagement at the time of renewaldecision. Determined most important attributesthat predict membership renewal. E.g. “Is the member currently certified?” “Is the member currently a chaptermember?#IMA16ACE45
Differences in the ModelsCritical Moment of Renewal What attributes are pertinent at the timemembers decide to renew?Member Lifetime Experience model Members who ever became certifiedwere more likely to renew.Member Renewal Decision model Members who were certified at the timeof renewal are more likely to renew.#IMA16ACE46
Answering a Business QuestionBusiness Question: Are there regional differencesin the factors that influence renewal and nonrenewal? Developed regional member renewal decisionmodels: North America EMEA Latin America Asia Pacific (China only and without China)#IMA16ACE47
Regional ModelsExcept for Asia-Pacific, the regional models did notreveal any major differences from the worldwide “allmembers "model. PMP certified is the first indicator for predictingrenewal in all regions except in Asia-Pacific Chapter size is the first indicator for predictingrenewal in Asia-Pacific.#IMA16ACE48
Key PMI Takeaways and Learnings1.2.3.4.Organization needs to buy-in.Start small.Some “anecdotal” beliefs were proved.Data “under the microscope” needs to beclean.5. Correlation does not lead to causation.6. Educated the organization on what ispossible.#IMA16ACE49
Correlation Does Not Lead toCausation#IMA16ACE50
Thanks!Tom Coghlan Thomas.coghlan@villanova.eduThom Rossi trossi@pmi.org#IMA16ACE51
Villanova Center for Business Analytics 13 The mission of the Center for Business Analytics is to spearhead the development of the analytical capabilities of business students, conduct applied research, and disseminate best practices to improve organizational performance.
