WIXLIB

ContentsHidden Value in Variable Interactions: Simpson’s ParadoxThe Combinatorial Explosion of InteractionsCorrelationsSpurious CorrelationsBack to CorrelationsCrosstabsChapter 4Data Visualization, Two or Higher Dimensions69ScatterplotsAnscombe’s QuartetScatterplot MatricesOverlaying the Target Variable in SummaryScatterplots in More Than Two Dimensions6971757678The Value of Statistical SignificancePulling It All Together into a Data AuditSummary808182Data PreparationVariable Cleaning8384Incorrect ValuesConsistency in Data FormatsOutliersMultidimensional OutliersMissing ValuesFixing Missing Data848585899091Feature Creation98Simple Variable TransformationsFixing SkewBinning Continuous VariablesNumeric Variable ScalingNominal Variable TransformationOrdinal Variable TransformationsDate and Time Variable FeaturesZIP Code FeaturesWhich Version of a Variable Is Best?Multidimensional FeaturesVariable Selection Prior to ModelingSamplingExample: Why Normalization Matters for K-Means ClusteringChapter Summary143Itemsets and Association RulesTerminology145146ConditionLeft-Hand-Side, Antecedent(s)Right-Hand-Side, Consequent, Output, ConclusionRule (Item Set)ftoc.indd 01:52:15:PM 03/28/2014147148148148Page xvxv

xviContentsSupportAntecedent SupportConfidence, AccuracyLiftParameter SettingsHow the Data Is OrganizedStandard Predictive Modeling Data FormatTransactional FormatVariable SelectionInteraction Variable Creation157157Problems with Association Rules158158158159Building Classification Rules from Association RulesSummary159161Descriptive ModelingData Preparation Issues with Descriptive ModelingPrincipal Component Analysis163164165Clustering AlgorithmsThe K-Means AlgorithmData Preparation for K-MeansSelecting the Number of ClustersThe Kohonen SOM AlgorithmVisualizing Kohonen MapsSimilarities with 197Interpreting Descriptive ModelsStandard Cluster Model Interpretation199199Problems with Interpretation MethodsIdentifying Key Variables in Forming Cluster ModelsCluster PrototypesCluster OutliersChapter 8151152154156The PCA AlgorithmApplying PCA to New DataPCA for Data InterpretationAdditional Considerations before Using PCAThe Effect of Variable Magnitude on PCA ModelsChapter 7151151Measures of Interesting RulesDeploying Association RulesRedundant RulesToo Many RulesToo Few RulesChapter 6149149150150202203209210Summary212Predictive ModelingDecision Trees213214The Decision Tree LandscapeBuilding Decision Treesftoc.indd 01:52:15:PM 03/28/2014Page xvi215218

ContentsDecision Tree Splitting MetricsDecision Tree Knobs and OptionsReweighting Records: PriorsReweighting Records: Misclassification CostsOther Practical Considerations for Decision TreesLogistic Regression230Interpreting Logistic Regression ModelsOther Practical Considerations for Logistic RegressionNeural NetworksK-Nearest Neighbor242244247249251252253253254The k-NN Learning AlgorithmDistance Metrics for k-NNOther Practical Considerations for k-NNNaïve Bayes254258259264Bayes’ TheoremThe Naïve Bayes ClassifierInterpreting Naïve Bayes ClassifiersOther Practical Considerations for Naïve BayesRegression ModelsLinear Regression264268268269270271Linear Regression AssumptionsVariable Selection in Linear RegressionInterpreting Linear Regression ModelsUsing Linear Regression for Classification274276278279Other Regression AlgorithmsSummary280281Assessing Predictive ModelsBatch Approach to Model Assessment283284Percent Correct ClassificationRank-Ordered Approach to Model AssessmentChapter 10233235240Building Blocks: The NeuronNeural Network TrainingThe Flexibility of Neural NetworksNeural Network SettingsNeural Network PruningInterpreting Neural NetworksNeural Network Decision BoundariesOther Practical Considerations for Neural NetworksChapter 9221222224224229284293Assessing Regression ModelsSummary301304Model EnsemblesMotivation for Ensembles307307The Wisdom of CrowdsBias Variance Tradeoff308309Bagging311ftoc.indd 01:52:15:PM 03/28/2014Page xviixvii

xviiiContentsBoostingImprovements to Bagging and BoostingRandom ForestsStochastic Gradient BoostingHeterogeneous EnsemblesChapter 11323323326Text MiningMotivation for Text MiningA Predictive Modeling Approach to Text MiningStructured vs. Unstructured DataWhy Text Mining Is Hard327328329329330Data Preparation StepsPOS TaggingTokensStop Word and Punctuation FiltersCharacter Length and Number FiltersStemmingDictionariesThe Sentiment Polarity Movie Data SetText Mining FeaturesTerm FrequencyInverse Document FrequencyTF-IDFCosine SimilarityMulti-Word Features: N-GramsReducing Keyword FeaturesGrouping TermsModeling with Text Mining FeaturesRegular ExpressionsUses of Regular Expressions in Text 6346347347347349351Summary352Model DeploymentGeneral Deployment Considerations353354Deployment StepsChapter 13320321321Model Ensembles and Occam’s RazorInterpreting Model EnsemblesSummaryText Mining ApplicationsData Sources for Text MiningChapter 12316320355Summary375Case StudiesSurvey Analysis Case Study: Overview377377Business Understanding: Defining the ProblemData UnderstandingData PreparationModelingftoc.indd 01:52:15:PM 03/28/2014Page xviii378380381385

ContentsDeployment: “What-If” AnalysisRevisit ModelsDeploymentSummary and Conclusions391392401401Help Desk Case Study402Data Understanding: Defining the DataData PreparationModelingRevisit Business UnderstandingDeploymentSummary and ConclusionsIndex403403405407409411413ftoc.indd 01:52:15:PM 03/28/2014Page xixxix

IntroductionThe methods behind predictive analytics have a rich history, combining disciplines of mathematics, statistics, social science, and computer science. The label“predictive analytics” is relatively new, however, coming to the forefront only inthe past decade. It stands on the shoulders of other analytics-centric fields suchas data mining, machine learning, statistics, and pattern recognition.This book describes the predictive modeling process from the perspectiveof a practitioner rather than a theoretician. Predictive analytics is both scienceand art. The science is relatively easy to describe but to do the subject justicerequires considerable knowledge of mathematics. I don’t believe a good practitioner needs to understand the mathematics of the algorithms to be able to applythem successfully, any more than a graphic designer needs to understand themathematics of image correction algorithms to apply sharpening filters well.However, the better practitioners understand the effects of changing modelingparameters, the implications of the assumptions of algorithms in predictions,and the limitations of the algorithms, the more successful they will be, especiallyin the most challenging modeling projects.Science is covered in this book, but not in the same depth you will find inacademic treatments of the subject. Even though you won’t find sections describing how to decompose matrices while building linear regression models, thisbook does not treat algorithms as black boxes.The book describes what I would be telling you if you were looking over myshoulder while I was solving a problem, especially when surprises occur inthe data. How can algorithms fool us into thinking their performance is excellent when they are actually brittle? Why did I bin a variable rather than justtransform it numerically? Why did I use logistic regression instead of a neuralnetwork, or vice versa? Why did I build a linear regression model to predict axxiflast.indd01:56:26:PM 03/28/2014Page xxi

xxiiIntroductionbinary outcome? These are the kinds of questions, the art of predictive modeling, that this book addresses directly and indirectly.I don’t claim that the approaches described in this book represent the only wayto solve problems. There are many contingencies you may encounter where theapproaches described in this book are not appropriate. I can hear the commentsalready from other experienced statisticians or data miners asking, “but whatabout . . . ?” or “have you tried doing this . . . ?” I’ve found over the years thatthere are often many ways to solve problems successfully, and the approach youtake depends on many factors, including your personal style in solving problems and the desires of the client for how to solve the problems. However, evenmore often than this, I’ve found that the biggest gains in successful modelingcome more from understanding data than from applying a more sophisticatedalgorithm.How This Book Is OrganizedThe book is organized around the Cross-Industry Standard Process Model forData Mining (CRISP-DM). While the name uses the term “data mining,” thesteps are the same in predictive analytics or any project that includes the building of predictive or statistical models.This isn’t the only framework you can use, but it a well-documented framework that has been around for more than 15 years. CRISP-DM should not beused as a recipe with checkboxes, but rather as a guide during your predictivemodeling projects.The six phases or steps in CRISP-DM:1. Business Understanding2. Data Understanding3. Data Preparation4. Modeling5. Evaluation6. DeploymentAfter an introductory chapter, Business Understanding, Data Understanding,and Data Preparation receive a chapter each in Chapters 2, 3, and 4. The DataUnderstanding and Data Preparation chapters represent more than one-quarterof the pages of technical content of the book, with Data Preparation the largestsingle chapter. This is appropriate because preparing data for predictive modeling is nearly always considered the most time-consuming stage in a predictivemodeling project.flast.indd01:56:26:PM 03/28/2014Page xxii

IntroductionChapter 5 describes association rules, usually considered a modeling method,but one that differs enough from other descriptive and predictive modelingtechniques that it warrants a separate chapter.The modeling methods—descriptive, predictive, and model ensembles—aredescribed in Chapters 6, 8, and 10. Sandwiched in between these are the modeling evaluation or assessment methods for descriptive and predictive modeling,in Chapters 7 and 9, respectively. This sequence is intended to connect moredirectly the modeling methods with how they are evaluated.Chapter 11 takes a side-turn into text mining, a method of analysis of unstructured data that is becoming more mainstream in predictive modeling. Softwarepackages are increasingly including text mining algorithms built into the software or as an add-on package. Chapter 12 describes the sixth phase of predictivemodeling: Model Deployment.Finally, Chapter 13 contains two case studies: one based on work I did as acontractor to Seer Analytics for the YMCA, and the second for a Fortune 500company. The YMCA case study is actually two case studies built into a singlenarrative because we took vastly different approaches to solve the problem.The case studies are written from the perspective of an analyst as he considersdifferent ways to solve the problem, including solutions that were good ideasbut didn’t work well.Throughout the book, I visit many of the problems in predictive modelingthat I have encountered in projects over the past 27 years, but the list certainly isnot exhaustive. Even with the inherent limitations of book format, the thoughtprocess and principles are more important than developing a complete list ofpossible approaches to solving problems.Who Should Read This BookThis book is intended to be read by anyone currently in the field of predictiveanalytics or its related fields, including data mining, statistics, machine learning, data science, and business analytics.For those who are just starting in the field, the book will provide a descriptionof the core principles every modeler needs to know.The book will also help those who wish to enter these fields but aren’t yet there.It does not require a mathematics background beyond pre-calculus to understand predictive analytics methods, though knowledge of calculus and linearalgebra will certainly help. The same applies to statistics. One can understandthe material in this book without a background in statistics; I, for one, have nevertaken a course in statistics. However, understanding statistics certainly providesadditional intuition and insight into the approaches described in this book.flast.indd01:56:26:PM 03/28/2014Page xxiiixxiii

xxivIntroductionTools You Will NeedNo predictive modeling software is needed to understand the concepts in thebook, and no examples are shown that require a particular predictive analyticssoftware package. This was deliberate because the principles and techniquesdescribed in this book are general, and apply to any software package. Whenapplicable, I describe which principles are usually available in software andwhich are rarely available in software.What’s on the WebsiteAll data sets used in examples throughout the textbook are included on theWiley website at www.wiley.com/go/appliedpredictiveanalytics and in theResources section of www.abbottanalytics.com. These data sets include: KDD Cup 1998 data, simplified from the full version Nasadata data set Iris data setIn addition, baseline scripts, workflows, streams, or other documents specificto predictive analytics software will be included as they become available.These will provide baseline processing steps to replicate analyses from thebook chapters.SummaryMy hope is that this book will encourage those who want to be more effectivepredictive modelers to continue to work at their craft. I’ve worked side-by-sidewith dozens of predictive modelers over the years.I always love watching how effective predictive modelers go about solvingproblems, perhaps in ways I had never considered before. For those with moreexperience, my hope is that this book will describe data preparation, modeling,evaluation, and deployment in a way that even experienced modelers haven’tthought of before.flast.indd01:56:26:PM 03/28/2014Page xxiv

CHAPTER1Overview of Predictive AnalyticsA small direct response company had developed dozens of programs in cooperation with major brands to sell books and DVDs. These affinity programswere very successful, but required considerable up-front work to develop thecreative content and determine which customers, already engaged with the brand,were worth the significant marketing spend to purchase the books or DVDson subscription. Typically, they first developed test mailings on a moderatelysized sample to determine if the expected response rates were high enough tojustify a larger program.One analyst with the company identified a way to help the company becomemore profitable. What if one could identify the key characteristics of those whoresponded to the test mailing? Furthermore, what if one could generate a scorefor these customers and determine what minimum score would result in a highenough response rate to make the campaign profitable? The analyst discoveredpredictive analytics techniques that could be used for both purposes, findingkey customer characteristics and using those characteristics to generate a scorethat could be used to determine which customers to mail.Two decades before, the owner of a small company in Virginia had a compelling idea: Improve the accuracy and flexibility of guided munitions usingoptimal control. The owner and president, Roger Barron, began the process ofderiving the complex mathematics behind optimal control using a techniqueknown as variational calculus and hired a graduate student to assist him in thetask. Programmers then implemented the mathematics in computer code so1c01.indd01:52:36:PM 03/28/2014Page 1

2Chapter 1 Overview of Predictive Analyticsthey could simulate thousands of scenarios. For each trajectory, the variationalcalculus minimized the miss distance while maximizing speed at impact aswell as the angle of impact.The variational calculus algorithm succeeded in identifying the optimalsequence of commands: how much the fins (control surfaces) needed to changethe path of the munition to follow the optimal path to the target. The conceptworked in simulation in the thousands of optimal trajectories that were run.Moreover, the mathematics worked on several munitions, one of which wasthe MK82 glide bomb, fitted (in simulation) with an inertial guidance unit tocontrol the fins: an early smart-bomb.There was a problem, however. The variational calculus was so computationally complex that the small computers on-board could not solve the problemin real time. But what if one could estimate the optimal guidance commands atany time during the flight from observable characteristics of the flight? Afterall, the guidance unit can compute where the bomb is in space, how fast it isgoing, and the distance of the target that was programmed into the unit when itwas launched. If the estimates of the optimum guidance commands were closeenough to the actual optimal path, it would be near optimal and still succeed.Predictive models were built to do exactly this. The system was called OptimalPath-to-Go guidance.These two programs designed by two different companies seemingly couldnot be more different. One program knows characteristics of people, such asdemographics and their level of engagement with a brand, and tries to predicta human decision. The second program knows locations of a bomb in space andtries to predict the best physical action for it to hit a target.But they share something in common: They both need to estimate values thatare unknown but tremendously useful. For the affinity programs, the modelsestimate whether or not an individual will respond to a campaign, and for theguidance program, the models estimate the best guidance command. In thissense, these two programs are very similar because they both involve predicting a value or values that are known historically, but are unknown at the timea decision is needed. Not only are these programs rela

of the top-ten predictive analytics blogs to follow. He is a regular speaker at Predictive Analytics World and other analytics . I am very grateful to Eric Siegel for inviting me to speak regularly at Predictive Acknowledgments. xii Acknowledgments ffi rs.indd 01:5