WIXLIB

Predictive Modeling using SASEnterprise Miner and SAS/STAT :Principles and Best PracticesCAROLYN OLSEN & DANIEL FUHRMANN

1OverviewThis presentation will: Provide a brief introduction of how to set upanalytics projects in SAS Enterprise Miner Highlight some of the main statisticalconcepts applied in a predictive modelingproject Outline three supervised learning techniquesIt will not:‐ Offer in‐depth coverage of advanced statistics‐ Be a complete tutorial for analytics in SAS EG or SAS EM‐ Be comprehensive by any stretch of imagination. Consider this presentation astarting point for your analytics journey.WIILSU 2016 ‐ OLSEN & FUHRMANN2

Outline1 Introduction2 Business questions & analysis process3 SEMMA methodology & project work flow4 Considerations for predictive modeling5 Selected supervised learning techniques6 Model Assessment

introductionWIILSU 2016 ‐ OLSEN & FUHRMANN4

1It all starts with WIILSU 2016 ‐ OLSEN & FUHRMANN5

1The answers might come via Predictive modeling: The use of known, historical data and mathematicaltechniques (statistical algorithms) to develop models that predict futureevents.Typically, the end result is to streamline decision‐making and to create newinsights that lead to better actions or outcomes. Therefore, predictivemodeling is widely used in industry to more accurately answer businessquestions that improve performance, increase revenues, or reduce costs.WIILSU 2016 ‐ OLSEN & FUHRMANN6

1Predictive Modeling Use Casesin Insurance Response model Default model (claims model) Retention model Cross‐sell / Up‐sell model Fraud model Lifetime Value model Process Failure model . and the list goes on.WIILSU 2016 ‐ OLSEN & FUHRMANN7

business questions &analysis processWIILSU 2016 ‐ OLSEN & FUHRMANN8

Characteristics of AnalyticBusiness QuestionsAnalysis ComplexityAnalysis paradigm2 Descriptive or advanced analytics, or both? Statistics or data mining?Data preparation paradigm As much data as possible or business knowledge first?Analysis method Supervised or unsupervised analysis?Scoring needed Yes or No?Periodicity of analysis One‐time deal or re‐run(s) anticipated?Historic data needed Yes or No?Data structure One row or multiple rows per subject?Analytics team complexity Communication and documentation needed?WIILSU 2016 ‐ OLSEN & FUHRMANN9

The Analysis Process (short)2Select an analysis methodIdentify data sourcesPrepare the dataExecute the analysisInterpret and evaluate the resultsAutomate data preparation and execution of analysisWIILSU 2016 ‐ OLSEN & FUHRMANN10

The Analysis Process tcasesGatherresultsExtractinput dataIntegratedeploymentValidateinput dataGeneratedeploymentmethodsRepairinput dataApplyanalysisTransforminput dataWIILSU 2016 ‐ OLSEN & FUHRMANN11

semma methodology &process flow in SAS EMWIILSU 2016 ‐ OLSEN & FUHRMANN12

3Getting Started in SAS EM

3Menu and Project PanelMenu/Task Bar and ShortcutsProject Panel – Displays your datasources, diagrams, and modelpackages.WIILSU 2016 ‐ OLSEN & FUHRMANN14

3Properties and HelpProperties Panel – User can makeadjustments to Node propertieshere.Help Panel – Displays brief yethelpful explanations of nodeproperties and settings.WIILSU 2016 ‐ OLSEN & FUHRMANN15

3Diagram Workspace &SEMMA Tools TabsDiagram Workspace – User creates alogical process flow for the data miningproject here. Information flows fromnode to node in the direction of theconnecting arrows.SEMMA & Tools Tabs– Functionalities tosample, explore, modify, model, andasses are found here along with anyadditional licensed utilities (e.g. TextMiner, Credit Scoring, etc.)WIILSU 2016 ‐ OLSEN & FUHRMANN16

3Reality bites WIILSU 2016 ‐ OLSEN & FUHRMANN17

3WIILSU 2016 ‐ OLSEN & FUHRMANN18

considerations forpredictive modelingWIILSU 2016 ‐ OLSEN & FUHRMANN419

OutlineI) Introduction2) Business questions & analysis process3) SEMMA methodology & project work flow in SAS Enterprise Miner4) Considerations for predictive modeling are outlined next, they includedata preparation (definitions of target and observation window, offsetwindow, sampling, missing value treatment, and variabletransformations), variable selection methods, overfitting, validationmethods, and criteria to assess model performance. This paper concludeswith an introduction to several supervised learning techniques(regression, decision trees, neural networks, and ensemble models) andapplications of these methods to a use case in the insurance industry. Theintended audience for this paper are beginning and intermediate‐level datascientists, analysts, and modelers. While the paper focuses on applicationsin SAS Enterprise Miner, it provides alternative approaches implemented inSAS/STAT, wherever possible.WIILSU 2016 ‐ OLSEN & FUHRMANN20

4Data Structures and DataModeling – Things to consider The Origin of Data Data Models Analysis Subjects and Multiple Observations Data Mart Structures No analysis subject available? For purposes of predictive modeling – it is important to decide thegrain or event you want to predict. i.e. the structure and design of your dependent variable(s)WIILSU 2016 ‐ OLSEN & FUHRMANN21

4Sample – How toSAS ENTERPRISE GUIDESAS ENTERPRISE MINER PROC SURVEYSELECT SAS CODE Node PROC IML Sample NodeWIILSU 2016 ‐ OLSEN & FUHRMANN22

4Definition of EDAExploratory Data Analysis (EDA) is an approach/philosophyfor data analysis that employs a variety of techniques(mostly graphical) to maximize insight into a data set: uncover underlying structure; extract important variables; detect outliers and anomalies; test underlying assumptions; develop parsimonious models; and determine optimal factor settings.WIILSU 2016 ‐ OLSEN & FUHRMANN23

4Explore – How toSAS ENTERPRISE MINERSAS ENTERPRISE GUIDE PROC UNIVARIATE & CORR SAS CODE Node PROC SGPLOT & MI StatExplore, GraphExplore, MultiplotWIILSU 2016 ‐ OLSEN & FUHRMANN24

4Suggested Steps after EDACheck for missing values (MCAR, MAR, MNAR) Possible actions: imputation, leave as is, missing value flags, etc.Look for univariate (and multivariate) outliers Possible actions: drop obs, Winsorizing, value correction if erroneousentry, etc.Investigate variable distributions for ill‐behaved items Possible actions: transformation, binning, etc.Check if multicollinearity is present Possible actions: drop input(s), create factors or principal components, donothingDiscuss any existing business rules around data with IT Expert andBusiness Client, ideally before taking any of the actions above.WIILSU 2016 ‐ OLSEN & FUHRMANN25

4Modify ‐ Set up ofIndependent VariablesBasic Variables Continuous or Interval Variables Categorical (nominal or ordinal) Dummy VariablesIntermediate Variables Interaction Effects Transformed (Power, Logarithmic, etc.) Binning Interval to Categorical VariablesWIILSU 2016 ‐ OLSEN & FUHRMANN26

4Modify ‐ Important VariableCriteriaSufficiency All potentially relevant information from the available source systemsis also in the analysis table.Efficiency Keep number of variables as small as possible (parsimony).Relevance Data are gathered/aggregated in such a form that the (derived)variables are suitable for the analysis and business question.Interpretability Variables that are used for analysis can be interpreted and aremeaningful from a business point of view.WIILSU 2016 ‐ OLSEN & FUHRMANN27

4Modify – How toSAS ENTERPRISE GUIDESAS ENTERPRISE MINER DATA step SAS CODE Node Specify interactions, polynomialswithin analytical tasksWIILSU 2016 ‐ OLSEN & FUHRMANN28

selected supervisedlearning techniquesWIILSU 2016 ‐ OLSEN & FUHRMANN29

5Logistic RegressionPURPOSEUse multiple x variables (inputs) to estimatep (output), the probability of an event y. Thisis a supervised technique that requireshistorical data with known outcomes.METHOD VISUALIZATIONFor a probability p of an event, the odds ofthe event are p/(1‐p). A logistic regressionmodels the log of the odds of the event,using a linear function:logit log(p/(1‐p)) β0 β1X1 β2X2 βPXPThe coefficients in this equation aredetermined using “maximum likelihoodestimation,” an iterative process that beginswith a tentative solution then repeatsrevision until improvement is minute – atwhich point the process is said to have“converged.”WIILSU 2016 ‐ OLSEN & FUHRMANN30

5Logistic RegressionIMPORTANT CONSIDERATIONSMODEL VALIDATIONThere are assumptions which mustbe met for a logistic regression to beunbiased, and there are importantconsiderations around data setstructure and defining the outcome.Additionally, with small samples,maximum likelihood estimation isknown to be biased.There is a variety of measures used tomeasure how strong a classificationmodel is: the confusion matrix, whichgives true and false positive rates andtrue and false negative rates; accuracy;misclassification rate; positivepredictive value (PPV), negativepredictive value (NPV); and others.Review the Receiver OperativeCharacteristic (ROC) curve.Logistic regression has the samelimitations in modeling non‐linearrelationships as linear regression.That is because the log odds is stillestimated using a linear equation.This requires extra attention totransformations and interactions.For a logistic regression, you can alsoreview a Likelihood‐Ratio Test as ameasure of goodness of fit; or you canreview AIC, BIC, or one of several“pseudo R2” values.WIILSU 2016 ‐ OLSEN & FUHRMANN31

5Decision TreePURPOSEMETHOD VISUALIZATIONUse multiple x variables to predicta categorical outcome y. Theoutcome can be two‐level(yes/no), or multi‐level. These canbe used either for predictivepurposes, or for data exploration.This is a supervised technique thatrequires historical data withknown outcomes.WIILSU 2016 ‐ OLSEN & FUHRMANN32

5Decision TreeIMPORTANT CONSIDERATIONSDecision trees do not have as strongassumptions as regressions do, but theyare many different ways to approachthem: algorithm choice, missing valuetreatment, stopping criteria, pruningmethods, etc. Additionally, thepredictive power of decision trees isgreatly increased by use of ensemblingtechniques.SAS Enterprise Miner uses its ownalgorithm which adopts aspects of thevarious decision tree algorithms andallows the user to set parameters formissing variable handling and stoppingcriteria. Outside of SAS EM, you willneed to choose an algorithm andhandle your data preparationaccordingly.MODEL VALIDATIONThere is a variety of measures usedto measure how strong aclassification model is: theconfusion matrix, which gives trueand false positive rates and trueand false negative rates; accuracy;misclassification rate; positivepredictive value (PPV) and negativepredictive value (NPV); and others.Often analysts will review theReceiver Operative Characteristic(ROC) curve to see how well themodel is doing and to reviewpossible trade‐offs of sensitivity andspecificity.WIILSU 2016 ‐ OLSEN & FUHRMANN33

5Decision tree output in SAS EMNode RulesTree Plot Node color events as % ofnode observations Line width observations inbranch as % oftotal observations Definitionsfor each leafnode View Model Node RulesVariable Importance View Model VariableImportanceWIILSU 2016 ‐ OLSEN & FUHRMANN34

Ensembles combine informationfrom multiple models to producea better, “ensembled” model5Model 1Model 2Data EnsembleModelModel kOften more robust and more accurate than a single model. Think“committee of experts.”Often, but not always, ensembles are of the same algorithm, e.g. alldecision tree or all neural net.WIILSU 2016 ‐ OLSEN & FUHRMANN35

5Research shows ensembles oftenoutperform single modelsEnsemblesSingleClassifiersFrom Zhuowen Tu, “Ensemble Classification Methods: Bagging, Boosting, and Random Forests”WIILSU 2016 ‐ OLSEN & FUHRMANN36