Transcription
Best Practices Integrating SAS andOpen SourceUsing Open Source with SAS 9.x and SAS ViyaMelodie RushGlobal Customer Success Principal Data ScientistConnect with me:LinkedIn: https://www.linkedin.com/in/melodierushTwitter: @Melodie RushC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceWhat are the integration points? The Base SAS Java Object - Call Open Sources from within SAS or Call SAS from withinOpen Source- Supported since 9.1.3 Client-side integration with R was released in July 2009 Delivered with SAS/IML for SAS 9.2 Administrative tool for managing and monitoring predictivemodels Support for models created with R was delivered with SAS9.3 in July 2011Support for models created with R and Python deliveredwith SAS Viya 3.4 in 2018 New architecture released October 2016Jupyter Notebook Model Manager SAS 9.4 Maintenance 3 (released July 2015)SAS Viya Server-side integration with R delivered with IML 9.22 inNovember 2010Extends support to Windows and Linux serverenvironmentsOpen Source Node that enables users to submit R code aspart of an EM process flow in December 2013Base SAS - PROC LUA SAS/IML SAS/IML StudioEnterprise Miner (in SAS 9.4) SAS 9-Initial release on Linux September 2016-Kernel for Windows and Mainframe released March 2017SAS Viya Oct 2016Python Functions SAS 9.4M6 May 2019 Using Python functions inside of SAS ProgramsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Environment*** Call Python;data null ;python pgm "&WORK DIR.\mppr python.py";python arg1 "&WORK DIR";python exec command "&python exec command";python call cat('"', trim(python exec command), '" "', trim(python pgm),'" "', trim(python arg1), '"');Dtaput python call ;call system(python call);run;Base SAS DATA stepC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceSAS/IMLCan run code in SAS Enterprise Guide, SASEnterprise Miner, SAS Studio and SAS IML Studio(need IML licensed and installed)Supported Since SAS 9.22C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS/IML R for data modeling4. Transfer results into SAS/IML vectors1. Read data into SAS/IML vectorsproc iml;use Sashelp.Class;read all var {Weight Height};close Sashelp.Class;2. Transfer data to R/* send matrices to R */call ExportMatrixToR(Weight, "w");call ExportMatrixToR(Height, "h");call ImportMatrixFromR(pe,"ParamEst");print pe[r {"Intercept" "Height"}];ht T( do(55, 70, 5) );A j(nrow(ht),1,1) ht;pred wt A * pe;print ht pred wt;3. Call R functions for data analysissubmit / R;Model - lm(w h,na.action "na.exclude") # aParamEst - coef(Model) # bPred - fitted(Model)Resid - residuals(Model)endsubmit;C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Enterprise Miner and Open SourceC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
R integrationSAS Enterprise Miner SAS Enterprise Miner Open Source Integration node Enables the execution of R code within an Enterprise Miner flow- Facilitates multitasking in R- Generates text and graphical output from R- Integrates both supervised and unsupervised learning tasks Transfers data, metadata, and results automatically between Enterprise Miner and RUses SAS/IML under the coversC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceSAS Enterprise MinerC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceDiscovery – R in Enterprise MinerR MODELSSAS MODELSC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceSAS and R Model ComparisonC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceSAS and R Model ComparisonC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Using R in SAS Enterprise MinerPMML output modelibrary(rpart)&EMR MODEL - rpart(&EMR CLASS TARGET &EMR CLASS INPUT &EMR NUM INPUT, data &EMR IMPORT DATA, method "class")C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Using R in SAS Enterprise MinerPMML output modelibrary(rpart)&EMR MODEL - rpart(&EMR CLASS TARGET &EMR CLASS INPUT &EMR NUM INPUT, data &EMR IMPORT DATA, method "class")SAS Enterprise Miner data sourceC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Using R in SAS Enterprise MinerPMML output modeSAS Enterprise Miner metadatalibrary(rpart)&EMR MODEL - rpart(&EMR CLASS TARGET &EMR CLASS INPUT &EMR NUM INPUT, data &EMR IMPORT DATA, method "class")SAS Enterprise Miner data sourceC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Using R in SAS Enterprise MinerPMML output modeSAS Enterprise Miner metadatalibrary(rpart)&EMR MODEL - rpart(&EMR CLASS TARGET &EMR CLASS INPUT &EMR NUM INPUT, data &EMR IMPORT DATA, method "class")R object translated to SAS DATAstep code using PMMLSAS Enterprise Miner data sourceC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Enterprise Miner and PythonC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python Example The code in SAS Code Nodes: Export the data to .csv files Execute a Python Program Import the predictions and merge with training datasetC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python ExampleFirst SAS Code NodeThe code in Data Export SAS Code Node (repeat for the validation and test data sets):data train;set &EM IMPORT DATA (keep %EM INPUT %EM TARGET);format all ;run;proc export data trainoutfile "&WORK DIR.\train.csv"dbms csvreplace;run;C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python ExampleSecond Code NodeThe code in Python Call SAS Code Node:*** Call Python;data null ;python pgm "&WORK DIR.\hmeq python dt.py";python arg1 "&WORK DIR";python exec command "&python exec command";python call cat('"', trim(python exec command), '" "', trim(python pgm), '" "', trim(python arg1), '"');put python call ;call system(python call);run;C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python Examplehmeq python dt.pyimport csv, os, sysimport pandas as pdfrom sklearn.tree import DecisionTreeClassifier#score training and validation datatrain predictions dt.predict proba(train features)validate predictions dt.predict proba(validate features)#read data into panda data framestrain pd.read csv(os.path.join(sys.argv[1], "train.csv"))validate pd.read csv(os.path.join(sys.argv[1], "validate.csv"))#export predictions to .csv filestrain predictions file open((os.path.join(sys.argv[1],"hmeq python train predictions dt.csv")), "w" , newline "")open file object csv.writer(train predictions file)open file object.writerow(["p BAD0","p BAD1"])open file object.writerows(train predictions)train predictions file.close()#set up target and features listtrain target train['BAD']train features train.drop('BAD', axis 1)validate features validate.drop('BAD', axis 1)#run decision treedt DecisionTreeClassifier(criterion 'gini',max depth 5, min samples leaf 5,random state 31415)dt.fit(train features,train target)validate predictions file open((os.path.join(sys.argv[1],"hmeq python validate predictions dt.csv")), "w", newline "")open file object csv.writer(validate predictions file)open file object.writerow(["p BAD0","p BAD1"])open file object.writerows(validate predictions)validate predictions file.close()C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python ExampleThird Code Node The code in Data Import SAS Code Node (repeat for the validation and test data sets):data predict py train;infile "&work dir.\hmeq python train predictions dt.csv"dsddelimiter ','firstobs 2;input p bad0 p bad1;run;data &EM EXPORT TRAIN;merge &EM IMPORT DATA predict py train;run;C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python ExampleModel Import Node Map Predicted Variables Model Import can be connected to Model Comparison nodeC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Python ExampleModel ComparisonC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPySAS 9.4Project headed by Jared Deanhttps://github.com/sassoftwareC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Open SourceExample: IDE Choice - Jupyter SAS Kernel The Jupyter Notebook is a web application that allows you to create andshare documents that contain live code, equations, visualizations andexplanatory text. Currently supports about 40 languages. SAS Kernel released asOpen source. Jupyter Notebooksintegration being includedin SAS University Edition More integration comingsoon https://github.com/sassoftware/sas kernelC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
What is the SAS Kernel?A Jupyter kernel for SAS. This opens up all the data manipulation andanalytics capabilities of your SAS system within a notebook interface. Usethe Jupyter Notebook interface to execute SAS code and view resultsinline.For SAS 9.4 and Python 3.X https://github.com/sassoftware/sas kernelC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPy Provides Python APIs to the SAS system Run SAS code from Python Run analytics from Python usingobject-oriented methods Can transfer between SAS data setsand Pandas data framesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPyC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPySAS Procedure vs SASPy Method for Linear RegressionC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Viewing the SAS Code Behind SASPyC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPyTechnical InformationThe SASPy project provides Python APIs to the SAS system. You can start aSAS session and run analytics from Python through a combination ofobject-oriented methods and Python magics. For SAS 9.4 and Python 3.x Works using SAS Kernel on LINUX WINDOWS UNIX https://github.com/sassoftware/saspyC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
What is SAS Pipefitter The SAS pipefitter package provides a Python API for developing pipelinesfor data transformation and model fitting as stages of a repeatableworkflow. Works with both SAS Viya (SWAT) and SAS 9.4 ter/overview.htmlC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPy and Pipefitter Available on GitHubC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SASPyDemoC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
PROC FCMPSubmit and execute functions written in Python from within a SASsession using the new Python objectC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
PROC FCMPWhat is it?SAS Function Compiler Enables you to create, test, and store SAS functions, CALL routines, andsubroutines before you use them in other SAS procedures or DATA steps PROC FCMP DocumentationUsing PROC FCMP to the Fullest: Getting Started and Doing MoreC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
PROC FCMPPython ObjectsPython objects enable you to embed and import Python functions intoSAS programs The Python code is not converted to SAS code. Instead, the Python coderuns in the Python interpreter of your choice and returns the results toSAS. With a small Python code modification, you can run your Python functionsfrom SAS and easily program in both languages at the same time. Available with SAS 9.4 M6 (May 2019) Using Python functions inside of SAS ProgramsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
PROC FCMPUsing Python Functions in 5 StepsPython Function Workflow1. Declare a Python object & a dictionary object2. Insert Python source code into SAS3. Publish Python source code4. Call the Python source code5. Return results from the dictionaryResultsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.proc fcmp;declare object py(python);submit into py;def PyProduct(var1, var2):"Output: MyKey"newvar var1 * var2return newvar,endsubmit;rc py.publish();rc py.call("PyProduct", 5, 10);MyResult py.results["MyKey"];put MyResult ;run;
PROC FCMPDemoC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
What is SAS Viya?SAS Viya is a cloud-enabled, in-memory analytics engine that provides quick,accurate and reliable analytical insights.C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Viya ProductsSAS Viya takes advantage of a cloud-enabled, open platform. Most offeringsinclude both a coding interface as well a visual interface. SAS Visual AnalyticsSAS Visual StatisticsSAS Visual Data Mining andMachine LearningSAS Visual ForecastingSAS Visual Text AnalyticsSAS OptimizationSAS EconometricsSAS Model Manager SAS Data PreparationSAS Visual InvestigatorSAS Business AnalyticsSAS Intelligent DecisioningSAS CybersecuritySAS Detection and InvestigationSAS Event Stream ProcessingAnd more C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Visual InterfacesProgramming InterfacesAPI InterfacesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
proc print data x.hmeq (obs 10);run;WorkersControllerAPIsdf s.CASTable(‘hmeq’)df.head(10)CAS Action[table.fetch]table.name “hmeq”from 1 to 10df - defCasTable(s, ‘hmeq’)head(df, 10)C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Cloud Analytic Services Language (CASL)CASL is the language specification that enables you to access the CASserver. Designed to offer easy access to SAS Viya functionality Modeled after DATA step and IML Available via PROC CAS C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
CAS Actions HierarchiesAction SetsActionsParametersOptionsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
PROC versus CAS Actionproc factmac data mycas.movlens nfactors 10 learnstep 0.15maxiter 20 outmodel mycas.factors;input userid itemid /level nominal;target rating /level interval;output out mycas.out1 copyvars (userid itemid rating);run;proc cas;action factmac result R / table {name "movlens"},outModel {name "factors out", replace true},inputs {"userid", "itemid"},nominals {"userid", "itemid"},target "rating",maxIter 20, nFactors 10, learnStep 0.15,output {casout {name "score out", replace "TRUE"},copyvars {"userid","itemid","rating"}};run;C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Scripting Wrapper for Analytics Transfer (SWAT)Access to SAS Viya from Python and R Integration of SAS Analytics in Pythonand R code R Studio and Jupyter Notebook support Issue tracking and collaboration indevelopment through GitHub project Open to SAS C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
The Python SWAT Package Gives unique Python functions to perform licensed CAS actions. The functionalitymimics the look and feel of Python syntax, making it easy for Python users to takeadvantage of CAS.Action stTrain(table dict(),target string,inputs value list,nominals value list,ntree int32,casOut dict())C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
An Example Flow for Python SWAT Interface to CASImport PackagesConnect to CAS SessionLoad data into CASImport CAS Action SetsUse CAS Action Sets forCardinalityUse Python to plotresulting tablesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SWAT PythonImport packages and connect to CAS SessionImport needed packagesConnect to SAS CAS SessionC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SWAT PythonLoad data into CASLoad dataTake a look at the dataC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SWAT PythonAction SetsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SWAT PythonImport CAS Action SetsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SWAT PythonUse SAS Action Sets for CardinalityC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SWAT PythonUse Python to plot resulting tablesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Open source integrationSAS in Hadoop from PythonC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Visual InterfacePipelinesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Visual Data Mining and Machine LearningPipelinesDrag-and-drop pipelines includingpreprocessing and machine learningtechniques Customizable and portable nodes andSAS best practice pipelines (Toolbox) Support for SAS coding (macro, datastep, procs, batch Enterprise Miner)within pipelines Collaboration through the use of the“Toolbox” – a collection of SAS BestPractice Pipelines, in addition to usergenerated templates Example Code for PipelineC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Visual Data Mining and Machine LearningPipelinesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaPipelinesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaPipelines – Supervised Learning – RC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaPipelines – Supervised Learning – RC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaPipelines - Supervised Learning – PythonC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaPipelines – Supervised Learning - Python CodeC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS ViyaPipelines – Data Mining Preprocessing – Python CodeC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Visual Interface DemoPipelinesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
ResourcesC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceOpen APIs to Access SAS – All Accessible on Github9.4“SASPy”“SWAT”DLPy – Deep LearningSASOPTPy – OptimizationESPPy – Streaming AnalyticsC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open Source ResourcesSAS 9.4 Blogs: Open Source Integration Using the Base SAS Java Objecthttps://communities.sas.com/docs/DOC-10746 Github Page: SAS Base OpenSrcIntegration n/blob/master/SAS Base OpenSrcIntegration/main caller.sas SASPy https://github.com/sassoftware/saspy Python-pipefitter https://github.com/sassoftware/python-pipefitter SAS Community tips (Cheat sheet for version numbers – installation-cheat-sheet/ta-p/223470C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open Source ResourcesEmpowering the Base SAS user SAS or Python? Why not use both? Using Python functions inside oceedings.wuss.org/2019/73 Final Paper PDF.pdf Using Python to run jobs in your SAS Grid NEW /19/using-python-to-run-jobs-in-your-sasgridC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open Source ResourcesEmpowering the SAS Enterprise Miner userVideo: Using R in SAS Enterprise Minerhttps://www.youtube.com/watch?v TbXo0xQCqDwBlogs: Spectral Clustering in SAS Enterprise Miner Using Open Source Integration : How to execute a Python script in SAS Enterprise Miner https://communities.sas.com/docs/DOC-10832Article: The Open Source Integration node installation cheat e Notes:http://support.sas.com/dsearch?Find Search&ct &qt open source&col suppprd&nh 25&qp &qc suppsas&ws 1&qm 1&st 1&lk 1&rf 0&oq &rq 0C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS and Open SourceViya CAS actions documentationSAS Github page for SWAT-PythonSAS Github page for SWAT-RMore example scripts for using SWAT-R & SWAT-PythonVideo - Viya and Python ExampleCall Python in PROC DS2 (PyMas)C o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
assoftwareC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Useful WebsitesDeveloper.sas.com, Communities.sas.comC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
https://www.sas.com/en us/events/sas-global-forum.htmlC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
Questions?Thank you for your time and attention!Connect with me:LinkedIn: https://www.linkedin.com/in/melodierushTwitter: @Melodie Rushsas.comC o p yr ig h t S A S I n s titu te I n c . A ll r ig h ts r e s e rved.
SAS Enterprise Miner Open Source Integration node Enables the execution of R code within an Enterprise Miner flow - Facilitates multitasking in R - Generates text and graphical output from R - Integrates both supervised and unsupervised learning tasks Transfers data, metadata, and results automatically between Enterprise Miner and R .
