{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# SAS Viya for Learners ML Pipeline (Python)\n---\n\nWelcome to the SAS Viya for Learners Challenge - where you will learn how to build machine learning models for a simulated telecommunication company. For the challenge, you will build a ML model to predict whether an existing customer will upgrade their existing service (upsell) or purchase another service (xsell).\n\nTo get you started, you can use this existing Python notebook that will walkthrough the steps of the machine learning (ML) process by building a simple model. By showing you the basics, you will learn how to program in Python using machine learning running in SAS Viya. However, since this is a simple exmaple, the challenge is to then add more to this example by adding more variables, engineering new features, and build better models to move up the leaderboard!","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"markdown","source":"#### Note - This notebook is meant to be run on the SAS Viya environment, not Kaggle! Download the notebook and upload to SAS Viya following the instructions on the Discussion forum: https://www.kaggle.com/c/sasviyaforlearners2020/discussion/168325\n\n#### Additionally, note that autotuning is disabled in the SAS VFL environment, but ","metadata":{}},{"cell_type":"markdown","source":"## Introduction\n\n### What is SAS Viya?\n\n---\n\n__SAS Viya__ is the latest analytics platform from SAS for building and developing machine learning models. At its core, SAS Viya has an in-memory analytics engine which has been designed to run analytics and ML procedures more efficiently, than if they were running on your local laptop. These procedures, known as __CAS actions__ can be called and executed using Python, R & SAS so that data scientists can use their preferred programming language of choice, but be able to run these ML procedures using the SAS Viya engine for more scalable performance.\n\nTo do this, we use the __SAS Scripting Wrapper for Analytics Transfer (SWAT)__, which is a library that allows our Jupyter Lab client to connect to SAS Viya. Using __SWAT__, you can analyse large data sets in-memory using the analytical engine and performance of SAS Viya but not be restricted by your choice of programming language.\n\nThe diagram below demonstrates this process, where you can have syntactically different code from three different programming languages (SAS, Python, R) but they are all calling the equivalent CAS Action on the SAS Viya server to execute. For this notebook specifically, we are using Python code in our JupyterLab notebook to call the CAS actions running on the SAS Viya server. ","metadata":{"slideshow":{"slide_type":"fragment"}}},{"cell_type":"markdown","source":"### Notebook Workflow\n\n---\n\nML is an iterative process involving three main steps:\n* 1. Data Processing (Exploring your data and pre-processing)\n* 2. Model Building (Building your machine learning models)\n* 3. Model Evaluation (Assessing the quality of your models)\n\nThis notebook will walk through this process:\n \n* 1. Data Processing\n    - Load Data\n    - Explore & Pre-process Data\n    - Partition Data\n* 2. Model Building\n    - Training a model algorithm on training set\n    - Scoring on validation set\n- 3. Model Evaluation & Submission\n    - Assess on the test dataset for leaderboard\n    - Save model files required for submission","metadata":{}},{"cell_type":"markdown","source":"## Setup\n\n### Package Import\n\n---\n\nThe first step is to import packages - packages contain the functions that we want to use for our analysis. The key packages are:\n\n- __SWAT__ -- This package allows our Jupyter Notebook to connect to SAS Viya and use the analytical capabilities that SAS Viya provides. See the introduction for a more detailed description.\n\n- __pandas, matplotlib__ -- These are Python packages for data manipulation and graphing respectively.","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"import sys\nimport os\nimport pandas as pd\nimport numpy as np\n\n### Import swat\nimport swat\n###\n\n# Set Graphing Options\nfrom matplotlib import pyplot as plt\n%matplotlib inline\n\n# Set column/row display options to be unlimited\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', None)","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Connect to SAS Viya\n\n---\n\nIn order to execute CAS actions, a connection must be made using SWAT from the OS client to the CAS server. Your login credentials and server details are required, but by default these are already saved as environment variables into the ```swat.CAS()``` statement.\n\nThis creates a 'connection object' represented by the Python variable 's', which represents the connection between Jupyter Notebook and SAS Viya. If the connection was successful, the connection object must be invoked in order to run CAS actions. Additionally, any function with the connection object in front of it indicates that it is running on SAS Viya.\n\nUpon successful connection, a __CAS session__ will be started allowing execution of CAS actions. The CAS session information will be returned below, if you print the connection object variable.","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"#Connect to CAS\ns = swat.CAS(os.environ['CASHOST'], os.environ['CASPORT'], None, os.environ.get(\"SAS_VIYA_TOKEN\"))\ns","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Caslibs\n\n---\n\nOn SAS Viya, all data are organised into __CAS Libraries__ (caslib). They represent a individual or group workspace where data is stored. For the challenge, the training data is located in the __DLUS34__ caslib so we will set that to our working Caslib.","metadata":{}},{"cell_type":"code","source":"s.sessionProp.setsessopt(caslib='KAGGLE')\n\n## We can also enable session metrics to view how long our procedures take to run\ns.session.metrics(on=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load CAS Action Sets\n\n---\n\nThe analytical procedures in SAS Viya are known as __CAS actions__ and are organised into __CAS action sets__. These __CAS action sets__ represent modular functionality (e.g. processing data, building models) and must be loaded after connecting to SAS Viya, similar to importing Python packages.\n\nAction sets can be loaded using the following command:\n\n```python\ns.loadactionset(action_set_name)\n```\n\nThe list of CAS Action Sets available can be found in the [documentation](https://go.documentation.sas.com/?docsetId=allprodsactions&docsetTarget=actionSetsByName.htm&docsetVersion=3.5&locale=en).\n\nThe cell below has been created to allow loading of multiple CAS action sets by specifying the names of the action sets required in the list variable _actionsets_:","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"# Load actionsets for analysis (for data prep, modelling, assessing)\nactionsets = ['cardinality', 'sampling', 'decisionTree', 'astore','autotune']\n[s.loadactionset(i) for i in actionsets]","metadata":{"slideshow":{"slide_type":"fragment"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Writing CAS Action Sets\n\n---\n\nNow that we have loaded the action sets, let's review the Python syntax for writing CAS actions. Every CAS action requires the connection object (in this case, 's') in front and then follows a specific order - action set, action, parameter, option:\n\n```python \nconnectionObject.actionSetName.actionSet(\n    parameter1={\n        option1=True,\n        option2=True,\n    },\n    parameter2=5\n)\n```\n\nFor example, the action to train a decision tree model is:\n```python\ns.decisionTree.dtreeTrain(\n    table = dict(name= input_data),\n    target = target,\n    inputs = input_variables,\n    nominals = nominal_variables,\n    casOut = dict(name='DecisionTreeModelOutput', replace=True),\n    code = dict(casout=dict(name='DecisionTreeModelCode', replace=True)\n)\n```\n\nNote, using the _dict()_ function is the same as using _{}_ to define a Python dictionary.","metadata":{"tags":[]}},{"cell_type":"markdown","source":"### Help & Documentation\n\n---\n\nJupyter Notebook provides in-line help and documentation via use of the help() function - this can be used to view how to use the __CAS action sets__. The action set must be loaded in the previous step.","metadata":{}},{"cell_type":"code","source":"s.help(actionSet='cardinality')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Data Processing\n\n### Accessing Data on SAS Viya\n\n---\n\nAll data in SAS Viya resides in-memory to perform analytics more efficiently. What that means is, after connecting, you must load the dataset you want to use into the SAS Viya memory. When data is in-memory in SAS Viya, it is represented as a __CAS table__.\n\nData can be loaded into SAS Viya from the client-side (local computer into SAS Viya) or server-side (server machine into SAS Viya engine). For this challenge, the data exists on the server machine (the same cloud environment SAS Viya is configured on) so we must load the tables into memory using the ```load_path()``` function.\n\nNote, that the client-side operations from SWAT are built on top of the __pandas__ functionality, meaning you can pass the same __pandas__ arguments for the equivalent function.","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"table_name = \"HEART_TRAINING_JULY2\"\n\ncastbl = s.load_path(\n  table_name+'.sas7bdat',\n  casOut=dict(name=table_name, replace=True)\n)\ncastbl.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Explore Data through Summary Statistics\n\n---\nWe can explore our data and generate summary statistics on our data set. This information can be used in combination with the Data tab view from the Kaggle competition page to assess how our input data is distributed. ","metadata":{}},{"cell_type":"code","source":"# Create table of summary statistics in SAS\ncastbl.cardinality.summarize(\n    cardinality=dict(name = 'full_data_card', replace = True)\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above action generates the summary statistics, and we can now view this summary information by bringing it locally into the Jupyter notebook client:","metadata":{}},{"cell_type":"code","source":"full_data_card = s.CASTable('full_data_card').to_frame() # bring the summary data locally\n\n# Modify SAS output table using Python to present summary statistics\nfull_data_card['_PCTMISS_'] = (full_data_card['_NMISS_']/full_data_card['_NOBS_'])*100\nprint('\\n', 'Summary Statistics'.center(90, ' '))\nfull_data_card[['_VARNAME_','_TYPE_','_PCTMISS_','_MIN_','_MAX_','_MEAN_','_STDDEV_','_SKEWNESS_','_KURTOSIS_']].round(2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Declare Input Variables\n\n---\n\nAs seen above, the dataset contains a lot of variables of varying characteristics, type & structure. \n\nWe can explicitly declare our variables by creating a list, and then subsetting our data based off a list, __input_vars__. Our subsetted input table we will use to build a model is __select_castbl__.\n\n**Note:** This time we're going to use a biger sample!","metadata":{}},{"cell_type":"code","source":"#Declare input variables\ntarget =  'Status'\ninput_vars = [ 'AgeCHDdiag', 'Sex', 'AgeAtStart',\n       'Height']\nvariables = [target] + input_vars\n\n              \nselect_castbl = castbl[variables]\nselect_castbl.head(20)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can perform the same set data exploration procedures as well to view the variables we have selected:","metadata":{}},{"cell_type":"code","source":"# Create table of summary statistics in SAS\nselect_castbl.cardinality.summarize(\n    varList=[\n        {'vars': input_vars}\n    ],\n    cardinality=dict(name = 'data_card', replace = True)\n)\n\ndf_data_card = s.CASTable('data_card').to_frame() # bring the summary data locally\n\n# Modify SAS output table using Python to present summary statistics\ndf_data_card['_PCTMISS_'] = (df_data_card['_NMISS_']/df_data_card['_NOBS_'])*100\nprint('\\n', 'Summary Statistics'.center(90, ' '))\ndf_data_card[['_VARNAME_','_TYPE_','_PCTMISS_','_MIN_','_MAX_','_MEAN_','_STDDEV_','_SKEWNESS_','_KURTOSIS_']].round(2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Note, you can set the following option to fetch more rows of data out Viya memory - this defaults to 10000 rows.\nswat.options.cas.dataset.max_rows_fetched=60000\nselect_castbl.hist(figsize = (15, 10));","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Transformation - Imputation\n\n---\n\nFrom viewing the summary statistics, we can see that one of our variables has missing values. Missing data introduces bias into our models and creates problems with analysis, so in the context of machine learning, we want to create a more helpful input. This is known as feature engineering, where we create new inputs, or features from the inputs in our data. \n\nOne of the methods we can use to deal with missing values is imputation. That is, instead of deleting entries and reducing our sample size, we can replace these missing values with a substituted value (imputation).\n\nThe choice of substituted value depends on your choice of imputation method but for this simple example we will impute with the median to demonstrate a simple data processing transformation.","metadata":{}},{"cell_type":"code","source":"# Plot missing values in matplotlib\ndf_data_miss = df_data_card[df_data_card['_PCTMISS_'] > 0]\ntbl_forplot  = pd.Series(list(df_data_miss['_PCTMISS_']), index = list(df_data_miss['_VARNAME_']))\nmissing_val  = tbl_forplot.plot(kind  = 'bar', title = 'Percentage of Missing Values', color = 'c', figsize = (10, 6));\nmissing_val.set_ylabel('Percent Missing')\nmissing_val.set_xlabel('Variable Names');","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"removeInputs = ['AgeCHDdiag']\nmissingInputs = ['Height']\n\ns.dataPreprocess.impute(\n    table = select_castbl,\n    outVarsNamePrefix = 'IMP',\n    methodContinuous  = 'MEDIAN',\n    inputs            = missingInputs,\n    copyAllVars       = True,\n    casOut            = dict(caslib= 'KAGGLE', name=table_name, replace=True)\n)\n\n# Print the first five rows with imputations\nimp_input_vars = ['IMP_' + s for s in missingInputs]\ntotal_inputs = input_vars + imp_input_vars\n\ntotal_inputs.remove('AgeCHDdiag')\n\n# total_inputs = input_vars\nselect_castbl = s.CASTable(table_name)[total_inputs]\nselect_castbl.head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Partitioning\n\n---\n\nNow that we have declared our input variables, we partition our data into a training set and validation set for modelling. The training set (70% of the data) will be used to build the model and evaluate initial performance. The validation set (remaining 30%) will be used to evaluate the model on unseen data that still fit the same distribution as the training set.\n\nPartitioning of the data is done to prevent overfitting the model to the training data. That is, the model will be fitted so tightly to the training data that it will inhibit it's ability to accurately predict whether new customers will upsell or xsell on their services. We want the model to be able to generalise and assessing on a validation set helps does this.\n\nHere, we create a partition using a CAS action:","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"# Create a 70/30 simple random sample split\nselect_castbl.sampling.srs(\n    samppct = 70,\n    partind = True,\n    seed    = 1,\n    output  = dict(\n        casOut = dict(\n            name=f'{table_name}',\n            replace=True), \n        copyVars = 'ALL'\n    ),\n    outputTables=dict(replace=True)\n)","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build Models\n\n### Set modelling shortcuts\n\n---\n\nAfter partitioning, we are ready to start the modelling. We are going to train two gradient boosting models, with one tuned.\n\nFirst, we declare the modelling variables and then create a Python dictionary to store the model results after training.","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"# Set key-word argument shortcuts (common model inputs)\n## For models that can handle missing values (decision tree, gradient boosting)\nimport collections \n\nparams = dict(\n    table    = dict(name = table_name, where = '_partind_ = 1'), \n    target   = target, \n    inputs   = total_inputs, \n    nominals = target,\n)\n\n# Algorithms to be trained\nmodels = collections.OrderedDict()\nmodels['DT'] = 'Decision Tree'\nmodels['GB'] = 'Gradient Boosting'","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Decision Tree\n\n---\n\nNow that the variables have been selected, the model training can start. As per above, three models are going to be trained:\n* Decision Tree\n* Gradient Boosting\n* Autotuned Gradient Boosting\n\nWe start with training a decision tree:\n* For any SAS model, we can specify the code parameter to generate SAS scoring code to use the model again\n* Below, we store the SAS scoring code in a CAS Table called 'DT_model_code'.","metadata":{}},{"cell_type":"code","source":"s.decisionTree.dtreeTrain(\n    **params,\n    casOut = dict(name='DT_model', replace=True),\n    code = dict(casout=dict(name='DT_model_code', replace=True)),\n    encodeName=True\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s.CASTable('DT_model').head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Gradient Boosting\n\n---\n\nAfter the decision tree, we start the training of the gradient boosting algorithm. We can specify specific hyperparameters to run for this model but for this example we are using the default:\n* Due to the complexity of ML models like gradient boosting, instead of saving SAS score code, we specify a \"savestate\" parameter that we save the model state to.\n* Through the savestate parameter, we save a CAS Table called 'save_gb' which contains the stored model logic. From this, we can output a combination of SAS score code & an analytic store (ASTORE) files which represent the serialised model.","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"s.decisionTree.gbtreeTrain(\n    **params, \n    seed = 1, \n    casOut = dict(name = 'GB_model', replace = True),\n    savestate=dict(name='save_gb', replace=True),\n    encodeName=True,\n)","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Score Model\n\n---\n\nWe can now score our models, by creating a generic scoring function and then iterating through our collection of models:","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"def score_model(model):\n    score = dict(\n        encodeName=True,\n        table      = table_name,\n        modelTable = model + '_model',\n        copyVars   = [target, '_partind_'],\n        casOut     = dict(name = '_scored_' + model, replace = True)\n    )\n    return score\n\n### Gradient Boosting\ns.decisionTree.dtreeScore(**score_model('DT'))\ns.decisionTree.gbtreeScore(**score_model('GB'))","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s.CASTable('_scored_GB').head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Assess Models \n\n---\n\nUsing the scores just generated, we can run model assessment on the models and output the assessment statistics. We then use this to compute which model performed best, so we can utilise it for our submission:","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"# Model assessment function\ndef assess_model(model):\n    assess = s.percentile.assess(\n        table    = dict(name = '_scored_' + model, where = '_partind_ = 0'),\n        inputs   = 'P_' + target + '1',      \n        response = target,\n        event    = '1',   \n    )\n    return assess\n\n# Loop through the models and append to the roc_df dataframe\nroc_df  = pd.DataFrame()\nfor i in range(len(models)):\n    tmp = assess_model(list(models)[i])\n    tmp.ROCInfo['Model'] = list(models.values())[i]\n    roc_df = pd.concat([roc_df, tmp.ROCInfo])\n\n# Display stacked confusion matrix using Python\nprint('\\n', 'Confusion Matrix Information'.center(42, ' '))\nroc_df[round(roc_df['CutOff'], 2) == 0.5][['Model', 'TP', 'FP', 'FN', 'TN']].reset_index(drop = True)","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display assessment statistics\n#roc_df","metadata":{"scrolled":true,"slideshow":{"slide_type":"subslide"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Display Misclassification Rate\n---","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"# Add misclassification rate calculation\nroc_df['Misclassification'] = 1 - roc_df['ACC']\n\nprint('\\n', 'Misclassification Rate Comparison'.center(37, ' '))\nmiss = roc_df[round(roc_df['CutOff'], 2) == 0.5][['Model', 'Misclassification']].reset_index(drop = True)\nmiss.sort_values('Misclassification')","metadata":{"scrolled":true,"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Comparison\n---\n\nWe can use the Python graphing library, __matplotlib__ once again to plot an ROC curve to compare the models we have created.","metadata":{"slideshow":{"slide_type":"slide"}}},{"cell_type":"code","source":"plt.figure(figsize = (7, 6))\nfor key, grp in roc_df.groupby(['Model']):\n    plt.plot(grp['FPR'], grp['Sensitivity'], label = key + ' (C = %0.2f)' % grp['C'].mean())\nplt.plot([0,1], [0,1], 'k--')\nplt.xlabel('False Postive Rate')\nplt.ylabel('True Positive Rate')\nplt.legend(loc='lower right')\nplt.title('ROC Curve (using validation data)');","metadata":{"slideshow":{"slide_type":"fragment"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluate & Create Submission\n\n### Evaluate on Test Set\n\n---\n\nNow that the model has been trained, we can evaluate it on the test set and create our submission. The first step is to score our model on the evaluation dataset - which is located in the _KAGGLE_ Caslib. \n\nAs stated in the Data section of the Kaggle competition https://www.kaggle.com/c/sasviyaforlearners2022/data, creating the submission CSV consists of two parts:\n1. Generating the column of predictions\n2. Appending prediction column to CSV containing the IDs of the test set (submissionsId.csv)\n\nThis is because Kaggle requires all submissions containing predictions to be sorted by a specific ID order.\n\nThe rest of this notebook will walkthrough how to generate the prediction column. \nTo start the scoring process to generate the prediction column we load the test set into memory and then score the model. We will need to impute our test data, since the decision tree model expects the imputed column and we will use the dtScore action to then score the model","metadata":{}},{"cell_type":"code","source":"## Describe ASTORE file\nm=s.describe(\n     rstore='save_gb',\n     epcode=True\n    )\n\n# Load into memory \ncastbls = s.load_path(\n  'HEART_TEST_JULY.sas7bdat',\n  caslib='KAGGLE',\n  casOut=dict(  name='testset', replace=True)\n)\n\n# Score\neval_model = castbls.score(\n    ds2code = m.epcode,\n    table='testset',\n    copyVars = [\"ID\",\"AgeAtStart\"],\n    out=dict(name='scored', replace=True),\n    # casOut = dict(name = 'scored', replace = True),\n    rstore='save_gb'\n)\neval_model","metadata":{"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s.table.fetch(table=dict(name='scored'))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keepcolumns = ['ID','I_Status']\n\n## Need to rename the column with the name Kaggle expects: \"Status\"\ns.table.alterTable(\n    columns=[{\n        \"name\": \"I_Status\",\n        \"rename\":\"Status\"\n    }],\n    name='scored'\n)\nevaluation = s.CASTable('scored')\n\n## Re-order column as how Kaggle submission expects\nevaluation = evaluation.ix[:, ['ID', \"Status\"]]\nevaluation.head(10)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# to bring across ID from testset to scored dataset\n\neval = evaluation.ix[:, ['Status']]\neval_df = eval.to_frame()\n\ntest = s.CASTable('testset')\ntestset_sml = test.ix[:, ['ID']]\ntestset_df = testset_sml.to_frame()\n# testset_df.head()\n\n# Merge two DataFrames by index using pandas.merge()\neval_df_id = pd.merge(testset_df, eval_df, left_index=True, right_index=True)\neval_df_id[['Status']] = eval_df_id[['Status']].apply(pd.to_numeric)\neval_df_id.head(10)\n# print(eval_df_id.dtypes)\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Output column as CSV - make sure you set the data download limit to be greater than 10000 rows since the test set has more than 10k samples.\nswat.options.cas.dataset.max_rows_fetched=60000\neval_df_id.to_csv('predictionColumn_Python.csv', index=False, float_format='%.12g')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Output Model Score Code Files\n\n---\n\nIn addition to the submission csv, being able to output your model score files is required for a valid submission to be able to reproduce results.\n\nTrained SAS Viya models are represented through:\n* SAS score code (Statistical models - e..g. linear regression, logistic regression, decision tree)\n* SAS score code & ASTORE (ML models - e.g. random forest, gradient boosting, neural network)\n\nSAS score code are text files consisting of IF-ELSE-THEN rules written in the SAS programming language (regardless of what other programming language was used to _train_ the model). ASTORE are binary objects containing the compressed model logic of complicated ML models to enable these files to be more portable. In this simple example, we trained a decision tree model so we can output the score code from the __CAS Table__ the model was stored in.","metadata":{}},{"cell_type":"markdown","source":"## Output Model Files\n---\n\n","metadata":{}},{"cell_type":"markdown","source":"For the gradient boosting model, there is a dedicated action set called 'aStore' that can be used to create the required scoring files from the save state. We can print the result of this action to view the metadata and information we have created:","metadata":{}},{"cell_type":"code","source":"m=s.describe(\n     rstore='save_gb',\n     epcode=True\n    )\nm","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To serialise this, we can write out both the SAS score code (given by epcode) and the ASTORE file physically:","metadata":{}},{"cell_type":"code","source":"# Score code file \nfile=open('GB_score.sas', 'w') \nfile.write(m.epcode)\nfile.close()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ASTORE file\nastore_file = s.download(rstore='save_gb')\nwith open(m.Key.Key[0].strip()+'.sasast','wb') as file:\n    file.write(astore_file['blob'])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Run this cell to view the score code for the gradient boosting model\n#sys.stdout.write(m.epcode)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Download Your Submission Files\n\n---\n\nWhen you have created your submission CSV and your model score code, your final submission must also include the training code used to build your model - in this case, this is the Jupyter notebook. Once you have all files, you may simply right-click on the selected file and click 'Download' which downloads the file to your local computer.\n\nAfter downloading the prediction column as a CSV, you should have also downloaded the \"submissionsId.csv\" from the Data section of the Kaggle competition. Open this file and append your prediction column onto this file and save - this is your submission CSV that can now be submitted via Kaggle onto the Leaderboard. Refer to the \"submissionExample.csv\" to see what your submission should look like.","metadata":{}},{"cell_type":"markdown","source":"## What's Next?\n\n---\n\nIf you have made it to the end, congratulations! You can make a submission to the Kaggle Leaderboard with the CSV you have created - the next steps are, how do you improve and do better?\n\n* Try with other input variables! We've suggested what could be important predictors, but there might be others in the dataset!\n* Try adjusting the autotuning procedure parameters - is 60 seconds enough to evaluate the optimal model? How many evaluations should be we make?\n* Beyond imputation, what other types of techniques could be used to create and engineer better features?","metadata":{}}]}