{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <center> A Neural Network Model for House Prices","metadata":{"_cell_guid":"c9d041f0-a7fe-46ea-8c84-821572ccd9a3","_uuid":"ca3198b1525214a53f33a933ddf0f22ac7963468"}},{"cell_type":"markdown","source":"The purpose of this notebook is to build a model (Deep Neural Network) with Tensorflow. We will see the differents steps to do that. This notebook is split in several parts:\n\n- I.    Importation & Devices Available\n- II.   Outliers\n- III.  Preprocessing\n- IV.   DNNRegressor for Contiunuous features\n- V.    Predictions\n- VI.   Example with Leaky Relu\n- VII.  DNNRegressor for Continuous and Categorial\n- VIII. Predictions bis\n- IX.   Shallow Neural Network\n- X.    Conclusion\n\nWe will expose 3 models. The first one will use just the continuous features, the second one we will add the categorical features and finally we will use a Neural Network with just one layer.\n\nTheey are no tuning and we will use DNNRegressor with Relu for all activations functions and the number of units by layer are: [200, 100, 50, 25, 12]. So we have 5 layers.\n\nIn the part VI I show how to use another activation function with the example of Leaky Relu. Finally I will try to use a Shallow Neural Network (just with one Hidden Layer) just for fun.\n\nIf you have an idea to improve the performance of the model: Share it ! Fork it ! And play with it !","metadata":{"_cell_guid":"0adcac33-8276-4f86-a647-f2199e9d5ccf","_uuid":"5608ad498d10e44e2126587e0b654dbbef918b42"}},{"cell_type":"markdown","source":"# <center> I. Importation & Devices Available","metadata":{"_cell_guid":"78d12eeb-72ea-43b0-b503-c8b9842e7d7b","_uuid":"bd03c1594867b43b3f3494591b788d8eb9eab920"}},{"cell_type":"markdown","source":"Before the importation I prefer to check the devices available. Sometimes we can have a problems with your GPU for example. And if you want to have a good performance you must do use GPU and not CPU. In our example we have just a CPU but now you have the code to check if your devices is detected.","metadata":{"_cell_guid":"cb0e6d26-0b33-4f57-9aa7-91bdb6e9d953","_uuid":"ff96a752268379cf111e275fee40722ad6780f18"}},{"cell_type":"code","source":"import os\nimport tensorflow as tf\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = \"99\"","metadata":{"_uuid":"14491f7cb884be2c1ac35e9a2aca023604d1aed0","collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.python.client import device_lib\ndevice_lib.list_local_devices()","metadata":{"_cell_guid":"90b67cbc-419a-4fc8-9a91-8d8dc7f34396","_uuid":"ff972448f235077543f3a0f6b9391bcea18329b6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have checked the devices available we will test them wth a simple computation. Here we have an example with the computation on the CPU. But you can split the computation on your gpu with '/gpu:0'. If you want more GPU you can do 'with tf.device('/gpu:1'): ', 'with tf.device('/gpu:2'): ' etc...\n\nIn our example I display the log information.","metadata":{"_cell_guid":"8932987f-35e8-4769-9feb-ad9916bf0109","_uuid":"8f79a27f340c23cc74add1419bb1c98cab99b183"}},{"cell_type":"code","source":"# Test with a simple computation\nimport tensorflow as tf\n\ntf.Session()\n\nwith tf.device('/cpu:0'):\n    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])\n# If you have gpu you can try this line to compute b with your GPU\n#with tf.device('/gpu:0'):    \n    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])\nc = tf.matmul(a, b)\n# Creates a session with log_device_placement set to True.\nsess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\n\nprint(sess.run(c))\n\n# Runs the op.\n# Log information\noptions = tf.RunOptions(output_partition_graphs=True)\nmetadata = tf.RunMetadata()\nc_val = sess.run(c, options=options, run_metadata=metadata)\n\nprint(metadata.partition_graphs)\n\nsess.close()","metadata":{"_cell_guid":"e14879c3-a5bd-4aee-8e06-139f643b7d18","_uuid":"03e668187944a3aa412b3bba2e2a06edb5eb9e20","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this tutorial our data is composed to 1460 row with 81 features. 38 continuous features and 43 categorical features. As exposed in the introduction we will use onlly the continuous features to build our first model.\n\nHere the objective is to predict the House Prices. In this case we have a regression model to build.\nSo our first data we will contain 37 features to explain the 'SalePrice'. We can see the list of features that we will use to build our first model.","metadata":{"_cell_guid":"cdb671d1-2803-4290-8e8b-a54a00d21d44","_uuid":"348514ad519851c2dc7e51dc04acdbe7d391d6fc"}},{"cell_type":"code","source":"from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport itertools\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom pylab import rcParams\nimport matplotlib\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import MinMaxScaler\n\ntf.logging.set_verbosity(tf.logging.INFO)\nsess = tf.InteractiveSession()\n\ntrain = pd.read_csv('../input/train.csv')\nprint('Shape of the train data with all features:', train.shape)\ntrain = train.select_dtypes(exclude=['object'])\nprint(\"\")\nprint('Shape of the train data with numerical features:', train.shape)\ntrain.drop('Id',axis = 1, inplace = True)\ntrain.fillna(0,inplace=True)\n\ntest = pd.read_csv('../input/test.csv')\ntest = test.select_dtypes(exclude=['object'])\nID = test.Id\ntest.fillna(0,inplace=True)\ntest.drop('Id',axis = 1, inplace = True)\n\nprint(\"\")\nprint(\"List of features contained our dataset:\",list(train.columns))","metadata":{"_cell_guid":"9e15ad43-8a3a-4840-9c0d-afa2ba2bd148","_uuid":"16485c4c885b6416be79eade3cfff6345861a8bd","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> II. Outliers","metadata":{"_cell_guid":"c675ba7f-3d0c-4641-b133-e80711a19cf1","_uuid":"6b90c9ce6240afc8a36106661afe100170ae738a"}},{"cell_type":"markdown","source":"In this small part we will isolate the outliers with an IsolationForest (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html). I tried with and without this step and I had a better performance removing these rows.\n\nI haven't analysed the test set but I suppose that our train set looks like more at our data test without these outliers.\n","metadata":{"_cell_guid":"7b049d56-52b0-4b6c-8c35-490a57eaea87","_uuid":"ae94ba5f35444c2b46bd5bec0acd4a45e084bfc8"}},{"cell_type":"code","source":"from sklearn.ensemble import IsolationForest\n\nclf = IsolationForest(max_samples = 100, random_state = 42)\nclf.fit(train)\ny_noano = clf.predict(train)\ny_noano = pd.DataFrame(y_noano, columns = ['Top'])\ny_noano[y_noano['Top'] == 1].index.values\n\ntrain = train.iloc[y_noano[y_noano['Top'] == 1].index.values]\ntrain.reset_index(drop = True, inplace = True)\nprint(\"Number of Outliers:\", y_noano[y_noano['Top'] == -1].shape[0])\nprint(\"Number of rows without outliers:\", train.shape[0])","metadata":{"_cell_guid":"599c148c-bee5-424e-80cc-afc1d7b1e3fb","_uuid":"08498903317073ce7632c3fbff677ae842ca8389","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(10)","metadata":{"_cell_guid":"b6056d39-23cd-4821-989a-e6769ad298d8","_uuid":"d80cb31a8c00f704879e1c4771f701dd5d8031d0","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> III. Preprocessing","metadata":{"_cell_guid":"96ba460b-8ca5-4479-b9a8-94c04bd08b53","_uuid":"546316d0afa2fad5717bb15a11186d2229d38cd7"}},{"cell_type":"markdown","source":"To rescale our data we will use the fonction MinMaxScaler of Scikit-learn. I am wondering if it is not interesting to use the same MinMaxScaler for Train and Test !","metadata":{"_cell_guid":"445cfd64-2d2f-4218-afb6-b5870d849fb1","_uuid":"4b5d09fd48475d5707699f1725810c01811b61a8"}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\ncol_train = list(train.columns)\ncol_train_bis = list(train.columns)\n\ncol_train_bis.remove('SalePrice')\n\nmat_train = np.matrix(train)\nmat_test  = np.matrix(test)\nmat_new = np.matrix(train.drop('SalePrice',axis = 1))\nmat_y = np.array(train.SalePrice).reshape((1314,1))\n\nprepro_y = MinMaxScaler()\nprepro_y.fit(mat_y)\n\nprepro = MinMaxScaler()\nprepro.fit(mat_train)\n\nprepro_test = MinMaxScaler()\nprepro_test.fit(mat_new)\n\ntrain = pd.DataFrame(prepro.transform(mat_train),columns = col_train)\ntest  = pd.DataFrame(prepro_test.transform(mat_test),columns = col_train_bis)\n\ntrain.head()","metadata":{"_cell_guid":"941c0846-55bf-45bb-a49d-be8084b9c056","_uuid":"bdbe7b6ceb041aace9b18fe3849e568f3ad37f99","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To use Tensorflow we need to transform our data (features) in a special format. As a reminder we have just the continuous features. So the first function used is: tf.contrib.layers.real_valued_column.\nMore information here: https://www.tensorflow.org/api_docs/python/tf/contrib/layers/real_valued_column\n\nThe others cells allowed to us to create a train set and test set with our training data set.","metadata":{"_cell_guid":"8343b430-aac9-4fb1-b752-cfa9f619a9c5","_uuid":"a24e009d75b66a0979ec4e87d34611b405ad8b85"}},{"cell_type":"code","source":"# List of features\nCOLUMNS = col_train\nFEATURES = col_train_bis\nLABEL = \"SalePrice\"\n\n# Columns for tensorflow\nfeature_cols = [tf.contrib.layers.real_valued_column(k) for k in FEATURES]\n\n# Training set and Prediction set with the features to predict\ntraining_set = train[COLUMNS]\nprediction_set = train.SalePrice\n\n# Train and Test \nx_train, x_test, y_train, y_test = train_test_split(training_set[FEATURES] , prediction_set, test_size=0.33, random_state=42)\ny_train = pd.DataFrame(y_train, columns = [LABEL])\ntraining_set = pd.DataFrame(x_train, columns = FEATURES).merge(y_train, left_index = True, right_index = True)\ntraining_set.head()\n\n# Training for submission\ntraining_sub = training_set[col_train]","metadata":{"_cell_guid":"0b40bdb6-f813-4b94-94c6-599880f59445","_uuid":"091aee81e454d14b95d61cae7e35a90e4a376ec2","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Same thing but for the test set\ny_test = pd.DataFrame(y_test, columns = [LABEL])\ntesting_set = pd.DataFrame(x_test, columns = FEATURES).merge(y_test, left_index = True, right_index = True)\ntesting_set.head()","metadata":{"_cell_guid":"0cd9a504-0fd9-47ca-a247-69a920911e35","_uuid":"9985b1ca4bfd4999989af752951676311c4f213e","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> IV. Deep Neural Network for continuous features","metadata":{"_cell_guid":"047f4ae6-7618-42d4-8ad0-4a19fd30ee81","_uuid":"44e0b00b9d8718876bbe814f68044918207faeab"}},{"cell_type":"markdown","source":"With tf.contrib.learn it is very easy to implement a Deep Neural Network. In our example we will have 5 hidden layers with repsectly 200, 100, 50, 25 and 12 units and the function of activation will be Relu.\n\nThe optimizer used in our case is an Adagrad optimizer (by default).","metadata":{"_cell_guid":"0158e8c3-20e7-4a99-ab3a-b470ad4d0f9d","_uuid":"b166945d4a4af37865003549116816cf5bd71062"}},{"cell_type":"code","source":"# Model\ntf.logging.set_verbosity(tf.logging.ERROR)\nregressor = tf.contrib.learn.DNNRegressor(feature_columns=feature_cols, \n                                          activation_fn = tf.nn.relu, hidden_units=[200, 100, 50, 25, 12])#,\n                                         #optimizer = tf.train.GradientDescentOptimizer( learning_rate= 0.1 ))","metadata":{"_cell_guid":"ed573d14-ffb1-4ae5-9f28-97057c20c6ce","_uuid":"6d2f589a8d2be2dd515929f9b9bdcf84736ac96e","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reset the index of training\ntraining_set.reset_index(drop = True, inplace =True)","metadata":{"_cell_guid":"cd61c1bb-0140-4b99-af7d-1feb99b43d9c","_uuid":"17cb0aa6571321dfd833e54de61fc08a27a724da","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def input_fn(data_set, pred = False):\n    \n    if pred == False:\n        \n        feature_cols = {k: tf.constant(data_set[k].values) for k in FEATURES}\n        labels = tf.constant(data_set[LABEL].values)\n        \n        return feature_cols, labels\n\n    if pred == True:\n        feature_cols = {k: tf.constant(data_set[k].values) for k in FEATURES}\n        \n        return feature_cols","metadata":{"_cell_guid":"4de80d3b-6f4d-4257-8b8c-763215173e28","_uuid":"7d121349efc922e5a69d3ef7e9aef69954e3b42a","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Deep Neural Network Regressor with the training set which contain the data split by train test split\nregressor.fit(input_fn=lambda: input_fn(training_set), steps=2000)","metadata":{"_cell_guid":"aed9be24-12ad-48ca-b2eb-bf7c11d6a54a","_uuid":"8324e3670b648792b4010d495eb3564554077d80","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluation on the test set created by train_test_split\nev = regressor.evaluate(input_fn=lambda: input_fn(testing_set), steps=1)","metadata":{"_cell_guid":"eb9cad74-6898-4bb5-a563-953a060f5448","_uuid":"3e21c5eaba39aa7ad09df92c051ef22c5c95fd0d","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the score on the testing set\n# 0.002X in average\nloss_score1 = ev[\"loss\"]\nprint(\"Final Loss on the testing set: {0:f}\".format(loss_score1))","metadata":{"_cell_guid":"0c163468-e38f-4340-ac6e-198f25739d12","_uuid":"fe7df945092e0de175d64056b23dfd9da28747f9","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predictions\ny = regressor.predict(input_fn=lambda: input_fn(testing_set))\npredictions = list(itertools.islice(y, testing_set.shape[0]))\n","metadata":{"_cell_guid":"c5c517b3-6011-4b05-9b34-ded41d82f8e4","_uuid":"6ba45c1abe834c716c5445dde3b794af05e1aeff","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> V. Predictions and submission","metadata":{"_cell_guid":"87746ee4-64f8-49ba-9a23-85d21261dc6b","_uuid":"d75c2f98f4524273133765d146f131624ae7569c"}},{"cell_type":"markdown","source":"Let's go to prepare our first submission ! Data Preprocessed: checked ! Outlier excluded: checked ! Model built: : checked! Next step: Used our model to make the predictions with the data set Test. And add one graphic to see the difference between the reality and the predictions.","metadata":{"_cell_guid":"1a08cbad-7b61-4c37-8686-5cc533d496ac","_uuid":"0cebdd59654ca804beddc4f1a590877e59aa4e8a"}},{"cell_type":"code","source":"predictions = pd.DataFrame(prepro_y.inverse_transform(np.array(predictions).reshape(434,1)),columns = ['Prediction'])","metadata":{"_cell_guid":"0fb8ed1b-c0df-41b9-a764-55d675e8aa7c","_uuid":"bf2281f14b4bcf56aac14ad691b1729e50cff5d8","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"reality = pd.DataFrame(prepro.inverse_transform(testing_set), columns = [COLUMNS]).SalePrice","metadata":{"_cell_guid":"e4e48b49-f405-4e57-b6cb-587607253aab","_uuid":"aa904a7f562129ac5f362e39089e6923f78d1acc","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matplotlib.rc('xtick', labelsize=30) \nmatplotlib.rc('ytick', labelsize=30) \n\nfig, ax = plt.subplots(figsize=(50, 40))\n\nplt.style.use('ggplot')\nplt.plot(predictions.values, reality.values, 'ro')\nplt.xlabel('Predictions', fontsize = 30)\nplt.ylabel('Reality', fontsize = 30)\nplt.title('Predictions x Reality on dataset Test', fontsize = 30)\nax.plot([reality.min(), reality.max()], [reality.min(), reality.max()], 'k--', lw=4)\nplt.show()","metadata":{"_cell_guid":"450559b2-bff3-427f-8dec-569191bd8c94","_uuid":"a11028c04b0a6981169196e2c6fe84c245799b8e","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_predict = regressor.predict(input_fn=lambda: input_fn(test, pred = True))\n\ndef to_submit(pred_y,name_out):\n    y_predict = list(itertools.islice(pred_y, test.shape[0]))\n    y_predict = pd.DataFrame(prepro_y.inverse_transform(np.array(y_predict).reshape(len(y_predict),1)), columns = ['SalePrice'])\n    y_predict = y_predict.join(ID)\n    y_predict.to_csv(name_out + '.csv',index=False)\n    \nto_submit(y_predict, \"submission_continuous\")","metadata":{"_cell_guid":"6008e5e1-29d7-42ac-b81a-d9e761a7e956","_uuid":"3be76f4c812b3676a6816d8eb9353f049948ac49","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> VI. Leaky Relu","metadata":{"_cell_guid":"75a14830-c565-4efa-a298-e45ebf11055a","_uuid":"b4ac8e0fa1f7ea486880e006a5a70b37f91dd44a"}},{"cell_type":"markdown","source":"An example with another activation function: Leaky Relu ! We can create this new function with Relu. As a reminder Relu is Max(x,0) and Leaky Relu is the function Max(x, delta*x). In our case we can take delta = 0.01","metadata":{"_cell_guid":"f2b740ac-bcbc-4eff-9026-94b5a286174f","_uuid":"e3174c3df652b712a2c0af3c1d2517b4ab44bfac"}},{"cell_type":"code","source":"def leaky_relu(x):\n    return tf.nn.relu(x) - 0.01 * tf.nn.relu(-x)","metadata":{"_cell_guid":"d697dbb2-54e2-49aa-93dd-0c7c0cd12996","_uuid":"51764360ba91ecdff96c6c89bb57142c6f19bac7","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Model\nregressor = tf.contrib.learn.DNNRegressor(feature_columns=feature_cols, \n                                          activation_fn = leaky_relu, hidden_units=[200, 100, 50, 25, 12])\n    \n# Deep Neural Network Regressor with the training set which contain the data split by train test split\nregressor.fit(input_fn=lambda: input_fn(training_set), steps=2000)\n\n# Evaluation on the test set created by train_test_split\nev = regressor.evaluate(input_fn=lambda: input_fn(testing_set), steps=1)","metadata":{"_cell_guid":"49fe448a-883e-4693-b5c0-50417488336f","_uuid":"940beb55da2e02f704c47a21d7ca4490a8852419","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the score on the testing set\n# 0.002X in average\nloss_score2 = ev[\"loss\"]\nprint(\"Final Loss on the testing set with Leaky Relu: {0:f}\".format(loss_score2))","metadata":{"_cell_guid":"c3885a56-a0a1-434e-b6f4-d33440d3b721","_uuid":"ff854c7ca8e0c03e0cf86edd3ad566fc79ebc05f","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predictions\ny_predict = regressor.predict(input_fn=lambda: input_fn(test, pred = True))\nto_submit(y_predict, \"Leaky_relu\")","metadata":{"_cell_guid":"4df61196-e5e6-400b-90b4-f1c9736f92ab","_uuid":"c96f4a907c614cd2bf2cfd649c8869e27aeac512","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Model\nregressor = tf.contrib.learn.DNNRegressor(feature_columns=feature_cols, \n                                          activation_fn = tf.nn.elu, hidden_units=[200, 100, 50, 25, 12])\n    \n# Deep Neural Network Regressor with the training set which contain the data split by train test split\nregressor.fit(input_fn=lambda: input_fn(training_set), steps=2000)\n\n# Evaluation on the test set created by train_test_split\nev = regressor.evaluate(input_fn=lambda: input_fn(testing_set), steps=1)\n\nloss_score3 = ev[\"loss\"]","metadata":{"_cell_guid":"d33bd73c-5ebb-41bf-911c-bd8e6a4e6817","_uuid":"83507853824f10fa77ce98e9f78d70dcbd857149","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Final Loss on the testing set with Elu: {0:f}\".format(loss_score3))","metadata":{"_cell_guid":"97a3110c-cd2f-47b1-a17e-2256ec224f55","_uuid":"7dd50347b565abc2d23486fda6488385116fe637","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predictions\ny_predict = regressor.predict(input_fn=lambda: input_fn(test, pred = True))\nto_submit(y_predict, \"Elu\")","metadata":{"_cell_guid":"ca9e2de3-12e8-4881-92ba-f51fe185c958","_uuid":"42ba57572d89bf477957f9d8e677759b77620f40","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we have 3 submissions with 3 differents activation functions. But we built ours models just with the continuous features. If you want to comapare the performance download the 3 submissions and submit to the leaderboard. \nNow we see how do build another model by adding a categorical features.","metadata":{"_cell_guid":"e49c1375-ff65-49e2-a594-b97c1cfffea1","_uuid":"c131e5e4af5379787e492e1eb2a1855c27e93d30"}},{"cell_type":"markdown","source":"# <center> VII. Deep Neural Network for continuous and categorical features","metadata":{"_cell_guid":"f8b3a45c-b978-4861-a44f-1236be9d4d2e","_uuid":"0ca9015e24c91d4ea1503baf7002a2ecdc888d1b"}},{"cell_type":"markdown","source":"For this part I repeat the same functions that you can find previously by adding a categorical features.\n","metadata":{"_cell_guid":"c607487a-83a1-4ba9-bd8f-ec9392fb1326","_uuid":"173e1941a581e7a12d3054238af555d1fec89183"}},{"cell_type":"code","source":"# Import and split\ntrain = pd.read_csv('../input/train.csv')\ntrain.drop('Id',axis = 1, inplace = True)\ntrain_numerical = train.select_dtypes(exclude=['object'])\ntrain_numerical.fillna(0,inplace = True)\ntrain_categoric = train.select_dtypes(include=['object'])\ntrain_categoric.fillna('NONE',inplace = True)\ntrain = train_numerical.merge(train_categoric, left_index = True, right_index = True) \n\ntest = pd.read_csv('../input/test.csv')\nID = test.Id\ntest.drop('Id',axis = 1, inplace = True)\ntest_numerical = test.select_dtypes(exclude=['object'])\ntest_numerical.fillna(0,inplace = True)\ntest_categoric = test.select_dtypes(include=['object'])\ntest_categoric.fillna('NONE',inplace = True)\ntest = test_numerical.merge(test_categoric, left_index = True, right_index = True) ","metadata":{"_cell_guid":"5ae69e94-625c-421b-b3b5-325f80aa0164","_uuid":"b536485388dc43bc383179d02e6c89b054f37159","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Removie the outliers\nfrom sklearn.ensemble import IsolationForest\n\nclf = IsolationForest(max_samples = 100, random_state = 42)\nclf.fit(train_numerical)\ny_noano = clf.predict(train_numerical)\ny_noano = pd.DataFrame(y_noano, columns = ['Top'])\ny_noano[y_noano['Top'] == 1].index.values\n\ntrain_numerical = train_numerical.iloc[y_noano[y_noano['Top'] == 1].index.values]\ntrain_numerical.reset_index(drop = True, inplace = True)\n\ntrain_categoric = train_categoric.iloc[y_noano[y_noano['Top'] == 1].index.values]\ntrain_categoric.reset_index(drop = True, inplace = True)\n\ntrain = train.iloc[y_noano[y_noano['Top'] == 1].index.values]\ntrain.reset_index(drop = True, inplace = True)","metadata":{"_cell_guid":"f67a2138-cdbc-4282-bd1d-55cc70c5c466","_uuid":"8b5200d119fb5f1e836fb746e63bf26a5cb53aea","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col_train_num = list(train_numerical.columns)\ncol_train_num_bis = list(train_numerical.columns)\n\ncol_train_cat = list(train_categoric.columns)\n\ncol_train_num_bis.remove('SalePrice')\n\nmat_train = np.matrix(train_numerical)\nmat_test  = np.matrix(test_numerical)\nmat_new = np.matrix(train_numerical.drop('SalePrice',axis = 1))\nmat_y = np.array(train.SalePrice)\n\nprepro_y = MinMaxScaler()\nprepro_y.fit(mat_y.reshape(1314,1))\n\nprepro = MinMaxScaler()\nprepro.fit(mat_train)\n\nprepro_test = MinMaxScaler()\nprepro_test.fit(mat_new)\n\ntrain_num_scale = pd.DataFrame(prepro.transform(mat_train),columns = col_train)\ntest_num_scale  = pd.DataFrame(prepro_test.transform(mat_test),columns = col_train_bis)","metadata":{"_cell_guid":"0639e24d-202b-461c-b708-0c7d5bc3aa18","_uuid":"b4b2a5534771950433404e618d6b002a913606a2","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[col_train_num] = pd.DataFrame(prepro.transform(mat_train),columns = col_train_num)\ntest[col_train_num_bis]  = test_num_scale","metadata":{"_cell_guid":"8ae00e60-3fe9-4297-a36e-91be9af657e0","_uuid":"666dc634a6f1b56d4e5492644ecb49d9fc444fc6","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The principal changements are here with the lines beginning by for categorical_features... It is possible to use other function to prepare your categorical data. ","metadata":{"_cell_guid":"06a63dfe-4b39-4281-a240-7620595dcb87","_uuid":"043062c7d11e3503cf2d60da1e3eb46639228a36"}},{"cell_type":"code","source":"# List of features\nCOLUMNS = col_train_num\nFEATURES = col_train_num_bis\nLABEL = \"SalePrice\"\n\nFEATURES_CAT = col_train_cat\n\nengineered_features = []\n\nfor continuous_feature in FEATURES:\n    engineered_features.append(\n        tf.contrib.layers.real_valued_column(continuous_feature))\n\nfor categorical_feature in FEATURES_CAT:\n    sparse_column = tf.contrib.layers.sparse_column_with_hash_bucket(\n        categorical_feature, hash_bucket_size=1000)\n\n    engineered_features.append(tf.contrib.layers.embedding_column(sparse_id_column=sparse_column, dimension=16,combiner=\"sum\"))\n                                 \n# Training set and Prediction set with the features to predict\ntraining_set = train[FEATURES + FEATURES_CAT]\nprediction_set = train.SalePrice\n\n# Train and Test \nx_train, x_test, y_train, y_test = train_test_split(training_set[FEATURES + FEATURES_CAT] ,\n                                                    prediction_set, test_size=0.33, random_state=42)\ny_train = pd.DataFrame(y_train, columns = [LABEL])\ntraining_set = pd.DataFrame(x_train, columns = FEATURES + FEATURES_CAT).merge(y_train, left_index = True, right_index = True)\n\n# Training for submission\ntraining_sub = training_set[FEATURES + FEATURES_CAT]\ntesting_sub = test[FEATURES + FEATURES_CAT]","metadata":{"_cell_guid":"268dc739-d13f-42e5-8eb0-f0f16e75b41b","_uuid":"dae02f368a2cc5b5ca248e55e5b63e1e3477993e","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Same thing but for the test set\ny_test = pd.DataFrame(y_test, columns = [LABEL])\ntesting_set = pd.DataFrame(x_test, columns = FEATURES + FEATURES_CAT).merge(y_test, left_index = True, right_index = True)","metadata":{"_cell_guid":"b7a855c8-82d4-4935-83c0-b05f6d917a7e","_uuid":"9be1327ff12a8c1e0a3ef6920cc482fea1a6b346","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_set[FEATURES_CAT] = training_set[FEATURES_CAT].applymap(str)\ntesting_set[FEATURES_CAT] = testing_set[FEATURES_CAT].applymap(str)\n\ndef input_fn_new(data_set, training = True):\n    continuous_cols = {k: tf.constant(data_set[k].values) for k in FEATURES}\n    \n    categorical_cols = {k: tf.SparseTensor(\n        indices=[[i, 0] for i in range(data_set[k].size)], values = data_set[k].values, dense_shape = [data_set[k].size, 1]) for k in FEATURES_CAT}\n\n    # Merges the two dictionaries into one.\n    feature_cols = dict(list(continuous_cols.items()) + list(categorical_cols.items()))\n    \n    if training == True:\n        # Converts the label column into a constant Tensor.\n        label = tf.constant(data_set[LABEL].values)\n\n        # Returns the feature columns and the label.\n        return feature_cols, label\n    \n    return feature_cols\n\n# Model\nregressor = tf.contrib.learn.DNNRegressor(feature_columns = engineered_features, \n                                          activation_fn = tf.nn.relu, hidden_units=[200, 100, 50, 25, 12])","metadata":{"_cell_guid":"f3df3f0c-9d5a-460b-9913-bd5768c3c665","_uuid":"8069ef0a0ba6e38f4139e3b376e4e4de5546816d","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_cols = {k: tf.SparseTensor(indices=[[i, 0] for i in range(training_set[k].size)], values = training_set[k].values, dense_shape = [training_set[k].size, 1]) for k in FEATURES_CAT}","metadata":{"_cell_guid":"b009c3cd-9048-4418-b73d-322268ddfe10","_uuid":"d7f907a9fc08d8af967c1f3f7acce50c722889fb","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Deep Neural Network Regressor with the training set which contain the data split by train test split\nregressor.fit(input_fn = lambda: input_fn_new(training_set) , steps=2000)","metadata":{"_cell_guid":"874777ec-071a-4e2f-9e60-dcbc72487487","_uuid":"71daf4fea970ed7a9232c67b5664fdaaf276203b","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ev = regressor.evaluate(input_fn=lambda: input_fn_new(testing_set, training = True), steps=1)","metadata":{"_cell_guid":"242a2c81-5736-4322-ae86-8ff474ac1438","_uuid":"343de6e9ea87548d71fb379d5d3455d769b1fdf8","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loss_score4 = ev[\"loss\"]\nprint(\"Final Loss on the testing set: {0:f}\".format(loss_score4))","metadata":{"_cell_guid":"bc5b50b5-5c58-471d-a4ac-18d5b3021ebf","_uuid":"efa77284a7415bd742573492cc86a37eeff977a2","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> VIII. Predictions bis","metadata":{"_cell_guid":"81cb38d1-5448-440c-af7f-92754947a1f3","_uuid":"ef71e7267dae00cbf4bc2cd88260e46bb9088ee1"}},{"cell_type":"code","source":"# Predictions\ny = regressor.predict(input_fn=lambda: input_fn_new(testing_set))\npredictions = list(itertools.islice(y, testing_set.shape[0]))\npredictions = pd.DataFrame(prepro_y.inverse_transform(np.array(predictions).reshape(434,1)))","metadata":{"_cell_guid":"3c1a3d70-26b9-4725-a431-09688c61b135","_uuid":"71b34cbf12b26059ea3ddab595ddae87b1e937b4","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matplotlib.rc('xtick', labelsize=30) \nmatplotlib.rc('ytick', labelsize=30) \n\nfig, ax = plt.subplots(figsize=(50, 40))\n\nplt.style.use('ggplot')\nplt.plot(predictions.values, reality.values, 'ro')\nplt.xlabel('Predictions', fontsize = 30)\nplt.ylabel('Reality', fontsize = 30)\nplt.title('Predictions x Reality on dataset Test', fontsize = 30)\nax.plot([reality.min(), reality.max()], [reality.min(), reality.max()], 'k--', lw=4)\nplt.show()","metadata":{"_cell_guid":"6e1fa3c8-6bc0-4f44-88bb-e102bbc15bab","_uuid":"f202b4cc9c46c2cec55f144ad457233e2d892cfa","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_predict = regressor.predict(input_fn=lambda: input_fn_new(testing_sub, training = False))","metadata":{"_cell_guid":"784393b4-b463-4245-9256-c14dffa30b12","_uuid":"9f13d3b60f511acbb91056cf1b12d037b60146be","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"to_submit(y_predict, \"submission_cont_categ\")","metadata":{"_cell_guid":"24bc5061-452d-4e05-9d13-97cb7aab8289","_uuid":"1c9106270fd38e149054c6fcd1576c67a2db3a48","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> IX. Shallow Network","metadata":{"_cell_guid":"7e016bd8-c2ce-46f2-aabd-6ab0c319de47","_uuid":"c38f966130fa61b824e432043818618a0e849fe5"}},{"cell_type":"markdown","source":"For this part we will expolore the architecture with just one Hidden Layer with several units. The question is: How many units do you need to have a good score on the leaderboard? We will try with 1000 units with the activation function Relu.","metadata":{"_cell_guid":"a669a46b-f1a9-463d-8b1b-7e45505ad178","_uuid":"a47468d133d7d191f770c3d08334864aa74d15d1"}},{"cell_type":"code","source":"# Model\nregressor = tf.contrib.learn.DNNRegressor(feature_columns = engineered_features, \n                                          activation_fn = tf.nn.relu, hidden_units=[1000])","metadata":{"_cell_guid":"62ea6c6e-da9a-43fd-b72c-a9ce5f04330c","_uuid":"827d3c0248410ceb027cb564c8cdee55965cf0e5","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Deep Neural Network Regressor with the training set which contain the data split by train test split\nregressor.fit(input_fn = lambda: input_fn_new(training_set) , steps=2000)","metadata":{"_cell_guid":"d06dc911-71e7-458a-9a6b-92ae6556fe77","_uuid":"a130ce41bfa507dcd3171eb97f1e56f559d9e444","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ev = regressor.evaluate(input_fn=lambda: input_fn_new(testing_set, training = True), steps=1)\nloss_score5 = ev[\"loss\"]","metadata":{"_cell_guid":"a071d27f-5257-4f3c-bccf-5e9a80daed0f","_uuid":"352fd136f75cab9e514d3fa8918385bb3ce768fb","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Final Loss on the testing set: {0:f}\".format(loss_score5))","metadata":{"_cell_guid":"3c76cf13-b147-414c-a2d0-4eee1b62b297","_uuid":"e5063613e9925962428ed861437449f458ff9d29","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_predict = regressor.predict(input_fn=lambda: input_fn_new(testing_sub, training = False))    \nto_submit(y_predict, \"submission_shallow\")","metadata":{"_cell_guid":"58bc0c60-456c-408b-a6d7-255a4506db36","_uuid":"87be0f6006d75564020d7967528cec97a6664e85","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <center> X. Conclusion","metadata":{"_cell_guid":"51ab5586-53f3-43e0-926b-e178f856f5a1","_uuid":"ee22d288772717d0f61fa1ca8db79b8f765ecccc"}},{"cell_type":"code","source":"list_score = [loss_score1, loss_score2, loss_score3, loss_score4,loss_score5]\nlist_model = ['Relu_cont', 'LRelu_cont', 'Elu_cont', 'Relu_cont_categ','Shallow_1ku']","metadata":{"_cell_guid":"9547c21a-4f23-49cf-8cc6-269c328b8ec5","_uuid":"548cadf701cde3e7850171dd6f52a24fd42add0f","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt; plt.rcdefaults()\n\nplt.style.use('ggplot')\nobjects = list_model\ny_pos = np.arange(len(objects))\nperformance = list_score\n \nplt.barh(y_pos, performance, align='center', alpha=0.9)\nplt.yticks(y_pos, objects)\nplt.xlabel('Loss ')\nplt.title('Model compared without hypertuning')\n \nplt.show()","metadata":{"_cell_guid":"5b1f7fb1-f8ba-4176-9bc6-a7c85d520e50","_uuid":"38ac30fe3da8cd6a6bd98e78c8aaadbfa16ccb7a","collapsed":true,"jupyter":{"outputs_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n","metadata":{"_cell_guid":"9b48d35b-9281-48de-8534-b0448b99918b","_uuid":"285871604d03aa1b18672e871c4641b0098a11a1"}}]}