{"cells":[{"metadata":{"_uuid":"a1c37e526e01069b76b113b30b4b9ff8b4b54fd0"},"cell_type":"markdown","source":"In this Kernel, I am focusing on exploring the available data, and adjusting SalePrice to inflation"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\n\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import Imputer\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.tree import DecisionTreeRegressor\n\nfrom xgboost import XGBRegressor\n\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d869c2c96a15e6c57d55098a6f4f5765670ad238"},"cell_type":"markdown","source":"Notes on things to look into:\n\n  1) Does price take into account inflation? (goes all the way to 1880, which could have a huge effect on the price difference)\n  \n  2) We did data cleaning on 1 variable, but looking at some other heavily correlated data (to SalePrice), this could be done for more variables, to limit as much as possible the effects of anormal data points"},{"metadata":{"_uuid":"3adc17cf4f6fc013275529bde46f79530d488484","trusted":true},"cell_type":"code","source":"# List of all the functions that we are going to use\n\ndef get_rmse(y_predicted,y_real):\n    return np.mean(np.sqrt((np.log(y_predicted)-np.log(y_real))**2))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_data = pd.read_csv('../input/train.csv')\ntest_data = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b34e884a7084e61df69a8166a1b6da8ac1e3dc10","trusted":true},"cell_type":"code","source":"train_data.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42ab7915bb35500873b6660868b059db9aa9b0f7"},"cell_type":"code","source":"train_data.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d0fb907ccee888249f27077b53ae56bba136cc2","trusted":true},"cell_type":"code","source":"#We want to visualize the dsitribution of our target, here the SalePrice:\n\nplt.hist(train_data.SalePrice, bins=100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20addabf5b1bd96162ff2237619ec78d73767a39"},"cell_type":"code","source":"#Data seems very scattered, so we will look at the log ot the Sale Price ditribution:\n\nlogged_price = np.log(train_data.SalePrice)\nplt.hist(logged_price, bins=100)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61d78d915cfa214a6c784cfe665febfa9b19e6e9","trusted":true},"cell_type":"code","source":"plt.hist(train_data.YearBuilt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d1686ac2a6920c3b633a3a2bf666b91d33865bb3"},"cell_type":"code","source":"plt.hist(train_data.YrSold)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b9b0586671fc6a97a400fffb8a586cf3dedb185"},"cell_type":"markdown","source":"We can see that the houses have been sold from 2006 to 2010.\n\nFrom this [website](https://www.in2013dollars.com/1860-dollars-in-2017?amount=1), we can see that from a basis of 1 dollar in 1860, it was 24.29 dollars and 26.17 dollars in 2010. That is on average a 7.74 per cent increase in the value of the dollar, in just those 4 years. The actual change over that period is:"},{"metadata":{"trusted":true,"_uuid":"e9418717c315bfbfcb717f73922133b67ecf8b20"},"cell_type":"code","source":"plt.hist(test_data.YrSold)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f0b6dc4b984521e5ac42a8ebdc8cb99305819f3c"},"cell_type":"markdown","source":"By looking at the test data, we can see that the 'YrSold' also goes from 2006 to 2010. It seems like it might be interesting to adjust inflation in the test set too.\n\nI simply am not sure if this is something we can do, and if we should to any modifcation to the test set at all? It seems like the proper thing to do too, so I also adjust SalePrice on the test set."},{"metadata":{"trusted":true,"_uuid":"f7cceab8251d311195255a655efb66d844939925"},"cell_type":"code","source":"inflation = pd.DataFrame(dict(value_by_1860_usd=[24.29, 24.98, 25.94, 25.85, 26.27],\n                          inflation_percent=[3.23, 2.85, 3.84, -0.36, 1.64]) ,\n                      index = ['2006', '2007', '2008', '2009', '2010'])\ninflation","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22d07e4a62fd7bfcf215f9b103babb1e1da6d160"},"cell_type":"markdown","source":"## Now we want to create a new dataset with SalePrice adjusted for inflation. \nSince the lastest year where any house was sold in this data set is 2010, this is what out reference will be. This means we are going to keep the value of the houses sold in 2010, but increase (since inflation devaluates the money) the SalePrice of the other houses.\n\nHow to do this:\n\n- Start by creating a new pd.df for each year, containing all the examples that were sold that particular year.\n- For each, multiply SalePrice by inflation factor (which has to be 2010_usd/200X_usd (>1, so it will increase the price)) of all examples\n- Concat all these pd.df into one, where each will have been adjusted for inflation accordingly\n- Check that we have:\n    - 1) Same nb of elements in the pd.df\n    - 2) Check we have the same nb of examples per year, this should be enough to guarantee enough that we have all the examples"},{"metadata":{"trusted":true,"_uuid":"930a53ee04d976d82035d3fbe5df85b3c3db19d7"},"cell_type":"code","source":"#Let's start by creating a new data set for each year:\ninfl_df_2006 = train_data.loc[train_data['YrSold'] == 2006]\ninfl_df_2007 = train_data.loc[train_data['YrSold'] == 2007]\ninfl_df_2008 = train_data.loc[train_data['YrSold'] == 2008]\ninfl_df_2009 = train_data.loc[train_data['YrSold'] == 2009]\ninfl_df_2010 = train_data.loc[train_data['YrSold'] == 2010] #this one will not be changed, just used for final concat\n\n#Since we tried adjusting prices for inflation, and the MAE turned out ot be worse, \n#we are going to try at different % of inflation correction: 25%, 50%, 75% and 100%\n\n#25%\ninfl_df_2006_25 = infl_df_2006\ninfl_df_2007_25 = infl_df_2007\ninfl_df_2008_25 = infl_df_2008\ninfl_df_2009_25 = infl_df_2009\ninfl_df_2010_25 = infl_df_2010\n\n#50%\ninfl_df_2006_50 = infl_df_2006\ninfl_df_2007_50 = infl_df_2007\ninfl_df_2008_50 = infl_df_2008\ninfl_df_2009_50 = infl_df_2009\ninfl_df_2010_50 = infl_df_2010\n\n#75%\ninfl_df_2006_75 = infl_df_2006\ninfl_df_2007_75 = infl_df_2007\ninfl_df_2008_75 = infl_df_2008\ninfl_df_2009_75 = infl_df_2009\ninfl_df_2010_75 = infl_df_2010\n\ninfl_df_2006_25.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7ef90eb71cc25aad6dd1a943f4ec4e1d3996f890"},"cell_type":"code","source":"#We get the value of the USD at a given year\nusd_2006 = inflation.loc['2006','value_by_1860_usd']\nusd_2007 = inflation.loc['2007','value_by_1860_usd']\nusd_2008 = inflation.loc['2008','value_by_1860_usd']\nusd_2009 = inflation.loc['2009','value_by_1860_usd']\n\nusd_2010 = inflation.loc['2010','value_by_1860_usd']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c3ff6f27ce81b666eae81474cf7df8b87b738f6e"},"cell_type":"code","source":"#We then get a factor of the value of the USD in a year compared to the 2010 USD:\ndivid_factor_06 = usd_2010 / usd_2006\ndivid_factor_07 = usd_2010 / usd_2007\ndivid_factor_08 = usd_2010 / usd_2008\ndivid_factor_09 = usd_2010 / usd_2009\n\nprint('divide factor for 2006 is {} '.format(divid_factor_06))\nprint('divide factor for 2007 is {} '.format(divid_factor_07))\nprint('divide factor for 2008 is {} '.format(divid_factor_08))\nprint('divide factor for 2009 is {} '.format(divid_factor_09))\n\n#25%:\ndivid_factor_06_25 = divid_factor_06/4\ndivid_factor_07_25 = divid_factor_07/4\ndivid_factor_08_25 = divid_factor_08/4\ndivid_factor_09_25 = divid_factor_09/4\n\n#50%\ndivid_factor_06_50 = divid_factor_06/2\ndivid_factor_07_50 = divid_factor_07/2\ndivid_factor_08_50 = divid_factor_08/2\ndivid_factor_09_50 = divid_factor_09/2\n\n#75%\ndivid_factor_06_75 = divid_factor_06/(4/3)\ndivid_factor_07_75 = divid_factor_07/(4/3)\ndivid_factor_08_75 = divid_factor_08/(4/3)\ndivid_factor_09_75 = divid_factor_09/(4/3)\n\nfloat(divid_factor_06)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f11e11b581b27a588f132cdbe7dd78c1b71cbf6a"},"cell_type":"code","source":"#We need to multiply the values of SalePrice by divid_factor\n\n#25%:\ninfl_df_2006_25['SalePrice'] = infl_df_2006_25['SalePrice'].apply(lambda x: x*float(divid_factor_06_25));\ninfl_df_2007_25['SalePrice'] = infl_df_2007_25['SalePrice'].apply(lambda x: x*divid_factor_07_25);\ninfl_df_2008_25['SalePrice'] = infl_df_2008_25['SalePrice'].apply(lambda x: x*divid_factor_08_25);\ninfl_df_2009_25['SalePrice'] = infl_df_2009_25['SalePrice'].apply(lambda x: x*divid_factor_09_25);\n\n#50%:\ninfl_df_2006_50['SalePrice'] = infl_df_2006['SalePrice'].apply(lambda x: x*divid_factor_06_50);\ninfl_df_2007_50['SalePrice'] = infl_df_2007['SalePrice'].apply(lambda x: x*divid_factor_07_50);\ninfl_df_2008_50['SalePrice'] = infl_df_2008['SalePrice'].apply(lambda x: x*divid_factor_08_50);\ninfl_df_2009_50['SalePrice'] = infl_df_2009['SalePrice'].apply(lambda x: x*divid_factor_09_50);\n\n#75%:\ninfl_df_2006_75['SalePrice'] = infl_df_2006['SalePrice'].apply(lambda x: x*divid_factor_06_75);\ninfl_df_2007_75['SalePrice'] = infl_df_2007['SalePrice'].apply(lambda x: x*divid_factor_07_75);\ninfl_df_2008_75['SalePrice'] = infl_df_2008['SalePrice'].apply(lambda x: x*divid_factor_08_75);\ninfl_df_2009_75['SalePrice'] = infl_df_2009['SalePrice'].apply(lambda x: x*divid_factor_09_75);\n\n#100%:\ninfl_df_2006['SalePrice'] = infl_df_2006['SalePrice'].apply(lambda x: x*divid_factor_06);\ninfl_df_2007['SalePrice'] = infl_df_2007['SalePrice'].apply(lambda x: x*divid_factor_07);\ninfl_df_2008['SalePrice'] = infl_df_2008['SalePrice'].apply(lambda x: x*divid_factor_08);\ninfl_df_2009['SalePrice'] = infl_df_2009['SalePrice'].apply(lambda x: x*divid_factor_09);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fa668113a3490360e037294f09c312c14053cfe"},"cell_type":"code","source":"#Now we need to concat all the sub DataFrames in one bigger one\n#25%:\nframes_25 = [infl_df_2006_25, infl_df_2007_25, infl_df_2008_25, infl_df_2009_25, infl_df_2010_25]\ninfl_train_data_25 = pd.concat(frames_25)\n\n#50%:\nframes_50 = [infl_df_2006_50, infl_df_2007_50, infl_df_2008_50, infl_df_2009_50, infl_df_2010_50]\ninfl_train_data_50 = pd.concat(frames_50)\n\n#75%:\nframes_75 = [infl_df_2006_75, infl_df_2007_75, infl_df_2008_75, infl_df_2009_75, infl_df_2010_75]\ninfl_train_data_75 = pd.concat(frames_75)\n\n#100:\nframes = [infl_df_2006, infl_df_2007, infl_df_2008, infl_df_2009, infl_df_2010]\ninfl_train_data = pd.concat(frames)\n\n\nprint('size of train data : ',train_data.size)\nprint('size of train data adjusted for inflation : ', infl_train_data.size)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d73c3500f203363be34bd2169670f50a4bfa730"},"cell_type":"markdown","source":"We have now created the same DataFrame, but with SalePrice adjusted for inflation. We can see that both 'train_data' and the adjusted for inflation DataFrame have the same size. The other difference is that they are not arranged in the same order, as the adjusted is order by 'YrSold', as it is a concat of sub-DataFrames divided by year"},{"metadata":{"_uuid":"71fb1bcdcacc8dc3d45cea110d9144d83a052c47","trusted":true},"cell_type":"code","source":"print('nb of missing data = {0}'.format(train_data.isnull().sum().max()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89e5ee3122513cc39207bba1bcb3e99792e65195","trusted":true},"cell_type":"code","source":"#Missing values:\ntotal = train_data.isnull().sum().sort_values(ascending=False)\npercent=(train_data.isnull().sum()/train_data.isnull().count()).sort_values(ascending=False)\n\nmissing_data=pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nmissing_data.head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f8e0a5d3c22a26b2a21e86dd36f84bf03e8bb25"},"cell_type":"markdown","source":"After looking at this information, we can see that:\n- 'PoolQc', 'MiscFeature', 'Alley', and 'Fence'are lacking from 99 tp 80% of their data. No real point in keeping that sort of data, we could just dump it, it will only make out database bigger for nothing to use dummies here.\n- All the 'Garage[...]' features seem to have the same data lacking. Later on, with the histrogram they look very correlated. We could keep only one of them maybe? \n- Same goes for 'Bsmt[...]' data.\n\nIn the [Kernel](http://www.kaggle.com/pmarcelino/comprehensive-data-exploration-with-python) most of this is based on, dummies are the last step. I guess this is once everything has already been smoothed out, which sounds (more) logical"},{"metadata":{"_uuid":"4e6aed2b034ca1795b15887aa4746f68ce2092fe","trusted":true},"cell_type":"code","source":"correlation = train_data.corr()\nprint('Most correlated columns to {0} are: '.format('SalePrice'),'\\n', correlation['SalePrice'].sort_values(ascending = False)[:10])\nprint('Least correlated columns to {0} are: '.format('SalePrice'),'\\n', correlation['SalePrice'].sort_values(ascending = False)[-10:])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc759e58de3ec31aed0ff716d4b1e7da6d62cb85","trusted":true},"cell_type":"code","source":"corrmat = train_data.corr()\nf, ax = plt.subplots(figsize=(12,9))\nsns.heatmap(corrmat, vmax = 0.8, square=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57c81b8164487679f5ebc5848dc4b389345e16f1","trusted":true},"cell_type":"code","source":"#Scatterplot\nsns.set()\ncols_scat = ['SalePrice','OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 'FullBath', 'YearBuilt']\nsns.pairplot(train_data[cols_scat])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2407f1726ac1283d4d152567728ba7e8836f720","trusted":true},"cell_type":"code","source":"#Now, we get rid of variables that are missing too much data (here, everything that is missing more than 1 variable)\ntrain_data = train_data.drop((missing_data[missing_data['Total']>1]).index, 1)\ninfl_train_data = infl_train_data.drop((missing_data[missing_data['Total']>1]).index, 1)\ninfl_train_data_25 = infl_train_data_25.drop((missing_data[missing_data['Total']>1]).index, 1)\ninfl_train_data_50 = infl_train_data_50.drop((missing_data[missing_data['Total']>1]).index, 1)\ninfl_train_data_75 = infl_train_data_75.drop((missing_data[missing_data['Total']>1]).index, 1)\n\n#We will delete the entry containing the missing data in the Electrical variable: \ntrain_data = train_data.drop(train_data.loc[train_data['Electrical'].isnull()].index)\ninfl_train_data = infl_train_data.drop(infl_train_data.loc[infl_train_data['Electrical'].isnull()].index)\ninfl_train_data_25 = infl_train_data_25.drop(infl_train_data_25.loc[infl_train_data_25['Electrical'].isnull()].index)\ninfl_train_data_50 = infl_train_data_50.drop(infl_train_data_50.loc[infl_train_data_50['Electrical'].isnull()].index)\ninfl_train_data_75 = infl_train_data_75.drop(infl_train_data_75.loc[infl_train_data_75['Electrical'].isnull()].index)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aae018e5005f72a933ac6523f10aff609a0cf1bf"},"cell_type":"markdown","source":"We should look more into the details of the data before deleting it, we should not base ourselves on the conclusions that we think might explain the irregularity of the data. Especially before deleting that data."},{"metadata":{"_uuid":"ab90fe870840c06aa640364c73b54001655eb9b3","trusted":true},"cell_type":"code","source":"print('Missing values in train data : ', train_data.isnull().sum().max())\nprint('Missing values in train data adjusted for inflation : ', infl_train_data.isnull().sum().max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a8581fbc83c8c85248b835b222dc8ee579a2637","trusted":true},"cell_type":"code","source":"var = 'GrLivArea'\ndata = pd.concat([train_data['SalePrice'], train_data[var]], axis=1)\ndata.plot.scatter(x=var, y='SalePrice', ylim=(0,800000))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb9a4c72928dccbd1235420d3ca50102b0db3cab","trusted":true},"cell_type":"code","source":"#We want to see the two points with highest GrLivArea, they seem ourliers\ntrain_data.sort_values(by = 'GrLivArea', ascending = False)[:2] #(from kernel found online)\n\n#I just checked if another technique worked, just for pratcise, it gives the same result (which is good news)\n#High_GrLivArea = train_data[train_data['GrLivArea']>4500]\n#High_GrLivArea.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2db2fc9405a143300ad639b6996d17debf68f87","trusted":true},"cell_type":"code","source":"train_data = train_data.drop(train_data[train_data[\"Id\"] == 524].index)\ntrain_data = train_data.drop(train_data[train_data[\"Id\"] == 1299].index)\n\ninfl_train_data = infl_train_data.drop(infl_train_data[infl_train_data['Id']==524].index)\ninfl_train_data = infl_train_data.drop(infl_train_data[infl_train_data['Id']==1299].index)\n\ninfl_train_data_25 = infl_train_data_25.drop(infl_train_data_25[infl_train_data_25['Id']==524].index)\ninfl_train_data_25 = infl_train_data_25.drop(infl_train_data_25[infl_train_data_25['Id']==1299].index)\n\ninfl_train_data_50 = infl_train_data_50.drop(infl_train_data_50[infl_train_data_50['Id']==524].index)\ninfl_train_data_50 = infl_train_data_50.drop(infl_train_data_50[infl_train_data_50['Id']==1299].index)\n\ninfl_train_data_75 = infl_train_data_75.drop(infl_train_data_75[infl_train_data_75['Id']==524].index)\ninfl_train_data_75 = infl_train_data_75.drop(infl_train_data_75[infl_train_data_75['Id']==1299].index)\n\n\ndata = pd.concat([train_data['SalePrice'], train_data[var]], axis=1)\ndata.plot.scatter(x=var, y='SalePrice', ylim=(0,800000), title='Train data')\n\ninfl_data = pd.concat([infl_train_data['SalePrice'], infl_train_data[var]], axis=1)\ninfl_data.plot.scatter(x=var, y='SalePrice', ylim=(0,800000), title='Adjusted train data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1187f7915410b1ddbbda88114d1d86bb331cf729","trusted":true},"cell_type":"code","source":"#One-hot encoding (using categorical data)\n\ncols_with_missing = [col for col in train_data.columns \n                                 if train_data[col].isnull().any()]                                  \ncandidate_train_predictors = train_data.drop(['Id', 'SalePrice'] + cols_with_missing, axis=1)\ncandidate_test_predictors = test_data.drop(['Id'] + cols_with_missing, axis=1)\n\n# \"cardinality\" means the number of unique values in a column.\n# We use it as our only way to select categorical columns here. This is convenient, though\n# a little arbitrary.\nlow_cardinality_cols = [cname for cname in candidate_train_predictors.columns if \n                                candidate_train_predictors[cname].nunique() < 10 and\n                                candidate_train_predictors[cname].dtype == \"object\"]\nnumeric_cols = [cname for cname in candidate_train_predictors.columns if \n                                candidate_train_predictors[cname].dtype in ['int64', 'float64']]\nmy_cols = low_cardinality_cols + numeric_cols\ntrain_predictors = candidate_train_predictors[my_cols]\ntest_predictors = candidate_test_predictors[my_cols]\n\none_hot_encoded_training_predictors = pd.get_dummies(train_predictors)\none_hot_encoded_test_predictors = pd.get_dummies(test_predictors)\nfinal_train, final_test = one_hot_encoded_training_predictors.align(one_hot_encoded_test_predictors,\n                                                                    join='left', \n                                                                    axis=1)\n\n#Same for adjusted set:\ninfl_candidate_train_predictors = infl_train_data.drop(['Id', 'SalePrice'] + cols_with_missing, axis=1)\n\ninfl_train_predictors = infl_candidate_train_predictors[my_cols]\n\ninfl_one_hot_encoded_training_predictors = pd.get_dummies(infl_train_predictors)\ninfl_final_train, final_test = infl_one_hot_encoded_training_predictors.align(one_hot_encoded_test_predictors,\n                                                                    join='left', \n                                                                    axis=1)\n\n#25:\ninfl_candidate_train_predictors_25 = infl_train_data_25.drop(['Id', 'SalePrice'] + cols_with_missing, axis=1)\n\ninfl_train_predictors_25 = infl_candidate_train_predictors_25[my_cols]\n\ninfl_one_hot_encoded_training_predictors_25 = pd.get_dummies(infl_train_predictors_25)\ninfl_final_train_25, final_test = infl_one_hot_encoded_training_predictors_25.align(one_hot_encoded_test_predictors,\n                                                                    join='left', \n                                                                    axis=1)\n\n#50:\ninfl_candidate_train_predictors_50 = infl_train_data_50.drop(['Id', 'SalePrice'] + cols_with_missing, axis=1)\n\ninfl_train_predictors_50 = infl_candidate_train_predictors_50[my_cols]\n\ninfl_one_hot_encoded_training_predictors_50 = pd.get_dummies(infl_train_predictors_50)\ninfl_final_train_50, final_test = infl_one_hot_encoded_training_predictors_50.align(one_hot_encoded_test_predictors,\n                                                                    join='left', \n                                                                    axis=1)\n\n#75:\ninfl_candidate_train_predictors_75 = infl_train_data_75.drop(['Id', 'SalePrice'] + cols_with_missing, axis=1)\n\ninfl_train_predictors_75 = infl_candidate_train_predictors_75[my_cols]\n\ninfl_one_hot_encoded_training_predictors_75 = pd.get_dummies(infl_train_predictors_75)\ninfl_final_train_75, final_test = infl_one_hot_encoded_training_predictors_75.align(one_hot_encoded_test_predictors,\n                                                                    join='left', \n                                                                    axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a20c1051a2bfe4ead26993cfc4b7d6a6417872a"},"cell_type":"markdown","source":"**Now we move to the model**"},{"metadata":{"_uuid":"f3784b339b7b72142e226ad999badebf41301dd3","trusted":true},"cell_type":"code","source":"X = np.array(final_train)\ny = train_data.SalePrice\n\ntrain_X, val_X, train_y, val_y = train_test_split(X, y)\n\n#We are going to compare the original data set, with the adjusted to inflation one:\ninfl_X = np.array(infl_final_train)\ninfl_y = infl_train_data.SalePrice\n\ninfl_train_X, infl_val_X, infl_train_y, infl_val_y = train_test_split(infl_X, infl_y)\n\n#25%:\ninfl_X_25 = np.array(infl_final_train_25)\ninfl_y_25 = infl_train_data_25.SalePrice\n\ninfl_train_X_25, infl_val_X_25, infl_train_y_25, infl_val_y_25 = train_test_split(infl_X_25, infl_y_25)\n\n#50%:\ninfl_X_50 = np.array(infl_final_train_50)\ninfl_y_50 = infl_train_data_50.SalePrice\n\ninfl_train_X_50, infl_val_X_50, infl_train_y_50, infl_val_y_50 = train_test_split(infl_X_50, infl_y_50)\n\n#75%:\ninfl_X_75 = np.array(infl_final_train_75)\ninfl_y_75 = infl_train_data_75.SalePrice\n\ninfl_train_X_75, infl_val_X_75, infl_train_y_75, infl_val_y_75 = train_test_split(infl_X_75, infl_y_75)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73195052bfb5b0a5d4c85050023e5980cf3322fb","trusted":true,"scrolled":true},"cell_type":"code","source":"best_learn_rate = 0.2\nbest_nb_est = 50 \n#These come from a previous Kernel, but for the purpose of just testing this, I will keep those values for now\n\n#we extract the year at which the predicted SalePrice have been sold\nyear_train = train_X[:,[32]]\nyear_test = val_X[:,[32]]\n\nmy_pipeline = make_pipeline(Imputer(), XGBRegressor(n_estimators=best_nb_est, learning_rate = best_learn_rate))\n\nmy_pipeline.fit(X,y)\ntrain_y_predicted = my_pipeline.predict(train_X)\nval_y_predicted = my_pipeline.predict(val_X)\n\nprint('Score on training set:',get_rmse(train_y_predicted,train_y))\nprint('Score on validation set:',get_rmse(val_y_predicted,val_y))\n\n#Adjusted dataset:\nmy_pipeline.fit(infl_X,infl_y)\ninfl_train_y_predicted = my_pipeline.predict(infl_train_X)\ninfl_val_y_predicted = my_pipeline.predict(infl_val_X)\n\n#25% adjusted dataset:\nmy_pipeline.fit(infl_X_25,infl_y_25)\ninfl_train_y_predicted_25 = my_pipeline.predict(infl_train_X_25)\ninfl_val_y_predicted_25 = my_pipeline.predict(infl_val_X_25)\n\n#50% adjusted dataset:\nmy_pipeline.fit(infl_X_50,infl_y_50)\ninfl_train_y_predicted_50 = my_pipeline.predict(infl_train_X_50)\ninfl_val_y_predicted_50 = my_pipeline.predict(infl_val_X_50)\n\n#75% adjusted dataset:\nmy_pipeline.fit(infl_X_75,infl_y_75)\ninfl_train_y_predicted_75 = my_pipeline.predict(infl_train_X_75)\ninfl_val_y_predicted_75 = my_pipeline.predict(infl_val_X_75)\n\n#Now we need to divide the predictions 'infl_train_y_predicted' by the divid_factor of each year accordingly:\n#start by making a copy of the predicted price dataset, just in case:\ny_predict = infl_train_y_predicted\ny_predict[year_train[:,0] == 2009] = y_predict[year_train[:,0] == 2009]/divid_factor_09\ny_predict[year_train[:,0] == 2008] = y_predict[year_train[:,0] == 2008]/divid_factor_08\ny_predict[year_train[:,0] == 2007] = y_predict[year_train[:,0] == 2007]/divid_factor_07\ny_predict[year_train[:,0] == 2006] = y_predict[year_train[:,0] == 2006]/divid_factor_06\ninfl_train_y_predicted = y_predict\n\nval_y_predict = infl_val_y_predicted\nval_y_predict[year_test[:,0] == 2009] = val_y_predict[year_test[:,0] == 2009]/divid_factor_09\nval_y_predict[year_test[:,0] == 2008] = val_y_predict[year_test[:,0] == 2008]/divid_factor_08\nval_y_predict[year_test[:,0] == 2007] = val_y_predict[year_test[:,0] == 2007]/divid_factor_07\nval_y_predict[year_test[:,0] == 2006] = val_y_predict[year_test[:,0] == 2006]/divid_factor_06\ninfl_val_y_predicted = val_y_predict\n\nprint('Score on adjusted training set:',get_rmse(infl_train_y_predicted,infl_train_y))\nprint('Score on adjusted validation set:',get_rmse(infl_val_y_predicted,infl_val_y))\n\n#25%:\ny_predict_25 = infl_train_y_predicted_25\ny_predict_25[year_train[:,0] == 2009] = y_predict_25[year_train[:,0] == 2009]/divid_factor_09_25\ny_predict_25[year_train[:,0] == 2008] = y_predict_25[year_train[:,0] == 2008]/divid_factor_08_25\ny_predict_25[year_train[:,0] == 2007] = y_predict_25[year_train[:,0] == 2007]/divid_factor_07_25\ny_predict_25[year_train[:,0] == 2006] = y_predict_25[year_train[:,0] == 2006]/divid_factor_06_25\ninfl_train_y_predicted_25 = y_predict_25\n\nval_y_predict_25 = infl_val_y_predicted_25\nval_y_predict_25[year_test[:,0] == 2009] = val_y_predict_25[year_test[:,0] == 2009]/divid_factor_09_25\nval_y_predict_25[year_test[:,0] == 2008] = val_y_predict_25[year_test[:,0] == 2008]/divid_factor_08_25\nval_y_predict_25[year_test[:,0] == 2007] = val_y_predict_25[year_test[:,0] == 2007]/divid_factor_07_25\nval_y_predict_25[year_test[:,0] == 2006] = val_y_predict_25[year_test[:,0] == 2006]/divid_factor_06_25\ninfl_val_y_predicted_25 = val_y_predict_25\n\nprint('Score on 25% adjusted training set:',get_rmse(infl_train_y_predicted_25,infl_train_y_25))\nprint('Score on 25% adjusted validation set:',get_rmse(infl_val_y_predicted_25,infl_val_y_25))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2900a9975721b973f5ae63a5b032dd128386de9f"},"cell_type":"markdown","source":"This priced are baed on the 2010 USD. Wee need to now convert back to their respective years. To do so, we are going to need to get 'YrSold' by index, because the predicted prices are just situated in an array with no index or year info. For this, we basically need to get an array with just 'YrSold', since it will be in the same order as our predicted prive array.\n\nThen, we need to divide the predicted SalePrice by the ratio of USD value depending on the YrSold"},{"metadata":{"_uuid":"2ea81d1275aeebefafd0b036f27caaeb9f254c59"},"cell_type":"markdown","source":"Score on training set is better than previous Kernel (0.06955...)\n\nScore on validation set is worse than previous Kernel (0.06391...)\n\nWe can notice a very slight improvement when adjusting for inflation, which is good news!"},{"metadata":{"_uuid":"22c8d51b5aa3a50610483c35477128027107b85b","trusted":true},"cell_type":"code","source":"predictions = my_pipeline.predict(final_test)\noutput = pd.DataFrame({'Id': test_data.Id,\n                       'SalePrice': predictions})\n\noutput.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de7e6b6f7784bd57577b4505546b615f54216154"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}