{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview\nThis was my first Kaggle competition with official submissions. My primary objectives here were to acquire new data science techniques and deepen recently learned techniques through exercise.\n\n## Learning objectives\n* Deepen understanding of recently learned subjects through applied usage:\n    * EDA: analyzing data, filling in null values, engineering new features\n    * Preprocessing  \n* Familiarizing myself with Kaggle and how it can be a helpful sandbox to \"learn by doing\".\n* Learn new skills with machine learning algorithms through hands on application.\n\n## Notable features of this notebook:\n* It trains up models for Lasso, Ridge, XGB Regressor, and a blended Lasso + XGB model. To date, this notebook has produced the best results with the Lasso model.\n* Feature selection for transformations is automated through a script that I wrote. It attempts to automatically select which columns to transform, and tries to pick the best transformation methods from log, square root, inverse, Box-Cox, and Yeo-Johnson\n* I also used some scripting around filling in null values that helped automate the process, reduced manual intervention, and increased repeatability.\n\n## Takeaways\n* Smart usage of scripts and variables makes experimentation within the notebook much easier. I've not yet personally delved into formal data pipelining techniques, but it's clear how powerful it can be to have some frameworks for these techniques.\n* I thoroughly enjoyed working with different ML algorithms. I was able to quickly get a sense of the surface area of what's possible, and where potential limits and future upside could be.","metadata":{"id":"FyhxtkB8JMhp"}},{"cell_type":"markdown","source":"# Setup\n## Importing libraries, loading the data, ","metadata":{"id":"UvEo_QhRJMhq"}},{"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport seaborn as sns\nfrom scipy import stats\nfrom sklearn.preprocessing import RobustScaler, MinMaxScaler, MaxAbsScaler, StandardScaler, PowerTransformer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LinearRegression, Lasso, Ridge, LassoCV\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error\nfrom xgboost import XGBRegressor\n\n# this will be used later, in the script for automating transformations\nimport warnings\nwarnings.filterwarnings(\"error\")\n\n# changing the default data display option, so that it is more legible\npd.set_option('display.float_format', lambda x: '%.5f' % x)","metadata":{"id":"VoZpaO47JMhq","execution":{"iopub.status.busy":"2022-07-09T04:54:25.240875Z","iopub.execute_input":"2022-07-09T04:54:25.241800Z","iopub.status.idle":"2022-07-09T04:54:25.250663Z","shell.execute_reply.started":"2022-07-09T04:54:25.241748Z","shell.execute_reply":"2022-07-09T04:54:25.249038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\ntest = pd.read_csv('../input/house-prices-advanced-regression-techniques/test.csv')","metadata":{"id":"UqMzW_gHJMhr","execution":{"iopub.status.busy":"2022-07-09T04:54:25.253135Z","iopub.execute_input":"2022-07-09T04:54:25.254335Z","iopub.status.idle":"2022-07-09T04:54:25.341612Z","shell.execute_reply.started":"2022-07-09T04:54:25.254249Z","shell.execute_reply":"2022-07-09T04:54:25.340403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"id":"B-sbk4rJJMhr","outputId":"0666e3b7-6920-4e4b-d99b-fc61830bb7c4","execution":{"iopub.status.busy":"2022-07-09T04:54:25.343122Z","iopub.execute_input":"2022-07-09T04:54:25.343941Z","iopub.status.idle":"2022-07-09T04:54:25.355393Z","shell.execute_reply.started":"2022-07-09T04:54:25.343895Z","shell.execute_reply":"2022-07-09T04:54:25.354349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.shape","metadata":{"id":"Y6Ugz7tTJMhs","outputId":"3ae690ed-4ac7-4a3c-f322-b5d1c53920d5","execution":{"iopub.status.busy":"2022-07-09T04:54:25.356899Z","iopub.execute_input":"2022-07-09T04:54:25.357568Z","iopub.status.idle":"2022-07-09T04:54:25.369049Z","shell.execute_reply.started":"2022-07-09T04:54:25.357536Z","shell.execute_reply":"2022-07-09T04:54:25.367787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ad hoc - dropping a couple of outlier cells\nBecause I later split off the *SalePrice* feature into it's own table, I wanted to do some analysis on this step to see if there are outliers that are worth dropping. Especially between the *GrLivArea* and *SalePrice*.\n\nThe scatter plot shows that there are some clear outliers worth dropping in the lower right corner. These are houses with massive square footager, but very low price.\n\nNote: this was actually a step I added very late in this notebook, and it dramatically improved the RMSE of my moedls.","metadata":{"id":"uTpZl3p7JMhs"}},{"cell_type":"code","source":"plt.scatter(train.GrLivArea, train.SalePrice)","metadata":{"id":"nw7vzzw2JMhs","outputId":"a58cebaa-01e3-419b-9397-1fec2a2fa058","execution":{"iopub.status.busy":"2022-07-09T04:54:25.372474Z","iopub.execute_input":"2022-07-09T04:54:25.373152Z","iopub.status.idle":"2022-07-09T04:54:25.611735Z","shell.execute_reply.started":"2022-07-09T04:54:25.373117Z","shell.execute_reply":"2022-07-09T04:54:25.610431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This isolates for values where GrLivArea is above 4,000 square feet and SalePrice is below $300,000.\ntrain = train.drop(train[(train['GrLivArea']>4000) & (train['SalePrice']<300000)].index)","metadata":{"id":"YO2_4CPcJMhs","execution":{"iopub.status.busy":"2022-07-09T04:54:25.613257Z","iopub.execute_input":"2022-07-09T04:54:25.613751Z","iopub.status.idle":"2022-07-09T04:54:25.626237Z","shell.execute_reply.started":"2022-07-09T04:54:25.613709Z","shell.execute_reply":"2022-07-09T04:54:25.625116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"id":"yQmCxo9sJMhs","outputId":"e724129f-7c78-4a1c-9408-188213e9c439","execution":{"iopub.status.busy":"2022-07-09T04:54:25.628069Z","iopub.execute_input":"2022-07-09T04:54:25.628996Z","iopub.status.idle":"2022-07-09T04:54:25.636684Z","shell.execute_reply.started":"2022-07-09T04:54:25.628931Z","shell.execute_reply":"2022-07-09T04:54:25.635674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Quickly checking shape shows that we've dropped 2 rows.","metadata":{"id":"4vHq0fnQJMhs"}},{"cell_type":"markdown","source":"## Stacking the train and test data\nSo that preprocessing activities can be consistently applied to both sets of data. SalePrice is also separated off into it's own table. ","metadata":{"id":"E7UtdAGHJMht"}},{"cell_type":"code","source":"y = pd.DataFrame(train['SalePrice']) # copying the SalePrice column into its own table\ny.describe()","metadata":{"id":"8lgw3fAlJMht","outputId":"e1d78b86-afd9-46d7-dd33-d615a9820556","execution":{"iopub.status.busy":"2022-07-09T04:54:25.638271Z","iopub.execute_input":"2022-07-09T04:54:25.639200Z","iopub.status.idle":"2022-07-09T04:54:25.668344Z","shell.execute_reply.started":"2022-07-09T04:54:25.639154Z","shell.execute_reply":"2022-07-09T04:54:25.666888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x = pd.concat([train.drop(columns='SalePrice'), test], axis = 0) # merging the training and test CSVs, while dropping the SalePrice column from the training data","metadata":{"id":"Xa49er86JMht","execution":{"iopub.status.busy":"2022-07-09T04:54:25.669923Z","iopub.execute_input":"2022-07-09T04:54:25.670662Z","iopub.status.idle":"2022-07-09T04:54:25.695397Z","shell.execute_reply.started":"2022-07-09T04:54:25.670615Z","shell.execute_reply":"2022-07-09T04:54:25.694449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x","metadata":{"id":"K_nrr7VFJMht","outputId":"94f24284-f8ae-47c6-c108-3b934114db18","execution":{"iopub.status.busy":"2022-07-09T04:54:25.696660Z","iopub.execute_input":"2022-07-09T04:54:25.697612Z","iopub.status.idle":"2022-07-09T04:54:25.731923Z","shell.execute_reply.started":"2022-07-09T04:54:25.697574Z","shell.execute_reply":"2022-07-09T04:54:25.730800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analysis","metadata":{"id":"6xMKKLbxJMht"}},{"cell_type":"markdown","source":"Quick takeaways when glancing through the summary details:\n* There are a LOT of features to work through. Even just calling up the info() method made for annoying scrolling through the notebook, because it was so long.\n* Understanding how the features work and interrelate will be hard. I constantly needed to have the data_description.txt file open next to me.\n* There are also a LOT of nulls throughout the data.]","metadata":{"id":"w85FA1GwJMht"}},{"cell_type":"code","source":"x.info()","metadata":{"id":"VVNu1RVzJMht","outputId":"abf1b9e4-237a-4dfb-ebf8-16361fa22261","execution":{"iopub.status.busy":"2022-07-09T04:54:25.733149Z","iopub.execute_input":"2022-07-09T04:54:25.733506Z","iopub.status.idle":"2022-07-09T04:54:25.773037Z","shell.execute_reply.started":"2022-07-09T04:54:25.733475Z","shell.execute_reply":"2022-07-09T04:54:25.771823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SalePrice analysis\nTaking a peek at the target feature yielded a few thoughts:\n* It has a lot of skew on the right tail, and will require some transformations (later on).","metadata":{"id":"MG2gjo7fJMhu"}},{"cell_type":"code","source":"y.hist(bins = 20)","metadata":{"id":"7ssEH9oZJMhu","outputId":"acf76c7a-2b97-4340-c80b-4391d1c7888f","execution":{"iopub.status.busy":"2022-07-09T04:54:25.774586Z","iopub.execute_input":"2022-07-09T04:54:25.775739Z","iopub.status.idle":"2022-07-09T04:54:26.031274Z","shell.execute_reply.started":"2022-07-09T04:54:25.775689Z","shell.execute_reply":"2022-07-09T04:54:26.030302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Filling in Nulls\n* While there are a large number of null values. When reading through the data_description.txt file, it became clear that many of the null values were actually \"Not present\". For example, PoolQC (\"Pool Quality\") has \"NA\" meaning \"No Pool\". Therefore, it actually does have a value. When reading through the data_description.txt file, it becomes clear that they are all \"Not present\".\n* Those cases cover the vast majority of null values. \n* I wrote a script in order to automatically fill in the remaining nulls\n\nAs you can see in the cell below, the initial amount of nulls is huge. Watch what happens after we fill in the values as \"Not present\".","metadata":{"id":"WH74zAp5JMhu"}},{"cell_type":"code","source":"x.isnull().sum().sum()","metadata":{"id":"sX8KcY9YJMhu","outputId":"dbefb87a-fd67-42ea-9e3e-28eee9e22e89","execution":{"iopub.status.busy":"2022-07-09T04:54:26.032831Z","iopub.execute_input":"2022-07-09T04:54:26.033193Z","iopub.status.idle":"2022-07-09T04:54:26.061482Z","shell.execute_reply.started":"2022-07-09T04:54:26.033163Z","shell.execute_reply":"2022-07-09T04:54:26.060551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Categorizing the nulls as \"Not present\"","metadata":{"id":"o_uxr6DMJMhu"}},{"cell_type":"code","source":"x.PoolQC.fillna('Not present', inplace=True) # PoolQC \nx.Fence.fillna('Not present', inplace=True) # Fence\nx.MiscFeature.fillna('Not present', inplace=True) # MiscFeature\nx.Alley.fillna('Not present', inplace=True) # Alley\nx.BsmtQual.fillna('Not present', inplace=True) # BsmtQual\nx.BsmtCond.fillna('Not present', inplace=True) # BsmtCond\nx.BsmtExposure.fillna('Not present', inplace=True) # BsmtExposure\nx.BsmtFinType1.fillna('Not present', inplace=True) # BsmtFinType1\nx.BsmtFinType2.fillna('Not present', inplace=True) # BsmtFinType2\nx.FireplaceQu.fillna('Not present', inplace=True) # FireplaceQu\nx.GarageType.fillna('Not present', inplace=True) # GarageType\nx.GarageYrBlt.fillna('Not present', inplace=True) # GarageYrBlt\nx.GarageQual.fillna('Not present', inplace=True) # GarageQual\nx.GarageCond.fillna('Not present', inplace=True) # GarageCond\nx.GarageFinish.fillna('Not present', inplace=True) # GarageFinish","metadata":{"id":"gmoMgnV8JMhu","execution":{"iopub.status.busy":"2022-07-09T04:54:26.066418Z","iopub.execute_input":"2022-07-09T04:54:26.066774Z","iopub.status.idle":"2022-07-09T04:54:26.088437Z","shell.execute_reply.started":"2022-07-09T04:54:26.066743Z","shell.execute_reply":"2022-07-09T04:54:26.087538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x.isnull().sum().sum()","metadata":{"id":"jU0IdMbyJMhu","outputId":"501acd92-1f10-47e2-b084-9459ee64c8f5","execution":{"iopub.status.busy":"2022-07-09T04:54:26.089974Z","iopub.execute_input":"2022-07-09T04:54:26.090905Z","iopub.status.idle":"2022-07-09T04:54:26.121156Z","shell.execute_reply.started":"2022-07-09T04:54:26.090860Z","shell.execute_reply":"2022-07-09T04:54:26.119658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dealing with the remaining nulls\nI wanted to analyze what the distribution of nulls were. There are a long tail of values where filling in the median or mode would be fine. However, I wanted to reserve any features with a \"large\"-ish amount of nulls, that I would want to take a look at more closely.\n\nWhen reviewing other notebooks, I do see that a lot of folks will take a very blunt approach in filling in nulls (ie just dropping in the median for ALL nulls). That probably is good enough. However, for learning purposes, I wanted to take a more granular approach.","metadata":{"id":"rlIfJVToJMhu"}},{"cell_type":"code","source":"isnull = x.isnull().sum().sort_values(ascending=False)\nisnull = isnull[isnull > 0]\nisnull","metadata":{"id":"e2E1P3wcJMhv","outputId":"9f294751-ca68-4618-eb79-c5549bd062d7","execution":{"iopub.status.busy":"2022-07-09T04:54:26.124386Z","iopub.execute_input":"2022-07-09T04:54:26.124942Z","iopub.status.idle":"2022-07-09T04:54:26.154044Z","shell.execute_reply.started":"2022-07-09T04:54:26.124909Z","shell.execute_reply":"2022-07-09T04:54:26.152762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Separating out \"low number of nulls\". ","metadata":{"id":"Zu8r2gRAJMhv"}},{"cell_type":"code","source":"isnull_under_100 = isnull[isnull < 100] # set the threshold at 100 nulls\nlist_isnull_under_100 = isnull_under_100.index.tolist()\n\nlist_isnull_under_100","metadata":{"id":"FeDzZXwUJMhv","outputId":"5700a4d2-d92a-4b4a-9839-8aeabff96649","execution":{"iopub.status.busy":"2022-07-09T04:54:26.155553Z","iopub.execute_input":"2022-07-09T04:54:26.156054Z","iopub.status.idle":"2022-07-09T04:54:26.163895Z","shell.execute_reply.started":"2022-07-09T04:54:26.156023Z","shell.execute_reply":"2022-07-09T04:54:26.162823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note:** I wanted to try to fill in the median here. However, given several of these features are not numericals, I used a fallback to filling in the mode.","metadata":{"id":"aL6jGIFcJMhv"}},{"cell_type":"code","source":"for i in list_isnull_under_100:\n  try:\n    x[i].fillna(x[i].median()[0], inplace=True)\n  except:\n    x[i].fillna(x[i].mode()[0], inplace=True)","metadata":{"id":"JoC73p41JMhv","execution":{"iopub.status.busy":"2022-07-09T04:54:26.165656Z","iopub.execute_input":"2022-07-09T04:54:26.166640Z","iopub.status.idle":"2022-07-09T04:54:26.202821Z","shell.execute_reply.started":"2022-07-09T04:54:26.166591Z","shell.execute_reply":"2022-07-09T04:54:26.201665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Checking the nulls now, I see there is just a small remnant of nulls to deal with. In this case, it was just LotFrontage, and so I just opted to group by the LotConfig and apply the median value. My intuition was that how a house's lot is configured would great impact on the amount of frontage it would have to a street. So for example, a corner lot configuration should theoretically have more frontage since it should have (at least) 2 sides facing the street. And also, one would expect a cul de sac facing lot to have a smaller frontage, due to the way the streets curve in a cul de sac. This is backed up by the groupby data.","metadata":{"id":"3U9cGPt6JMhv"}},{"cell_type":"code","source":"isnull = x.isnull().sum().sort_values(ascending=False)\nisnull = isnull[isnull > 0]\nisnull","metadata":{"id":"DBueTxE0JMhv","outputId":"5a601d7e-6771-4fe4-e6d9-0d40cde9147f","execution":{"iopub.status.busy":"2022-07-09T04:54:26.204292Z","iopub.execute_input":"2022-07-09T04:54:26.204683Z","iopub.status.idle":"2022-07-09T04:54:26.235246Z","shell.execute_reply.started":"2022-07-09T04:54:26.204651Z","shell.execute_reply":"2022-07-09T04:54:26.234086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x.groupby('LotConfig')['LotFrontage'].median()","metadata":{"id":"0vn_kCdHJMhv","outputId":"500554d2-ebc9-4b52-da16-c343758ca1b9","execution":{"iopub.status.busy":"2022-07-09T04:54:26.236952Z","iopub.execute_input":"2022-07-09T04:54:26.238041Z","iopub.status.idle":"2022-07-09T04:54:26.249289Z","shell.execute_reply.started":"2022-07-09T04:54:26.237976Z","shell.execute_reply":"2022-07-09T04:54:26.248115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x.LotFrontage.fillna(x.groupby('LotConfig')['LotFrontage'].transform('median'),inplace=True)","metadata":{"id":"sXALHKDlJMhv","execution":{"iopub.status.busy":"2022-07-09T04:54:26.250941Z","iopub.execute_input":"2022-07-09T04:54:26.252062Z","iopub.status.idle":"2022-07-09T04:54:26.261818Z","shell.execute_reply.started":"2022-07-09T04:54:26.252015Z","shell.execute_reply":"2022-07-09T04:54:26.260434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We have no more nulls!**","metadata":{"id":"fdlSHB8aJMhv"}},{"cell_type":"code","source":"isnull = x.isnull().sum().sort_values(ascending=False)\nisnull = isnull[isnull > 0]\nisnull","metadata":{"id":"GwuGkgqhJMhv","outputId":"a90adccd-d400-4fa7-c13e-8208383d4e39","execution":{"iopub.status.busy":"2022-07-09T04:54:26.263605Z","iopub.execute_input":"2022-07-09T04:54:26.264139Z","iopub.status.idle":"2022-07-09T04:54:26.295846Z","shell.execute_reply.started":"2022-07-09T04:54:26.264079Z","shell.execute_reply":"2022-07-09T04:54:26.294802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering\n* Total square footage (sum all of the sq fts) - the dataset breaks up the actual livable square footage across a handful of columns. If we add them up, we get a more comprehensive total square footage. \n* Total # of bathrooms - Bathrooms are broken up over a handful of features. If we add them up, we get a more comprehensive total number of bathrooms. This is more consistent with how # of bathrooms is shown on listings.\n* Total porch square footage - there are multiple individual measures for porch square footage (if those features are present). Adding them up yields a more comprehensive total square footage for porch space.\n* Does it have a pool? - Boolean true or false.\n* Does it have a 2nd floor? - Boolean true or false.\n* Does it have a garage? - Boolean true or false.\n* Does it have a basement? - Boolean true or false.\n* Does it have a fireplace? - Boolean true or false.","metadata":{"id":"nakGgpbiJMhw"}},{"cell_type":"code","source":"# Total square footage\nx['TotalLivableSF'] = x['1stFlrSF'] + x['2ndFlrSF'] + x['BsmtFinSF1'] + x['BsmtFinSF2']\n# Total # of bathrooms\nx['TotalBaths'] = x['BsmtFullBath'] + (x['BsmtHalfBath'] * .5) + x['FullBath'] + (x['HalfBath'] * .5)\n# Total porch square footage\nx[\"TotalPorchSF\"] = x.WoodDeckSF + x.OpenPorchSF + x.EnclosedPorch + x['3SsnPorch']\n# Does it have a pool (2906)\nx['HasPool'] = x['PoolArea'].apply(lambda x: 1 if x >0 else 0).astype('category')\n# Does it have a 2nd floor\nx['Has2ndFloor'] = x['2ndFlrSF'].apply(lambda x: 1 if x > 0 else 0).astype('category')\n# Does it have a garage\nx['HasGarage'] = x['GarageArea'].apply(lambda x: 1 if x > 0 else 0).astype('category')\n# Does it have a basement\nx['HasBasement'] = x['TotalBsmtSF'].apply(lambda x: 1 if x > 0 else 0).astype('category')\n# Does it have a fireplace?\nx['HasFireplace'] = x['Fireplaces'].apply(lambda x: 1 if x > 0 else 0).astype('category')","metadata":{"id":"wcFRbAzRJMhw","execution":{"iopub.status.busy":"2022-07-09T04:54:26.297878Z","iopub.execute_input":"2022-07-09T04:54:26.298232Z","iopub.status.idle":"2022-07-09T04:54:26.331134Z","shell.execute_reply.started":"2022-07-09T04:54:26.298202Z","shell.execute_reply":"2022-07-09T04:54:26.330145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing\n* Changing objects to numericals - a number of features are coded as objects, but may be better suited to be turned into numericals. \n* Changing numbericals to objects - inverse of the above.\n\n(**Note**: upon running the notebook, it became clear that some of the above changes either hand no impact or negative on the results. I have preserved them for future experimentation purposes, but have commented them out)\n\n* Transformations - I wrote up a script in order to automate which columns to transform, and what was the best transformer to use.\n* Dropping columns\n* Scaling\n* Encoding categoricals","metadata":{"id":"FKhi1tjYJMhw"}},{"cell_type":"code","source":"xcat = x.select_dtypes(include='object')","metadata":{"id":"d9t0Z5ZyJMhw","execution":{"iopub.status.busy":"2022-07-09T04:54:26.332505Z","iopub.execute_input":"2022-07-09T04:54:26.333053Z","iopub.status.idle":"2022-07-09T04:54:26.348036Z","shell.execute_reply.started":"2022-07-09T04:54:26.333012Z","shell.execute_reply":"2022-07-09T04:54:26.346998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Changing quality values to numbers\n* These are features with \"Ex, Gd, TA, FA, PO, NA\"\n  * ExterQual\n  * ExterCond\n  * BsmtQual\n  * BsmtCond\n  * HeatingQC\n  * KitchenQual\n  * FireplaceQu\n  * GarageQual\n  * GarageCond\n  * PoolQC","metadata":{"id":"u-5QiyVbJMhw"}},{"cell_type":"code","source":"# # Changing all of the columns with \"Ex, Gd, TA, FA, PO, NA\"\n# qual_dict = {\n#             \"Ex\": '6',\n#             \"Gd\": '5',\n#             \"TA\": '4',\n#             \"Fa\": '3',\n#             \"Po\": '2', \n#             \"Not present\": '1'\n#             }\n\n# x.replace({'ExterQual': qual_dict}, inplace=True)\n# x.replace({'ExterCond': qual_dict}, inplace=True)\n# x.replace({'BsmtQual': qual_dict}, inplace=True)\n# x.replace({'BsmtCond': qual_dict}, inplace=True)\n# x.replace({'HeatingQC': qual_dict}, inplace=True)\n# x.replace({'KitchenQual': qual_dict}, inplace=True)\n# x.replace({'FireplaceQu': qual_dict}, inplace=True)\n# x.replace({'GarageQual': qual_dict}, inplace=True)\n# x.replace({'GarageCond': qual_dict}, inplace=True)\n# x.replace({'PoolQC': qual_dict}, inplace=True)\n\n# x.ExterQual = x.ExterQual.astype('int')\n# x.ExterCond = x.ExterCond.astype('int')\n# x.BsmtQual = x.BsmtQual.astype('int')\n# x.BsmtCond = x.BsmtCond.astype('int')\n# x.HeatingQC = x.HeatingQC.astype('int')\n# x.KitchenQual = x.KitchenQual.astype('int')\n# x.FireplaceQu = x.FireplaceQu.astype('int')\n# x.GarageQual = x.GarageQual.astype('int')\n# x.GarageCond = x.GarageCond.astype('int')\n# x.PoolQC = x.PoolQC.astype('int')","metadata":{"id":"8FxoLKt3JMhw","execution":{"iopub.status.busy":"2022-07-09T04:54:26.349556Z","iopub.execute_input":"2022-07-09T04:54:26.350127Z","iopub.status.idle":"2022-07-09T04:54:26.355727Z","shell.execute_reply.started":"2022-07-09T04:54:26.350086Z","shell.execute_reply":"2022-07-09T04:54:26.354439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* These have \"GLQ, ALQ, BLQ, Rec, LwQ, Unf, NA\"\n  * BsmtFinType1\n  * BsmtFinType2","metadata":{"id":"Y5fGK7E1JMhw"}},{"cell_type":"code","source":"# # Changing the Basement Finishes\n# fin_dict = {\n#             \"GLQ\":'7',\n#             \"ALQ\":'6',\n#             \"BLQ\":'5',\n#             \"Rec\":'4',\n#             \"LwQ\":'3',\n#             \"Unf\":'2',\n#             \"Not present\":'1'\n#             }\n\n# x.replace({'BsmtFinType1': fin_dict}, inplace=True)\n# x.replace({'BsmtFinType2': fin_dict}, inplace=True)\n\n# x.BsmtFinType1 = x.BsmtFinType1.astype('int')\n# x.BsmtFinType2 = x.BsmtFinType2.astype('int')","metadata":{"id":"E7MDQZ7GJMhw","execution":{"iopub.status.busy":"2022-07-09T04:54:26.357684Z","iopub.execute_input":"2022-07-09T04:54:26.358305Z","iopub.status.idle":"2022-07-09T04:54:26.371260Z","shell.execute_reply.started":"2022-07-09T04:54:26.358248Z","shell.execute_reply":"2022-07-09T04:54:26.369917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Basement exposure","metadata":{"id":"JpgKA-twJMhw"}},{"cell_type":"code","source":"# # Change basement exposure\n# exp_dict = {\n#             \"Gd\":'5',\n#             \"Av\":'4',\n#             \"Mn\":'3',\n#             \"No\":'2',\n#             \"Not present\":'1'\n#             }\n\n# x.replace({'BsmtExposure': exp_dict}, inplace=True)\n\n# x.BsmtExposure = x.BsmtExposure.astype('int')","metadata":{"id":"wGswsLGtJMhw","execution":{"iopub.status.busy":"2022-07-09T04:54:26.372785Z","iopub.execute_input":"2022-07-09T04:54:26.373350Z","iopub.status.idle":"2022-07-09T04:54:26.381612Z","shell.execute_reply.started":"2022-07-09T04:54:26.373320Z","shell.execute_reply":"2022-07-09T04:54:26.380740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Functional Deduction","metadata":{"id":"FI9tc4MvJMhx"}},{"cell_type":"code","source":"# Functional category\n# func_dict = {\n#             \"Typ\":'8',\n#             \"Min1\":'7',\n#             \"Min2\":'6',\n#             \"Mod\":'5',\n#             \"Maj1\":'4',\n#             \"Maj2\":'3',\n#             \"Sev\":'2',\n#             \"Sal\":'1'\n#             }\n\n# x.replace({'Functional': func_dict}, inplace=True)\n# x.Functional = x.Functional.astype('int')","metadata":{"id":"qY6PKPLQJMhx","execution":{"iopub.status.busy":"2022-07-09T04:54:26.382688Z","iopub.execute_input":"2022-07-09T04:54:26.383210Z","iopub.status.idle":"2022-07-09T04:54:26.394338Z","shell.execute_reply.started":"2022-07-09T04:54:26.383177Z","shell.execute_reply":"2022-07-09T04:54:26.393129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CentralAir should be a Boolean","metadata":{"id":"1nEqd3ZJJMhx"}},{"cell_type":"code","source":"x['CentralAir'] = x['CentralAir'].apply(lambda x: 1 if x == 'Y' else 0)\nx['CentralAir'] = x['CentralAir'].astype(int)","metadata":{"id":"qWrvMRtLJMhx","execution":{"iopub.status.busy":"2022-07-09T04:54:26.395931Z","iopub.execute_input":"2022-07-09T04:54:26.396318Z","iopub.status.idle":"2022-07-09T04:54:26.415097Z","shell.execute_reply.started":"2022-07-09T04:54:26.396287Z","shell.execute_reply":"2022-07-09T04:54:26.414123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Changing numericals to objects\n* Change MSSubClass to categoricals. The numbers are codes, not ordered.\n* Change YrSold and MoSold. Those aren't ordered","metadata":{"id":"HZWfVHLPJMhx"}},{"cell_type":"code","source":"xnum = x.select_dtypes(include = 'number')\n\n# Changing MSSubClass to categoricals\nx['MSSubClass'] = x[\"MSSubClass\"].astype('object')\n\n# Changing YrSold to categoricals\nx['YrSold'] = x[\"YrSold\"].astype('object')\n\n# Changing MoSold to categoricals\nx['MoSold'] = x[\"MoSold\"].astype('object')\n\nxnum = x.select_dtypes(include = 'number') # updating this variable because it is used for a later script","metadata":{"id":"krgjOOA8JMhx","execution":{"iopub.status.busy":"2022-07-09T04:54:26.416613Z","iopub.execute_input":"2022-07-09T04:54:26.417193Z","iopub.status.idle":"2022-07-09T04:54:26.436195Z","shell.execute_reply.started":"2022-07-09T04:54:26.417158Z","shell.execute_reply":"2022-07-09T04:54:26.435258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transformations\n* I started off manually assessing which columns I wanted to transform. I started off by visually inspecting histograms, and then would look at skew numbers to have a more rigorous approach for determining which columns were worth transforming.\n* I then wrote a script where I would set a threshold for skewness, and would only transform those columns with the best transformers.","metadata":{"id":"bbYMALjbJMhx"}},{"cell_type":"code","source":"xnum_to_transform = [] # create a list for columns to transform\nskew_threshold = .5 # setting the skewness threshold to append to the list\n\n# iterate through the columns in xnum (numerical columns of x), if skew is over the threshold, add them to the list.\nfor (xcolname, xcol) in xnum.iteritems():\n  skew = xcol.skew()\n  if abs(skew) > skew_threshold:\n    xnum_to_transform.append(xcolname)","metadata":{"id":"dYLavxIRJMhx","execution":{"iopub.status.busy":"2022-07-09T04:54:26.437597Z","iopub.execute_input":"2022-07-09T04:54:26.438431Z","iopub.status.idle":"2022-07-09T04:54:26.454331Z","shell.execute_reply.started":"2022-07-09T04:54:26.438383Z","shell.execute_reply":"2022-07-09T04:54:26.453437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below, are all of the columns that require transformation","metadata":{"id":"Uj4IJBQwJMhx"}},{"cell_type":"code","source":"xnum_to_transform","metadata":{"id":"93N3w2HlJMhx","outputId":"128d7b1a-8558-48f0-f545-67a2ca47de67","execution":{"iopub.status.busy":"2022-07-09T04:54:26.455787Z","iopub.execute_input":"2022-07-09T04:54:26.456937Z","iopub.status.idle":"2022-07-09T04:54:26.464616Z","shell.execute_reply.started":"2022-07-09T04:54:26.456888Z","shell.execute_reply":"2022-07-09T04:54:26.463435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I wrote a script in order to automate the selection of which transformer to use. I looked for the transformer that yielded the lowest skew absolute value (since skew can be negative).\n\n**Note:** Since some of the columns can generate errors for the different transformers (for ex. some of the columns have negative values, and some of the transformers do not work with negative values), I created try / except blocks to catch those instances.","metadata":{"id":"te5ShJQOJMhx"}},{"cell_type":"code","source":"xnum_transformed_skews = pd.DataFrame()\n\nfor i in xnum_to_transform:  # for loop across the xnums_to_transform\n  dict = {} # create a dictionary\n  try: \n    log_skew = np.log(x[i]).skew() # log transformation\n  except: \n    log_skew = 999\n  try: \n    sqrt_skew = np.sqrt(x[i]).skew() # sqrt transformation\n  except: \n    srt_skew_skew = 999\n  try:\n    inverse_skew = (1/x[i]).skew() # inverse transformation\n  except: \n    inverse_skew = 999\n  try: \n    boxcox_skew = pd.DataFrame(list(stats.boxcox(x[i])[0])).skew()[0] # boxcox transformation\n  except: \n    boxcox_skew = 999\n  try:\n    yj_skew = pd.DataFrame(list(stats.yeojohnson(x[i])[0])).skew()[0]# yj skew\n  except: \n    yj_skew = 999\n  dict['log skew'] = abs(log_skew) # append log to dict\n  dict['sqrt skew'] = abs(sqrt_skew) # append sqrt to dict\n  dict['inverse skew'] = abs(inverse_skew) # append inverted to dict\n  dict['boxcox skew'] = abs(boxcox_skew) # append boxcox to dict\n  dict['yj skew'] = abs(yj_skew) # append yj to dict\n  best_skew_key = min(dict, key = dict.get)\n  if best_skew_key == 'log skew':\n    x[i] = np.log(x[i])\n  elif best_skew_key == 'sqrt skew':\n    x[i] = np.sqrt(x[i])\n  elif best_skew_key == 'inverse skew':\n    x[i] = (1/x[i])\n  elif best_skew_key == 'boxcox skew':\n    x[i] = list(stats.boxcox(x[i])[0])\n  else:\n    x[i] = list(stats.yeojohnson(x[i])[0])","metadata":{"id":"JebQXuF9JMhy","execution":{"iopub.status.busy":"2022-07-09T04:54:26.466107Z","iopub.execute_input":"2022-07-09T04:54:26.466963Z","iopub.status.idle":"2022-07-09T04:54:26.961290Z","shell.execute_reply.started":"2022-07-09T04:54:26.466894Z","shell.execute_reply":"2022-07-09T04:54:26.959804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Following the automated step, we see if there are still columns that are above the skew threshold. My original intent was to manually look for opportunities to transform them. However, to save time, I originally opted to drop those columns.","metadata":{"id":"M-oujpYFJMhy"}},{"cell_type":"code","source":"xnum = x.select_dtypes(include = 'number')\n\nxnum_to_transform_2 = []\nskew_threshold = 0.60\n\nfor (xcolname, xcol) in xnum.iteritems():\n  skew = xcol.skew()\n  if abs(skew) > skew_threshold:\n    xnum_to_transform_2.append(xcolname)\n\nlen(xnum_to_transform_2)","metadata":{"id":"f2bkLle6JMhy","outputId":"2c512192-d346-4b60-e713-595721ee2a9c","execution":{"iopub.status.busy":"2022-07-09T04:54:26.962830Z","iopub.execute_input":"2022-07-09T04:54:26.963201Z","iopub.status.idle":"2022-07-09T04:54:26.985632Z","shell.execute_reply.started":"2022-07-09T04:54:26.963167Z","shell.execute_reply":"2022-07-09T04:54:26.984433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this shows all of the remaining \"high skew\" columns\nfor i in xnum_to_transform_2:\n  skew = x[i].skew()\n  print(i, \"{:2f}\".format(skew))","metadata":{"id":"k_nT5LEzJMhy","outputId":"2a00a7eb-349c-4836-f6cd-cffe450ebad6","execution":{"iopub.status.busy":"2022-07-09T04:54:26.987307Z","iopub.execute_input":"2022-07-09T04:54:26.988046Z","iopub.status.idle":"2022-07-09T04:54:26.999116Z","shell.execute_reply.started":"2022-07-09T04:54:26.988001Z","shell.execute_reply":"2022-07-09T04:54:26.997288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x.shape # see what the dataset shape is before dropping","metadata":{"id":"hwlZttrEJMhy","outputId":"56b69ad9-671e-4dd2-cd14-18059b58447d","execution":{"iopub.status.busy":"2022-07-09T04:54:27.000939Z","iopub.execute_input":"2022-07-09T04:54:27.001298Z","iopub.status.idle":"2022-07-09T04:54:27.010434Z","shell.execute_reply.started":"2022-07-09T04:54:27.001265Z","shell.execute_reply":"2022-07-09T04:54:27.009078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Dropping all of the high skew columns\nx.drop(columns=xnum_to_transform_2, inplace=True)","metadata":{"id":"vdS_5mY0JMhy","execution":{"iopub.status.busy":"2022-07-09T04:54:27.012517Z","iopub.execute_input":"2022-07-09T04:54:27.012928Z","iopub.status.idle":"2022-07-09T04:54:27.023924Z","shell.execute_reply.started":"2022-07-09T04:54:27.012899Z","shell.execute_reply":"2022-07-09T04:54:27.022651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x.shape #seeing how many columns were dropped","metadata":{"id":"wVMQLHeSJMhy","outputId":"1d6c83ed-c1a9-4914-8427-14b91b92ada0","execution":{"iopub.status.busy":"2022-07-09T04:54:27.025341Z","iopub.execute_input":"2022-07-09T04:54:27.025957Z","iopub.status.idle":"2022-07-09T04:54:27.038341Z","shell.execute_reply.started":"2022-07-09T04:54:27.025919Z","shell.execute_reply":"2022-07-09T04:54:27.036916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Transforming the SalePrice\n* Because we separated out the SalePrice, we also need to transform the target feature. As noted earlier, SalePrice is highly skewed.\n* While we will see that Yeo-Johnson is the best transformation technique, I initially used a log transformer because that was the easiest to invert once the model predictions were run. PowerTransformer has special syntax and quirks that took some effort to learn through iteration.\n* Inverting Yeo-Johnson requires use of the PowerTransformer from sklearn, which took some time to learn. [Documentation here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html)\n* In retrospect, while it was helpful to learn how to use PowerTransformer in order to later invert the Yeo-Johnson transformation, it does not look like YJ was materially better than a log transformer when assessing the RMSEs.","metadata":{"id":"xro2-CJ2JMhy"}},{"cell_type":"code","source":"y.skew() # baseline the original skew, to be compared further down","metadata":{"id":"9Immp5OZJMhy","outputId":"05f464e4-9cb1-435a-b897-12eddbb2ffbb","execution":{"iopub.status.busy":"2022-07-09T04:54:27.041428Z","iopub.execute_input":"2022-07-09T04:54:27.042120Z","iopub.status.idle":"2022-07-09T04:54:27.054188Z","shell.execute_reply.started":"2022-07-09T04:54:27.042073Z","shell.execute_reply":"2022-07-09T04:54:27.053046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('y log skew', np.log(y).skew())\nprint('y sqrt skew', np.sqrt(y).skew())\nprint('y inverse skew', (1/y).skew())\nprint('y YJ skew', pd.DataFrame(list(stats.yeojohnson(y))[0]).skew()[0])","metadata":{"id":"7fc1adRNJMhy","outputId":"14d455ad-4ca3-43fd-e5f5-8c4dcb208c81","execution":{"iopub.status.busy":"2022-07-09T04:54:27.055951Z","iopub.execute_input":"2022-07-09T04:54:27.056613Z","iopub.status.idle":"2022-07-09T04:54:27.081214Z","shell.execute_reply.started":"2022-07-09T04:54:27.056567Z","shell.execute_reply":"2022-07-09T04:54:27.079814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pt = PowerTransformer(method = 'yeo-johnson')\npt.fit(y)","metadata":{"id":"khb0bkUgJMhy","outputId":"6f5b6d68-010f-448d-fd7a-f3afd437321f","execution":{"iopub.status.busy":"2022-07-09T04:54:27.089858Z","iopub.execute_input":"2022-07-09T04:54:27.090248Z","iopub.status.idle":"2022-07-09T04:54:27.108747Z","shell.execute_reply.started":"2022-07-09T04:54:27.090217Z","shell.execute_reply":"2022-07-09T04:54:27.107550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y.SalePrice = pt.transform(y)","metadata":{"id":"6aLeX9a0JMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.110212Z","iopub.execute_input":"2022-07-09T04:54:27.110555Z","iopub.status.idle":"2022-07-09T04:54:27.118117Z","shell.execute_reply.started":"2022-07-09T04:54:27.110525Z","shell.execute_reply":"2022-07-09T04:54:27.117196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y.skew() # should be much lower after the transformation","metadata":{"id":"AaFWQvz-JMhz","outputId":"98dd5005-5a59-4e51-dde8-ed82b30df8c8","execution":{"iopub.status.busy":"2022-07-09T04:54:27.119313Z","iopub.execute_input":"2022-07-09T04:54:27.119992Z","iopub.status.idle":"2022-07-09T04:54:27.133115Z","shell.execute_reply.started":"2022-07-09T04:54:27.119956Z","shell.execute_reply":"2022-07-09T04:54:27.132091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dropping columns\n* I only manually dropped the Id column","metadata":{"id":"rB2P5JuTJMhz"}},{"cell_type":"code","source":"x.drop(columns = 'Id', inplace=True)","metadata":{"id":"zPfwo2OnJMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.134776Z","iopub.execute_input":"2022-07-09T04:54:27.135150Z","iopub.status.idle":"2022-07-09T04:54:27.144982Z","shell.execute_reply.started":"2022-07-09T04:54:27.135119Z","shell.execute_reply":"2022-07-09T04:54:27.143873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Scaling\n* I wanted to identify which columns were numericals. I also needed to separate out the boolean columns from that.\n* Columns that needed to be scaled were then isolated.\n* I iteratively changed back and forth between Robust, Standard, and MinMax scalers. Over several iterations, Standard delivered the best RMSEs, and so has been what has been used most recently.","metadata":{"id":"MES1TT_OJMhz"}},{"cell_type":"code","source":"bool_list = [] # initiate a list\nfor (xcolname, xcol) in x.iteritems(): # iterate through columns that\n  bool_test = xcol.value_counts().index.tolist() # find the number of unique values per column\n  if 1 and 0 in bool_test and len(bool_test) == 2: # if there are only 2 unique values, add them to the list\n    bool_list.append(xcolname)\n\nprint(bool_list)# print the list","metadata":{"id":"xNRMhxuWJMhz","outputId":"188a6baf-ebdd-49d1-9db3-cd986f9074d5","execution":{"iopub.status.busy":"2022-07-09T04:54:27.146140Z","iopub.execute_input":"2022-07-09T04:54:27.146879Z","iopub.status.idle":"2022-07-09T04:54:27.215287Z","shell.execute_reply.started":"2022-07-09T04:54:27.146838Z","shell.execute_reply":"2022-07-09T04:54:27.214147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in bool_list:\n  x[i] = x[i].astype('bool')","metadata":{"id":"hoIvM_J9JMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.216625Z","iopub.execute_input":"2022-07-09T04:54:27.217710Z","iopub.status.idle":"2022-07-09T04:54:27.226756Z","shell.execute_reply.started":"2022-07-09T04:54:27.217673Z","shell.execute_reply":"2022-07-09T04:54:27.225670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_cols_to_scale = x.select_dtypes(exclude = ['object', 'bool']).columns.values.tolist()\nnum_cols_to_scale","metadata":{"id":"SGxjO0ZXJMhz","outputId":"ff65b9b8-c5e1-41c3-96c5-aed52d03d32e","execution":{"iopub.status.busy":"2022-07-09T04:54:27.227971Z","iopub.execute_input":"2022-07-09T04:54:27.228683Z","iopub.status.idle":"2022-07-09T04:54:27.245816Z","shell.execute_reply.started":"2022-07-09T04:54:27.228641Z","shell.execute_reply":"2022-07-09T04:54:27.244802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Scaling\nI set up Robust, Standard, and MinMax scalers and will have two of the three commented out at a time. Currently, Standard scaler works the best.\n\n### Robust Scaler","metadata":{"id":"wBTS47CsJMhz"}},{"cell_type":"code","source":"# robust_scaler = RobustScaler().fit(x[num_cols_to_scale])\n# x[num_cols_to_scale] = robust_scaler.transform(x[num_cols_to_scale])","metadata":{"id":"KyojV_YCJMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.247298Z","iopub.execute_input":"2022-07-09T04:54:27.248301Z","iopub.status.idle":"2022-07-09T04:54:27.253828Z","shell.execute_reply.started":"2022-07-09T04:54:27.248262Z","shell.execute_reply":"2022-07-09T04:54:27.252817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Standard Scaler","metadata":{"id":"mhjqw_hoJMhz"}},{"cell_type":"code","source":"standard_scaler = StandardScaler().fit(x[num_cols_to_scale])\nx[num_cols_to_scale] = standard_scaler.transform(x[num_cols_to_scale])","metadata":{"id":"KeWvR_53JMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.255257Z","iopub.execute_input":"2022-07-09T04:54:27.255881Z","iopub.status.idle":"2022-07-09T04:54:27.277688Z","shell.execute_reply.started":"2022-07-09T04:54:27.255846Z","shell.execute_reply":"2022-07-09T04:54:27.276326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### MinMax Scaler","metadata":{"id":"BXOyt-N5JMhz"}},{"cell_type":"code","source":"# minmax_scaler = MinMaxScaler().fit(x[num_cols_to_scale])\n# x[num_cols_to_scale] = minmax_scaler.transform(x[num_cols_to_scale])","metadata":{"id":"xrYUFjSnJMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.279435Z","iopub.execute_input":"2022-07-09T04:54:27.279886Z","iopub.status.idle":"2022-07-09T04:54:27.284799Z","shell.execute_reply.started":"2022-07-09T04:54:27.279847Z","shell.execute_reply":"2022-07-09T04:54:27.283467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Encoding","metadata":{"id":"yh63RS7DJMhz"}},{"cell_type":"code","source":"x = pd.get_dummies(x)","metadata":{"id":"SuX_EmraJMhz","execution":{"iopub.status.busy":"2022-07-09T04:54:27.285951Z","iopub.execute_input":"2022-07-09T04:54:27.287115Z","iopub.status.idle":"2022-07-09T04:54:27.358139Z","shell.execute_reply.started":"2022-07-09T04:54:27.287055Z","shell.execute_reply":"2022-07-09T04:54:27.357177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training and Predicting\n* I started off with using LinearRegression. However, I ran into some issues when untransforming the target variable. It appears that untransformation was creating some \"INF\" values. This was likely caused by some features over impacting the prediction.\n* Then I tried using Lasso and that seems to have done a very good job of dampening the impact of features through regularization.\n* I also tried using Ridge to test out L2 regularization. However, it does not seem to have delivered as good results for RMSE as Lasso.\n* Finally, as a bonus, I also attempted to learn how to use XGBoost Regressor. I mainly used this as a way to try to learn my way around the function and the parameters. I wanted to familiarize myself with \"the state of the art\". However, it still did not beat out Lasso.\n* I even attempted a blended XGB and Lasso result. Interestingly, while that has occasionally yielded lower RMSEs than Lasso, upon submission to Kaggle, it usually yields worst results in terms of ranking in the competition.\n* Regarding untransforming the target variable in the training and test data, we use PowerTransformer inversion to reverse the YJ transformation.","metadata":{"id":"veWHiSDLJMh0"}},{"cell_type":"markdown","source":"## Prepping the training data\nBecause we had stacked the training and test datasets earlier, we now need to unstack them.\n\nThen we split the training data into testing and training pairs for training the regression models. This does lead to some confusion between what is the \"test\" data set for submitting to Kaggle, and what is the \"test\" dataset used for evaluating the trained models. Bear with me.","metadata":{"id":"phgTtKi4JMh0"}},{"cell_type":"code","source":"total_copy = x\nnew_test = test.shape[0]\ntrain2 = total_copy[:train.shape[0]]\ntest2 = total_copy[train.shape[0]:test.shape[0] + (new_test+1)]\nxtrain2, xtest2, ytrain2, ytest2 = train_test_split(train2, y.SalePrice, test_size = .33, random_state = 7)\nprint('xtrain2', xtrain2.shape)\nprint('xtest2', xtest2.shape)\nprint('ytrain2', ytrain2.shape)\nprint('ytest2', ytest2.shape)","metadata":{"id":"VWycAmjcJMh0","outputId":"c75feb44-9015-46a8-f284-b32031752a66","execution":{"iopub.status.busy":"2022-07-09T04:54:27.359478Z","iopub.execute_input":"2022-07-09T04:54:27.359944Z","iopub.status.idle":"2022-07-09T04:54:27.378185Z","shell.execute_reply.started":"2022-07-09T04:54:27.359904Z","shell.execute_reply":"2022-07-09T04:54:27.377191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#untransform ytest data\nytest2 = pd.DataFrame(ytest2)\nytest2_untrans = pt.inverse_transform(ytest2)","metadata":{"id":"PFSZxb70JMh0","execution":{"iopub.status.busy":"2022-07-09T04:54:27.379545Z","iopub.execute_input":"2022-07-09T04:54:27.381009Z","iopub.status.idle":"2022-07-09T04:54:27.390520Z","shell.execute_reply.started":"2022-07-09T04:54:27.380958Z","shell.execute_reply":"2022-07-09T04:54:27.389545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lasso Regression","metadata":{"id":"4t1gCAocJMh0"}},{"cell_type":"markdown","source":"For Lasso, an alpha value has to be selected as a parameter. I initially iteratively searched for the value manually. However, using the LassoCV function, it's possible to use cross validation to automatically find an optimal alpha value.","metadata":{"id":"GLLby8HXJMh0"}},{"cell_type":"code","source":"#Lasso alpha optimization\nmodel = LassoCV(cv=5, random_state=0, max_iter=10000)\nmodel.fit(xtrain2, ytrain2)\nmodel.alpha_","metadata":{"id":"Mbn7nuKmJMh0","outputId":"4b36ecdc-ab45-4b62-fbb4-1acb20ddf280","execution":{"iopub.status.busy":"2022-07-09T04:54:27.391819Z","iopub.execute_input":"2022-07-09T04:54:27.392719Z","iopub.status.idle":"2022-07-09T04:54:28.702422Z","shell.execute_reply.started":"2022-07-09T04:54:27.392680Z","shell.execute_reply":"2022-07-09T04:54:28.701123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"las = Lasso(alpha = model.alpha_).fit(xtrain2, ytrain2)\nypred2_las = las.predict(xtest2)\nypred2_las = pd.DataFrame(ypred2_las, columns = ['SalePrice'])\nypred2_las_untrans = pt.inverse_transform(ypred2_las)","metadata":{"id":"BmXTElKCJMh0","execution":{"iopub.status.busy":"2022-07-09T04:54:28.708834Z","iopub.execute_input":"2022-07-09T04:54:28.713385Z","iopub.status.idle":"2022-07-09T04:54:28.952007Z","shell.execute_reply.started":"2022-07-09T04:54:28.713293Z","shell.execute_reply":"2022-07-09T04:54:28.950639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analyzing the Lasso results","metadata":{"id":"t3NyCbAuJMh0"}},{"cell_type":"code","source":"las_mse = mean_squared_error(ytest2_untrans, ypred2_las_untrans)\nlas_rmse = np.sqrt(las_mse)\nlas_mae = mean_absolute_error(ytest2_untrans, ypred2_las_untrans)\n\nplt.scatter(ypred2_las_untrans, ytest2_untrans)\nprint('Mean Squared Error is', '{:,.2f}'.format(las_mse))\nprint('Root Mean Squared Error is', '{:,.2f}'.format(las_rmse))\nprint('Mean Absolute Error is', '{:,.2f}'.format(las_mae))","metadata":{"id":"PaQu7bjtJMh0","outputId":"52796b84-d775-4082-ce98-a065545f2536","execution":{"iopub.status.busy":"2022-07-09T04:54:28.953963Z","iopub.execute_input":"2022-07-09T04:54:28.955396Z","iopub.status.idle":"2022-07-09T04:54:29.159340Z","shell.execute_reply.started":"2022-07-09T04:54:28.955328Z","shell.execute_reply":"2022-07-09T04:54:29.156406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting the submission results for Lasso","metadata":{"id":"jbuzaqW9JMh0"}},{"cell_type":"code","source":"test = pd.read_csv('../input/house-prices-advanced-regression-techniques/test.csv')","metadata":{"id":"1X6-c5Q6JMh0","execution":{"iopub.status.busy":"2022-07-09T04:54:29.160736Z","iopub.execute_input":"2022-07-09T04:54:29.161109Z","iopub.status.idle":"2022-07-09T04:54:29.187811Z","shell.execute_reply.started":"2022-07-09T04:54:29.161076Z","shell.execute_reply":"2022-07-09T04:54:29.186473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_pred_las = las.predict(test2)\nsubmit_pred_las = pd.DataFrame(submit_pred_las, columns = ['SalePrice'])\nsubmit_pred_untrans_las = pt.inverse_transform(submit_pred_las)","metadata":{"id":"nObpO0UYJMh0","execution":{"iopub.status.busy":"2022-07-09T04:54:29.189620Z","iopub.execute_input":"2022-07-09T04:54:29.190062Z","iopub.status.idle":"2022-07-09T04:54:29.298135Z","shell.execute_reply.started":"2022-07-09T04:54:29.190018Z","shell.execute_reply":"2022-07-09T04:54:29.296628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_to_submit_las = pd.DataFrame(test['Id'])\ntest_to_submit_las['SalePrice'] = submit_pred_untrans_las\ntest_to_submit_las.to_csv('./predictions(lasso).csv', index=False)","metadata":{"id":"GN-BPKMyJMh0","execution":{"iopub.status.busy":"2022-07-09T04:54:29.300251Z","iopub.execute_input":"2022-07-09T04:54:29.301575Z","iopub.status.idle":"2022-07-09T04:54:29.328371Z","shell.execute_reply.started":"2022-07-09T04:54:29.301508Z","shell.execute_reply":"2022-07-09T04:54:29.326612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ridge Regression","metadata":{"id":"yZtyiHvgJMh0"}},{"cell_type":"code","source":"ridge = Ridge(alpha = 10).fit(xtrain2, ytrain2)\nypred2_ridge = ridge.predict(xtest2)\nypred2_ridge = pd.DataFrame(ypred2_ridge, columns = ['SalePrice'])\nypred2_ridge_untrans = pt.inverse_transform(ypred2_ridge)","metadata":{"id":"Q6cvbsgtJMh0","execution":{"iopub.status.busy":"2022-07-09T04:54:29.336344Z","iopub.execute_input":"2022-07-09T04:54:29.337473Z","iopub.status.idle":"2022-07-09T04:54:29.555139Z","shell.execute_reply.started":"2022-07-09T04:54:29.337412Z","shell.execute_reply":"2022-07-09T04:54:29.553705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analyzing the results for ridge","metadata":{"id":"EVHSVHaMJMh1"}},{"cell_type":"code","source":"ridge_mse = mean_squared_error(ytest2_untrans, ypred2_ridge_untrans)\nridge_rmse = np.sqrt(ridge_mse)\nridge_mae = mean_absolute_error(ytest2_untrans, ypred2_ridge_untrans)\n\nplt.scatter(ypred2_ridge_untrans, ytest2_untrans)\nprint('Mean Squared Error is', '{:,.2f}'.format(ridge_mse))\nprint('Root Mean Squared Error is', '{:,.2f}'.format(ridge_rmse))\nprint('Mean Absolute Error is', '{:,.2f}'.format(ridge_mae))","metadata":{"id":"95tVTzr4JMh1","outputId":"f7e3e2fc-bce5-428c-c716-85d1b98d2535","execution":{"iopub.status.busy":"2022-07-09T04:54:29.557400Z","iopub.execute_input":"2022-07-09T04:54:29.558465Z","iopub.status.idle":"2022-07-09T04:54:29.772537Z","shell.execute_reply.started":"2022-07-09T04:54:29.558409Z","shell.execute_reply":"2022-07-09T04:54:29.771238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting the submission results for Ridge","metadata":{"id":"0HKh_kgaKBM5"}},{"cell_type":"code","source":"submit_pred_ridge = ridge.predict(test2)\nsubmit_pred_ridge = pd.DataFrame(submit_pred_ridge, columns = ['SalePrice'])\nsubmit_pred_ridge_untrans = pt.inverse_transform(submit_pred_ridge)","metadata":{"id":"1aAU0kHlJMh1","execution":{"iopub.status.busy":"2022-07-09T04:54:29.776152Z","iopub.execute_input":"2022-07-09T04:54:29.776533Z","iopub.status.idle":"2022-07-09T04:54:29.872994Z","shell.execute_reply.started":"2022-07-09T04:54:29.776493Z","shell.execute_reply":"2022-07-09T04:54:29.871635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_to_submit_ridge = pd.DataFrame(test['Id'])\ntest_to_submit_ridge['SalePrice'] = submit_pred_ridge_untrans\ntest_to_submit_ridge.to_csv('./predictions(ridge).csv', index=False)","metadata":{"id":"UMg530gaJMh1","execution":{"iopub.status.busy":"2022-07-09T04:54:29.875074Z","iopub.execute_input":"2022-07-09T04:54:29.875886Z","iopub.status.idle":"2022-07-09T04:54:29.895859Z","shell.execute_reply.started":"2022-07-09T04:54:29.875843Z","shell.execute_reply":"2022-07-09T04:54:29.894395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## XG Boost","metadata":{"id":"f1zbc_djJMh1"}},{"cell_type":"markdown","source":"* In reviewing other notebooks, I noticed that folks were getting good results from using XG Boost, and I wanted to learn a new function.","metadata":{"id":"fDeO8vUwJMh1"}},{"cell_type":"code","source":"xgb = XGBRegressor(n_estimators=10000, learning_rate=0.02, early_stopping_rounds=3)\nxgb.fit(xtrain2, ytrain2, verbose=False, eval_set=[(xtrain2, ytrain2), (xtest2, ytest2)])\nypred2_xgb = xgb.predict(xtest2)","metadata":{"id":"2TFkzKmyJMh1","outputId":"b4ab078f-b3c4-455d-ad62-29d8a8b0b466","execution":{"iopub.status.busy":"2022-07-09T04:54:29.897963Z","iopub.execute_input":"2022-07-09T04:54:29.898875Z","iopub.status.idle":"2022-07-09T04:54:37.887497Z","shell.execute_reply.started":"2022-07-09T04:54:29.898826Z","shell.execute_reply":"2022-07-09T04:54:37.886427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ypred2_xgb = pd.DataFrame(ypred2_xgb, columns=['SalePrice'])\nypred2_xgb_untrans = pt.inverse_transform(ypred2_xgb)","metadata":{"id":"yQ4g0hrWJMh1","execution":{"iopub.status.busy":"2022-07-09T04:54:37.888752Z","iopub.execute_input":"2022-07-09T04:54:37.889468Z","iopub.status.idle":"2022-07-09T04:54:37.902664Z","shell.execute_reply.started":"2022-07-09T04:54:37.889433Z","shell.execute_reply":"2022-07-09T04:54:37.900859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analyzing the results using XGB","metadata":{"id":"xj5VZxfgJMh1"}},{"cell_type":"code","source":"xgb_mse = mean_squared_error(ytest2_untrans, ypred2_xgb_untrans)\nxgb_rmse = np.sqrt(xgb_mse)\nxgb_mae = mean_absolute_error(ytest2_untrans, ypred2_xgb_untrans)\n\nplt.scatter(ypred2_xgb_untrans, ytest2_untrans)\nprint('Mean Squared Error is', '{:,.2f}'.format(xgb_mse))\nprint('Root Mean Squared Error is', '{:,.2f}'.format(xgb_rmse))\nprint('Mean Absolute Error is', '{:,.2f}'.format(xgb_mae))","metadata":{"id":"YOeGLhj-JMh1","outputId":"1265f31c-ce46-4c7f-873d-a74df6e69ca5","execution":{"iopub.status.busy":"2022-07-09T04:54:37.904157Z","iopub.execute_input":"2022-07-09T04:54:37.904659Z","iopub.status.idle":"2022-07-09T04:54:38.091863Z","shell.execute_reply.started":"2022-07-09T04:54:37.904625Z","shell.execute_reply":"2022-07-09T04:54:38.090597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting the submission results for XG Boost","metadata":{"id":"-ogaWAKiKLLG"}},{"cell_type":"code","source":"submit_pred_xgb = xgb.predict(test2)\nsubmit_pred_xgb = pd.DataFrame(submit_pred_xgb, columns = ['SalePrice'])\nsubmit_pred_untrans_xgb = pt.inverse_transform(submit_pred_xgb)","metadata":{"id":"xgRrF1YjJMh1","execution":{"iopub.status.busy":"2022-07-09T04:54:38.093324Z","iopub.execute_input":"2022-07-09T04:54:38.093721Z","iopub.status.idle":"2022-07-09T04:54:38.204252Z","shell.execute_reply.started":"2022-07-09T04:54:38.093692Z","shell.execute_reply":"2022-07-09T04:54:38.203180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_to_subamit_ridge = pd.DataFrame(test['Id'])\ntest_to_submit_ridge['SalePrice'] = submit_pred_ridge_untrans\ntest_to_submit_ridge.to_csv('./predictions(xgb).csv', index=False)","metadata":{"id":"K8iTq9TTJMh1","execution":{"iopub.status.busy":"2022-07-09T04:54:38.205545Z","iopub.execute_input":"2022-07-09T04:54:38.205858Z","iopub.status.idle":"2022-07-09T04:54:38.218927Z","shell.execute_reply.started":"2022-07-09T04:54:38.205829Z","shell.execute_reply":"2022-07-09T04:54:38.218096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Blend XG Boost and Lasso\n* I thought that by blending the two values, it would be possible to dampen some of the outliers between the two.","metadata":{"id":"3bXpLT8TJMh1"}},{"cell_type":"code","source":"blend_las_weight = .9\nypred2_blend = ((1 - blend_las_weight) * ypred2_xgb_untrans) + (blend_las_weight * ypred2_las_untrans)","metadata":{"id":"UssHOjslJMh1","execution":{"iopub.status.busy":"2022-07-09T04:54:38.220352Z","iopub.execute_input":"2022-07-09T04:54:38.220911Z","iopub.status.idle":"2022-07-09T04:54:38.225892Z","shell.execute_reply.started":"2022-07-09T04:54:38.220880Z","shell.execute_reply":"2022-07-09T04:54:38.224324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analyzing the results using XGB","metadata":{"id":"eKR0126ZJMh2"}},{"cell_type":"code","source":"blend_mse = mean_squared_error(ytest2_untrans, ypred2_blend)\nblend_rmse = np.sqrt(blend_mse)\nblend_mae = mean_absolute_error(ytest2_untrans, ypred2_blend)\n\nplt.scatter(ypred2_blend, ytest2_untrans)\nprint('Mean Squared Error is', '{:,.2f}'.format(blend_mse))\nprint('Root Mean Squared Error is', '{:,.2f}'.format(blend_rmse))\nprint('Mean Absolute Error is', '{:,.2f}'.format(blend_mae))","metadata":{"id":"m9V0teNKJMh2","outputId":"389c6109-441b-40d6-f256-8a6ba89df92a","execution":{"iopub.status.busy":"2022-07-09T04:54:38.227539Z","iopub.execute_input":"2022-07-09T04:54:38.228538Z","iopub.status.idle":"2022-07-09T04:54:38.412820Z","shell.execute_reply.started":"2022-07-09T04:54:38.228493Z","shell.execute_reply":"2022-07-09T04:54:38.411781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting and submitting the blended results","metadata":{"id":"fQUmCNNNJMh2"}},{"cell_type":"code","source":"submit_blend_las_xgb = (submit_pred_untrans_xgb * (1-blend_las_weight)) + (submit_pred_untrans_las * blend_las_weight)","metadata":{"id":"fAEXDoK-JMh2","execution":{"iopub.status.busy":"2022-07-09T04:54:38.414612Z","iopub.execute_input":"2022-07-09T04:54:38.414955Z","iopub.status.idle":"2022-07-09T04:54:38.419772Z","shell.execute_reply.started":"2022-07-09T04:54:38.414924Z","shell.execute_reply":"2022-07-09T04:54:38.418754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_to_submit_blend = pd.DataFrame(test['Id'])\ntest_to_submit_blend['SalePrice'] = submit_blend_las_xgb\ntest_to_submit_ridge.to_csv('./predictions(blend).csv', index=False)","metadata":{"id":"1BkifTfcJMh2","execution":{"iopub.status.busy":"2022-07-09T04:54:38.421601Z","iopub.execute_input":"2022-07-09T04:54:38.421916Z","iopub.status.idle":"2022-07-09T04:54:38.438314Z","shell.execute_reply.started":"2022-07-09T04:54:38.421887Z","shell.execute_reply":"2022-07-09T04:54:38.437425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Summarized RMSE results\n* Pulling just the RMSE values here in order to better visualize the results","metadata":{"id":"dq-gtF8SJMh2"}},{"cell_type":"code","source":"results = {'Method': ['Lasso', 'Ridge','XGB', 'Blend - Lasso/XGB'],\n           'RMSE':[las_rmse, ridge_rmse, xgb_rmse, blend_rmse]\n          }\n\nresults = pd.DataFrame(results)\nresults","metadata":{"id":"GUIV1MUqJMh2","outputId":"422f441e-64b9-4e69-e9b4-df77f2efdd37","execution":{"iopub.status.busy":"2022-07-09T04:54:38.439771Z","iopub.execute_input":"2022-07-09T04:54:38.440242Z","iopub.status.idle":"2022-07-09T04:54:38.461769Z","shell.execute_reply.started":"2022-07-09T04:54:38.440212Z","shell.execute_reply":"2022-07-09T04:54:38.460570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (10,5));\nplt.bar(results.Method, results.RMSE);\nplt.xlabel(\"Method\");\nplt.ylabel(\"RMSE\");\nplt.ylim(15000,25000);","metadata":{"id":"9qNe1FBVLHJv","outputId":"d82f5dd1-6e62-41c7-b73a-55b61af36e81","execution":{"iopub.status.busy":"2022-07-09T04:54:38.463531Z","iopub.execute_input":"2022-07-09T04:54:38.463883Z","iopub.status.idle":"2022-07-09T04:54:38.632204Z","shell.execute_reply.started":"2022-07-09T04:54:38.463854Z","shell.execute_reply":"2022-07-09T04:54:38.630163Z"},"trusted":true},"execution_count":null,"outputs":[]}]}