{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\nš® Predicting the Sales Price of Bulldozers š®
\n![](\"https://render.fineartamerica.com/images/images-profile-flow/400/images/artworkimages/mediumlarge/2/4-bulldozer-csa-images.jpg\")
\n
\n
\n\n","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.016138,"end_time":"2022-07-11T03:25:32.705004","exception":false,"start_time":"2022-07-11T03:25:32.688866","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"We're going to go through an example machine learning project with the goal of predicting the sale price of bulldozers. \n\n### 1) Problem Definition\n\n> How we can we predict the future sale price of a bulldozer, given its characteristics and previous examples of how much similar bulldozers have been sold for?\n\n### 2) Data\n\n* `train.csv` is the training set, which contains data through the end of 2011.\n* `test.csv` Contains data from May 1, 2012 - November 2012. Test set score determines your final rank for the competition.\n* `valid.csv` The validation set, which contains data from January 1, 2012 - April 30, 2012 You make predictions on this set throughout the majority of the competition. Your score on this set is used to create the public leaderboard.\n* [Kaggle Competition](https://www.kaggle.com/competitions/bluebook-for-bulldozers/data)\n\n### 3) Evaluation\n\nThe evaluation metric for this competition is the RMSLE (root mean squared log error) between the actual predicted auction prices.\n\nFor more information on the evaluation of this project check: [Bluebook for Bulldozers Evaluation](https://www.kaggle.com/competitions/bluebook-for-bulldozers/overview/evaluation).\n\n**Note:** The goal for most regression evaluation metrics is to minimize the error. For example, our goal for this project will be to build a machine learning model which minimizes RMSLE. \n\n### 4) Features\n\nKaggle provides a data dictionary detailing all of the features of the dataset. You can view this data [here](https://www.kaggle.com/competitions/bluebook-for-bulldozers/data).\n\n### 5) Modeling\n\n\n#### Libraries šā¬","metadata":{"papermill":{"duration":0.014101,"end_time":"2022-07-11T03:25:32.733845","exception":false,"start_time":"2022-07-11T03:25:32.719744","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Import modules\nimport numpy as np\nimport pandas as pd\nimport matplotlib .pyplot as plt\nimport sklearn\nimport seaborn as sns\nimport time\n\n# Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"papermill":{"duration":1.809873,"end_time":"2022-07-11T03:25:34.558234","exception":false,"start_time":"2022-07-11T03:25:32.748361","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-18T23:31:38.851331Z","iopub.execute_input":"2022-07-18T23:31:38.851946Z","iopub.status.idle":"2022-07-18T23:31:38.861196Z","shell.execute_reply.started":"2022-07-18T23:31:38.851895Z","shell.execute_reply":"2022-07-18T23:31:38.858895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import training and validation data\ndf = pd.read_csv(\"../input/bluebook-for-bulldozers/TrainAndValid.csv\")\ndf_test = pd.read_csv(\"../input/bluebook-for-bulldozers/Valid.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:31:40.839971Z","iopub.execute_input":"2022-07-18T23:31:40.840697Z","iopub.status.idle":"2022-07-18T23:31:43.813963Z","shell.execute_reply.started":"2022-07-18T23:31:40.840652Z","shell.execute_reply":"2022-07-18T23:31:43.812916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Evaluation\n\nThings to evalute for this data\n\n* `df.head()`\n* `df.info()`\n* `df.describe()`\n* `df.isnull().sum()`","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10,6))\nax.scatter(df['saledate'][:1000], df['SalePrice'][:1000]);","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:41:04.505543Z","iopub.execute_input":"2022-07-18T23:41:04.505988Z","iopub.status.idle":"2022-07-18T23:41:04.781943Z","shell.execute_reply.started":"2022-07-18T23:41:04.505951Z","shell.execute_reply":"2022-07-18T23:41:04.780592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.SalePrice.plot.hist(figsize=(10,6));","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:41:07.854148Z","iopub.execute_input":"2022-07-18T23:41:07.854658Z","iopub.status.idle":"2022-07-18T23:41:08.158152Z","shell.execute_reply.started":"2022-07-18T23:41:07.854611Z","shell.execute_reply":"2022-07-18T23:41:08.157297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### When we work with time series data, we want to enrich the time & date component as much as possible.\n\nWe can do that by telling pandas which of our columns has dates in it using `parse_dates` parameter.","metadata":{}},{"cell_type":"code","source":"# Import data again but this time parse dates\ndf = pd.read_csv(\"../input/bluebook-for-bulldozers/TrainAndValid.csv\",\n parse_dates=['saledate'])","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:41:26.813696Z","iopub.execute_input":"2022-07-18T23:41:26.814100Z","iopub.status.idle":"2022-07-18T23:41:29.828490Z","shell.execute_reply.started":"2022-07-18T23:41:26.814068Z","shell.execute_reply":"2022-07-18T23:41:29.827411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Evaluate the updated import\n\n* `df.saledate.dtype`\n* `df.saledate[:1000]`\n* `df.saledate.head(20)`","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:42:25.936099Z","iopub.execute_input":"2022-07-18T23:42:25.936839Z","iopub.status.idle":"2022-07-18T23:42:25.944009Z","shell.execute_reply.started":"2022-07-18T23:42:25.936801Z","shell.execute_reply":"2022-07-18T23:42:25.942407Z"}}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,7))\nax.scatter(df['saledate'][:1000], df[\"SalePrice\"][:1000]);","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:42:38.304215Z","iopub.execute_input":"2022-07-18T23:42:38.305046Z","iopub.status.idle":"2022-07-18T23:42:38.579698Z","shell.execute_reply.started":"2022-07-18T23:42:38.304997Z","shell.execute_reply":"2022-07-18T23:42:38.578364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sort DataFrame by saledate\n\nWhen working with time series data, it's a good idea to sort by date.","metadata":{}},{"cell_type":"code","source":"# Sort DataFrame in date order\ndf.sort_values(by=[\"saledate\"], inplace=True, ascending=True) # inplace=True is used so we don't need to do df = df.sort_values\n# This will sort our DataFrame by saledate, starting with the earliest date first\ndf.saledate.head(20)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:43:06.077387Z","iopub.execute_input":"2022-07-18T23:43:06.078183Z","iopub.status.idle":"2022-07-18T23:43:06.504108Z","shell.execute_reply.started":"2022-07-18T23:43:06.078144Z","shell.execute_reply":"2022-07-18T23:43:06.502984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Make a copy of the original DataFrame \n\nMake a copy of the original DataFrame so when we manipulate the copied version we've still got the original data","metadata":{}},{"cell_type":"code","source":"# Make a copy of the original DataFrame\ndf_tmp = df.copy()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:43:45.467260Z","iopub.execute_input":"2022-07-18T23:43:45.467660Z","iopub.status.idle":"2022-07-18T23:43:45.781335Z","shell.execute_reply.started":"2022-07-18T23:43:45.467626Z","shell.execute_reply":"2022-07-18T23:43:45.780156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Add datetime parameters for `saledate` column","metadata":{}},{"cell_type":"code","source":"df_tmp[\"saleYear\"] = df_tmp.saledate.dt.year\ndf_tmp[\"saleMonth\"] = df_tmp.saledate.dt.month\ndf_tmp[\"saleDay\"] = df_tmp.saledate.dt.day\ndf_tmp[\"saleDayOfWeek\"] = df_tmp.saledate.dt.dayofweek\ndf_tmp[\"saleDayOfYear\"] = df_tmp.saledate.dt.dayofyear","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:43:49.681327Z","iopub.execute_input":"2022-07-18T23:43:49.681715Z","iopub.status.idle":"2022-07-18T23:43:49.885309Z","shell.execute_reply.started":"2022-07-18T23:43:49.681680Z","shell.execute_reply":"2022-07-18T23:43:49.883911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now we've enriched our DataFrame with date time features, we can remove 'saledate'\ndf_tmp.drop('saledate', axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:43:53.381317Z","iopub.execute_input":"2022-07-18T23:43:53.381708Z","iopub.status.idle":"2022-07-18T23:43:53.624096Z","shell.execute_reply.started":"2022-07-18T23:43:53.381675Z","shell.execute_reply":"2022-07-18T23:43:53.622953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Modeling\n\nStarting model-driven EDA","metadata":{}},{"cell_type":"code","source":"df_tmp.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:44:32.809330Z","iopub.execute_input":"2022-07-18T23:44:32.809743Z","iopub.status.idle":"2022-07-18T23:44:32.838587Z","shell.execute_reply.started":"2022-07-18T23:44:32.809709Z","shell.execute_reply":"2022-07-18T23:44:32.837640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's build a machine learning model\nfrom sklearn.ensemble import RandomForestRegressor\n\nmodel = RandomForestRegressor(n_jobs=-1,\n random_state=42)\n\nmodel.fit(df_tmp.drop('SalePrice', axis=1), df_tmp[\"SalePrice\"])","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:44:37.067603Z","iopub.execute_input":"2022-07-18T23:44:37.067970Z","iopub.status.idle":"2022-07-18T23:44:38.252294Z","shell.execute_reply.started":"2022-07-18T23:44:37.067940Z","shell.execute_reply":"2022-07-18T23:44:38.250545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.api.types.is_string_dtype(df_tmp[\"UsageBand\"])","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:44:43.403699Z","iopub.execute_input":"2022-07-18T23:44:43.404137Z","iopub.status.idle":"2022-07-18T23:44:43.412335Z","shell.execute_reply.started":"2022-07-18T23:44:43.404086Z","shell.execute_reply":"2022-07-18T23:44:43.411209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Build a for loop that finds columns that contain strings\nfor label, content in df_tmp.items():\n if pd.api.types.is_string_dtype(content):\n print(label)\n\n# This shows all of the columns that are strings","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:44:46.682945Z","iopub.execute_input":"2022-07-18T23:44:46.683349Z","iopub.status.idle":"2022-07-18T23:44:46.693913Z","shell.execute_reply.started":"2022-07-18T23:44:46.683313Z","shell.execute_reply":"2022-07-18T23:44:46.692653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This will turn all of the string values into category values\nfor label, content in df_tmp.items():\n if pd.api.types.is_string_dtype(content):\n df_tmp[label] = content.astype('category').cat.as_ordered()\n \n# as_ordered puts things in alphabetical order","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:45:33.755423Z","iopub.execute_input":"2022-07-18T23:45:33.756112Z","iopub.status.idle":"2022-07-18T23:45:39.901644Z","shell.execute_reply.started":"2022-07-18T23:45:33.756068Z","shell.execute_reply":"2022-07-18T23:45:39.900564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We have changed all of the columns that were strings into categories\ndf_tmp.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:45:43.165796Z","iopub.execute_input":"2022-07-18T23:45:43.166608Z","iopub.status.idle":"2022-07-18T23:45:43.248033Z","shell.execute_reply.started":"2022-07-18T23:45:43.166571Z","shell.execute_reply":"2022-07-18T23:45:43.246731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thanks to pandas categories we now have a way to access all of our data in the form of numbers\n\nBut we still have a bunch of missing data...","metadata":{}},{"cell_type":"code","source":"# Check missing data\ndf_tmp.isnull().sum()/len(df_tmp) # divide by the length of the DataFrame so we see a % of missing data","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:01.790432Z","iopub.execute_input":"2022-07-18T23:46:01.791623Z","iopub.status.idle":"2022-07-18T23:46:01.843438Z","shell.execute_reply.started":"2022-07-18T23:46:01.791579Z","shell.execute_reply":"2022-07-18T23:46:01.841938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Save preprocessed data","metadata":{}},{"cell_type":"code","source":"# Export current tmp DataFrame\ndf_tmp.to_csv(\"train_tmp.csv\",\n index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:06.238681Z","iopub.execute_input":"2022-07-18T23:46:06.239324Z","iopub.status.idle":"2022-07-18T23:46:14.793250Z","shell.execute_reply.started":"2022-07-18T23:46:06.239285Z","shell.execute_reply":"2022-07-18T23:46:14.792030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import preprocessed data\ndf_tmp.to_csv = pd.read_csv(\"train_tmp.csv\",\n low_memory=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:19.212523Z","iopub.execute_input":"2022-07-18T23:46:19.212898Z","iopub.status.idle":"2022-07-18T23:46:23.212132Z","shell.execute_reply.started":"2022-07-18T23:46:19.212868Z","shell.execute_reply":"2022-07-18T23:46:23.210773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_tmp.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:27.431340Z","iopub.execute_input":"2022-07-18T23:46:27.431744Z","iopub.status.idle":"2022-07-18T23:46:27.480925Z","shell.execute_reply.started":"2022-07-18T23:46:27.431711Z","shell.execute_reply":"2022-07-18T23:46:27.479907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fill Missing Values\n\n### Fill numerical missing values first\n","metadata":{}},{"cell_type":"code","source":"for label, content in df_tmp.items():\n if pd.api.types.is_numeric_dtype(content): # IF the content in the columns is numberic ... print out the label\n print(label)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:30.724588Z","iopub.execute_input":"2022-07-18T23:46:30.724955Z","iopub.status.idle":"2022-07-18T23:46:30.731666Z","shell.execute_reply.started":"2022-07-18T23:46:30.724924Z","shell.execute_reply":"2022-07-18T23:46:30.730485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for which numeric columns have null values\nfor label, content in df_tmp.items():\n if pd.api.types.is_numeric_dtype(content):\n if pd.isnull(content).sum():\n print(label)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:34.950030Z","iopub.execute_input":"2022-07-18T23:46:34.950830Z","iopub.status.idle":"2022-07-18T23:46:34.969849Z","shell.execute_reply.started":"2022-07-18T23:46:34.950785Z","shell.execute_reply":"2022-07-18T23:46:34.968830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill the numberic rows with the median\nfor label, content in df_tmp.items():\n if pd.api.types.is_numeric_dtype(content):\n if pd.isnull(content).sum():\n # Add a binary column which tells us if the data was missing or not\n df_tmp[label+\"_is_missing\"] = pd.isnull(content)\n # Fill missing numberic values with median\n df_tmp[label] = content.fillna(content.median())","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:46:40.258958Z","iopub.execute_input":"2022-07-18T23:46:40.259591Z","iopub.status.idle":"2022-07-18T23:46:40.304171Z","shell.execute_reply.started":"2022-07-18T23:46:40.259541Z","shell.execute_reply":"2022-07-18T23:46:40.302824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check if there's any null numberic values\nfor label, content in df_tmp.items():\n if pd.api.types.is_numeric_dtype(content):\n if pd.isnull(content).sum():\n print(label)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:25.205688Z","iopub.execute_input":"2022-07-18T23:47:25.206145Z","iopub.status.idle":"2022-07-18T23:47:25.286982Z","shell.execute_reply.started":"2022-07-18T23:47:25.206094Z","shell.execute_reply":"2022-07-18T23:47:25.285845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check to see how many examples were missing\ndf_tmp.auctioneerID_is_missing.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:31.300739Z","iopub.execute_input":"2022-07-18T23:47:31.301158Z","iopub.status.idle":"2022-07-18T23:47:31.313619Z","shell.execute_reply.started":"2022-07-18T23:47:31.301108Z","shell.execute_reply":"2022-07-18T23:47:31.312431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Filling and turning categorical variables into numbers","metadata":{}},{"cell_type":"code","source":"# Check for columns which aren't numberic\nfor label, content in df_tmp.items():\n if not pd.api.types.is_numeric_dtype(content):\n print(label)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:08.277463Z","iopub.execute_input":"2022-07-18T23:47:08.278221Z","iopub.status.idle":"2022-07-18T23:47:08.286710Z","shell.execute_reply.started":"2022-07-18T23:47:08.278182Z","shell.execute_reply":"2022-07-18T23:47:08.285866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Turn categorical variables into numbers and fill missing\nfor label, content in df_tmp.items():\n if not pd.api.types.is_numeric_dtype(content):\n # Add binary column to indicate whether sample had missing value\n df_tmp[label+\"_is_missing\"] = pd.isnull(content)\n # Turn categories into numbers and add +1\n df_tmp[label] = pd.Categorical(content).codes +1","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:46.504380Z","iopub.execute_input":"2022-07-18T23:47:46.504834Z","iopub.status.idle":"2022-07-18T23:47:46.512437Z","shell.execute_reply.started":"2022-07-18T23:47:46.504789Z","shell.execute_reply":"2022-07-18T23:47:46.511223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.Categorical(df_tmp['state']).codes+1","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:50.392418Z","iopub.execute_input":"2022-07-18T23:47:50.393296Z","iopub.status.idle":"2022-07-18T23:47:50.408735Z","shell.execute_reply.started":"2022-07-18T23:47:50.393251Z","shell.execute_reply":"2022-07-18T23:47:50.406977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_tmp.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:51.855074Z","iopub.execute_input":"2022-07-18T23:47:51.855874Z","iopub.status.idle":"2022-07-18T23:47:51.869706Z","shell.execute_reply.started":"2022-07-18T23:47:51.855830Z","shell.execute_reply":"2022-07-18T23:47:51.868400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can see all of our categorical missing columns now\ndf_tmp.head().T","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:53.165037Z","iopub.execute_input":"2022-07-18T23:47:53.166188Z","iopub.status.idle":"2022-07-18T23:47:53.187614Z","shell.execute_reply.started":"2022-07-18T23:47:53.166144Z","shell.execute_reply":"2022-07-18T23:47:53.186649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_tmp.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:56.283312Z","iopub.execute_input":"2022-07-18T23:47:56.283949Z","iopub.status.idle":"2022-07-18T23:47:56.362506Z","shell.execute_reply.started":"2022-07-18T23:47:56.283913Z","shell.execute_reply":"2022-07-18T23:47:56.361302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that all of our data is numberic as well as our DataFrame has no missing values, we should be able to build a machine learning model.","metadata":{}},{"cell_type":"code","source":"df_tmp.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:47:58.346435Z","iopub.execute_input":"2022-07-18T23:47:58.347109Z","iopub.status.idle":"2022-07-18T23:47:58.379108Z","shell.execute_reply.started":"2022-07-18T23:47:58.347073Z","shell.execute_reply":"2022-07-18T23:47:58.377833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Instantiate model on all 412,000 rows\nmodel = RandomForestRegressor(n_jobs=-1,\n random_state=42)\n\n# Fit the model\nmodel.fit(df_tmp.drop('SalePrice', axis=1), df_tmp['SalePrice'])\n\n","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:48:15.547947Z","iopub.execute_input":"2022-07-18T23:48:15.548571Z","iopub.status.idle":"2022-07-18T23:53:30.435271Z","shell.execute_reply.started":"2022-07-18T23:48:15.548520Z","shell.execute_reply":"2022-07-18T23:53:30.433993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Score the model\nmodel.score(df_tmp.drop('SalePrice', axis=1), df_tmp['SalePrice'])","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:54:27.787456Z","iopub.execute_input":"2022-07-18T23:54:27.788237Z","iopub.status.idle":"2022-07-18T23:54:40.449262Z","shell.execute_reply.started":"2022-07-18T23:54:27.788200Z","shell.execute_reply":"2022-07-18T23:54:40.447902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Splitting data into train / validation sets","metadata":{}},{"cell_type":"code","source":"df_tmp.saleYear","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:55:44.491222Z","iopub.execute_input":"2022-07-18T23:55:44.491677Z","iopub.status.idle":"2022-07-18T23:55:44.500756Z","shell.execute_reply.started":"2022-07-18T23:55:44.491644Z","shell.execute_reply":"2022-07-18T23:55:44.499425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_tmp.saleYear.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:55:46.918959Z","iopub.execute_input":"2022-07-18T23:55:46.919424Z","iopub.status.idle":"2022-07-18T23:55:46.931487Z","shell.execute_reply.started":"2022-07-18T23:55:46.919388Z","shell.execute_reply":"2022-07-18T23:55:46.930073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split data into training and validation\ndf_val = df_tmp[df_tmp.saleYear == 2012]\ndf_train = df_tmp[df_tmp.saleYear != 2012]\n\nlen(df_val), len(df_train)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:55:50.054974Z","iopub.execute_input":"2022-07-18T23:55:50.055702Z","iopub.status.idle":"2022-07-18T23:55:50.155794Z","shell.execute_reply.started":"2022-07-18T23:55:50.055664Z","shell.execute_reply":"2022-07-18T23:55:50.153522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The validation set contains all data where the sale year is in 2012 (as per Kaggle competition rules)\n# The train set is everything that is not 2012 for the sale year","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:55:52.282256Z","iopub.execute_input":"2022-07-18T23:55:52.283425Z","iopub.status.idle":"2022-07-18T23:55:52.287610Z","shell.execute_reply.started":"2022-07-18T23:55:52.283383Z","shell.execute_reply":"2022-07-18T23:55:52.286590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the data into X & y\nX_train, y_train = df_train.drop('SalePrice', axis =1 ), df_train.SalePrice\nX_valid, y_valid = df_val.drop('SalePrice', axis=1), df_val.SalePrice\n\nX_train.shape, y_train.shape, X_valid.shape, y_valid.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:55:54.328630Z","iopub.execute_input":"2022-07-18T23:55:54.329676Z","iopub.status.idle":"2022-07-18T23:55:54.385511Z","shell.execute_reply.started":"2022-07-18T23:55:54.329618Z","shell.execute_reply":"2022-07-18T23:55:54.384170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Building an evaluation function","metadata":{}},{"cell_type":"code","source":"# Create a valuiation function (the competition uses RMSLE)\nfrom sklearn.metrics import mean_squared_log_error, mean_absolute_error, r2_score\n\ndef rmsle(y_test, y_preds):\n \"\"\"\n Calculates root mean squared log error between predictions and true labels.\n \"\"\"\n return np.sqrt(mean_squared_log_error(y_test, y_preds))\n\n# Create function to evaluate model on a few different levels\ndef show_scores(model):\n train_preds = model.predict(X_train)\n val_preds = model.predict(X_valid)\n scores = {\"Training MAE\": mean_absolute_error(y_train, train_preds),\n 'Valid MAE': mean_absolute_error(y_valid, val_preds),\n 'Training RMSLE': rmsle(y_train, train_preds),\n \"Valid RMSLE\": rmsle(y_valid, val_preds),\n \"Training R^2\": r2_score(y_train, train_preds),\n 'Valid R^2': r2_score(y_valid, val_preds)}\n return scores","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:56:07.471161Z","iopub.execute_input":"2022-07-18T23:56:07.472038Z","iopub.status.idle":"2022-07-18T23:56:07.481476Z","shell.execute_reply.started":"2022-07-18T23:56:07.471989Z","shell.execute_reply":"2022-07-18T23:56:07.480350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing our model on a subset (to tune the hyperparameters)\n","metadata":{}},{"cell_type":"code","source":"# Change max_samples value\nmodel = RandomForestRegressor(n_jobs=-1,\n random_state=42,\n max_samples=10000)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:56:10.440846Z","iopub.execute_input":"2022-07-18T23:56:10.441331Z","iopub.status.idle":"2022-07-18T23:56:10.447099Z","shell.execute_reply.started":"2022-07-18T23:56:10.441283Z","shell.execute_reply":"2022-07-18T23:56:10.445859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Cutting down on the max number of samples each estimator can see improves training time\nmodel.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:56:12.276157Z","iopub.execute_input":"2022-07-18T23:56:12.276566Z","iopub.status.idle":"2022-07-18T23:56:27.398744Z","shell.execute_reply.started":"2022-07-18T23:56:12.276533Z","shell.execute_reply":"2022-07-18T23:56:27.397490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_scores(model)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:56:29.472542Z","iopub.execute_input":"2022-07-18T23:56:29.472934Z","iopub.status.idle":"2022-07-18T23:56:38.471624Z","shell.execute_reply.started":"2022-07-18T23:56:29.472901Z","shell.execute_reply":"2022-07-18T23:56:38.470440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Hyperparameter tuning with RandomizedSearchCV","metadata":{}},{"cell_type":"code","source":"%%time\nfrom sklearn.model_selection import RandomizedSearchCV\n\n# Different RandomForestRegressor Hyperparameters\nrf_grid = {\"n_estimators\": np.arange(10, 100, 10),\n \"max_depth\": [None, 3, 5, 10],\n \"min_samples_split\": np.arange(2, 20, 2),\n \"min_samples_leaf\": np.arange(1, 20, 2),\n \"max_features\": [0.5, 1, 'sqrt', 'auto'],\n \"max_samples\": [10000]}\n\n# Instantiate RandomizedSearchCV Model\nrs_model = RandomizedSearchCV(RandomForestRegressor(n_jobs=-1,\n random_state=42),\n param_distributions=rf_grid,\n n_iter=10,\n cv=5,\n verbose=True)\n\n# Fit the RandomizedSearchCV Model\nrs_model.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-07-18T23:56:41.538935Z","iopub.execute_input":"2022-07-18T23:56:41.539949Z","iopub.status.idle":"2022-07-19T00:03:05.350535Z","shell.execute_reply.started":"2022-07-18T23:56:41.539905Z","shell.execute_reply":"2022-07-19T00:03:05.349248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Find the best model hyperparameters \nrs_model.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:04:00.229457Z","iopub.execute_input":"2022-07-19T00:04:00.229907Z","iopub.status.idle":"2022-07-19T00:04:00.237142Z","shell.execute_reply.started":"2022-07-19T00:04:00.229868Z","shell.execute_reply":"2022-07-19T00:04:00.236280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_scores(rs_model)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:04:02.137872Z","iopub.execute_input":"2022-07-19T00:04:02.138280Z","iopub.status.idle":"2022-07-19T00:04:08.009308Z","shell.execute_reply.started":"2022-07-19T00:04:02.138245Z","shell.execute_reply":"2022-07-19T00:04:08.008165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_scores(model)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:04:47.299164Z","iopub.execute_input":"2022-07-19T00:04:47.299590Z","iopub.status.idle":"2022-07-19T00:04:56.277283Z","shell.execute_reply.started":"2022-07-19T00:04:47.299555Z","shell.execute_reply":"2022-07-19T00:04:56.276100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Most ideal hyperparameters\nideal_model = RandomForestRegressor(n_estimators=40,\n min_samples_leaf=1,\n min_samples_split=14,\n max_features=0.5,\n n_jobs=-1,\n max_samples=None,\n random_state=42)\n\n# Fit the RandomizedSearchCV Model\nideal_model.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:05:23.471708Z","iopub.execute_input":"2022-07-19T00:05:23.472145Z","iopub.status.idle":"2022-07-19T00:06:22.793603Z","shell.execute_reply.started":"2022-07-19T00:05:23.472095Z","shell.execute_reply":"2022-07-19T00:06:22.792281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Scores for ideal_model trained on all of the data\nshow_scores(ideal_model)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:07:31.384960Z","iopub.execute_input":"2022-07-19T00:07:31.385406Z","iopub.status.idle":"2022-07-19T00:07:38.802760Z","shell.execute_reply.started":"2022-07-19T00:07:31.385369Z","shell.execute_reply":"2022-07-19T00:07:38.801234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Scores on rs_model only trainedon 10,000 examples\nshow_scores(rs_model)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:07:40.451611Z","iopub.execute_input":"2022-07-19T00:07:40.452312Z","iopub.status.idle":"2022-07-19T00:07:46.319591Z","shell.execute_reply.started":"2022-07-19T00:07:40.452262Z","shell.execute_reply":"2022-07-19T00:07:46.318342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = pd.read_csv('../input/bluebook-for-bulldozers/Test.csv',\n low_memory=False,\n parse_dates=[\"saledate\"])\n\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:24.516069Z","iopub.execute_input":"2022-07-19T00:10:24.517182Z","iopub.status.idle":"2022-07-19T00:10:24.715659Z","shell.execute_reply.started":"2022-07-19T00:10:24.517134Z","shell.execute_reply":"2022-07-19T00:10:24.714304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Preprocessing the data (getting the test dataset in the same format as our training dataset)","metadata":{}},{"cell_type":"code","source":"def preprocess_data(df):\n \"\"\"\n Performs transformations on df and returns transformed df.\n \"\"\"\n df[\"saleYear\"] = df.saledate.dt.year\n df[\"saleMonth\"] = df.saledate.dt.month\n df[\"saleDay\"] = df.saledate.dt.day\n df[\"saleDayOfWeek\"] = df.saledate.dt.dayofweek\n df[\"saleDayOfYear\"] = df.saledate.dt.dayofyear\n \n df.drop(\"saledate\", axis=1, inplace=True)\n \n # Fill the numeric rows with median\n for label, content in df.items():\n if pd.api.types.is_numeric_dtype(content):\n if pd.isnull(content).sum():\n # Add a binary column which tells us if the data was missing or not\n df[label+\"_is_missing\"] = pd.isnull(content)\n # Fill missing numeric values with median\n df[label] = content.fillna(content.median())\n \n # Filled categorical missing data and turn categories into numbers\n if not pd.api.types.is_numeric_dtype(content):\n df[label+\"_is_missing\"] = pd.isnull(content)\n # We add +1 to the category code because pandas encodes missing categories as -1\n df[label] = pd.Categorical(content).codes+1\n \n return df","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:42.756662Z","iopub.execute_input":"2022-07-19T00:10:42.757409Z","iopub.status.idle":"2022-07-19T00:10:42.767556Z","shell.execute_reply.started":"2022-07-19T00:10:42.757368Z","shell.execute_reply":"2022-07-19T00:10:42.766564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Process the test data \ndf_test = preprocess_data(df_test)\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:45.165261Z","iopub.execute_input":"2022-07-19T00:10:45.165651Z","iopub.status.idle":"2022-07-19T00:10:45.440146Z","shell.execute_reply.started":"2022-07-19T00:10:45.165620Z","shell.execute_reply":"2022-07-19T00:10:45.438588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We can find how the columns differ using sets\nset(X_train.columns) - set(df_test.columns)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:51.473034Z","iopub.execute_input":"2022-07-19T00:10:51.474322Z","iopub.status.idle":"2022-07-19T00:10:51.482859Z","shell.execute_reply.started":"2022-07-19T00:10:51.474267Z","shell.execute_reply":"2022-07-19T00:10:51.481702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test['auctioneerID_is_missing'] = False\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:53.301246Z","iopub.execute_input":"2022-07-19T00:10:53.302613Z","iopub.status.idle":"2022-07-19T00:10:53.332685Z","shell.execute_reply.started":"2022-07-19T00:10:53.302561Z","shell.execute_reply":"2022-07-19T00:10:53.331774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make predictions on the test data\ntest_preds = ideal_model.predict(df_test)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:55.100155Z","iopub.execute_input":"2022-07-19T00:10:55.101140Z","iopub.status.idle":"2022-07-19T00:10:55.365336Z","shell.execute_reply.started":"2022-07-19T00:10:55.101080Z","shell.execute_reply":"2022-07-19T00:10:55.364189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_preds","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:10:59.662783Z","iopub.execute_input":"2022-07-19T00:10:59.663223Z","iopub.status.idle":"2022-07-19T00:10:59.670774Z","shell.execute_reply.started":"2022-07-19T00:10:59.663186Z","shell.execute_reply":"2022-07-19T00:10:59.669480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Format predictions into the same format Kaggle is after\ndf_preds = pd.DataFrame()\ndf_preds['SalesID'] = df_test['SalesID']\ndf_preds['SalesPrice'] = test_preds\ndf_preds","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:11:01.409894Z","iopub.execute_input":"2022-07-19T00:11:01.410636Z","iopub.status.idle":"2022-07-19T00:11:01.429204Z","shell.execute_reply.started":"2022-07-19T00:11:01.410582Z","shell.execute_reply":"2022-07-19T00:11:01.427747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_preds.to_csv('./predictions.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:25:04.731329Z","iopub.execute_input":"2022-07-19T00:25:04.731812Z","iopub.status.idle":"2022-07-19T00:25:04.790548Z","shell.execute_reply.started":"2022-07-19T00:25:04.731763Z","shell.execute_reply":"2022-07-19T00:25:04.789268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Find feature importance of best model\nideal_model.feature_importances_","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:25:06.835438Z","iopub.execute_input":"2022-07-19T00:25:06.835875Z","iopub.status.idle":"2022-07-19T00:25:07.051944Z","shell.execute_reply.started":"2022-07-19T00:25:06.835839Z","shell.execute_reply":"2022-07-19T00:25:07.050595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Helper function for plotting feature importance\ndef plot_features(columns, importances, n=20):\n df = (pd.DataFrame({\"features\": columns,\n \"feature_importances\": importances})\n .sort_values(\"feature_importances\", ascending=False)\n .reset_index(drop=True))\n \n # Plot the dataframe\n fig, ax = plt.subplots()\n ax.barh(df[\"features\"][:n], df[\"feature_importances\"][:20])\n ax.set_ylabel(\"Features\")\n ax.set_xlabel(\"Feature importance\")\n ax.invert_yaxis()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:25:09.372673Z","iopub.execute_input":"2022-07-19T00:25:09.373077Z","iopub.status.idle":"2022-07-19T00:25:09.382524Z","shell.execute_reply.started":"2022-07-19T00:25:09.373046Z","shell.execute_reply":"2022-07-19T00:25:09.380824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_features(X_train.columns, ideal_model.feature_importances_)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T00:25:49.788866Z","iopub.execute_input":"2022-07-19T00:25:49.789346Z","iopub.status.idle":"2022-07-19T00:25:50.209033Z","shell.execute_reply.started":"2022-07-19T00:25:49.789296Z","shell.execute_reply":"2022-07-19T00:25:50.207821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}