{"cells":[{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"# KDD First EDA\n\n

Kernel description:

\nThis my first try\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents:\n\n**1. [Problem Definition](#id1)**
\n**2. [Get the Data (Collect / Obtain)](#id2)**
\n**3. [Load the Dataset](#id3)**
\n**4. [Data Pre-processing](#id4)**
\n**5. [Model](#id5)**
\n**6. [Visualization and Analysis of Results](#id6)**
\n**7. [Submittion](#id7)**
\n**8. [References](#ref)**
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **1. Problem Definition:** \n\nThe objective of this competition is to predict the unavailability of cars in a car rental agency.
\nWhen a car is unavailable, its status can be either “in maintenance” or “being washed”. The goal is to predict the number of cars entering and leaving each of the two status, for each of the four shifts in a day","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **2. About the Data (Collect / Obtain):** \n\n The training set, built using real data from 2017 and 2018, contains both input and output columns.
\n In the test set, containing data from 2019, only the input columns are provided.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **3. Load the Dataset** \n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport itertools\nimport gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### All Functions Used in this Kernel","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df\n\n\ndef rmse(predictions, targets):\n return np.sqrt(((predictions - targets) ** 2).mean())\n\n#from https://www.kaggle.com/marlesson/simple-pytorch-model don't know if is right\nweigths = [1]*16 + [0.75]*16 + [0.6]*16 + [0.5]*16 + [0.43]*16 + [0.38]*16 + [0.33]*16\ndef wrmse(predictions, targets, weigths):\n return np.sqrt((((predictions - targets) ** 2)*weigths).mean())\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Session Parameters\n\nThis parameters controls the size of train/test. Only for test speed up\n* `columns_size`: number or None(use all columns)\n* `rows_size` : numer or None(user all rows)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"columns_size = None\nrows_size = 1000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path='/kaggle/input/kddbr-2020/'\n\ndef get_files(years=[],months=12,pre=''):\n files = []\n \n for year in years:\n for month in range(1,months + 1):\n file = F'0{month}' if month < 10 else str(month)\n files.append( F'{path}/{pre}{year}{file}.csv' )\n return files\n\nbase = [pd.read_csv(file,parse_dates=['date']) for file in get_files( ['2018'], 12) ]\nbase = pd.concat(base, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if (columns_size != None or rows_size != None):\n inputs = list( base.columns[ base.columns.str.contains('input')][:columns_size])\n inputs.sort()\n output = list(base.columns[ base.columns.str.contains('output')] )\n cols = list( base[ inputs + output ].columns )\n cols.append('id')\n cols.append('date')\n base = base[cols].copy()[:rows_size]\n\nbase.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(base.shape)\nbase.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base = reduce_mem_usage(base)\n#test = reduce_mem_usage(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check Categorical Variables\n\nDescription: The input variables are 557 features
\nThey are labeled from 0 up to 556, with the first four being categorical variables, and the rest being numerical.
\nChecking Categorical Variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Check if date was parsed\nbase[['date']].info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#check first 4 feature\nfor i in range(4):\n col = 'input_'+str(i)\n print(col,' : ', base[col].shape[0], base[col].nunique() )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_columns = base.columns[base.columns.str.contains('input') ]\noutput_columns = base.columns[base.columns.str.contains('output') ]\n\nprint(F'Inputs: {input_columns.shape} Outputs: {output_columns.shape}')\ninput_columns, output_columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Let's dive into the date\n\n### First, let's check if input is correlated with month. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"For the numerical variables, all the previous fourteen (14) values are provided (one for each day in two weeks before the tuple date). The input column names follow the syntax below:\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(base.columns[ base.columns.str.contains('input_4_') ])\nfor i in range(2000):\n \n if F'input_4_{i}' in input_columns:\n print(F'input_1_{i}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(36, 22))\nplt.subplots_adjust(top=1.2, hspace = 0.8)\nsns.set_palette(\"husl\")\npalette = itertools.cycle(sns.color_palette())\nfor i in range(1,15):\n plt.subplot( 5, 3 ,i)\n col = F'input_4_{i}'\n plot = base.groupby( base['date'].dt.date )[col].mean().reset_index()\n sns.lineplot( plot.date, plot[col] , color=next(palette))\n plt.xticks(rotation=45,ha='right')\n plt.title(F\" Mean of {col} distributed per month \")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, next to vacation months (January, June, July, August, December), features like `input_4` raises.
\n\nMaybe later we can take an advantage of this information. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Different Features\n\nNow let's check for different features (input_4_1 to input_20_1).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(36, 22))\nplt.subplots_adjust(top=1.2, hspace = 0.8)\nsns.set_palette(\"husl\")\npalette = itertools.cycle(sns.color_palette())\nfor i in range(4,20):\n plt.subplot( 5, 4 ,i - 3)\n col = F'input_{i}_1'\n plot = base.groupby( base['date'].dt.date )[col].mean().reset_index()\n sns.lineplot( plot.date, plot[col] , color=next(palette))\n plt.xticks(rotation=45,ha='right')\n plt.title(F\" Mean of {col} distributed per month \")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# **4. Data Pre-processing** ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_features(df):\n df['input_month'] = df.date.dt.month\n df['input_year'] = df.date.dt.year\n df['input_day'] = df.date.dt.day\n df['input_dt_sin_quarter'] = np.sin(2*np.pi*df.date.dt.quarter/4)\n df['input_dt_sin_day_of_week'] = np.sin(2*np.pi*df.date.dt.dayofweek/6)\n df['input_dt_sin_day_of_year'] = np.sin(2*np.pi*df.date.dt.dayofyear/365)\n df['input_dt_sin_day'] = np.sin(2*np.pi*df.date.dt.day/30)\n df['input_dt_sin_month'] = np.sin(2*np.pi*df.date.dt.month/12)\n \n return df\n\n\ncreate_features(base)\n\ninput_columns = base.columns[base.columns.str.contains('input') ]\noutput_columns = base.columns[base.columns.str.contains('output') ]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# **5. Model** ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X = base[input_columns[:columns_size] ].fillna(0).values\nY = base[output_columns].fillna(0).values\n\ndel base\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nx_train, x_val, y_train, y_val = train_test_split(X, Y, test_size=0.20, random_state=42)\nx_train.shape, y_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Random Forest","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfrom sklearn.ensemble import RandomForestRegressor\nmodel = RandomForestRegressor(n_estimators = 1000, max_depth=10, random_state=0)\n\n# fit model\nmodel.fit(x_train,y_train)\n#LB = 0.79503","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = model.predict(x_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print( rmse(y_pred, y_val) )\nprint( wrmse(y_pred, y_val, weigths) )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# **6. Visualization and Analysis of Results** \n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"input_columns[:-1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# **7. Submittion** ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Loading test file\ntest = [pd.read_csv(file,parse_dates=['date']) for file in get_files(['2019'], 12,'public' )]\ntest = pd.concat(test, ignore_index=True).fillna(0)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = create_features(test)\ninput_columns = test.columns[test.columns.str.contains('input') ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = model.predict(test[input_columns].values)\npred_sub = pd.DataFrame(pred)\npred_sub.columns = output_columns\npred_sub['id'] = test['id']\npred_sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = []\nfor i, row in pred_sub.iterrows():\n for column, value in zip(output_columns, row.values):\n id = \"{}_{}\".format(int(row.id), column)\n submission.append([id, value])\n\nsubmission = pd.DataFrame(submission)\nsubmission.columns = ['id', 'value']\nsubmission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# **8. References** \n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> **Nota**: This Kernel is used as a WorkFlow template.
\nMade by[@AdrianoAvelar](https://www.kaggle.com/adrianoavelar)
\nFork and Code.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}