{"cells":[{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"# Set-Single Output with LGB\n\n

Kernel description:

\nIn this kernel it will be investigated the Single Ouptut strategy.
\nThere are a total of 112 outputs in this problem, we can face the problem as `one multioutput` or a `set of single outputs` problem. I already made a kernel with multioutput using a simple RandomForest with only part of train dataset, check this out: \n\nIn this kernel I will user a LGB with single output, then I'll group together all outputs as a matrix.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents:\n\n**1. [Problem Definition](#id1)**
\n**2. [Get the Data (Collect / Obtain)](#id2)**
\n**3. [Load the Dataset](#id3)**
\n**4. [Data Pre-processing](#id4)**
\n**5. [Model](#id5)**
\n**6. [Visualization and Analysis of Results](#id6)**
\n**7. [Submittion](#id7)**
\n**8. [References](#ref)**
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **1. Problem Definition:** \n\nThe objective of this competition is to predict the unavailability of cars in a car rental agency.
\nWhen a car is unavailable, its status can be either “in maintenance” or “being washed”. The goal is to predict the number of cars entering and leaving each of the two status, for each of the four shifts in a day","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **2. About the Data (Collect / Obtain):** \n\n The training set, built using real data from 2017 and 2018, contains both input and output columns.
\n In the test set, containing data from 2019, only the input columns are provided.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **3. Load the Dataset** \n\n\n","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport itertools\nimport gc\n\n\nimport lightgbm as lgb\nimport xgboost as xgb\n\nimport time\n\nfrom sklearn.model_selection import KFold\nfrom sklearn import metrics\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### All Functions Used in this Kernel","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n start_mem = df.memory_usage().sum() / 1024**2 \n for col in df.columns:\n col_type = df[col].dtypes\n if col_type in numerics:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64) \n end_mem = df.memory_usage().sum() / 1024**2\n if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n return df\n\n\ndef rmse(predictions, targets):\n return np.sqrt(((predictions - targets) ** 2).mean())\n\n#from https://www.kaggle.com/marlesson/simple-pytorch-model don't know if is right\nweigths = [1]*16 + [0.75]*16 + [0.6]*16 + [0.5]*16 + [0.43]*16 + [0.38]*16 + [0.33]*16\ndef wrmse(predictions, targets, weigths):\n return np.sqrt((((predictions - targets) ** 2)*weigths).mean())\n\n\ndef train_model_regression(X, y, X_test, params, folds, model_type='lgb', eval_metric='rmse', columns=None, plot_feature_importance=False, model=None,\n verbose=10000, early_stopping_rounds=200, n_estimators=50000):\n \"\"\"\n A function to train a variety of regression models.\n Returns dictionary with test predictions, scores and, if necessary, feature importances.\n \n :params: X - training data, can be pd.DataFrame or np.ndarray (after normalizing)\n \n :params: y - target\n :params: folds - folds to split data\n :params: model_type - type of model to use\n :params: eval_metric - metric to use\n :params: columns - columns to use. If None - use all columns\n :params: plot_feature_importance - whether to plot feature importance of LGB\n :params: model - sklearn model, works only for \"sklearn\" model type\n \n \"\"\"\n columns = X.columns if columns is None else columns\n X_test = X_test[columns]\n \n # averaged predictions on train data\n prediction = np.zeros((len(X_test)))\n \n # to set up scoring parameters\n metrics_dict = {'mae': {'lgb_metric_name': 'mae',\n 'catboost_metric_name': 'MAE',\n 'sklearn_scoring_function': metrics.mean_absolute_error},\n 'rmse': {'lgb_metric_name': rmse,\n 'scoring_function': metrics.mean_squared_error\n },\n \n }\n\n \n result_dict = {}\n \n \n \n # list of scores on folds\n scores = []\n model = None\n feature_importance = pd.DataFrame()\n \n # split and train on folds\n for fold_n, (train_index, valid_index) in enumerate(folds.split(X)):\n print(f'Fold {fold_n + 1} started at {time.ctime()}')\n if type(X) == np.ndarray:\n X_train, X_valid = X[columns][train_index], X[columns][valid_index]\n y_train, y_valid = y[train_index], y[valid_index]\n else:\n X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n \n if model_type == 'lgb':\n model = lgb.LGBMRegressor(**params, n_estimators = n_estimators, n_jobs = -1)\n model.fit(X_train, y_train, eval_metric=metrics_dict[eval_metric]['lgb_metric_name'],\n verbose=verbose)\n \n y_pred_valid = model.predict(X_valid)\n y_pred = model.predict(X_test, num_iteration=model.best_iteration_)\n \n scores.append(metrics_dict[eval_metric]['scoring_function'](y_valid, y_pred_valid))\n\n prediction += y_pred \n \n if model_type == 'lgb' and plot_feature_importance:\n # feature importance\n fold_importance = pd.DataFrame()\n fold_importance[\"feature\"] = columns\n fold_importance[\"importance\"] = model.feature_importances_\n fold_importance[\"fold\"] = fold_n + 1\n feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n\n prediction /= folds.n_splits\n \n print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))\n\n result_dict['prediction'] = prediction\n result_dict['scores'] = scores\n result_dict['model'] = model\n \n if model_type == 'lgb':\n if plot_feature_importance:\n feature_importance[\"importance\"] /= folds.n_splits\n cols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n by=\"importance\", ascending=False)[:50].index\n\n best_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\n #plt.figure(figsize=(16, 12));\n #sns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\n #plt.title('LGB Features (avg over folds)');\n \n result_dict['feature_importance'] = feature_importance\n \n return result_dict\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Session Parameters\n\nThis parameters controls the size of train/test. Only for test speed up\n* `columns_size`: number or None(use all columns)\n* `rows_size_frac` : select percentagem of dataset (number between 0 and 1)","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"columns_size = None\nrows_size_frac = 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"path='/kaggle/input/kddbr-2020/'\n\ndef get_files(years=[],months=12,pre=''):\n files = []\n \n for year in years:\n for month in range(1,months + 1):\n file = F'0{month}' if month < 10 else str(month)\n files.append( F'{path}/{pre}{year}{file}.csv' )\n return files\n\nbase = [pd.read_csv(file,parse_dates=['date']) for file in get_files( ['2017','2018'], 12) ]\nbase = pd.concat(base, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"if (columns_size != None or rows_size_frac != None):\n inputs = list( base.columns[ base.columns.str.contains('input')][:columns_size])\n inputs.sort()\n output = list(base.columns[ base.columns.str.contains('output')] )\n cols = list( base[ inputs + output ].columns )\n cols.append('id')\n cols.append('date')\n base = base[cols].copy().sample(frac=rows_size_frac)\n \n\nbase.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print(base.shape)\nbase.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\n\n#base = reduce_mem_usage(base)\n#test = reduce_mem_usage(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check Categorical Variables\n\nDescription: The input variables are 557 features
\nThey are labeled from 0 up to 556, with the first four being categorical variables, and the rest being numerical.
\nChecking Categorical Variables","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"#Check if date was parsed\nbase[['date']].info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#check first 4 feature\nfor i in range(4):\n col = 'input_'+str(i)\n print(col,' : ', base[col].shape[0], base[col].nunique() )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"input_columns = base.columns[base.columns.str.contains('input') ]\noutput_columns = base.columns[base.columns.str.contains('output') ]\n\nprint(F'Inputs: {input_columns.shape} Outputs: {output_columns.shape}')\ninput_columns, output_columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Let's dive into the date\n\n### First, let's check if input is correlated with month. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"For the numerical variables, all the previous fourteen (14) values are provided (one for each day in two weeks before the tuple date). The input column names follow the syntax below:\n\n","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"len(base.columns[ base.columns.str.contains('input_4_') ])\nfor i in range(2000):\n \n if F'input_4_{i}' in input_columns:\n print(F'input_1_{i}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.figure(figsize=(36, 22))\nplt.subplots_adjust(top=1.2, hspace = 0.8)\nsns.set_palette(\"husl\")\npalette = itertools.cycle(sns.color_palette())\nfor i in range(1,15):\n plt.subplot( 5, 3 ,i)\n col = F'input_4_{i}'\n plot = base.groupby( base['date'].dt.date )[col].mean().reset_index()\n sns.lineplot( plot.date, plot[col] , color=next(palette))\n plt.xticks(rotation=45,ha='right')\n plt.title(F\" Mean of {col} distributed per month \")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, next to vacation months (January, June, July, August, December), features like `input_4` raises.
\n\nMaybe later we can take an advantage of this information. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Different Features\n\nNow let's check for different features (input_4_1 to input_20_1).","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.figure(figsize=(36, 22))\nplt.subplots_adjust(top=1.2, hspace = 0.8)\nsns.set_palette(\"husl\")\npalette = itertools.cycle(sns.color_palette())\nfor i in range(4,20):\n plt.subplot( 5, 4 ,i - 3)\n col = F'input_{i}_1'\n plot = base.groupby( base['date'].dt.date )[col].mean().reset_index()\n sns.lineplot( plot.date, plot[col] , color=next(palette))\n plt.xticks(rotation=45,ha='right')\n plt.title(F\" Mean of {col} distributed per month \")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nbase_ = base[ base.columns[base.columns.str.contains('output')] ] \n\nsns.boxplot(x=\"variable\", y=\"value\", data=pd.melt(base_))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Remove somes Outputs.\n\nThere're a lot of outputs. Make we can find a way to remove some of then in order low the complexity of the model.","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"base_ = base[ base.columns[base.columns.str.contains('output')] ] \ncolumn_mean = {}\nfor i in base_.columns:\n #print(F'mean: {base_[i].mean()}')\n column_mean[i] = ( base_[i].mean(), base_[i].std() )\n\ncolumn_mean = sorted(column_mean.items(), key=lambda kv: kv[1])\ncolumn_mean","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"corr = base_.corr()\ncorr","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"\n \nmask = np.triu(np.ones_like(corr, dtype=np.bool))\n\nf, ax = plt.subplots(figsize=(20, 20))\ncmap = sns.diverging_palette(220, 10, as_cmap=True)\n\nsns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,\n square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see there the correlation between outputs variables is not strong.
\nLet's check this numericaly. This information will help us to remove some of then to improve our performance.
\nLet-s find columns with correlation is grater than 0.9. If we find lot's of columns with this caracteristics maybe we can delete those.","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"for col in list( corr.columns ):\n for i in range(corr.shape[0]) :\n if(corr[ col].iloc[i] > 0.9 and corr[ col].iloc[i] < 1):\n print( col, corr[col].iloc[i] )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" As we can see, there's fews columns like this. The impact in performance should not be considerable. So we decide to keep those columns.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# **4. Data Pre-processing** ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_features(df):\n df['input_month'] = df.date.dt.month\n df['input_year'] = df.date.dt.year\n df['input_day'] = df.date.dt.day\n df['input_quarter'] = df.date.dt.quarter\n df['input_dayofweek'] = df.date.dt.dayofweek\n df['input_dt_sin_quarter'] = np.sin(2*np.pi*df.date.dt.quarter/4)\n df['input_dt_sin_day_of_week'] = np.sin(2*np.pi*df.date.dt.dayofweek/6)\n df['input_dt_sin_day_of_year'] = np.sin(2*np.pi*df.date.dt.dayofyear/365)\n df['input_dt_sin_day'] = np.sin(2*np.pi*df.date.dt.day/30)\n df['input_dt_sin_month'] = np.sin(2*np.pi*df.date.dt.month/12)\n \n return df\n\n\ncreate_features(base)\n\ninput_columns = base.columns[base.columns.str.contains('input') ]\noutput_columns = base.columns[base.columns.str.contains('output') ]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# **5. Model** ","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"X = base[input_columns[:columns_size]].fillna(0)\nY = base[output_columns].fillna(0)\n\ndel base\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#Loading test file\ntest = [pd.read_csv(file,parse_dates=['date']) for file in get_files(['2019'], 12,'public' )]\ntest = pd.concat(test, ignore_index=True).fillna(0)\ntest = create_features(test)\ntest","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"best_features = pd.read_csv('/kaggle/input/kdd-bestfeaturesv1/best_features_v2.csv',index_col=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"best_features","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"%%time\nn_fold = 5\nfolds = KFold(n_splits=n_fold, shuffle=True, random_state=12)\n\nparams = {'num_leaves': 128,\n 'min_child_samples': 79,\n 'objective': 'regression',\n 'max_depth': 9,\n 'learning_rate': 0.01,\n \"boosting_type\": \"gbdt\",\n \"subsample_freq\": 1,\n \"subsample\": 0.9,\n \"bagging_seed\": 11,\n 'reg_alpha': 0.1,\n 'reg_lambda': 0.3,\n 'colsample_bytree': 1.0\n }\n\noutput_file = 'output_111.csv'\n\ntotal_features = 500\npred = {}\ntotal_score = 0\nfor n,feature in enumerate(output_columns[111]):\n print(F'{n}:Traning model for {feature}')\n best_features_cols = best_features[feature].to_list()[:total_features] \n result_dict_lgb = train_model_regression(X=X[ best_features_cols ] , y=Y[ feature ], X_test = test[ best_features_cols ] , params=params, \n folds=folds, model_type='lgb', eval_metric='rmse', \n plot_feature_importance=True, verbose=10, early_stopping_rounds=200, n_estimators=400)\n pred[feature] = result_dict_lgb['prediction']\n total_score += np.mean(result_dict_lgb['scores'])/len(output_columns)\n \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"print('rmse: %f' % total_score)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Submission History

\n\n| | Version | RMSE | LB | Desciption | Run Time |\n|-|-|-|-|-|-| \n| 0 | 3 | 0.368135 | 0.69252 | 500 Features; 100 Estimators; 100% Base 2018 | -x- |\n| 1 | 4 | 0.380798 (2017) | 0.69119 | 500 Features; 100 Estimators; Blended 2017 e 2018; | -x- |\n| 2 | 9 | 0.367238 | 0.69272 | 700 Features; 100 Estimators; 2018; | 7h |\n| 3 | 10 | 0.345896 | 0.67794 | 500 Features; 200 Estimators; 100% Base 2018; | 3h 30min |\n| 4 | 11 | 0.351631 | ? | 500 Featues; 300 Estimators; 100% Base 2017; | 3h 44min |\n| 5 | 12 | 0.351392 | 0.67161 | 500 Features; 300 Estimators; 100% Base 2017 e 2018; | 7h 44min |","execution_count":null},{"metadata":{"trusted":false},"cell_type":"code","source":"pred_sub = pd.DataFrame(pred)\npred_sub['id'] = test['id']\npred_sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_sub.to_csv(output_file,index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"'''submission = []\nfor i, row in pred_sub.iterrows():\n for column, value in zip(output_columns, row.values):\n id = \"{}_{}\".format(int(row.id), column)\n submission.append([id, value])\n\nsubmission = pd.DataFrame(submission)\nsubmission.columns = ['id', 'value']\nsubmission'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"#submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}