{"cells":[{"metadata":{},"cell_type":"markdown","source":"## 1. Preface :\n1. In this Notebook , I Have tried to find Best Hyperparameters for Light Gbm Model using Bayesian Optimization \n2. I have Taken only 2 million Data Points ! I will try to check out what is best size of Data Points \n"},{"metadata":{},"cell_type":"markdown","source":"## 2. Acknowledgements \n1. To Understand Bayesian Optimization I followed This [Blog](https://medium.com/vantageai/bringing-back-the-time-spent-on-hyperparameter-tuning-with-bayesian-optimisation-2e21a3198afb) , it is very detailed and Explain , Everything Needed for Bayesian Optimization\n\n2. For Code on Bayesian Optimization and Hyperparameters Tuning I refer This  [Kernel](https://www.kaggle.com/somang1418/tuning-hyperparameters-under-10-minutes-lgbm) by Somang So Han ! \n\n3. For Basic Feature Engineering I refered this [Notebook](https://www.kaggle.com/jsylas/riiid-lgbm-starter) by Sylas"},{"metadata":{},"cell_type":"markdown","source":"#### Please Do Upvote This Notebook If You Find it Useful It will motivate me to produce More Public Work for this and Upcoming Competitions in Future !"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport warnings\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nfrom bayes_opt import BayesianOptimization\nfrom skopt  import BayesSearchCV \n\n#graph, plots\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n#building models\nimport lightgbm as lgb\nimport xgboost as xgb\nfrom sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score\nimport time\nimport sys\nfrom sklearn.metrics import roc_auc_score, roc_curve\nimport shap\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np \nimport matplotlib.pyplot as plt\nimport dask.dataframe as dd\nfrom sklearn.metrics import roc_auc_score\nimport riiideducation\nenv = riiideducation.make_env()\ntrain = pd.read_csv('../input/riiid-test-answer-prediction/train.csv' , usecols = [1,2,3,4,7,8,9] , \n                   dtype ={'timestamp':'int64' , 'user_id':'int32' , 'content_id':'int16' ,'content_type_id':'int8',\n                          'answered_correctly':'int8','prior_question_elapsed_time': 'float32','prior_question_had_explanation': 'boolean'} ,nrows = 2000000)\n\nprint(\"Shape of train set: \",train.shape)\ntrain = train[train.content_type_id == False] # only take question data\n\ntrain.sort_values(['timestamp'] , ascending = True)\n\ntrain.drop(columns = ['timestamp' , 'content_type_id'] , axis = 1 ,inplace = True) # drop useless columns\n\nresults_c = train[['content_id','answered_correctly']].groupby('content_id').agg('mean') # content_id based mean\n\nresults_c.columns = ['answered_correctly_content']\n\n\nresults_u = train[['user_id','answered_correctly']].groupby('user_id').agg(['mean' , 'sum'])\nresults_u.columns = [\"answered_correctly_user\", 'sum']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Questions.csv \n\nquestions_df = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv' , usecols = [0,1,3,4] , dtype = {'question_id': 'int16',\n                              'part': 'int8','bundle_id': 'int8','tags': 'str'\n})\ntags = questions_df['tags'].str.split(\" \" , n = 10 , expand = True )\n# Simply Converts Tags to 5 Columns for Each Questions\n# Each Entry in Tag Denotes Value of Tag","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tags.columns = ['tags1','tags2','tags3','tags4','tags5','tags6']\nquestions_df =  pd.concat([questions_df,tags],axis=1)\nquestions_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Simply Converting Each Tag to Numeric \n# Coerce is use to set invalid tag values to Nan\nquestions_df['tags1'] = pd.to_numeric(questions_df['tags1'], errors='coerce')\nquestions_df['tags2'] = pd.to_numeric(questions_df['tags2'], errors='coerce')\nquestions_df['tags3'] = pd.to_numeric(questions_df['tags3'], errors='coerce')\nquestions_df['tags4'] = pd.to_numeric(questions_df['tags4'], errors='coerce')\nquestions_df['tags5'] = pd.to_numeric(questions_df['tags5'], errors='coerce')\nquestions_df['tags6'] = pd.to_numeric(questions_df['tags6'], errors='coerce')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train and Validation\nX = train.iloc[: , : ]\nX['prior_question_had_explanation'].fillna(False , inplace = True)\nX  = pd.merge(X , results_c , on = ['content_id'] , how = 'left')\nX  = pd.merge(X , results_u , on = ['user_id'] , how = 'left')\nX = pd.merge(X , questions_df , left_on = 'content_id' , right_on ='question_id' , how = 'left')\nX=X[X.answered_correctly!= -1 ] # only take non Null action Data Points\nX  = X.sort_values(['user_id'])\nY = X[['answered_correctly']]\nX = X.drop([\"answered_correctly\"], axis=1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfrom  sklearn.model_selection import train_test_split\n\nlb_make = LabelEncoder()\nX[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(X[\"prior_question_had_explanation\"])\n\n\nX = X[['answered_correctly_user', 'answered_correctly_content', 'sum','bundle_id','part','prior_question_elapsed_time','prior_question_had_explanation_enc','tags1','tags2','tags3','tags4','tags5','tags6']] \nX.fillna(0.5 , inplace = True)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndef bayes_parameter_opt_lgb(X, y, init_round=15, opt_round=25, n_folds=3, random_seed=6,n_estimators=10000, output_process=False):\n    # prepare data\n    train_data = lgb.Dataset(data=X, label=Y, free_raw_data=False)\n    # parameters\n    def lgb_eval(learning_rate,num_leaves, feature_fraction, bagging_fraction, max_depth, max_bin, min_data_in_leaf,min_sum_hessian_in_leaf,subsample):\n        params = {'application':'binary', 'metric':'auc'}\n        params['learning_rate'] = max(min(learning_rate, 1), 0)\n        params[\"num_leaves\"] = int(round(num_leaves))\n        params['feature_fraction'] = max(min(feature_fraction, 1), 0)\n        params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)\n        params['max_depth'] = int(round(max_depth))\n        params['max_bin'] = int(round(max_depth))\n        params['min_data_in_leaf'] = int(round(min_data_in_leaf))\n        params['min_sum_hessian_in_leaf'] = min_sum_hessian_in_leaf\n        params['subsample'] = max(min(subsample, 1), 0)\n        \n        cv_result = lgb.cv(params, train_data, nfold=n_folds, seed=random_seed, stratified=True, verbose_eval =200, metrics=['auc'])\n        return max(cv_result['auc-mean'])\n    lgbBO = BayesianOptimization(lgb_eval, {'learning_rate': (0.01, 1.0),\n                                            'num_leaves': (24, 80),\n                                            'feature_fraction': (0.1, 0.9),\n                                            'bagging_fraction': (0.8, 1),\n                                            'max_depth': (5, 30),\n                                            'max_bin':(20,90),\n                                            'min_data_in_leaf': (20, 80),\n                                            'min_sum_hessian_in_leaf':(0,100),\n                                           'subsample': (0.01, 1.0)}, random_state=200)\n\n    \n    #n_iter: How many steps of bayesian optimization you want to perform. The more steps the more likely to find a good maximum you are.\n    #init_points: How many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space.\n    \n    lgbBO.maximize(init_points=init_round, n_iter=opt_round)\n    \n    model_auc=[]\n    for model in range(len( lgbBO.res)):\n        model_auc.append(lgbBO.res[model]['target'])\n    \n    # return best parameters\n    return lgbBO.res[pd.Series(model_auc).idxmax()]['target'],lgbBO.res[pd.Series(model_auc).idxmax()]['params']\n\nopt_params = bayes_parameter_opt_lgb(X, Y, init_round=5, opt_round=10, n_folds=3, random_seed=6,n_estimators=10000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"opt_params[1][\"num_leaves\"] = int(round(opt_params[1][\"num_leaves\"]))\nopt_params[1]['max_depth'] = int(round(opt_params[1]['max_depth']))\nopt_params[1]['min_data_in_leaf'] = int(round(opt_params[1]['min_data_in_leaf']))\nopt_params[1]['max_bin'] = int(round(opt_params[1]['max_bin']))\nopt_params[1]['objective']='binary'\nopt_params[1]['metric']='auc'\nopt_params[1]['is_unbalance']=True\nopt_params[1]['boost_from_average']=False\nopt_params=opt_params[1]\nopt_params","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n\ntarget= Y['answered_correctly']\nfeatures= [c for c in X.columns]\n\n\nfolds = StratifiedKFold(n_splits=10, shuffle=True, random_state=31416)\noof = np.zeros(len(X))\n#predictions = np.zeros(len(test))\nfeature_importance_df = pd.DataFrame()\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(X.values, target.values)):\n    print(\"Fold {}\".format(fold_))\n    trn_data = lgb.Dataset(X.iloc[trn_idx][features], label=target.iloc[trn_idx])\n    val_data = lgb.Dataset(X.iloc[val_idx][features], label=target.iloc[val_idx])\n\n    num_round = 15000\n    clf = lgb.train(opt_params, trn_data, num_round, valid_sets = [trn_data, val_data], verbose_eval=500, early_stopping_rounds = 250)\n    oof[val_idx] = clf.predict(X.iloc[val_idx][features], num_iteration=clf.best_iteration)\n    \nprint(\"CV score: {:<8.5f}\".format(roc_auc_score(target, oof)))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\nfilename = 'lgbm_2m_bayesian_hyperopt_tuned.sav'\npickle.dump(clf, open(filename, 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making Final Predictions on Dtest\niter_test = env.iter_test()\nbatch = 0 \nfor (test_df , sample_prediction_df) in iter_test:\n    test_df = test_df.sort_values(['user_id','timestamp'], ascending=False)\n    test_df['answer_time'] = test_df.groupby(['user_id'])['prior_question_elapsed_time'].shift(1)\n    # Features for Test Set\n    test_df = pd.merge(test_df , results_u , on = ['user_id'] , how = 'left')\n    test_df = pd.merge(test_df , results_c , on = ['content_id'] , how = 'left')\n    test_df = pd.merge(test_df, questions_df, left_on = 'content_id', right_on = 'question_id', how = 'left')  \n    # Imputations and Transformations\n    test_df['answered_correctly_user'].fillna(0.5, inplace=True)\n    test_df['answered_correctly_content'].fillna(0.5, inplace=True)\n    test_df['sum'].fillna(0, inplace=True)\n    test_df['prior_question_had_explanation'].fillna(False, inplace=True)\n    test_df[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(test_df[\"prior_question_had_explanation\"])\n    # Dtest Predictions\n    test_df['answered_correctly'] =  clf.predict(test_df[['answered_correctly_user', 'answered_correctly_content', 'sum','bundle_id','part','prior_question_elapsed_time','prior_question_had_explanation_enc',\n                                                           'tags1','tags2','tags3','tags4','tags5','tags6']])\n    \n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n    print(\"Batch Done : \", batch)\n    batch += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Please Do Checkout Some of My Other Works in this Competition !\n1. EDA and Problem Understanding Kernel [Here](https://www.kaggle.com/sayedathar11/riiid-eda-let-s-explore) \n2. Simple User Id Mean Correctness  Baseline [Here](https://www.kaggle.com/sayedathar11/riiid-mean-user-accuracy-baseline) "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}