{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":" # Introduction to Light GBM\n## A Women in Kaggle Philly Workshop\n\n\nIn this notebook, we will go through a kaggle competition together. This tutorial assumes that you have some basic knowlege of Kaggle and its competitions, statistical models, and Python. By the end of this notebook, you should be able to submit a prediction using Light GBM to the Talkdata competition. \n\nPlease fork this notebook so that you can edit the codes to do the exercises.\n\n## What we will do today:\n1. Brief Introduction \n2. Loading large dataset\n3. Feature Engineering\n4. Modeling & Evaluation\n5. Feature Importance\n6. Create Submission File\n\n## Target today:\nSubmit your prediction file to the kaggle competition.\n\n## References\nThis notebook is written for a workshop organized by the Women in Kaggle Philly Meetup group.\n\nThis notebook is built upon ideas from: \n\nhttps://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm\n\nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask"},{"metadata":{"_cell_guid":"04489bfe-6c90-46f9-9176-9e28f9663a8b","_uuid":"5420ea3f4c62ff6c1b50630f2df8f4a823674f75"},"cell_type":"markdown","source":"## 1. Brief Introduction\n### 1.1 About the competition\n\nFor this competition, your objective is to predict whether a user will download an app after clicking a mobile app advertisement. The training dataset is 1.21 GB, incuding records of 180 milion clicks with ip, app, device, os, channel, click_time as main features, and is_attributed (whether the app is downloaded or not) as the target variable to be predicted. Please refer to the [competition overview](https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection) for details.\n\nThis is a very big dataset and kaggle kernel cannot handle training with the full data. If you have a super computer, you can download the full dataset and train locally. However, it is still possible to participate the competition with a laptop. That is why we are going to learn about Light GBM today.\n\n### 1.2 A very brief introduction of Light GBM\n\nTo put it simple, Light GBM, or the Light Gradient Boosting Model, grows trees vertically while the other tree_based algorithems (like XGB) grow trees horizontally. Light GBM is designed for handling very large datasets with faster speed and lower memory (hence \"light\"). I do not recommend using light GBM on small datasets (e.g. fewer than 10,000 rows) because it tends to overfit. \n\nFor more information on Light GBM, I recommend reading the following posts:\n\nhttps://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc\n\nhttps://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/\n\nhttps://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db\n\n"},{"metadata":{"_cell_guid":"0dc511d5-b0fc-4b0e-8695-fe4415f008d8","_uuid":"6dee8b010792767b170525feeccab30d9d8277c4"},"cell_type":"markdown","source":"## 2. Loading Large Dataset\n\nFirst we are going to load all libraries needed for this notebook. There are a few simple tricks to consider for loading very large datasets. "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-output":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"# Load libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing\nfrom pandas import Series, DataFrame # to deal with time data\nimport gc # To collect RAM garbage\nimport time # To get current time, used to calculate model training time\nfrom sklearn.model_selection import train_test_split # To split training and validation datasets\nimport matplotlib.pyplot as plt # For plotting feature importance\nimport lightgbm as lgb # Light gbm model\nimport warnings\nwarnings.filterwarnings('ignore') # Toignore warnings","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"223b23c8-d708-4e59-828d-a196b7362083","_uuid":"b90536c5239b7ff48881d724c7975ef7f3bc66e1"},"cell_type":"markdown","source":"### Trick 1: Set a debug mode\n\nThis is a really really large dataset. When we write our codes, we need to debug from time to time, but it would be extremely slow if we test every line of code with the whole dataset. In this case, we will set a \"debugging mode\" under which we only import a small part of the dataset for faster performance. Please make sure to set debug=1 throughout the workshop. "},{"metadata":{"_cell_guid":"49fbd021-fb7c-401b-a696-d97c33d71107","_kg_hide-output":true,"_uuid":"3427e70d0671608e753c4a04d344f6738c8a6775","collapsed":true,"trusted":true},"cell_type":"code","source":"# Set debug mode. When debug=1, we'll only be importing a few lines.\n# When debug=0, we'll import a much larger dataset to do serious training.\n# You'll see how to set this up in a later code block. \n# Make sure to set debug=1 throughout the workshop! \n\ndebug=1","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e8b55e5c-fbe8-4882-8321-3b1a350dde58","_uuid":"de28e73d08973221a3227d2ace2fb70e3596466e"},"cell_type":"markdown","source":"### Trick 2: Define data types before importing\n\nIf pandas does not know the data type for each feature, it will need to assign more RAM to handle them. Therefore, assinging data types beforehand will help save a lot of computational power. "},{"metadata":{"_cell_guid":"9f2ed241-84e2-4254-9812-3ce81598c09f","_kg_hide-output":true,"_uuid":"0046d92ea5f5ea1b39b5ad830e66fcad8d2be41f","collapsed":true,"trusted":true},"cell_type":"code","source":"# Define data types\n# uint32 is an unsigned integer with 32 bit \n# which means that you can represent 2^32 numbers (0-4294967295)\ndtypes = {\n            'ip'            : 'uint32',\n            'app'           : 'uint16',\n            'device'        : 'uint8',\n            'os'            : 'uint8',\n            'channel'       : 'uint16',\n            'is_attributed' : 'uint8',\n            'click_id'      : 'uint32',\n            }","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"82deb188-5b66-417e-a9fa-543d1581b406","_uuid":"120a79a0a18c64ffd2889b04e0ff2c0b0b65932a"},"cell_type":"markdown","source":"### Trick 3: Select columns before importing\n\n"},{"metadata":{"_cell_guid":"82a18ae2-74d5-493c-80c5-26966615497f","_kg_hide-output":true,"_uuid":"e51b85a8432336f827b0accaac5bb42afa77aa83","collapsed":true,"trusted":true},"cell_type":"code","source":"# Only import columns you need: create a list before you actually read the data\ntrain_cols=['ip','app','device','os', 'channel', 'click_time', 'is_attributed']\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f82cbcb6-1baa-4a1b-a02b-714f717394d7","_uuid":"24241a5f66c09b99850b4a8bdccf70eacca90c66","collapsed":true,"trusted":true},"cell_type":"code","source":"# Exercise: create a list called test_cols with the following feature: \n# 'ip','app','device','os', 'channel', 'click_time', 'click_id'\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"27d58987-4b40-4727-b939-be64244dcf08","_uuid":"f9daefecbf72257d8c07176ea3f4a46ee53d30f4"},"cell_type":"markdown","source":"Now, set up debug mode and import data "},{"metadata":{"_cell_guid":"0b962b68-3294-41da-9c4e-bb697fd744a0","_kg_hide-output":true,"_uuid":"a7e9d6cafbe62cb969009f8b08ea1fce6a9fd1da","collapsed":true,"trusted":true},"cell_type":"code","source":"# It takes a long time to load a large dataset, so we print a mark here just to keep track of the process\nprint(\"Loading training data...\")\n\n# Now reading the training and test data.\n# Load only a few lines if debug=1; Load a much larger part of the dataset if debug=0\nif debug:\n    train = pd.read_csv(\"../input/train.csv\", dtype=dtypes, parse_dates=['click_time'], \n                        nrows=100000, usecols=train_cols)\n    test = pd.read_csv(\"../input/test.csv\", dtype=dtypes, parse_dates=['click_time'], \n                       nrows=100000, usecols=test_cols)\nelse: \n    train =  pd.read_csv(\"../input/train.csv\", dtype=dtypes, parse_dates=['click_time'], \n                         # skiprows=range(1,129903891), this will skip the first n rows\n                         nrows=1000000, usecols=train_cols)\n    test = pd.read_csv(\"../input/test.csv\", dtype=dtypes, parse_dates=['click_time'], \n                       usecols=test_cols)\nprint (\"Loading finished\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ef034787-f020-4df7-95af-c7750972cc9b","_uuid":"f6f6ce9e7d85733aa3b490fe60c6e51a7fc87470"},"cell_type":"markdown","source":"## 3. Feature Engineering\nFrom now on we are going to do some feature engineering. Whenever we add a new feature in the trainining dataset, we have to create the same feature in the test dataset. To avoid duplicate coding, we will combine the training and test datasets before we do anything with the features. "},{"metadata":{"_cell_guid":"6221e28f-a0b5-4f97-8b2b-9b6b7f2f7985","_uuid":"b2f7312eb2bdc9617ac4a98ff8c0e84207a96e5e","collapsed":true,"trusted":true},"cell_type":"code","source":"# Exercise: Print a sentence to indicate that we are now \"processing data\"\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"446c6b53-50a7-4542-8f9b-e7fc003a77af","_uuid":"704043abc0c2473da0343efe0985c5ece5deb337"},"cell_type":"markdown","source":"### 3.1 Combine training and test data before feature engineering"},{"metadata":{"_cell_guid":"1e069eeb-2904-4d73-8bf5-af91591ea7a6","_kg_hide-output":true,"_uuid":"8ecc32fd20cfdd134c41ebdad94cdb9705744870","collapsed":true,"scrolled":true,"trusted":true},"cell_type":"code","source":"# First, we have to get the length of the training data. \n# We'll need this number when we have to split the training and test data again\nlen_train = len(train)    \n\n# Now append test data to training data and make a new data frame called full\nfull=train.append(test)   ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e132c89c-4bc5-4efa-a1e9-730af3c8273c","_uuid":"d116ddbc01c041b656fc4acebb57ceadbc8e042d"},"cell_type":"markdown","source":"### Trick 4: Delete large objects and use gc.collect()"},{"metadata":{"_cell_guid":"fb0f9335-607b-4cee-8238-1a4e3b8d6cb1","_kg_hide-output":true,"_uuid":"366e6ced6376469b0b0bc0cfcf48b4376e454ae4","collapsed":true,"trusted":true},"cell_type":"code","source":"# Now we have stored both training and test data in a new dataframe called full\n# We can delete training and test data because we don't need them anymore and they are very large\ndel test  \ndel train \n\n# Collect any other temp garbage in the RAM. \n# It's a good habit to gc.collect() from time to time when you deal with large datasets\ngc.collect() ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"72bc39ea-c428-403f-8ddf-4186443af945","_kg_hide-output":true,"_uuid":"e6d5802d6ea67f9b5a01d12830407a69c2f8ade7","collapsed":true,"trusted":true},"cell_type":"code","source":"# Assign a list for predictors and target. We'll use these two lists very soon\npredictors = [\"ip\",\"app\",\"device\",\"os\",\"channel\"]\ntarget = \"is_attributed\"\n\n# Create a list with names of categorical variables. LGBM can handle categorical variables smoothly.\ncategorical = [\"ip\", \"app\", \"device\",\"os\", \"channel\"]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9fefb2b6-0704-47b1-b4cd-cc9122268888","_uuid":"582f9a8da1153b74bd4742faab2916b8afa2e992"},"cell_type":"markdown","source":"### 3.2 Extract time features\n\nThe first thing we can do with this dataset is to extract the time features. Remember \"click_time\"? It marked the day, hour, minute, and second when each click happened. We can extract those time features seperately. "},{"metadata":{"_cell_guid":"3bb9eeed-f706-498d-96af-d5a68e23d50c","_kg_hide-output":true,"_uuid":"fb6d2d006fe913c582062705b4f6ba479040f2c8","collapsed":true,"trusted":true},"cell_type":"code","source":"# Getting time features\n# Get \"day\" from \"click_time\" and add it to \"full\" as a new feature\nfull['day'] = pd.to_datetime(full.click_time).dt.day.astype('int8')\n# Append \"day\" to the predictor list\npredictors.append('day')\n\n# Get \"hour\" from \"click_time\" and add it to \"full\" as a new feature\nfull['hour'] = pd.to_datetime(full.click_time).dt.hour.astype('int8')\n# Append \"hour\" to the predictor list\npredictors.append('hour')\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"22fb1155-5691-4123-90af-5401d6f983e4","_uuid":"953bbad4c0547366e4f0f6c158eec9675ea10ebb","collapsed":true,"trusted":true},"cell_type":"code","source":"# Exercise: get \"minute\" and \"second\" from \"click_time\" and add them to \"full\" as new features\n\n\n# Exercise: Append \"minute\" and \"second\" to the predictor list\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9220abbd-2ff8-49be-b4ef-6cb5324fc71b","_uuid":"e6975972aaea7fa38c4520c27d07770c797598ca"},"cell_type":"markdown","source":"Note: sometimes it doesn't make sense to include all time features. For example, in this dataset, the \"day\" feature in the training and test datasets do not overlap at all (Monday to Thursday in train, Friday in test). Also, it is hard to image how \"minute\" and \"second\" would influence the probability of downloading. \"Hour\", on the other hand, may play a role (e.g. people may be more likely to download an app after working hours). This is just for practice. You can try to drop some of the time features later to see how the final score changes. "},{"metadata":{"_cell_guid":"86ce3ce2-ce5c-4976-bede-24e0cb21555e","_uuid":"4ae97be95dc5df045d330776834eedc57515639a"},"cell_type":"markdown","source":"### 3.3 Extracting time difference between two clicks\n\nThe next step is a little bit tricky. We're extracting one more feature called \"next click\", which calculates the time difference between next click and the current click from the same ip, same os, same device, and same channel (therefore very likely to be the same person). We doubt that if this time difference is too short (two clicks happen very fast), then the current click is more likely to be a fake one. "},{"metadata":{"_cell_guid":"db593152-5e0b-4cff-8847-79fdaf100525","_kg_hide-output":true,"_uuid":"b46e1cedbb338c45245585431043f6c52497c01f","collapsed":true,"trusted":true},"cell_type":"code","source":"# shift means we are shifting the whole column up by one row, which basically means we're getting the next value in line\n# We subtract the time of current click from the time of next click, so we get the time difference between two clicks\n# We convert this difference to seconds, and claim that its data type is \"float32\", which means real number with 32 bit\nsame=[\"channel\", \"app\", \"os\", \"device\",\"ip\"]\nfull['next_click'] = (full.groupby(same).click_time.shift(-1) - full.click_time).dt.seconds.astype('float32')\n\n# Append \"next_click\" to the predictor list\npredictors.append('next_click')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2796e165-cdc9-406d-9169-778da592dcd1","_uuid":"f52b5fedad20e0cc7602c2b834f09bc9bde08944","collapsed":true,"trusted":true},"cell_type":"code","source":"# Exercise: using very similar method, create a new feature called \"prev_click\", \n# indicating the time difference between the current click and the previuos click\n# Hint: just change -1 to +1, and switch the two click_time\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"041686f1-1c63-47d7-8a3a-238c6b5d83c7","_uuid":"7419e996d188f483a71bea2bae4e13c03f4fb25a"},"cell_type":"markdown","source":"### 3.4 Creating grouped count features\n\nNext we will create a feature calculating the number of clicks for each ip in the same hour on the same day. We assume that if there are too many clicks from the same ip, then these clicks might be fake. "},{"metadata":{"_cell_guid":"6ac1ef2f-271b-4969-8a80-85f931fd7544","_kg_hide-output":true,"_uuid":"7fe95d8a136b8c435685c0013ffb32cf55e81c51","collapsed":true,"trusted":true},"cell_type":"code","source":"# Set the group. You can have other combinations\ngroup = ['ip','day','hour']\n\n# group by ip+day+hour, choose one column (click_time) to count the number of rows, \n# fill this number into the click_time variable, then change the column name to ip_day_hour\ngp = full.groupby(group)[\"click_time\"].count().reset_index().rename(index=str, columns={'click_time':\"ip_day_hour\"})\n\n# merge back with full data\nfull = full.merge(gp, on=group, how='left')\n\n# Append new variable name to the list \"prdictors\"\npredictors.append(\"ip_day_hour\")\n\n# Delete gp and collect garbage\ndel gp\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d3aaaa70-cc4f-45b6-b187-b6cbe293a98f","_uuid":"5f2da76e86541777d3861b8de8df8b2d031e2760","collapsed":true,"trusted":true},"cell_type":"code","source":"# Exercise: Very similarly, create a new variable called ip_app_channel, \n# which calculates the total number of clicks for the same ip + app + channel\n# Delete and collect garbage\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"534742ed-eb5e-4776-945b-cfd9dd4f0f96","_uuid":"4236eb8ba695d4e2312de611f716b2024f2e9b5a"},"cell_type":"markdown","source":"Note: you can also get grouped mean and variance features by changing count() to mean() or var(). We'll skip this stage today, but you are encouraged to try different features to see how they may affect the final score. "},{"metadata":{"_cell_guid":"49acb584-0265-44fa-822e-8f07d890472a","_uuid":"090ffe75ca62a97cac6ba2a4a58c40333245ae79"},"cell_type":"markdown","source":"### 3.5 Splitting training and test datasets \n\nWe have finished our feature engineering today. If you want to create more features, do it before next step.\n\nRemember that before feature engineering, we have merged the training and test datasets? Now it's time to split them again.\n"},{"metadata":{"_cell_guid":"78664c7c-8985-443f-a608-c3dbaeb102d7","_kg_hide-output":true,"_uuid":"404f556326d8c1eaa9d51ad37a2f72a52f9832c1","collapsed":true,"trusted":true},"cell_type":"code","source":"# Split training and test data\ntrain = full[:len_train]\ntest = full[len_train:]\n\n# Set X(predictors) and y(target)\nX = train[predictors]\ny = train[target]\n    \n# Delete unused parts \ndel train\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2a126d8d-995d-4483-a50a-cedd27fa744a","_uuid":"a33bddbcf5944cbf3020ffaf06cfc342a92227ad"},"cell_type":"markdown","source":"### 3.6 Splitting training and validation datasets\n\nThe next step is to split the training dataset into training and validation. This is to avoid overfitting. "},{"metadata":{"_cell_guid":"569dd852-23ef-41a7-9909-bd3f8c3bb90f","_kg_hide-output":true,"_uuid":"62c8e61302565804a2b634d4f63805717a4ce1cf","collapsed":true,"trusted":true},"cell_type":"code","source":"# Split training and validation data using train_test_split\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size = 0.2, random_state = 30)\n\n# delete X and y since we don't need them anymore\ndel X, y\n\n    ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"de8e20fc-fff0-443b-adb9-3a8771d1cb16","_uuid":"d67803fd1c3be8b6d5d1401b4a09e7fabff9dc4e"},"cell_type":"markdown","source":"## 4. Modelling with Light GBM \n\nFinally we are ready for Light GBM! For an introduction of how this model works, please refer to [this link](https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/). \n\nFirst, we'll define training and validation dataset again in the way that the lgbm library can understand: "},{"metadata":{"_cell_guid":"805c7686-622a-435d-be36-dc264aa90f6f","_kg_hide-output":true,"_uuid":"8ab88dceaac98cb259a1e191a23d26a7a447a4e5","collapsed":true,"trusted":true},"cell_type":"code","source":"# lgb.Dataset defines the training and validation dataset\n# this is a little bit confusing, but in lgb.Dataset, \"label\" means y(the target variable), \n# because our prediction is actually \"labelling\" the target\n# We also define the feature names for feature importance plotting afterwards\nxgtrain = lgb.Dataset(X_train.values, label=y_train.values, feature_name=predictors, categorical_feature = categorical)                          \nxgvalid = lgb.Dataset(X_val.values, label=y_val.values, feature_name=predictor,categorical_feature = categoricals)\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a2627fd0-644d-419d-8335-7597bee8c230","_uuid":"a6af35afcf9ce673f1e12a87abac958afe8a4ad3"},"cell_type":"markdown","source":"Light GBM is a very complex model with tons of parameters to tune. You can leave them as default, but we'll have a brief look at those parameters here. \n\nFor a very clear introduction (that speaks English) of what those parameters mean and what is the best approach, please refer to [this website](https://sites.google.com/view/lauraepp/parameters). Also check out [here](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst). \n\nNext block sets up basic lgb parameters. You can leave them as default if you don't know what to do with them. "},{"metadata":{"_cell_guid":"0674c6dd-bdfc-48a3-9f26-7efc23cffb4f","_kg_hide-output":true,"_uuid":"68ad9bbb280f90821369da2ac0c073ff3f805ab1","collapsed":true,"trusted":true},"cell_type":"code","source":"# Setting lgb model parameters; not mandatory\nlgb_params = {\n        'boosting_type': 'gbdt', # Gradient Boosted Decision Trees\n        'objective': 'binary', # Because we are predicting 0 and 1\n        'metric': 'auc', # Method to evaluate the model, auc means \"area under the curve\". The lower the better\n        'learning_rate': 0.03, #Basically the weight for each boosting iteration. A smaller learning_rate may increase accuracy but lead to slower training speed. \n        'num_leaves': 31,  # we should let it be smaller than 2^(max_depth)\n        'max_depth': -1,  # -1 means no limit. Too deep may lead to overfitting. \n        'min_child_samples': 20,  # Minimum number of data need in a child(min_data_in_leaf)\n        'max_bin': 255,  # Number of bucketed bin for feature values\n        'subsample': 0.6,  # Subsample ratio of the training instance.\n        'subsample_freq': 0,  # frequence of subsample, <=0 means no enable\n        'colsample_bytree': 0.3,  # Subsample ratio of columns when constructing each tree.\n        'min_child_weight': 5,  # Minimum sum of instance weight(hessian) needed in a child(leaf)\n        'subsample_for_bin': 200000,  # Number of samples for constructing bin\n        'min_split_gain': 0,  # lambda_l1, lambda_l2 and min_gain_to_split to regularization\n        'reg_alpha': 0,  # L1 regularization term on weights\n        'reg_lambda': 0,  # L2 regularization term on weights\n        'nthread': 8, # Number of threads using for training models, better to set it large for large dataset\n        'verbose': 0, # Do not affect training, just affect how detailed the information produced during training would be\n        'random_state': 42\n    }","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"50661277-6c2b-4e55-964f-2b48c1e6b09a","_uuid":"7fe877e6bcd90a9d5d81e492c64589105e5a0eae"},"cell_type":"markdown","source":"Let's officially start training! "},{"metadata":{"_cell_guid":"869ec405-cca6-425d-92a8-88ce01c6ba35","_kg_hide-output":true,"_uuid":"afa2d8d1d83cc44691504bf83dff89cdf260197c","trusted":true},"cell_type":"code","source":"# Training start! Print a marker for it. \nprint(\"Training...\")\n\n# Set start_time as current time\nstart_time = time.time()\n\n# Create an empty data frame for writing the evaluation results. \nevals_results={}\n\n# This is the real lgb model training process\n# There are more parameters to tune! I put those parameters that we are most likely to change here\n\nmodel_lgb = lgb.train(lgb_params, # The list of parameters that we have already set\n                xgtrain,  # The training dataset\n                valid_sets= [xgtrain, xgvalid], # We produce evaluation score for both the training and validation dataset\n                valid_names=['train','valid'],  # Assign names to the training and validation dataset\n                early_stopping_rounds=100, # If there's no improvement after 10 rounds, then stop\n                verbose_eval=50,  # Print evaluation scores every 10 rounds\n                num_boost_round=5000, # Maximum 200 rounds, even if it does not meet the early_stopping_round requirement\n                evals_result=evals_results) # Write evalution results into evals_results\n                \n# Print current time - start_time, this is the time used for training the model    \nprint('Model training time: {} seconds'.format(time.time() - start_time))\n\ngc.collect()\n\n# Exercise: change early_Stopping_round, verbose_eval, and num_boost_round, and train the model again\n# Print our how much time the training took\n# Note: you may need to run the revious block to reset xgtrain and xgvalid because lgbm has already processed categorical data","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"b562d0a0-af65-4baf-8848-10e6cf084b44","_uuid":"01f648a9fb158775de1d9c9c8b6a1b06bc967e93"},"cell_type":"markdown","source":"## 5. Feature Importance\n\nAfter training the model, we can actually examine how important each feature is. \n\nIn Light GBM, there are two ways to evaluate the importance of a feature: \n\n 1. “split”: the number of times a feature is used in a model \n 2. “gain”: the total gains of splits which use the feature\n\nWhen there are not too many features, gain is usually better.  It's possible that all models used a certain feature (therefore a high score for \"split\"),  but they only used it once for split in each model (therefore a low score for \"gain\"). \n"},{"metadata":{"_cell_guid":"742a57d1-2c40-4644-9be4-6ac5c4deaf38","_kg_hide-output":true,"_uuid":"e96b4736b46c2c000a95352500e1f0253e4dc521","collapsed":true,"trusted":true},"cell_type":"code","source":"# List feature importance for all features\nprint(\"Features importance...\")\ngain = model_lgb.feature_importance('gain')\nft = pd.DataFrame({'feature':model_lgb.feature_name(), \n                   'split':model_lgb.feature_importance('split'), \n                   'gain':100 * gain / gain.sum()}).sort_values('gain', ascending=False)\nprint(ft)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9dd109a8-6f78-48e6-821f-63bb6c2eb815","_kg_hide-output":true,"_uuid":"ba49a4599bbead5d2b4e2b73b13b435171bca86b","collapsed":true,"trusted":true},"cell_type":"code","source":"# Plot feature importance using the \"split\" method\nsplit = lgb.plot_importance(model_lgb, importance_type=\"split\")\nplt.show(split) # Show the plot\nplt.savefig(\"feature_importance_split.png\") # Save the plot in the output","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"87a36478-9b24-45ee-8635-e92327829069","_uuid":"017540479955ae0f538ae1bf14d33bc223aaa73c","collapsed":true,"trusted":true},"cell_type":"code","source":"# Exercise: Plot feature importance using the \"gain\" method\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"49b8c511-91fc-40dc-8a9f-1618c17b6c19","_uuid":"b4ee3a74385e816b6707d248f2f85433e836ba14"},"cell_type":"markdown","source":"## 6. Predicting and submission\n\n### 6.1 Predicting for the test dataset\n\nNow that we have trained our model, we will use it to predict the target variable using the predictors in the test dataset. "},{"metadata":{"_cell_guid":"46634e36-7427-4ca3-936a-89ba2cdebc63","_kg_hide-output":true,"_uuid":"16183c00125fd9790688f38d41be7046e28fa50d","collapsed":true,"trusted":true},"cell_type":"code","source":"# Print a mark here\nprint (\"Predicting test data...\")\n\n# Creat X_test, which includes all features in the test dataset\nX_test = test[predictors]\n\n# Feed X_test to our trained \"model_lgb\" to predict the target variable (y) in the test dataset\n# Store our prediction in ypred\nypred = model_lgb.predict(X_test,num_iteration=model_lgb.best_iteration)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e19dc013-3c88-44f2-8ae1-fe372414e922","_uuid":"d563726adedf38eb4be9c7a3b7f843bf01277777"},"cell_type":"markdown","source":"### 6.2 Writing the submission file\n\nWe are ready to write the submission file! \n\nHowever, remember that if you are in the debug mode, your predicted test file will be much shorter than the real test dataset, therefore you will get a warning message \"Length of values does not match length of index\" . To submit and get a score, you will need to set debug=0. That may take up to 2 hours to train the model, depending on the size of your training data, and how expensive your laptop is. \n\nChange debug = 0, adjust the nrows in your training data based on your laptop RAM and CPU, and hit \"commit & run\". You'll be able to see your output after a while. Submit it to the competition and see where you are on the Leader Board!\n"},{"metadata":{"_cell_guid":"5c2274a1-e5c8-44e6-ab68-06398773638c","_kg_hide-output":true,"_uuid":"bd330f1eec2719350aac607e62c3a536e7fed27c","collapsed":true,"trusted":true},"cell_type":"code","source":"# Print a mark\nprint (\"Writing submission file...\")\n\n# Read the sample submission file\nsubmission = pd.read_csv(\"../input/sample_submission.csv\")\n\n# Change the value in the prediction column into our prediction \"ypred\"\nsubmission[\"is_attributed\"] = ypred\n\n# Write it into a csv file\nsubmission.to_csv(\"submission.csv\", index = False)\n\n# Print a final mark\nprint (\"Mission Completed\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"963938fd-5036-4031-9cfb-94b98d52ba5f","_uuid":"035294afd0846523f04a431d4fd3913216129d97"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}