{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":8540,"databundleVersionId":862041,"sourceType":"competition"}],"dockerImageVersionId":30035,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

TalkingData AdTracking EDA

\n\n\n\n\n\n# Content\n\n- Introduction \n- Load packages \n- Parameters \n- Read the data \n- Check the data \n - Glimpse the data \n - Check missing data\n - Check data unbalance\n- Data exploration\n- Data engineering\n - Extract date and time data\n - Additional data engineering \n - Data unbalance between train and test data\n- Model \n - Prepare the model \n - Train the model \n- Predict and submission\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\nFraud risk is everywhere, but for companies that advertise online, click fraud can happen at an overwhelming volume, resulting in misleading click data and wasted money. Ad channels can drive up costs by simply clicking on the ad at a large scale. With over 1 billion smart mobile devices in active use every month, China is the largest mobile market in the world and therefore suffers from huge volumes of fraudulent traffic.\nFounded in 2011, TalkingData is China’s largest independent big data service platform. \n\nThey handle 3 billion clicks per day, of which 90% are potentially fraudulent. Their current approach to prevent click fraud for app developers is to measure the journey of a user’s click across their portfolio, and flag IP addresses who produce lots of clicks, but never end up installing apps. With this information, they've built an IP blacklist and device blacklist.\nWhile successful, they wanted to always be one step ahead of fraudsters and have turned to the Kaggle community for help in further developing their solution. In their 2nd competition with Kaggle, they challenge participants to build an algorithm that predicts whether a user will download an app after clicking a mobile app ad. To support the modeling, they have provided a generous dataset covering approximately 200 million clicks over 4 days!\n","metadata":{}},{"cell_type":"markdown","source":"# Load packages\n\nHere we load the libraries and set few parameters.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom sklearn.impute import SimpleImputer \nimputer = SimpleImputer(missing_values=np.nan, strategy='mean')\nfrom sklearn.preprocessing import PolynomialFeatures\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.feature_selection import VarianceThreshold\nfrom sklearn.feature_selection import SelectFromModel\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import cross_val_score\n\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier\nfrom sklearn.linear_model import LogisticRegression\n\n\nimport lightgbm as lgb\nimport gc # memory \nfrom datetime import datetime # train time checking\n\npd.set_option('display.max_columns', 100)","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:14.953570Z","iopub.execute_input":"2024-04-21T15:50:14.954133Z","iopub.status.idle":"2024-04-21T15:50:17.507773Z","shell.execute_reply.started":"2024-04-21T15:50:14.954096Z","shell.execute_reply":"2024-04-21T15:50:17.506549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parameters\n\nHere we set most of the parameters used in this Notebook.\n\n","metadata":{}},{"cell_type":"code","source":"#VALIDATION\nVALIDATE = False #validation using train_test_split\nVALID_SIZE = 0.90 # simple validation using train_test_split\n\n#CROSS-VALIDATION\nVALIDATE_KFOLDS = True #cross-validation using KFolds\nNUMBER_KFOLDS = 5 #number of KFolds for cross-validation\n\nSAMPLE = True #True: use train.sample (100,000 rows) False: use full training set (train)\nRANDOM_STATE = 2018\n\nMAX_ROUNDS = 1000 #lgb iterations\nEARLY_STOP = 50 #lgb early stop \nOPT_ROUNDS = 650 #To be adjusted based on best validation rounds\nskiprows = range(1,109903891) #\nnrows = 75000000\n#USE SAMPLE FROM FULL TRAIN SET\nSAMPLE_SIZE = 1 # use a subsample of the train set\noutput_filename = 'submission.csv'\n\nIS_LOCAL = True\n\n\nif (IS_LOCAL):\n PATH = '../input/talkingdata-adtracking-fraud-detection/'\nelse:\n PATH = '../input/'\nprint(os.listdir(PATH))","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:17.510064Z","iopub.execute_input":"2024-04-21T15:50:17.510577Z","iopub.status.idle":"2024-04-21T15:50:17.522750Z","shell.execute_reply.started":"2024-04-21T15:50:17.510498Z","shell.execute_reply":"2024-04-21T15:50:17.521739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read the data\n\nWe read the data with the options set in the **Parameters** section.\n","metadata":{}},{"cell_type":"code","source":"dtypes = {\n 'ip' : 'uint32',\n 'app' : 'uint16',\n 'device' : 'uint16',\n 'os' : 'uint16',\n 'channel' : 'uint16',\n 'is_attributed' : 'uint8',\n 'click_id' : 'uint32'\n }\n\ntrain_cols = ['ip','app','device','os', 'channel', 'click_time', 'is_attributed']\n\nif SAMPLE:\n trainset = pd.read_csv(PATH+\"train_sample.csv\", dtype=dtypes, usecols=train_cols) \nelse:\n trainset = pd.read_csv(PATH+\"train.csv\", skiprows=skiprows, nrows=nrows,dtype=dtypes, usecols=train_cols)\n trainset = trainset.sample(frac=SAMPLE_SIZE)\n\ntestset = pd.read_csv(PATH+\"test.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:17.524570Z","iopub.execute_input":"2024-04-21T15:50:17.525100Z","iopub.status.idle":"2024-04-21T15:50:48.073672Z","shell.execute_reply.started":"2024-04-21T15:50:17.525053Z","shell.execute_reply":"2024-04-21T15:50:48.072562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check the data\n\n## Glimpse the data\n\nLet's check the train and test set structure.\n","metadata":{}},{"cell_type":"code","source":"print(\"train - rows:\",trainset.shape[0],\" columns:\", trainset.shape[1])\nprint(\"test - rows:\",testset.shape[0],\" columns:\", testset.shape[1])\n","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:48.075294Z","iopub.execute_input":"2024-04-21T15:50:48.075779Z","iopub.status.idle":"2024-04-21T15:50:48.084193Z","shell.execute_reply.started":"2024-04-21T15:50:48.075723Z","shell.execute_reply":"2024-04-21T15:50:48.082657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainset.head()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:48.091312Z","iopub.execute_input":"2024-04-21T15:50:48.091738Z","iopub.status.idle":"2024-04-21T15:50:48.123914Z","shell.execute_reply.started":"2024-04-21T15:50:48.091701Z","shell.execute_reply":"2024-04-21T15:50:48.122870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"testset.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:48.127566Z","iopub.execute_input":"2024-04-21T15:50:48.128045Z","iopub.status.idle":"2024-04-21T15:50:48.141459Z","shell.execute_reply.started":"2024-04-21T15:50:48.128009Z","shell.execute_reply":"2024-04-21T15:50:48.140491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The trainset columns are:\n* **ip**: ip address of click (numeric);\n* **app**: app id for marketing (category);\n* **device**: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.) (category);\n* **os**: os version id of user mobile phone (category);\n* **channel**: channel id of mobile ad publisher (category);\n* **click_time**: timestamp of click (UTC) (time);\n* **attributed_time**: if user download the app for after clicking an ad, this is the time of the app download (time);\n* **is_attributed**: the target that is to be predicted, indicating the app was downloaded (binary);\n\nThe testset differs with three columns:\n* **click_id** reference for making predictions (additional column) (numeric);\n* **attributed_time**: missing column;\n* **is_attributed**: missing column (target column) (binary).\n\nLet's see what is the dimmension of the train and test set.\n\n","metadata":{}},{"cell_type":"markdown","source":"## Check missing data\n\nLet's check missing data in train and test.\n","metadata":{}},{"cell_type":"code","source":"def missing_data(data):\n total = data.isnull().sum().sort_values(ascending = False)\n percent = (data.isnull().sum()/data.isnull().count()*100).sort_values(ascending = False)\n return pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:48.143078Z","iopub.execute_input":"2024-04-21T15:50:48.143443Z","iopub.status.idle":"2024-04-21T15:50:48.152152Z","shell.execute_reply.started":"2024-04-21T15:50:48.143407Z","shell.execute_reply":"2024-04-21T15:50:48.150886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Train data:","metadata":{}},{"cell_type":"code","source":"missing_data(trainset)","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:48.153725Z","iopub.execute_input":"2024-04-21T15:50:48.154256Z","iopub.status.idle":"2024-04-21T15:50:48.221887Z","shell.execute_reply.started":"2024-04-21T15:50:48.154213Z","shell.execute_reply":"2024-04-21T15:50:48.220628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Test data:","metadata":{}},{"cell_type":"code","source":"missing_data(testset)","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:48.223167Z","iopub.execute_input":"2024-04-21T15:50:48.223534Z","iopub.status.idle":"2024-04-21T15:50:55.239873Z","shell.execute_reply.started":"2024-04-21T15:50:48.223476Z","shell.execute_reply":"2024-04-21T15:50:55.238824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check data unbalance\n\nWe will check the **target** (**is_attributed**) data unbalance. ","metadata":{}},{"cell_type":"code","source":"plt.figure()\nfig, ax = plt.subplots(figsize=(6,6))\nx = trainset['is_attributed'].value_counts().index.values\ny = trainset[\"is_attributed\"].value_counts().values\n# Bar plot\n# Order the bars descending on target mean\nsns.barplot(ax=ax, x=x, y=y)\nplt.ylabel('Number of values', fontsize=12)\nplt.xlabel('is_attributed value', fontsize=12)\nplt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:55.241258Z","iopub.execute_input":"2024-04-21T15:50:55.241614Z","iopub.status.idle":"2024-04-21T15:50:55.440705Z","shell.execute_reply.started":"2024-04-21T15:50:55.241581Z","shell.execute_reply":"2024-04-21T15:50:55.439544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only a very small part of the **is_attributed** data have 1 value (227 out of 100,000 values, 176,627 out of 75 M values). This means that the training dataset is highly imbalanced (0.23% have **is_attributed** = 1). Usually is used either undersample the records with **is_attributed** = 0 or oversample records with **is_attributed** = 1; because is a large dataset, it is a good option to do undersampling of records with **is_attributed** = 0.","metadata":{}},{"cell_type":"markdown","source":"# Data exploration","metadata":{}},{"cell_type":"code","source":"trainset.describe()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:55.442181Z","iopub.execute_input":"2024-04-21T15:50:55.442515Z","iopub.status.idle":"2024-04-21T15:50:55.498313Z","shell.execute_reply.started":"2024-04-21T15:50:55.442470Z","shell.execute_reply":"2024-04-21T15:50:55.497012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe that, while ip is a number with a wide range of values, **app**, **device**, **os** and **channel** are categorical values with a smaller range of variation. Let's separate between values with **is_attributed** with value `1` and value `0`.","metadata":{}},{"cell_type":"code","source":"t1 = trainset.loc[trainset['is_attributed'] != 0.]\nt0 = trainset.loc[trainset['is_attributed'] == 0.]\n\nt0.describe()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:55.499810Z","iopub.execute_input":"2024-04-21T15:50:55.500246Z","iopub.status.idle":"2024-04-21T15:50:55.566341Z","shell.execute_reply.started":"2024-04-21T15:50:55.500209Z","shell.execute_reply":"2024-04-21T15:50:55.565114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"t1.describe()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:55.567942Z","iopub.execute_input":"2024-04-21T15:50:55.568263Z","iopub.status.idle":"2024-04-21T15:50:55.604376Z","shell.execute_reply.started":"2024-04-21T15:50:55.568232Z","shell.execute_reply":"2024-04-21T15:50:55.603412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe that in the two sets, **ip** and **os** and **channel** have values in the same ranges for both **is_attributed** values (0 and 1). Maximum values for **app** is almost double and maximum value for **device** is four times larger for **is_attributed** = 0.\n\nLet's visualize the distribution of values of **app**, **device**, **os** and **channel**, grouped on **is_attributed**.","metadata":{}},{"cell_type":"code","source":"var = ['app','device','os','channel']\n\ni = 0\nt1 = trainset.loc[trainset['is_attributed'] != 0]\nt0 = trainset.loc[trainset['is_attributed'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax = plt.subplots(1,4,figsize=(16,4))\n\nfor feature in var:\n i += 1\n plt.subplot(1,4,i)\n sns.kdeplot(t1[feature], bw=0.5,label=\"is_attributed = 1\")\n sns.kdeplot(t0[feature], bw=0.5,label=\"is_attributed = 0\")\n plt.ylabel('Density plot', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:55.606001Z","iopub.execute_input":"2024-04-21T15:50:55.606323Z","iopub.status.idle":"2024-04-21T15:50:56.902363Z","shell.execute_reply.started":"2024-04-21T15:50:55.606292Z","shell.execute_reply":"2024-04-21T15:50:56.901420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data engineering\n\n## Extract date and time data","metadata":{}},{"cell_type":"code","source":"trainset['year'] = pd.to_datetime(trainset.click_time).dt.year\ntrainset['month'] = pd.to_datetime(trainset.click_time).dt.month\ntrainset['day'] = pd.to_datetime(trainset.click_time).dt.day\ntrainset['hour'] = pd.to_datetime(trainset.click_time).dt.hour\ntrainset['min'] = pd.to_datetime(trainset.click_time).dt.minute\ntrainset['sec'] = pd.to_datetime(trainset.click_time).dt.second\ntrainset.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:56.903708Z","iopub.execute_input":"2024-04-21T15:50:56.904228Z","iopub.status.idle":"2024-04-21T15:50:57.210048Z","shell.execute_reply.started":"2024-04-21T15:50:56.904190Z","shell.execute_reply":"2024-04-21T15:50:57.209081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainset.describe()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:57.211219Z","iopub.execute_input":"2024-04-21T15:50:57.211680Z","iopub.status.idle":"2024-04-21T15:50:57.306649Z","shell.execute_reply.started":"2024-04-21T15:50:57.211645Z","shell.execute_reply":"2024-04-21T15:50:57.305776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = ['day','hour']\n\ni = 0\nt1 = trainset.loc[trainset['is_attributed'] != 0]\nt0 = trainset.loc[trainset['is_attributed'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax = plt.subplots(1,2,figsize=(12,4))\n\nfor feature in var:\n i += 1\n plt.subplot(1,2,i)\n sns.kdeplot(t1[feature], bw=0.5,label=\"is_attributed = 1\")\n sns.kdeplot(t0[feature], bw=0.5,label=\"is_attributed = 0\")\n plt.ylabel('Density plot', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:57.308152Z","iopub.execute_input":"2024-04-21T15:50:57.308495Z","iopub.status.idle":"2024-04-21T15:50:58.050334Z","shell.execute_reply.started":"2024-04-21T15:50:57.308462Z","shell.execute_reply":"2024-04-21T15:50:58.049120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One can observe that the distribution for **true** (**is_attributed = 1**) clicks is more diverse (hour, min, sec) compared with **false** (**is_attributed = 0**). This might be explained in two ways: one explanation can be that due to the reduced number of **true** cases, the distribution is less uniform. Another explanation might be (to be verified with larger number of cases) that due to programatic nature of *artificial* (**false**) clicks, their distribution is more uniform. One observation, related to the density plot for hours: for both **true** and **false** there is a certain hourly profile, with a plateau between 1 and 16, a saddle between 16 and 20 and a peak between 21 and 22. The plateau for the artificial (**false**) clicks shows an additional pattern, with oscilant profile.","metadata":{}},{"cell_type":"markdown","source":"Let's represent the hour distribution with an alternative method, using barplots to show percent from all data of the **true** data.","metadata":{}},{"cell_type":"code","source":"var = ['day','hour']\n\nfor feature in var:\n fig, ax = plt.subplots(figsize=(16,6))\n # Calculate the percentage of target=1 per category value\n cat_perc = trainset[[feature, 'is_attributed']].groupby([feature],as_index=False).mean()\n cat_perc.sort_values(by='is_attributed', ascending=False, inplace=True)\n # Bar plot\n #sns.barplot(ax=ax,x=feature, y='is_attributed', data=cat_perc, order=cat_perc[feature]) #for ordered bars\n sns.barplot(ax=ax,x=feature, y='is_attributed', data=cat_perc)\n plt.ylabel('Percent of `is_attributed` with value 1 [%]', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n plt.tick_params(axis='both', which='major', labelsize=12)\n plt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:58.051702Z","iopub.execute_input":"2024-04-21T15:50:58.052043Z","iopub.status.idle":"2024-04-21T15:50:58.876747Z","shell.execute_reply.started":"2024-04-21T15:50:58.052011Z","shell.execute_reply":"2024-04-21T15:50:58.875517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Additional feature engineering\n\nDefine several functions for calculation of derived features.","metadata":{}},{"cell_type":"code","source":"#------------------------------------------------------------------------------\ndef show_max_clean(df,gp,agg_name,agg_type,show_max):\n#------------------------------------------------------------------------------ \n del gp\n if show_max:\n print( agg_name + \" max value = \", df[agg_name].max() )\n df[agg_name] = df[agg_name].astype(agg_type)\n gc.collect()\n return( df )\n \n#------------------------------------------------------------------------------\ndef perform_count( df, group_cols, agg_name, agg_type='uint32', show_max=False, show_agg=True ):\n#------------------------------------------------------------------------------\n if show_agg:\n print( \"Aggregating by \", group_cols , '...' )\n gp = df[group_cols][group_cols].groupby(group_cols).size().rename(agg_name).to_frame().reset_index()\n df = df.merge(gp, on=group_cols, how='left')\n return (show_max_clean(df,gp,agg_name,agg_type,show_max))\n\n#------------------------------------------------------------------------------\ndef perform_countuniq( df, group_cols, counted, agg_name, agg_type='uint32', show_max=False, show_agg=True ):\n#------------------------------------------------------------------------------ \n if show_agg:\n print( \"Counting unique \", counted, \" by \", group_cols , '...' )\n gp = df[group_cols+[counted]].groupby(group_cols)[counted].nunique().reset_index().rename(columns={counted:agg_name})\n df = df.merge(gp, on=group_cols, how='left')\n return (show_max_clean(df,gp,agg_name,agg_type,show_max))\n\n#------------------------------------------------------------------------------ \ndef perform_cumcount( df, group_cols, counted, agg_name, agg_type='uint32', show_max=False, show_agg=True ):\n#------------------------------------------------------------------------------ \n if show_agg:\n print( \"Cumulative count by \", group_cols , '...' )\n gp = df[group_cols+[counted]].groupby(group_cols)[counted].cumcount()\n df[agg_name]=gp.values\n return (show_max_clean(df,gp,agg_name,agg_type,show_max))\n\n#------------------------------------------------------------------------------\ndef perform_mean( df, group_cols, counted, agg_name, agg_type='float32', show_max=False, show_agg=True ):\n#------------------------------------------------------------------------------ \n if show_agg:\n print( \"Calculating mean of \", counted, \" by \", group_cols , '...' )\n gp = df[group_cols+[counted]].groupby(group_cols)[counted].mean().reset_index().rename(columns={counted:agg_name})\n df = df.merge(gp, on=group_cols, how='left')\n return (show_max_clean(df,gp,agg_name,agg_type,show_max))\n\n#------------------------------------------------------------------------------\ndef perform_var( df, group_cols, counted, agg_name, agg_type='float32', show_max=False, show_agg=True ):\n#------------------------------------------------------------------------------ \n if show_agg:\n print( \"Calculating variance of \", counted, \" by \", group_cols , '...' )\n gp = df[group_cols+[counted]].groupby(group_cols)[counted].var().reset_index().rename(columns={counted:agg_name})\n df = df.merge(gp, on=group_cols, how='left')\n return (show_max_clean(df,gp,agg_name,agg_type,show_max))","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:58.878266Z","iopub.execute_input":"2024-04-21T15:50:58.878620Z","iopub.status.idle":"2024-04-21T15:50:58.907480Z","shell.execute_reply.started":"2024-04-21T15:50:58.878587Z","shell.execute_reply":"2024-04-21T15:50:58.906277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we call the routines for additional features calculation.","metadata":{}},{"cell_type":"code","source":"trainset = perform_countuniq( trainset, ['ip'], 'channel', 'X0', 'uint8', show_max=True ); gc.collect()\ntrainset = perform_cumcount( trainset, ['ip', 'device', 'os'], 'app', 'X1', show_max=True ); gc.collect()\ntrainset = perform_countuniq( trainset, ['ip', 'day'], 'hour', 'X2', 'uint8', show_max=True ); gc.collect()\ntrainset = perform_countuniq( trainset, ['ip'], 'app', 'X3', 'uint8', show_max=True ); gc.collect()\ntrainset = perform_countuniq( trainset, ['ip', 'app'], 'os', 'X4', 'uint8', show_max=True ); gc.collect()\ntrainset = perform_countuniq( trainset, ['ip'], 'device', 'X5', 'uint16', show_max=True ); gc.collect()\ntrainset = perform_countuniq( trainset, ['app'], 'channel', 'X6', show_max=True ); gc.collect()\ntrainset = perform_cumcount( trainset, ['ip'], 'os', 'X7', show_max=True ); gc.collect()\ntrainset = perform_countuniq( trainset, ['ip', 'device', 'os'], 'app', 'X8', show_max=True ); gc.collect()\ntrainset = perform_count( trainset, ['ip', 'day', 'hour'], 'ip_tcount', show_max=True ); gc.collect()\ntrainset = perform_count( trainset, ['ip', 'app'], 'ip_app_count', show_max=True ); gc.collect()\ntrainset = perform_count( trainset, ['ip', 'app', 'os'], 'ip_app_os_count', 'uint16', show_max=True ); gc.collect()\ntrainset = perform_var( trainset, ['ip', 'day', 'channel'], 'hour', 'ip_tchan_count', show_max=True ); gc.collect()\ntrainset = perform_var( trainset, ['ip', 'app', 'os'], 'hour', 'ip_app_os_var', show_max=True ); gc.collect()\ntrainset = perform_var( trainset, ['ip', 'app', 'channel'], 'day', 'ip_app_channel_var_day', show_max=True ); gc.collect()\ntrainset = perform_mean( trainset, ['ip', 'app', 'channel'], 'hour', 'ip_app_channel_mean_hour', show_max=True ); gc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:50:58.909030Z","iopub.execute_input":"2024-04-21T15:50:58.909444Z","iopub.status.idle":"2024-04-21T15:51:03.457137Z","shell.execute_reply.started":"2024-04-21T15:50:58.909407Z","shell.execute_reply":"2024-04-21T15:51:03.455993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainset.head(5)","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:51:03.458977Z","iopub.execute_input":"2024-04-21T15:51:03.459469Z","iopub.status.idle":"2024-04-21T15:51:03.490064Z","shell.execute_reply.started":"2024-04-21T15:51:03.459418Z","shell.execute_reply":"2024-04-21T15:51:03.488950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = ['X0','X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X7']\n\ni = 0\nt1 = trainset.loc[trainset['is_attributed'] != 0]\nt0 = trainset.loc[trainset['is_attributed'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax = plt.subplots(2,4,figsize=(14,8))\n\nfor feature in var:\n i += 1\n plt.subplot(2,4,i)\n sns.kdeplot(t1[feature], bw=0.5,label=\"is_attributed = 1\")\n sns.kdeplot(t0[feature], bw=0.5,label=\"is_attributed = 0\")\n plt.ylabel('Density plot', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:51:03.491399Z","iopub.execute_input":"2024-04-21T15:51:03.491737Z","iopub.status.idle":"2024-04-21T15:51:06.092901Z","shell.execute_reply.started":"2024-04-21T15:51:03.491705Z","shell.execute_reply":"2024-04-21T15:51:06.091652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = ['X8', 'ip_tcount', 'ip_app_count','ip_app_os_count', \n 'ip_tchan_count','ip_app_os_var','ip_app_channel_var_day', 'ip_app_channel_mean_hour']\n\ni = 0\nt1 = trainset.loc[trainset['is_attributed'] != 0]\nt0 = trainset.loc[trainset['is_attributed'] == 0]\n\nsns.set_style('whitegrid')\nplt.figure()\nfig, ax = plt.subplots(2,4,figsize=(14,8))\n\nfor feature in var:\n i += 1\n plt.subplot(2,4,i)\n sns.kdeplot(t1[feature], bw=0.5,label=\"is_attributed = 1\")\n sns.kdeplot(t0[feature], bw=0.5,label=\"is_attributed = 0\")\n plt.ylabel('Density plot', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:51:06.094139Z","iopub.execute_input":"2024-04-21T15:51:06.094486Z","iopub.status.idle":"2024-04-21T15:51:08.718979Z","shell.execute_reply.started":"2024-04-21T15:51:06.094442Z","shell.execute_reply":"2024-04-21T15:51:08.717889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data unbalance between train and test data\n\nLet's compare the distribution of the features in the train and test datasets.\n\n\nLet's start with the data about application, devide, operating system and channel.","metadata":{}},{"cell_type":"code","source":"var = ['app','device','os','channel']\n\n# Bar plot\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax = plt.subplots(1,4,figsize=(16,4))\ni = 0\nfor feature in var:\n i = i + 1\n plt.subplot(1,4,i)\n sns.kdeplot(trainset[feature], bw=0.5, label=\"train\")\n sns.kdeplot(testset[feature], bw=0.5, label=\"test\")\n plt.ylabel('Distribution', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n #plt.setp(labels, rotation=90)\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:51:08.720777Z","iopub.execute_input":"2024-04-21T15:51:08.721219Z","iopub.status.idle":"2024-04-21T15:51:12.962887Z","shell.execute_reply.started":"2024-04-21T15:51:08.721179Z","shell.execute_reply":"2024-04-21T15:51:12.961588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's continue with the data on time. First we will have to do the date and time extraction operation on testset data.","metadata":{}},{"cell_type":"code","source":"testset['year'] = pd.to_datetime(testset.click_time).dt.year\ntestset['month'] = pd.to_datetime(testset.click_time).dt.month\ntestset['day'] = pd.to_datetime(testset.click_time).dt.day\ntestset['hour'] = pd.to_datetime(testset.click_time).dt.hour\ntestset['min'] = pd.to_datetime(testset.click_time).dt.minute\ntestset['sec'] = pd.to_datetime(testset.click_time).dt.second\ntestset.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:51:12.964867Z","iopub.execute_input":"2024-04-21T15:51:12.965333Z","iopub.status.idle":"2024-04-21T15:51:52.520568Z","shell.execute_reply.started":"2024-04-21T15:51:12.965286Z","shell.execute_reply":"2024-04-21T15:51:52.519730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"testset.describe()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:51:52.521970Z","iopub.execute_input":"2024-04-21T15:51:52.522525Z","iopub.status.idle":"2024-04-21T15:52:02.111254Z","shell.execute_reply.started":"2024-04-21T15:51:52.522468Z","shell.execute_reply":"2024-04-21T15:52:02.109855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = ['day','hour']\n\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax = plt.subplots(1,2,figsize=(12,4))\ni = 0\nfor feature in var:\n i = i + 1\n plt.subplot(1,2,i)\n sns.kdeplot(trainset[feature], bw=0.5, label=\"train\")\n sns.kdeplot(testset[feature], bw=0.5, label=\"test\")\n plt.ylabel('Distribution', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n #plt.setp(labels, rotation=90)\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:52:02.113181Z","iopub.execute_input":"2024-04-21T15:52:02.113718Z","iopub.status.idle":"2024-04-21T15:52:03.866674Z","shell.execute_reply.started":"2024-04-21T15:52:02.113667Z","shell.execute_reply":"2024-04-21T15:52:03.865573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the day for the train data is for 6-9 November 2017, the test data is only from 10 November 2017. As well, the hours distribution are very different between train and test data.\n","metadata":{}},{"cell_type":"markdown","source":"Let's calculate the derived features as well for the test set.","metadata":{}},{"cell_type":"code","source":"testset = perform_countuniq( testset, ['ip'], 'channel', 'X0', 'uint8', show_max=True ); gc.collect()\ntestset = perform_cumcount( testset, ['ip', 'device', 'os'], 'app', 'X1', show_max=True ); gc.collect()\ntestset = perform_countuniq( testset, ['ip', 'day'], 'hour', 'X2', 'uint8', show_max=True ); gc.collect()\ntestset = perform_countuniq( testset, ['ip'], 'app', 'X3', 'uint8', show_max=True ); gc.collect()\ntestset = perform_countuniq( testset, ['ip', 'app'], 'os', 'X4', 'uint8', show_max=True ); gc.collect()\ntestset = perform_countuniq( testset, ['ip'], 'device', 'X5', 'uint16', show_max=True ); gc.collect()\ntestset = perform_countuniq( testset, ['app'], 'channel', 'X6', show_max=True ); gc.collect()\ntestset = perform_cumcount( testset, ['ip'], 'os', 'X7', show_max=True ); gc.collect()\ntestset = perform_countuniq( testset, ['ip', 'device', 'os'], 'app', 'X8', show_max=True ); gc.collect()\ntestset = perform_count( testset, ['ip', 'day', 'hour'], 'ip_tcount', show_max=True ); gc.collect()\ntestset = perform_count( testset, ['ip', 'app'], 'ip_app_count', show_max=True ); gc.collect()\ntestset = perform_count( testset, ['ip', 'app', 'os'], 'ip_app_os_count', 'uint16', show_max=True ); gc.collect()\ntestset = perform_var( testset, ['ip', 'day', 'channel'], 'hour', 'ip_tchan_count', show_max=True ); gc.collect()\ntestset = perform_var( testset, ['ip', 'app', 'os'], 'hour', 'ip_app_os_var', show_max=True ); gc.collect()\ntestset = perform_var( testset, ['ip', 'app', 'channel'], 'day', 'ip_app_channel_var_day', show_max=True ); gc.collect()\ntestset = perform_mean( testset, ['ip', 'app', 'channel'], 'hour', 'ip_app_channel_mean_hour', show_max=True ); gc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:52:03.868799Z","iopub.execute_input":"2024-04-21T15:52:03.869263Z","iopub.status.idle":"2024-04-21T15:56:44.404086Z","shell.execute_reply.started":"2024-04-21T15:52:03.869215Z","shell.execute_reply":"2024-04-21T15:56:44.403176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = ['X0','X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X7']\n\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax = plt.subplots(2,4,figsize=(14,8))\ni = 0\nfor feature in var:\n i = i + 1\n plt.subplot(2,4,i)\n sns.kdeplot(trainset[feature], bw=0.5, label=\"train\")\n sns.kdeplot(testset[feature], bw=0.5, label=\"test\")\n plt.ylabel('Distribution', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n #plt.setp(labels, rotation=90)\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:56:44.405586Z","iopub.execute_input":"2024-04-21T15:56:44.405941Z","iopub.status.idle":"2024-04-21T15:56:52.917841Z","shell.execute_reply.started":"2024-04-21T15:56:44.405906Z","shell.execute_reply":"2024-04-21T15:56:52.916586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"var = ['X8', 'ip_tcount', 'ip_app_count','ip_app_os_count', \n 'ip_tchan_count', 'ip_app_os_var','ip_app_channel_var_day', 'ip_app_channel_mean_hour']\n\nsns.set_style('whitegrid')\n\nplt.figure()\nfig, ax = plt.subplots(2,4,figsize=(14,8))\ni = 0\nfor feature in var:\n i = i + 1\n plt.subplot(2,4,i)\n sns.kdeplot(trainset[feature], bw=0.5, label=\"train\")\n sns.kdeplot(testset[feature], bw=0.5, label=\"test\")\n plt.ylabel('Distribution', fontsize=12)\n plt.xlabel(feature, fontsize=12)\n locs, labels = plt.xticks()\n #plt.setp(labels, rotation=90)\n plt.tick_params(axis='both', which='major', labelsize=12)\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:56:52.919467Z","iopub.execute_input":"2024-04-21T15:56:52.919878Z","iopub.status.idle":"2024-04-21T15:57:00.926092Z","shell.execute_reply.started":"2024-04-21T15:56:52.919840Z","shell.execute_reply":"2024-04-21T15:57:00.924995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"markdown","source":"## Prepare the model","metadata":{}},{"cell_type":"code","source":"start = datetime.now()\n\nlen_train = len(trainset)\ngc.collect()\n\nmost_freq_hours_in_test_data = [4, 5, 9, 10, 13, 14]\nleast_freq_hours_in_test_data = [6, 11, 15]\n\ndef prep_data( df ):\n \n df['hour'] = pd.to_datetime(df.click_time).dt.hour.astype('uint8')\n df['day'] = pd.to_datetime(df.click_time).dt.day.astype('uint8')\n df.drop(['click_time'], axis=1, inplace=True)\n gc.collect()\n \n df['in_test_hh'] = ( 3 \n - 2*df['hour'].isin( most_freq_hours_in_test_data ) \n - 1*df['hour'].isin( least_freq_hours_in_test_data ) ).astype('uint8')\n gp = df[['ip', 'day', 'in_test_hh', 'channel']].groupby(by=['ip', 'day', 'in_test_hh'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'nip_day_test_hh'})\n df = df.merge(gp, on=['ip','day','in_test_hh'], how='left')\n df.drop(['in_test_hh'], axis=1, inplace=True)\n df['nip_day_test_hh'] = df['nip_day_test_hh'].astype('uint32')\n del gp\n gc.collect()\n\n gp = df[['ip', 'day', 'hour', 'channel']].groupby(by=['ip', 'day', 'hour'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'nip_day_hh'})\n df = df.merge(gp, on=['ip','day','hour'], how='left')\n df['nip_day_hh'] = df['nip_day_hh'].astype('uint16')\n del gp\n gc.collect()\n \n gp = df[['ip', 'os', 'hour', 'channel']].groupby(by=['ip', 'os', 'hour'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'nip_hh_os'})\n df = df.merge(gp, on=['ip','os','hour'], how='left')\n df['nip_hh_os'] = df['nip_hh_os'].astype('uint16')\n del gp\n gc.collect()\n\n gp = df[['ip', 'app', 'hour', 'channel']].groupby(by=['ip', 'app', 'hour'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'nip_hh_app'})\n df = df.merge(gp, on=['ip','app','hour'], how='left')\n df['nip_hh_app'] = df['nip_hh_app'].astype('uint16')\n del gp\n gc.collect()\n\n gp = df[['ip', 'device', 'hour', 'channel']].groupby(by=['ip', 'device', 'hour'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'nip_hh_dev'})\n df = df.merge(gp, on=['ip','device','hour'], how='left')\n df['nip_hh_dev'] = df['nip_hh_dev'].astype('uint32')\n del gp\n gc.collect()\n\n df.drop( ['ip','day'], axis=1, inplace=True )\n gc.collect()\n return df","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:57:00.927964Z","iopub.execute_input":"2024-04-21T15:57:00.928718Z","iopub.status.idle":"2024-04-21T15:57:01.073195Z","shell.execute_reply.started":"2024-04-21T15:57:00.928669Z","shell.execute_reply":"2024-04-21T15:57:01.071999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainset = prep_data(trainset)\ngc.collect()\n\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'binary',\n 'metric':'auc',\n 'learning_rate': 0.1,\n 'num_leaves': 9, # we should let it be smaller than 2^(max_depth)\n 'max_depth': 5, # -1 means no limit\n 'min_child_samples': 100, # Minimum number of data need in a child(min_data_in_leaf)\n 'max_bin': 100, # Number of bucketed bin for feature values\n 'subsample': 0.9, # Subsample ratio of the training instance.\n 'subsample_freq': 1, # frequence of subsample, <=0 means no enable\n 'colsample_bytree': 0.7, # Subsample ratio of columns when constructing each tree.\n 'min_child_weight': 0, # Minimum sum of instance weight(hessian) needed in a child(leaf)\n 'min_split_gain': 0, # lambda_l1, lambda_l2 and min_gain_to_split to regularization\n 'nthread': 8,\n 'verbose': 0,\n 'scale_pos_weight':99.7, # because training data is extremely unbalanced \n }\n\ntarget = 'is_attributed'\npredictors = ['app','device','os', 'channel', 'hour', 'nip_day_test_hh', 'nip_day_hh', 'nip_hh_os', 'nip_hh_app', 'nip_hh_dev']\ncategorical = ['app', 'device', 'os', 'channel', 'hour']\n","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:57:01.077559Z","iopub.execute_input":"2024-04-21T15:57:01.077951Z","iopub.status.idle":"2024-04-21T15:57:02.901894Z","shell.execute_reply.started":"2024-04-21T15:57:01.077912Z","shell.execute_reply":"2024-04-21T15:57:02.900634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train the model","metadata":{}},{"cell_type":"code","source":"if VALIDATE:\n\n train_df, val_df = train_test_split(trainset, test_size=VALID_SIZE, random_state=RANDOM_STATE, shuffle=True )\n \n dtrain = lgb.Dataset(train_df[predictors].values, \n label=train_df[target].values,\n feature_name=predictors,\n categorical_feature=categorical)\n del train_df\n gc.collect()\n\n dvalid = lgb.Dataset(val_df[predictors].values,\n label=val_df[target].values,\n feature_name=predictors,\n categorical_feature=categorical)\n del val_df\n gc.collect()\n\n evals_results = {}\n\n model = lgb.train(params, \n dtrain, \n valid_sets=[dtrain, dvalid], \n valid_names=['train','valid'], \n evals_result=evals_results, \n num_boost_round=MAX_ROUNDS,\n early_stopping_rounds=EARLY_STOP,\n verbose_eval=50, \n feval=None)\n\n del dvalid\n \nelif VALIDATE_KFOLDS:\n kf = KFold(n_splits = NUMBER_KFOLDS, random_state = RANDOM_STATE, shuffle = True)\n for train_index, test_index in kf.split(trainset):\n train_X, valid_X = trainset.iloc[train_index], trainset.iloc[test_index]\n\n dtrain = lgb.Dataset(train_X[predictors].values, label=train_X[target].values,\n feature_name=predictors, categorical_feature=categorical)\n \n dvalid = lgb.Dataset(valid_X[predictors].values, label=valid_X[target].values,\n feature_name=predictors, categorical_feature=categorical)\n \n evals_results = {}\n model = lgb.train(params, \n dtrain, \n valid_sets=[dtrain, dvalid], \n valid_names=['train','valid'], \n evals_result=evals_results, \n num_boost_round=MAX_ROUNDS,\n early_stopping_rounds=EARLY_STOP,\n verbose_eval=50, \n feval=None)\n \nelse:\n\n gc.collect()\n dtrain = lgb.Dataset(train_df[predictors].values, label=train_df[target].values,\n feature_name=predictors,\n categorical_feature=categorical\n )\n del train_df\n gc.collect()\n\n evals_results = {}\n\n model = lgb.train(params, \n dtrain, \n valid_sets=[dtrain], \n valid_names=['train'], \n evals_result=evals_results, \n num_boost_round=OPT_ROUNDS,\n verbose_eval=50,\n feval=None)\n \ndel dtrain\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:57:02.903923Z","iopub.execute_input":"2024-04-21T15:57:02.904286Z","iopub.status.idle":"2024-04-21T15:57:09.740219Z","shell.execute_reply.started":"2024-04-21T15:57:02.904251Z","shell.execute_reply":"2024-04-21T15:57:09.737859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction and submission","metadata":{}},{"cell_type":"code","source":"test_cols = ['ip','app','device','os', 'channel', 'click_time', 'click_id']\n\ntest_df = prep_data(testset)\ngc.collect()\n\nsub = pd.DataFrame()\nsub['click_id'] = test_df['click_id']\nsub['is_attributed'] = model.predict(test_df[predictors])\nsub.to_csv(output_filename, index=False, float_format='%.9f')\n","metadata":{"execution":{"iopub.status.busy":"2024-04-21T15:57:09.742571Z","iopub.execute_input":"2024-04-21T15:57:09.743361Z","iopub.status.idle":"2024-04-21T16:01:13.001541Z","shell.execute_reply.started":"2024-04-21T15:57:09.743299Z","shell.execute_reply":"2024-04-21T16:01:13.000160Z"},"trusted":true},"execution_count":null,"outputs":[]}]}