{ "id": 56262, "title": "My brief summary,a mainly NN based solution(3th)", "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/56262", "author_name": "bestfitting", "post_date": "2018-05-08T04:24:32.118000", "votes": 265, "comment_count": 108, "views": 0, "content": "

Congrats to all the winners(['flowlight', 'komaki'].shuffle(),PPP is already in use) and all the kagglers who have worked hard and have learned a lot of from this competition.

\n\n

Thanks to TalkingData and Kaggle for such an interesting competition.

\n\n

Here is the summary of my solution,it is mainly based on NN models.

\n\n

It was very hard for me to choose among Landmark Competitions and TalkingData's,so I started them at the same time,when I found I can get decent NN models, I focus on this one,it is always interesting to solve a problem other than CV/NLP using NN models.

\n\n

My models mainly based on 23 features,by using these features,my single LGBM model scored 0.9817 on public LB,it is not a good one compared to other kagglers,this is the first time that I used LGBM in kaggle competition indeed,so there are a lot to learn from you!

\n\n

I designed NN models based on those 23 features,and prepared the features to fed them into network carefully[NA,out of vocabulary,log,scale],my NN model can reach 0.9820 on public LB.As we all know, the click delta is important,so I fed deltas of last 5 and next 5 click_times to the network and designed a model with RNN cell to find the patterns of the click series,my model can reach 0.9821 on public LB and 0.9830 on private LB.

\n\n

Then I designed different NN models to add diversities,they are very simple,for example,adding some res-links to dense layers.I don’t know the single model performance of these 4 models because I judged them only by diversities.

\n\n

After have ensembled my NN models and LGBM models by weighted average,I can get 0.9827 on public LB and 0.9835 on private LB.

\n\n

Then,I predicted full set of train data on my n-fold models,it’s a little time consuming,but it’s a relatively small dataset for me when compared to other datasets I have met,I can train and predict a fold of my model in 2.x hours on a 1080i GPU.

\n\n

I trained second level NN models using predictions from the whole train and test dataset,and added some group by features based on IP,app-os-channel,my ensemble score improved to 0.9833 on public LB and 0.9840 on private LB,which is a huge improvement.

\n\n

This improvement happened at 30 hours before the competition end,I hope I could get such an improvement two days earlier,because I had no time to solve some limitations/weaknesses of my NN models,as I simulated on part of the data,I found NN models may led to a small drop(0.0003 also) on private dataset in some situation,which can explain my minor drop on private LB.So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :) I had thought I will be very happy if I am still in top 5 when the private LB revealed

\n\n

Sorry for not so detailed,I feel quite sleepy now.

\n\n

------------------------------------Some Details------------------------------
\nThe features:

\n\n
channel                                  1011\nos                                        544\nhour                                      472\napp                                       468\nip_app_os_device_day_click_time_next_1     320\napp_channel_os_mean_is_attributed         189\nip_app_mean_is_attributed                 124\nip_app_os_device_day_click_time_next_2     120\nip_os_device_count_click_id               113\nip_var_hour                                94\nip_day_hour_count_click_id                 91\nip_mean_is_attributed                      74\nip_count_click_id                          73\nip_app_os_device_day_click_time_lag1       67\napp_mean_is_attributed                     67\nip_nunique_os_device                       65\nip_nunique_app                             63\nip_nunique_os                              51\nip_nunique_app_channel                     49\nip_os_device_mean_is_attributed            46\ndevice                                     41\napp_channel_os_count_click_id              37\nip_hour_mean_is_attributed                 21\n
\n\n

a simple GRU network:

\n\n
class GRU_V0a():\n    def __init__(self, **kw):\n        super(GRU_V0a, self).__init__(**kw)\n    self.categorical=['app', 'device', 'os', 'channel', 'hour']\n    self.continous=[col for col in features if col not in self.categorical]\n        self.categorical_num = {\n            'app': (769, 16),\n            'device': (4228, 16),\n            'os': (957, 16),\n            'channel': (501, 8),\n            'hour': (24, 8),\n        }\n    def build_model(self):\n        categorial_inp = Input(shape=(len(self.categorical),))\n        cat_embeds = []\n        for idx, col in enumerate(self.categorical):\n            x = Lambda(lambda x: x[:, idx,None])(categorial_inp)\n            x = Embedding(self.categorical_num[col][0], self.categorical_num[col][1],input_length=1)(x)\n            cat_embeds.append(x)\n        embeds = concatenate(cat_embeds, axis=2)\n        embeds = GaussianDropout(0.2)(embeds)\n        continous_inp = Input(shape=(len(self.continous),))\n        cx = Reshape([1,len(self.continous)])(continous_inp)\n        x = concatenate([embeds, cx], axis=2)\n        x = CuDNNGRU(128)(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(64)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(32)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.05)(x)\n        outp = Dense(1, activation='sigmoid')(x)\n        model = Model(inputs=[categorial_inp, continous_inp], output=outp)\n        print(model.summary())\n        return model\n
\n\n

Thanks @aharless publish the code which I used as a reference,such as GaussianDropout.
\nIt's really simple? :)And the devil is in detail:
\nTo make NN model work,preprocess is very important.
\nFill-NA:Fill max value for delta feature.Fill mean for other feature.
\nLog of delta features,part of count features when the values are large and all the nunique features.
\nStandardScale

\n\n

It is not a tranditional RNN,the input_lenght=1 indeed,so I just used the function of CudnnGRU to get pattern of clicks which can be expressed like the following:

\n\n
zx=sigmoid(K.dot(Wz,X)\nhx=tanh(K.dot(W,X)\nh=zx*hx\n
\n\n

As we want to know the theory behind the stucture,and I have no time to prove it or write a paper,\nwe can have a look at the paper by Google:Searching for Activation Functions Swish:x · σ(βx), where σ(z) = (1 + exp(−z))−1,we can get some ideas to explain my setting.

\n\n

As to private score estimation,it's always a interesting part of my competition :).
\nThere is not certain method to do so,I just bear in mind:
\nDistribution variances lead to score variances.
\nFor example,in this competition,some category values in test-set are not in trainset,so I changed the same ratio of category value of validation set to values unseen in trainset.And the App19 is a very important app with high ratios of download and is imbalance in train and test-set.What's more,the ratio is differenct between the public and private set,so I tried to keep the ratio of my validatition as test set...We can also use a submission which I believe it is stable as True label,and caculate AUC based on it,if the public and private score as close,then we can believe it too. \n
\nThanks for all the congrats to me ,I will upvote your comment and will not reply to everyone to save space of this page.\n

", "messages": [ { "id": 325054, "postDate": "2018-05-08T04:24:32.120Z", "content": "

Congrats to all the winners(['flowlight', 'komaki'].shuffle(),PPP is already in use) and all the kagglers who have worked hard and have learned a lot of from this competition.

\n\n

Thanks to TalkingData and Kaggle for such an interesting competition.

\n\n

Here is the summary of my solution,it is mainly based on NN models.

\n\n

It was very hard for me to choose among Landmark Competitions and TalkingData's,so I started them at the same time,when I found I can get decent NN models, I focus on this one,it is always interesting to solve a problem other than CV/NLP using NN models.

\n\n

My models mainly based on 23 features,by using these features,my single LGBM model scored 0.9817 on public LB,it is not a good one compared to other kagglers,this is the first time that I used LGBM in kaggle competition indeed,so there are a lot to learn from you!

\n\n

I designed NN models based on those 23 features,and prepared the features to fed them into network carefully[NA,out of vocabulary,log,scale],my NN model can reach 0.9820 on public LB.As we all know, the click delta is important,so I fed deltas of last 5 and next 5 click_times to the network and designed a model with RNN cell to find the patterns of the click series,my model can reach 0.9821 on public LB and 0.9830 on private LB.

\n\n

Then I designed different NN models to add diversities,they are very simple,for example,adding some res-links to dense layers.I don’t know the single model performance of these 4 models because I judged them only by diversities.

\n\n

After have ensembled my NN models and LGBM models by weighted average,I can get 0.9827 on public LB and 0.9835 on private LB.

\n\n

Then,I predicted full set of train data on my n-fold models,it’s a little time consuming,but it’s a relatively small dataset for me when compared to other datasets I have met,I can train and predict a fold of my model in 2.x hours on a 1080i GPU.

\n\n

I trained second level NN models using predictions from the whole train and test dataset,and added some group by features based on IP,app-os-channel,my ensemble score improved to 0.9833 on public LB and 0.9840 on private LB,which is a huge improvement.

\n\n

This improvement happened at 30 hours before the competition end,I hope I could get such an improvement two days earlier,because I had no time to solve some limitations/weaknesses of my NN models,as I simulated on part of the data,I found NN models may led to a small drop(0.0003 also) on private dataset in some situation,which can explain my minor drop on private LB.So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :) I had thought I will be very happy if I am still in top 5 when the private LB revealed

\n\n

Sorry for not so detailed,I feel quite sleepy now.

\n\n

------------------------------------Some Details------------------------------
\nThe features:

\n\n
channel                                  1011\nos                                        544\nhour                                      472\napp                                       468\nip_app_os_device_day_click_time_next_1     320\napp_channel_os_mean_is_attributed         189\nip_app_mean_is_attributed                 124\nip_app_os_device_day_click_time_next_2     120\nip_os_device_count_click_id               113\nip_var_hour                                94\nip_day_hour_count_click_id                 91\nip_mean_is_attributed                      74\nip_count_click_id                          73\nip_app_os_device_day_click_time_lag1       67\napp_mean_is_attributed                     67\nip_nunique_os_device                       65\nip_nunique_app                             63\nip_nunique_os                              51\nip_nunique_app_channel                     49\nip_os_device_mean_is_attributed            46\ndevice                                     41\napp_channel_os_count_click_id              37\nip_hour_mean_is_attributed                 21\n
\n\n

a simple GRU network:

\n\n
class GRU_V0a():\n    def __init__(self, **kw):\n        super(GRU_V0a, self).__init__(**kw)\n    self.categorical=['app', 'device', 'os', 'channel', 'hour']\n    self.continous=[col for col in features if col not in self.categorical]\n        self.categorical_num = {\n            'app': (769, 16),\n            'device': (4228, 16),\n            'os': (957, 16),\n            'channel': (501, 8),\n            'hour': (24, 8),\n        }\n    def build_model(self):\n        categorial_inp = Input(shape=(len(self.categorical),))\n        cat_embeds = []\n        for idx, col in enumerate(self.categorical):\n            x = Lambda(lambda x: x[:, idx,None])(categorial_inp)\n            x = Embedding(self.categorical_num[col][0], self.categorical_num[col][1],input_length=1)(x)\n            cat_embeds.append(x)\n        embeds = concatenate(cat_embeds, axis=2)\n        embeds = GaussianDropout(0.2)(embeds)\n        continous_inp = Input(shape=(len(self.continous),))\n        cx = Reshape([1,len(self.continous)])(continous_inp)\n        x = concatenate([embeds, cx], axis=2)\n        x = CuDNNGRU(128)(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(64)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(32)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.05)(x)\n        outp = Dense(1, activation='sigmoid')(x)\n        model = Model(inputs=[categorial_inp, continous_inp], output=outp)\n        print(model.summary())\n        return model\n
\n\n

Thanks @aharless publish the code which I used as a reference,such as GaussianDropout.
\nIt's really simple? :)And the devil is in detail:
\nTo make NN model work,preprocess is very important.
\nFill-NA:Fill max value for delta feature.Fill mean for other feature.
\nLog of delta features,part of count features when the values are large and all the nunique features.
\nStandardScale

\n\n

It is not a tranditional RNN,the input_lenght=1 indeed,so I just used the function of CudnnGRU to get pattern of clicks which can be expressed like the following:

\n\n
zx=sigmoid(K.dot(Wz,X)\nhx=tanh(K.dot(W,X)\nh=zx*hx\n
\n\n

As we want to know the theory behind the stucture,and I have no time to prove it or write a paper,\nwe can have a look at the paper by Google:Searching for Activation Functions Swish:x · σ(βx), where σ(z) = (1 + exp(−z))−1,we can get some ideas to explain my setting.

\n\n

As to private score estimation,it's always a interesting part of my competition :).
\nThere is not certain method to do so,I just bear in mind:
\nDistribution variances lead to score variances.
\nFor example,in this competition,some category values in test-set are not in trainset,so I changed the same ratio of category value of validation set to values unseen in trainset.And the App19 is a very important app with high ratios of download and is imbalance in train and test-set.What's more,the ratio is differenct between the public and private set,so I tried to keep the ratio of my validatition as test set...We can also use a submission which I believe it is stable as True label,and caculate AUC based on it,if the public and private score as close,then we can believe it too. \n
\nThanks for all the congrats to me ,I will upvote your comment and will not reply to everyone to save space of this page.\n

", "rawMarkdown": "Congrats to all the winners(['flowlight', 'komaki'].shuffle(),PPP is already in use) and all the kagglers who have worked hard and have learned a lot of from this competition.
\n\nThanks to TalkingData and Kaggle for such an interesting competition.
\n\nHere is the summary of my solution,it is mainly based on NN models.
\n\nIt was very hard for me to choose among Landmark Competitions and TalkingData's,so I started them at the same time,when I found I can get decent NN models, I focus on this one,it is always interesting to solve a problem other than CV/NLP using NN models.
\n\nMy models mainly based on 23 features,by using these features,my single LGBM model scored 0.9817 on public LB,it is not a good one compared to other kagglers,this is the first time that I used LGBM in kaggle competition indeed,so there are a lot to learn from you!
\n\nI designed NN models based on those 23 features,and prepared the features to fed them into network carefully[NA,out of vocabulary,log,scale],my NN model can reach 0.9820 on public LB.As we all know, the click delta is important,so I fed deltas of last 5 and next 5 click_times to the network and designed a model with RNN cell to find the patterns of the click series,my model can reach 0.9821 on public LB and 0.9830 on private LB.
\n\nThen I designed different NN models to add diversities,they are very simple,for example,adding some res-links to dense layers.I don’t know the single model performance of these 4 models because I judged them only by diversities.
\n\nAfter have ensembled my NN models and LGBM models by weighted average,I can get 0.9827 on public LB and 0.9835 on private LB.
\n\nThen,I predicted full set of train data on my n-fold models,it’s a little time consuming,but it’s a relatively small dataset for me when compared to other datasets I have met,I can train and predict a fold of my model in 2.x hours on a 1080i GPU.
\n\nI trained second level NN models using predictions from the whole train and test dataset,and added some group by features based on IP,app-os-channel,my ensemble score improved to 0.9833 on public LB and 0.9840 on private LB,which is a huge improvement.
\n\nThis improvement happened at 30 hours before the competition end,I hope I could get such an improvement two days earlier,because I had no time to solve some limitations/weaknesses of my NN models,as I simulated on part of the data,I found NN models may led to a small drop(0.0003 also) on private dataset in some situation,which can explain my minor drop on private LB.So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :) I had thought I will be very happy if I am still in top 5 when the private LB revealed
\n\nSorry for not so detailed,I feel quite sleepy now.
\n\n------------------------------------Some Details------------------------------
\nThe features:
\n
channel                                  1011\nos                                        544\nhour                                      472\napp                                       468\nip_app_os_device_day_click_time_next_1     320\napp_channel_os_mean_is_attributed         189\nip_app_mean_is_attributed                 124\nip_app_os_device_day_click_time_next_2     120\nip_os_device_count_click_id               113\nip_var_hour                                94\nip_day_hour_count_click_id                 91\nip_mean_is_attributed                      74\nip_count_click_id                          73\nip_app_os_device_day_click_time_lag1       67\napp_mean_is_attributed                     67\nip_nunique_os_device                       65\nip_nunique_app                             63\nip_nunique_os                              51\nip_nunique_app_channel                     49\nip_os_device_mean_is_attributed            46\ndevice                                     41\napp_channel_os_count_click_id              37\nip_hour_mean_is_attributed                 21\n
\n\na simple GRU network:
\n
class GRU_V0a():\n    def __init__(self, **kw):\n        super(GRU_V0a, self).__init__(**kw)\n\tself.categorical=['app', 'device', 'os', 'channel', 'hour']\n\tself.continous=[col for col in features if col not in self.categorical]\n        self.categorical_num = {\n            'app': (769, 16),\n            'device': (4228, 16),\n            'os': (957, 16),\n            'channel': (501, 8),\n            'hour': (24, 8),\n        }\n    def build_model(self):\n        categorial_inp = Input(shape=(len(self.categorical),))\n        cat_embeds = []\n        for idx, col in enumerate(self.categorical):\n            x = Lambda(lambda x: x[:, idx,None])(categorial_inp)\n            x = Embedding(self.categorical_num[col][0], self.categorical_num[col][1],input_length=1)(x)\n            cat_embeds.append(x)\n        embeds = concatenate(cat_embeds, axis=2)\n        embeds = GaussianDropout(0.2)(embeds)\n        continous_inp = Input(shape=(len(self.continous),))\n        cx = Reshape([1,len(self.continous)])(continous_inp)\n        x = concatenate([embeds, cx], axis=2)\n        x = CuDNNGRU(128)(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(64)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(32)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.05)(x)\n        outp = Dense(1, activation='sigmoid')(x)\n        model = Model(inputs=[categorial_inp, continous_inp], output=outp)\n        print(model.summary())\n        return model\n
\nThanks [@aharless][1] publish the code which I used as a reference,such as GaussianDropout.
\nIt's really simple? :)And the devil is in detail:
\nTo make NN model work,preprocess is very important.
\nFill-NA:Fill max value for delta feature.Fill mean for other feature.
\nLog of delta features,part of count features when the values are large and all the nunique features.
\nStandardScale
\n\nIt is not a tranditional RNN,the input_lenght=1 indeed,so I just used the function of CudnnGRU to get pattern of clicks which can be expressed like the following:\n
zx=sigmoid(K.dot(Wz,X)\nhx=tanh(K.dot(W,X)\nh=zx*hx\n
\n\nAs we want to know the theory behind the stucture,and I have no time to prove it or write a paper,\nwe can have a look at the paper by Google:[Searching for Activation Functions][2] Swish:x · σ(βx), where σ(z) = (1 + exp(−z))−1,we can get some ideas to explain my setting.\n\nAs to private score estimation,it's always a interesting part of my competition :).
\nThere is not certain method to do so,I just bear in mind:
\nDistribution variances lead to score variances.
\nFor example,in this competition,some category values in test-set are not in trainset,so I changed the same ratio of category value of validation set to values unseen in trainset.And the App19 is a very important app with high ratios of download and is imbalance in train and test-set.What's more,the ratio is differenct between the public and private set,so I tried to keep the ratio of my validatition as test set...We can also use a submission which I believe it is stable as True label,and caculate AUC based on it,if the public and private score as close,then we can believe it too. \n
\nThanks for all the congrats to me ,I will upvote your comment and will not reply to everyone to save space of this page.\n
\n\n\n [1]: https://www.kaggle.com/aharless/gpu-nn-validation-more-features/code\n [2]: http://Searching%20for%20Activation%20Functions", "votes": 265 }, { "id": 325195, "postDate": "2018-05-08T08:02:58.430Z", "content": "

wow brilliant, never see RNN performed so well

", "rawMarkdown": "wow brilliant, never see RNN performed so well", "votes": 6 }, { "id": 325193, "postDate": "2018-05-08T08:01:41.843Z", "content": "

Congrats! I am very impressed by the performances of your NN models. I wonder what are the detailed structures of your NNs? You have mentioned RNN and the res-links so I am very curious about the details.

", "rawMarkdown": "Congrats! I am very impressed by the performances of your NN models. I wonder what are the detailed structures of your NNs? You have mentioned RNN and the res-links so I am very curious about the details. ", "votes": 6 }, { "id": 325317, "postDate": "2018-05-08T09:50:36.780Z", "content": "

Congrats @bestfitting and thanks for sharing the approach.

", "rawMarkdown": "Congrats @bestfitting and thanks for sharing the approach. ", "votes": 3 }, { "id": 328716, "postDate": "2018-05-15T00:13:01.303Z", "content": "

Truly good stuff!!

", "rawMarkdown": "Truly good stuff!!", "votes": 1 }, { "id": 328404, "postDate": "2018-05-14T08:16:33.053Z", "content": "

Thanks for sharing.\nIt's too great resource to learn.

", "rawMarkdown": "Thanks for sharing.\nIt's too great resource to learn.", "votes": 1 }, { "id": 327957, "postDate": "2018-05-13T03:09:34.453Z", "content": "

Congratulations and thank you so much for sharing! \nI have a little question when you talk about preprocessing for NN, what make you decide to use \n'Fill-NA:Fill max value for delta feature.Fill mean for other feature'\ninstead of filling mean for all features?

", "rawMarkdown": "Congratulations and thank you so much for sharing! \nI have a little question when you talk about preprocessing for NN, what make you decide to use \n'Fill-NA:Fill max value for delta feature.Fill mean for other feature'\ninstead of filling mean for all features?", "votes": 1, "replies": [ { "id": 327961, "postDate": "2018-05-13T03:15:14.510Z", "content": "

When we use NN models in such kind of problem,the data preprocess is very important.
\nIf there is no click from a clicktime,the next_click_feature will be NA,we can not fill mean or zero to NA,max delta is more reasonable.
\nOn the other side,for other features fill mean to NA item is more reasonable.

", "rawMarkdown": "When we use NN models in such kind of problem,the data preprocess is very important.
\nIf there is no click from a clicktime,the next_click_feature will be NA,we can not fill mean or zero to NA,max delta is more reasonable.
\nOn the other side,for other features fill mean to NA item is more reasonable.
\n", "votes": 2 }, { "id": 327967, "postDate": "2018-05-13T03:19:06.113Z", "content": "

Thanks! Learn a lot from you.

", "rawMarkdown": "Thanks! Learn a lot from you." } ] }, { "id": 327272, "postDate": "2018-05-11T06:49:42.290Z", "content": "

cool

", "rawMarkdown": "cool", "votes": 1 }, { "id": 326626, "postDate": "2018-05-10T04:26:12.007Z", "content": "

Thanks for sharing your solution, I finally lost to you, but really enjoyed competing with you :)
\nbtw, I couldn't understand \"I trained second level NN models using predictions from the whole train and test dataset\".\nDoes it mean you did pseudo-labelling, or is it just Stacking? could you please tell me what you use for training data and target?

", "rawMarkdown": "Thanks for sharing your solution, I finally lost to you, but really enjoyed competing with you :)
\nbtw, I couldn't understand \"I trained second level NN models using predictions from the whole train and test dataset\".\nDoes it mean you did pseudo-labelling, or is it just Stacking? could you please tell me what you use for training data and target?", "votes": 1, "replies": [ { "id": 326742, "postDate": "2018-05-10T08:59:13.683Z", "content": "

@mamasinkgs,you've a great job,although we can not get good position only by luck,we do need it sometimes.As there are so many competitions on kaggle,you can get better rank for sure.
\nIt's stacking,I tried to avoid use terms like OOF but I found that I should use them :)
\nplease refer to here

", "rawMarkdown": "@mamasinkgs,you've a great job,although we can not get good position only by luck,we do need it sometimes.As there are so many competitions on kaggle,you can get better rank for sure.
\nIt's stacking,I tried to avoid use terms like OOF but I found that I should use them :)
\nplease refer to [here][1]\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56262#326739", "votes": 1 }, { "id": 326823, "postDate": "2018-05-10T11:41:50.530Z", "content": "

I got what you mean.
\nThanks @bestfitting, I'll try harder and get better rank :)

", "rawMarkdown": "I got what you mean.
\nThanks @bestfitting, I'll try harder and get better rank :)", "votes": 2 }, { "id": 328913, "postDate": "2018-05-15T10:59:10.600Z", "content": "

btw, I'm curious about how much the score will become better if we combine the best submission. I think your unique methods will give a big score up when ensembling. could you upload your best submission if possible? 1st & 5th & 6th has already uploaded. https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56423

", "rawMarkdown": "btw, I'm curious about how much the score will become better if we combine the best submission. I think your unique methods will give a big score up when ensembling. could you upload your best submission if possible? 1st & 5th & 6th has already uploaded. https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56423" } ] }, { "id": 326087, "postDate": "2018-05-09T08:52:22.687Z", "content": "

Very thanks for your sharing。I have some small questions and look forward to your rely。How do you decide your NN models hyperparameters and your NN architecture？

", "rawMarkdown": "Very thanks for your sharing。I have some small questions and look forward to your rely。How do you decide your NN models hyperparameters and your NN architecture？", "votes": 1, "replies": [ { "id": 326567, "postDate": "2018-05-09T23:37:57.753Z", "content": "

As I said in an interview by kaggle recently,I always try to find related solutions and papers and learn from them and then modify the setting and parameters based on experiments,by doing so, I can save a lot of time,we can try to stand on the shoulders of gaints.I read papers everyday and implement some of them as a practice and get better understanding of them , when a competition started I will select some of them to use and find more related.

", "rawMarkdown": "As I said in an interview by kaggle recently,I always try to find related solutions and papers and learn from them and then modify the setting and parameters based on experiments,by doing so, I can save a lot of time,we can try to stand on the shoulders of gaints.I read papers everyday and implement some of them as a practice and get better understanding of them , when a competition started I will select some of them to use and find more related.", "votes": 17 }, { "id": 326622, "postDate": "2018-05-10T04:00:59.150Z", "rawMarkdown": "", "isDeleted": true }, { "id": 326624, "postDate": "2018-05-10T04:16:04.523Z", "content": "

@the1own not sure what repo is bestfitting using but I believe https://arxiv.org/list/cs.AI/recent to be the most comprehensive one. There's also this one https://towardsdatascience.com/ but that's not necessary a papers repo but more like \"trying to understand\" the papers repo ;-)

", "rawMarkdown": "@the1own not sure what repo is bestfitting using but I believe https://arxiv.org/list/cs.AI/recent to be the most comprehensive one. There's also this one https://towardsdatascience.com/ but that's not necessary a papers repo but more like \"trying to understand\" the papers repo ;-)", "votes": 1 }, { "id": 326625, "postDate": "2018-05-10T04:17:15.520Z", "content": "

＠the1owl,do you like this one?\nWhen we find a good papers,we can follow the papers it refered,and the papers list as best papers of a conference is a good place to find the good papers.:)

", "rawMarkdown": "＠the1owl,[do you like this one?][1]\nWhen we find a good papers,we can follow the papers it refered,and the papers list as best papers of a conference is a good place to find the good papers.:)\n [1]: https://github.com/dennybritz/deeplearning-papernotes", "votes": 6 }, { "id": 326660, "postDate": "2018-05-10T05:37:53.783Z", "content": "

@bestfitting Thanks a lot for this share! I think that it solves the main problem of arxiv ... too many papers published everyday :D

", "rawMarkdown": "@bestfitting Thanks a lot for this share! I think that it solves the main problem of arxiv ... too many papers published everyday :D", "votes": 1 }, { "id": 326809, "postDate": "2018-05-10T11:21:30.550Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 326015, "postDate": "2018-05-09T06:59:13.603Z", "content": "

Congratulations

", "rawMarkdown": "Congratulations", "votes": 1 }, { "id": 325962, "postDate": "2018-05-09T05:58:39.393Z", "content": "

Congratulations on your 3rd place. \nI had a feeling that NN will not do so well in this problem. But it is nice to see it doing so well. Thanks for sharing your approach.

", "rawMarkdown": "Congratulations on your 3rd place. \nI had a feeling that NN will not do so well in this problem. But it is nice to see it doing so well. Thanks for sharing your approach. ", "votes": 1 }, { "id": 325938, "postDate": "2018-05-09T05:03:14.467Z", "content": "

Thank you for your sharing. I did my last submission on the train to Changsha. Just happened to find that you work at the city.

", "rawMarkdown": "Thank you for your sharing. I did my last submission on the train to Changsha. Just happened to find that you work at the city.", "votes": 1 }, { "id": 325906, "postDate": "2018-05-09T03:21:48.950Z", "content": "

Congratulations. I made a plan to use RNN to realise it at first, but failed. Could you opensource your program and share some more experience? Thanks.

", "rawMarkdown": "Congratulations. I made a plan to use RNN to realise it at first, but failed. Could you opensource your program and share some more experience? Thanks.", "votes": 1, "replies": [ { "id": 326564, "postDate": "2018-05-09T23:29:15.743Z", "content": "

Thanks,you can refer to the network structure I posted and used the setting I mentioned　here,as I must am very busy in next two weeks I can not provide full set of my codes. sorry.

", "rawMarkdown": "Thanks,you can refer to the network structure I posted and used the setting I mentioned　[here][1],as I must am very busy in next two weeks I can not provide full set of my codes. sorry.\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56262#325825" }, { "id": 326803, "postDate": "2018-05-10T11:12:17.053Z", "content": "

OK，thanks for your reply. I will try again. If needed, look forward to your help. Thank you.

", "rawMarkdown": "OK，thanks for your reply. I will try again. If needed, look forward to your help. Thank you." } ] }, { "id": 325878, "postDate": "2018-05-09T02:12:36.480Z", "content": "

wow brilliant. I have a question about target encoder: how do you calculate the features like \"ip_app_mean_is_attributed\"? \n some methods like calculate target mean of yesterday or some other skills?? thanks

", "rawMarkdown": " wow brilliant. I have a question about target encoder: how do you calculate the features like \"ip_app_mean_is_attributed\"? \n some methods like calculate target mean of yesterday or some other skills?? thanks ", "votes": 1, "replies": [ { "id": 326562, "postDate": "2018-05-09T23:24:23.837Z", "content": "

I used this kind of setting:7 8->9 and 7 9 ->8 and 8,9->7 in trainset and 7 8 9 for 10.

", "rawMarkdown": "I used this kind of setting:7 8->9 and 7 9 ->8 and 8,9->7 in trainset and 7 8 9 for 10." }, { "id": 326578, "postDate": "2018-05-10T01:19:34.603Z", "content": "

thanks for your reply(duo xie)

", "rawMarkdown": "thanks for your reply(duo xie)" } ] }, { "id": 325825, "postDate": "2018-05-08T23:14:27.970Z", "content": "

Congratulations @bestfitting and thank you very much for sharing your solution. I could barely beat @Andy Harless NN's performance so I gave up on my NN model. You have shown me how best to use it with this data. That is a lesson learnt for life :-) I will certainly look into the activation function described, I if i have any ideas, I will pm you.

\n\n

Happy Kaggling and see u in the next one!!!

", "rawMarkdown": "Congratulations @bestfitting and thank you very much for sharing your solution. I could barely beat @Andy Harless NN's performance so I gave up on my NN model. You have shown me how best to use it with this data. That is a lesson learnt for life :-) I will certainly look into the activation function described, I if i have any ideas, I will pm you. \n\nHappy Kaggling and see u in the next one!!!", "votes": 1, "replies": [ { "id": 326561, "postDate": "2018-05-09T23:22:41.310Z", "content": "

I hope you can verify my finding in this competition,my setting:batchsize:100000,Adam,LR=0.01 for first 10 epoch.　

", "rawMarkdown": "I hope you can verify my finding in this competition,my setting:batchsize:100000,Adam,LR=0.01 for first 10 epoch.　", "votes": 2 }, { "id": 326617, "postDate": "2018-05-10T03:18:04.410Z", "content": "

I will look into it when I have a breather perhaps in a couple of weeks. Will keep you posted if I find useful insights.

", "rawMarkdown": "I will look into it when I have a breather perhaps in a couple of weeks. Will keep you posted if I find useful insights." } ] }, { "id": 325542, "postDate": "2018-05-08T14:23:40.643Z", "content": "

Congrats and thanks for sharing! I have 2 small questions.

\n\n
    \n
  1. What is res-links? (I'm not good at deep learning, so maybe it is a stupy question...
    2. \n
  2. How big is your video memory? Intuitively, video memory should be not able to load this data set. So it is hard to use GPU in my opinion. Could you explain it?
    3. \n
\n\n

Thanks~

", "rawMarkdown": "Congrats and thanks for sharing! I have 2 small questions.\n\n1. What is res-links? (I'm not good at deep learning, so maybe it is a stupy question...\n2. How big is your video memory? Intuitively, video memory should be not able to load this data set. So it is hard to use GPU in my opinion. Could you explain it?\n\nThanks~", "votes": 1, "replies": [ { "id": 325555, "postDate": "2018-05-08T14:42:12.387Z", "content": "

@Shawn Xiao

\n\n
    \n
  1. You kind of need to understand how a NN works in order to understand residuals - it's basically a short-cut between layers where the input to a set of layers is added after the processing done by the layers
    2. \n
  2. GPU memory is only relevant from the point of view of how much data can you use for a batch - the more video memory you have the bigger the batch that goes through the network. In NN is not necessary to use all the data in a pass - there are papers suggesting various batch sizing methods to optimize learning process. It might be counter intuitive, but sometimes, not passing all the data through the NN in one go generates better results.
    3. \n
", "rawMarkdown": "@Shawn Xiao\n\n1. You kind of need to understand how a NN works in order to understand residuals - it's basically a short-cut between layers where the input to a set of layers is added after the processing done by the layers \n2. GPU memory is only relevant from the point of view of how much data can you use for a batch - the more video memory you have the bigger the batch that goes through the network. In NN is not necessary to use all the data in a pass - there are papers suggesting various batch sizing methods to optimize learning process. It might be counter intuitive, but sometimes, not passing all the data through the NN in one go generates better results.", "votes": 1 }, { "id": 325601, "postDate": "2018-05-08T16:08:28.777Z", "content": "

@Mihai Cvasnievschi

\n\n

Thanks for answering. I still have something confused.

\n\n
    \n
  1. You mean that res-link is residuals? Maybe I would learn more about deep learning.
    2. \n
  2. My bad. I thougth bestfitting trains lgb model using GPU, which is strange for me. If he trains NN model using GPU, I understand that it could using GPU. Your explanation is very well.
    3. \n
", "rawMarkdown": "@Mihai Cvasnievschi\n\nThanks for answering. I still have something confused.\n\n1. You mean that res-link is residuals? Maybe I would learn more about deep learning.\n2. My bad. I thougth bestfitting trains lgb model using GPU, which is strange for me. If he trains NN model using GPU, I understand that it could using GPU. Your explanation is very well." }, { "id": 325609, "postDate": "2018-05-08T16:17:49.700Z", "content": "

@Shawn Xiao

\n\n
    \n
  1. Think about it like this
    2. \n
\n\n

B1 = I1 -> L1 -> L2 -> L3 => L3 + I1 = I2

\n\n

B2 = I2 -> L4 -> L5 => L5 + I2 = I3

\n\n

where B res block, I = input, L = NN layer\nYou can stack as many blocks as you want, having the shortcuts would simplify the back-propagation of gradients through the network and would force each block to learn a residual value. I think this article details the idea better than me - https://towardsdatascience.com/an-overview-of-resnet-and-its-variants-5281e2f56035?gi=cd366098c148

\n\n

Just keep in mind it doesn't have to apply to image processing and it can be applied for any kind of NN

\n\n

2 LightGBM can be used with GPU without having to load everything into GPU ram - see here https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html

", "rawMarkdown": "@Shawn Xiao\n\n1. Think about it like this \n\nB1 = I1 -> L1 -> L2 -> L3 => L3 + I1 = I2\n\nB2 = I2 -> L4 -> L5 => L5 + I2 = I3\n\nwhere B res block, I = input, L = NN layer\nYou can stack as many blocks as you want, having the shortcuts would simplify the back-propagation of gradients through the network and would force each block to learn a residual value. I think this article details the idea better than me - https://towardsdatascience.com/an-overview-of-resnet-and-its-variants-5281e2f56035?gi=cd366098c148\n\nJust keep in mind it doesn't have to apply to image processing and it can be applied for any kind of NN\n\n2 LightGBM can be used with GPU without having to load everything into GPU ram - see here https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html\n\n\n\n", "votes": 3 }, { "id": 325714, "postDate": "2018-05-08T19:21:37.437Z", "content": "

@Mihai Cvasnievschi,your comment is just want I want to write,thank you.

", "rawMarkdown": "@Mihai Cvasnievschi,your comment is just want I want to write,thank you.", "votes": 2 }, { "id": 325759, "postDate": "2018-05-08T20:39:04.497Z", "content": "

My pleasure, my intuition told me an NN can perform well and I've followed it. My approach is a bit different. Yet I didn't wanted to use any features based on data not know from the point of view of a row.

\n\n

So my NN variants would only relay on cumcounts, previous values and variations. One of this days I would spend some time to implement all the features used by top performers and would be curious to see the result.

\n\n

Edited ... tried to show you my code but I'm too new to this forum and can't manage to get the markdown to work :D

", "rawMarkdown": "My pleasure, my intuition told me an NN can perform well and I've followed it. My approach is a bit different. Yet I didn't wanted to use any features based on data not know from the point of view of a row. \n\nSo my NN variants would only relay on cumcounts, previous values and variations. One of this days I would spend some time to implement all the features used by top performers and would be curious to see the result.\n\nEdited ... tried to show you my code but I'm too new to this forum and can't manage to get the markdown to work :D \n" }, { "id": 325769, "postDate": "2018-05-08T20:56:28.800Z", "content": "

I have recieved the code in my email.

", "rawMarkdown": "I have recieved the code in my email." }, { "id": 325779, "postDate": "2018-05-08T21:07:47.970Z", "content": "

Err sorry for spam then :-) I kind of hate the spam nature of Kaggle.

", "rawMarkdown": "Err sorry for spam then :-) I kind of hate the spam nature of Kaggle." }, { "id": 325881, "postDate": "2018-05-09T02:16:23.960Z", "content": "

@bestfitting @Mihai Cvasnievschi\nThank you all very much. Learn a lot from you!

", "rawMarkdown": "@bestfitting @Mihai Cvasnievschi\nThank you all very much. Learn a lot from you!", "votes": 1 } ] }, { "id": 325349, "postDate": "2018-05-08T10:23:09.520Z", "content": "

Congrats! I'm happy to see NN performing so well, gives me hope I can improve my NN model ;-)

", "rawMarkdown": "Congrats! I'm happy to see NN performing so well, gives me hope I can improve my NN model ;-)", "votes": 1 }, { "id": 325296, "postDate": "2018-05-08T09:30:47.667Z", "content": "

Thanks for the write up bestfitting - I was wondering if anyone would try RNN on this dataset :)

", "rawMarkdown": "Thanks for the write up bestfitting - I was wondering if anyone would try RNN on this dataset :) ", "votes": 1 }, { "id": 325225, "postDate": "2018-05-08T08:37:18.043Z", "content": "

Congratulations on the approach and the result. Using RNN on click series is brillant, and using oof on all data is also a great approach.

", "rawMarkdown": "Congratulations on the approach and the result. Using RNN on click series is brillant, and using oof on all data is also a great approach.", "votes": 1, "replies": [ { "id": 325718, "postDate": "2018-05-08T19:31:47.007Z", "content": "

As I can train an epoch in 120s-150s also,25 epoches is enough although I trained 30-35 epoches,OOF is not unacceptable,if we use binary format file instead of pandas, it is very quick.

", "rawMarkdown": "As I can train an epoch in 120s-150s also,25 epoches is enough although I trained 30-35 epoches,OOF is not unacceptable,if we use binary format file instead of pandas, it is very quick.", "votes": 1 } ] }, { "id": 325185, "postDate": "2018-05-08T07:47:30.667Z", "content": "

Thanks for sharing and congratulations! It's nice to see a NN model act so well.
\nI wonder how to use RNN in this problem, is there any sequence data? Can't wait to see more details about this!

", "rawMarkdown": "Thanks for sharing and congratulations! It's nice to see a NN model act so well. \nI wonder how to use RNN in this problem, is there any sequence data? Can't wait to see more details about this!", "votes": 1, "replies": [ { "id": 325340, "postDate": "2018-05-08T10:16:08.410Z", "content": "

@rocuku - data is actually a sequence of clicks and you can consider (ip, device, os) to be a \"user\". You can use an RNN to predict the next click for a given (ip, device, os) ... or to predict if the next click is_attributed ...

\n\n

My NN uses a pseudo RNN model, meaning I've created additional features that would store prev1_app, prev2_app, prev_click_time, prev1_click_time for (user = ip, device, os) where prev1 is the previous app/click_time, prev2 is previous previous ... This way I'm able to have a mini-history for the \"user\". I've created a sigmoid based attention model to act like gating for the vectors and it actually improved my score a lot.

\n\n

Initially I've considered running a full RNN but in the end I've considered having the data within the dataset and do a naive approach that would achieve similar results.

", "rawMarkdown": "@rocuku - data is actually a sequence of clicks and you can consider (ip, device, os) to be a \"user\". You can use an RNN to predict the next click for a given (ip, device, os) ... or to predict if the next click is_attributed ...\n\nMy NN uses a pseudo RNN model, meaning I've created additional features that would store prev1_app, prev2_app, prev_click_time, prev1_click_time for (user = ip, device, os) where prev1 is the previous app/click_time, prev2 is previous previous ... This way I'm able to have a mini-history for the \"user\". I've created a sigmoid based attention model to act like gating for the vectors and it actually improved my score a lot.\n\nInitially I've considered running a full RNN but in the end I've considered having the data within the dataset and do a naive approach that would achieve similar results. ", "votes": 3 }, { "id": 325844, "postDate": "2018-05-09T00:37:52.273Z", "content": "

Thanks for sharing, very inspiring!
\nI have never thought about RNN and attention model can use like this.

", "rawMarkdown": "Thanks for sharing, very inspiring! \nI have never thought about RNN and attention model can use like this.", "votes": 1 } ] }, { "id": 325180, "postDate": "2018-05-08T07:41:46.663Z", "content": "

That's simply amazing! \nIt will be very enlightening to see your solution. As I am beggining with NN that would be great amount of knowledge ;)

\n\n

Congratulations!

", "rawMarkdown": "That's simply amazing! \nIt will be very enlightening to see your solution. As I am beggining with NN that would be great amount of knowledge ;)\n\nCongratulations!", "votes": 1 }, { "id": 325175, "postDate": "2018-05-08T07:32:46.617Z", "content": "

Wow simply amazing what you did! You are truely by far the best Kaggler at the moment mastering any kind of competition! I need to read your solution carefully and try to catch up a bit :) Congrats!!!

", "rawMarkdown": "Wow simply amazing what you did! You are truely by far the best Kaggler at the moment mastering any kind of competition! I need to read your solution carefully and try to catch up a bit :) Congrats!!!", "votes": 1 }, { "id": 325170, "postDate": "2018-05-08T07:27:24.737Z", "content": "

Congrats for the 3rd place @bestfitting. Just curious how a RNN could be implemented on this scenario. Chaining a series clicks of a specific ip? Would you mind sharing something more about the structure of the NN/RNN model?A short code snippet would be helpful :)

", "rawMarkdown": "Congrats for the 3rd place @bestfitting. Just curious how a RNN could be implemented on this scenario. Chaining a series clicks of a specific ip? Would you mind sharing something more about the structure of the NN/RNN model?A short code snippet would be helpful :)", "votes": 1 }, { "id": 325139, "postDate": "2018-05-08T06:54:43.343Z", "content": "

Congratulations..I was eagerly waiting for NN basee solutions to see.. Glad you mentioned it properly to some extend.. Thank you very much..

", "rawMarkdown": "Congratulations..I was eagerly waiting for NN basee solutions to see.. Glad you mentioned it properly to some extend.. Thank you very much..", "votes": 1 }, { "id": 325135, "postDate": "2018-05-08T06:50:54.097Z", "content": "

@bestfitting, thanks for sharing your solution. I am noob on NN and would like to gain more knowledge. Could you share more details about your CNN solutions and give some piece of code? Thanks in advance and well done for your 3rd position even though I imagine that droping from first to 3rd could be somehow frustrating. But for new comers like me, being 3rd is already a BIG ACHIEVEMENT and you should be proud of yourself!

", "rawMarkdown": "[@bestfitting](https://www.kaggle.com/bestfitting), thanks for sharing your solution. I am noob on NN and would like to gain more knowledge. Could you share more details about your CNN solutions and give some piece of code? Thanks in advance and well done for your 3rd position even though I imagine that droping from first to 3rd could be somehow frustrating. But for new comers like me, being 3rd is already a BIG ACHIEVEMENT and you should be proud of yourself!", "votes": 1, "replies": [ { "id": 326559, "postDate": "2018-05-09T23:17:51.217Z", "content": "

@eric I edited the post fo answer you question,I hope it will be helpful.Yes,I am quite happy with the top 3 result.Thank you!

", "rawMarkdown": "@eric I edited the post fo answer you question,I hope it will be helpful.Yes,I am quite happy with the top 3 result.Thank you!", "votes": 1 }, { "id": 327097, "postDate": "2018-05-10T20:08:40.130Z", "content": "

Thanks Bestfitting. You are an amazing competitor and I admire your fairness. You are my hero!

", "rawMarkdown": "Thanks Bestfitting. You are an amazing competitor and I admire your fairness. You are my hero!", "votes": 1 } ] }, { "id": 325114, "postDate": "2018-05-08T06:15:12.650Z", "content": "

Can u share the feature engineering in this competition?

", "rawMarkdown": "Can u share the feature engineering in this competition?", "votes": 1, "replies": [ { "id": 325732, "postDate": "2018-05-08T19:49:23.137Z", "content": "

I suggest you read the great thread of @CPMP,I hope I can use his features in my model.

", "rawMarkdown": "I suggest you read the great thread of [@CPMP][1],I hope I can use his features in my model.\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56283", "votes": 1 }, { "id": 326091, "postDate": "2018-05-09T09:03:24.103Z", "content": "

@bestfitting, thanks for referring to me. Let us know if you can improve your model with more features.

", "rawMarkdown": "@bestfitting, thanks for referring to me. Let us know if you can improve your model with more features.", "votes": 1 } ] }, { "id": 325103, "postDate": "2018-05-08T05:58:38.453Z", "content": "

@bestfitting, thanks for sharing and congratulations for your 3rd place !

", "rawMarkdown": "@bestfitting, thanks for sharing and congratulations for your 3rd place !", "votes": 1 }, { "id": 325096, "postDate": "2018-05-08T05:45:02.033Z", "content": "

Congrats @bestfitting and thanks for sharing. How can you view the private LB score before the competition ends?

", "rawMarkdown": "Congrats @bestfitting and thanks for sharing. How can you view the private LB score before the competition ends?", "votes": 1 }, { "id": 325091, "postDate": "2018-05-08T05:36:54.567Z", "content": "

Thanks for sharing! I'll definitely spend more time on NN models.

", "rawMarkdown": "Thanks for sharing! I'll definitely spend more time on NN models.", "votes": 1 }, { "id": 325073, "postDate": "2018-05-08T05:05:07.040Z", "content": "

Thanks for sharing and congratulations! It's great to hear from the winners.

", "rawMarkdown": "Thanks for sharing and congratulations! It's great to hear from the winners.", "votes": 1 }, { "id": 325068, "postDate": "2018-05-08T04:54:32.403Z", "content": "

Encouraging result for deep learning enthusiast!

", "rawMarkdown": "Encouraging result for deep learning enthusiast!", "votes": 1 }, { "id": 325062, "postDate": "2018-05-08T04:39:23.817Z", "content": "

Congrats and thanks for sharing! \ndoes nn model is better than lgb for this competition？

", "rawMarkdown": "Congrats and thanks for sharing! \ndoes nn model is better than lgb for this competition？", "votes": 1 }, { "id": 325058, "postDate": "2018-05-08T04:33:31.240Z", "content": "

god-like performance.

", "rawMarkdown": "god-like performance.", "votes": 1 }, { "id": 325612, "postDate": "2018-05-08T16:20:44.777Z", "content": "

Awesome to know an NN solution could be used to solve a problem that people mostly used LGB for. Thanks for your write up.

", "rawMarkdown": "Awesome to know an NN solution could be used to solve a problem that people mostly used LGB for. Thanks for your write up.", "votes": 2 }, { "id": 325202, "postDate": "2018-05-08T08:11:10.280Z", "content": "

Congrats bestfitting ...Truly impressed by your talent.

\n\n
\n

So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :)

\n
\n\n

If you managed to achieve it...you wouldn't have no excuse to do not share it on Bojan's thread ^^

", "rawMarkdown": "Congrats bestfitting ...Truly impressed by your talent.\n\n\n> So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :) \n\nIf you managed to achieve it...you wouldn't have no excuse to do not share it on Bojan's thread ^^", "votes": 2, "replies": [ { "id": 325724, "postDate": "2018-05-08T19:38:25.583Z", "content": "

I was expecting the idle_speculation's 0.9835 with 1 submission at that time. :)

", "rawMarkdown": "I was expecting the idle_speculation's 0.9835 with 1 submission at that time. :)", "votes": 3 } ] }, { "id": 325130, "postDate": "2018-05-08T06:43:51.717Z", "content": "

Congratulations and nice work!

\n\n

I have some questions:

\n\n
    \n
  1. How did you choice features?
    2. \n
  2. NN? You mean full connection neuron network with multi layers?
    3. \n
  3. RNN? I do not know how to apply RNN(Recurrent Neural Network) on this conpetition.How to organize the features to apply RNN.
    4. \n
  4. You do not mention ensemble, you do not ensemble your results?
    5. \n
", "rawMarkdown": "Congratulations and nice work!\n\nI have some questions:\n\n 1. How did you choice features? \n 2. NN? You mean full connection neuron network with multi layers?\n 3. RNN? I do not know how to apply RNN(Recurrent Neural Network) on this conpetition.How to organize the features to apply RNN.\n 4. You do not mention ensemble, you do not ensemble your results?", "votes": 2, "replies": [ { "id": 325731, "postDate": "2018-05-08T19:48:03.113Z", "content": "

@StudyExchange

\n\n

1.My features is quite simple,I suggest you read the great thread of CPMP
\n2.Yes
\n3.I have add some details,please refer to the last part of my post.
\n4.I use out of fold data to build ensemble models.

", "rawMarkdown": "@StudyExchange\n\n1.My features is quite simple,I suggest you read the great thread of [CPMP][1]
\n2.Yes
\n3.I have add some details,please refer to the last part of my post.
\n4.I use out of fold data to build ensemble models.
\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56283", "votes": 3 }, { "id": 325758, "postDate": "2018-05-08T20:36:25.073Z", "content": "

Hi, @bestfitting, it is very interesting to see RNN used here. Could you elaborate more the intuition behind using GRU unit with sequence_length = 1 as comparing to using just regular dense layer? Or have you tried both and the first option gives you better results?

\n\n

Thanks for the detailed solution. learnt a lot.

", "rawMarkdown": "Hi, @bestfitting, it is very interesting to see RNN used here. Could you elaborate more the intuition behind using GRU unit with sequence_length = 1 as comparing to using just regular dense layer? Or have you tried both and the first option gives you better results?\n\nThanks for the detailed solution. learnt a lot. " }, { "id": 325766, "postDate": "2018-05-08T20:47:33.723Z", "content": "

Sure,I did experiments on it,if I add 5 click before and after a click time,and if I used dense layer,the result was much worse than this setting.
If we examine the struture of the GRU,this kind of setting is different than dense with tanh,dot,sigmoid......We may say,it's not true RNN,but this kind of structure did work.

", "rawMarkdown": "Sure,I did experiments on it,if I add 5 click before and after a click time,and if I used dense layer,the result was much worse than this setting.
If we examine the struture of the GRU,this kind of setting is different than dense with tanh,dot,sigmoid......We may say,it's not true RNN,but this kind of structure did work.\n", "votes": 2 }, { "id": 325781, "postDate": "2018-05-08T21:08:23.850Z", "content": "

@bestfitting, Very cool, never thought about that we can do it that way, learnt something new.

\n\n

Just curious, have you tried more than one time step in your RNN? I would image the computation will grow dramatically, and it might be tricky to preprocess or define what is the sequence.

", "rawMarkdown": "@bestfitting, Very cool, never thought about that we can do it that way, learnt something new. \n\nJust curious, have you tried more than one time step in your RNN? I would image the computation will grow dramatically, and it might be tricky to preprocess or define what is the sequence. " }, { "id": 325796, "postDate": "2018-05-08T21:45:13.613Z", "content": "

I did not use multi-step setting because it's time consuming.And I have just modified the post and add a paper related to help you know why I used this kind of structure:\nSearching for Activation Functions

", "rawMarkdown": "I did not use multi-step setting because it's time consuming.And I have just modified the post and add a paper related to help you know why I used this kind of structure:\n[Searching for Activation Functions][1]\n\n\n [1]: https://arxiv.org/abs/1710.05941", "votes": 1 }, { "id": 325953, "postDate": "2018-05-09T05:32:07.107Z", "content": "
    \n
  1. NN and RNN aplly on this competition broaden my horizon, I will have a try! Give thanks for additional elaborate!
    2. \n
  2. Why not try ensemble, beyond KFold cross validation? Other competitor only use average the result, and got better score. More over, there are lots of ensemble method, like LIghtGBM boost the cv result once more.
    3. \n
", "rawMarkdown": " 1. NN and RNN aplly on this competition broaden my horizon, I will have a try! Give thanks for additional elaborate!\n 2. Why not try ensemble, beyond KFold cross validation? Other competitor only use average the result, and got better score. More over, there are lots of ensemble method, like LIghtGBM boost the cv result once more." }, { "id": 326558, "postDate": "2018-05-09T23:15:10.423Z", "content": "

@StudyExchange,I used ensemble indeed,I predict the train and test dataset and built model on it indeed,which helped me a lot.

", "rawMarkdown": "@StudyExchange,I used ensemble indeed,I predict the train and test dataset and built model on it indeed,which helped me a lot." }, { "id": 326665, "postDate": "2018-05-10T05:57:33.300Z", "content": "

1th layer:

\n\n\n\n
\n

Question:

\n
\n\n\n\n

2nd layer(ensemble=avg):

\n\n

2.1 weighted average 1.1, 1.2, 1.3, KFold >> LB0.9827

\n\n

3th layer(ensemble=pseudo label)

\n\n\n\n
\n

Did I understand your workflow right?

\n
", "rawMarkdown": "1th layer:\n----------\n\n- 1.1 23 feature, LightGBM, KFold >> LB0.9817\n- 1.2 23 feature, Full_Connection_NN(NN), binary_crossentropy, KFold >> LB0.9820\n- 1.3 23 feature+delta click_time, RNN, binary_crossentropy, KFold >> LB0.9821\n\n> Question:\n\n- Q1.2 Data process is important for NN and RNN: [NA,out of vocabulary,log,scale]. Did you apply this process on data of LightGBM? (PS: I think you did!)\n- Q1.2 I find some body use delta click_time at LightGBM, did you use it?\n\n2nd layer(ensemble=avg):\n--------------------\n\n2.1 weighted average 1.1, 1.2, 1.3, KFold >> LB0.9827\n\n3th layer(ensemble=pseudo label)\n---------------------------------\n\n- data: whole train and test data\n- features: prediction from 2.1(as 1 feature), IP, app-os-channel, so 3 feature in total\n- train: Full_Connection_NN(NN), binary_crossentropy, KFold\n\n> Did I understand your workflow right?\n", "votes": 2 }, { "id": 326739, "postDate": "2018-05-10T08:49:57.340Z", "content": "

Q1.2:I did not use them on LGBM models although it helped to improve 0.0001 to 0.0002,as my NN models were trained after LGBM models,I did not retrain and predict again
\nQ1.2: As you can see in my features list,I used it.
\n3th layers<>pseudo label，NN model indeed
\nＸ:all the predictions from N former layer models and the predictions grouped by IP_Day and app-os-channel
\ny:the is_attributed value of train-set.

", "rawMarkdown": "Q1.2:I did not use them on LGBM models although it helped to improve 0.0001 to 0.0002,as my NN models were trained after LGBM models,I did not retrain and predict again
\nQ1.2: As you can see in my features list,I used it.
\n3th layers<>pseudo label，NN model indeed
\nＸ:all the predictions from N former layer models and the predictions grouped by IP_Day and app-os-channel
\ny:the is_attributed value of train-set.
", "votes": 1 }, { "id": 327096, "postDate": "2018-05-10T20:06:18.247Z", "content": "

click_time, I find it. Thank you! Another questions:

\n\n
Ｘ:all the predictions from N former layer models and the predictions grouped by IP_Day and app-os-channel\n
\n\n

You mean, you do not use \"prediction from 2.1(as 1 feature) + IP + app-os-channel, so 3 feature in total\", but :

\n\n
    \n

  1. prediction is from \"N former layer\": 1.1, 1.2, 1.3 and 2.1, all of them?

    2. \n

  2. do not concatenate \"prediction, IP and app-os-channel\" and get 3 feature, but group the prediction by IP and app-os-channel and get 2 feature. The \"X\" have only 2 feature?

    3. \n
", "rawMarkdown": "**click_time**, I find it. Thank you! Another questions:\n\n Ｘ:all the predictions from N former layer models and the predictions grouped by IP_Day and app-os-channel\nYou mean, you do not use \"prediction from 2.1(as 1 feature) + IP + app-os-channel, so 3 feature in total\", but :\n\n1. prediction is from \"**N former layer**\": 1.1, 1.2, 1.3 and 2.1, all of them?\n\n2. do not concatenate \"prediction, IP and app-os-channel\" and get 3 feature, but group the prediction by IP and app-os-channel and get 2 feature. The \"**X**\" have only 2 feature?" }, { "id": 327703, "postDate": "2018-05-12T08:08:33.490Z", "content": "

Hi,@StudyExchange so for late reply.
\nprediction from former layers,if we have 3 models,then the X will be:
\nm1,m2,m3,mean_m1_by_ip_day,mean_m2_by_ip_day,mean_m3_by_ip_day,mean_m1_by_app-os-channel_day,mean_m2_by_app-os-channel_day,mean_m3_by_app-os-channel_day

", "rawMarkdown": "Hi,@StudyExchange so for late reply.
\nprediction from former layers,if we have 3 models,then the X will be:
\nm1,m2,m3,mean_m1_by_ip_day,mean_m2_by_ip_day,mean_m3_by_ip_day,mean_m1_by_app-os-channel_day,mean_m2_by_app-os-channel_day,mean_m3_by_app-os-channel_day\n", "votes": 2 }, { "id": 327821, "postDate": "2018-05-12T15:08:30.790Z", "content": "

Thank you very much for replies, very clear and helpful!

", "rawMarkdown": "Thank you very much for replies, very clear and helpful!" } ] }, { "id": 325095, "postDate": "2018-05-08T05:44:45.107Z", "content": "

Congratulations and good work!

", "rawMarkdown": "Congratulations and good work!", "votes": 2 }, { "id": 325057, "postDate": "2018-05-08T04:31:37.683Z", "content": "

Congrats and thanks for sharing! Absolutely god-like performance. Have a nice dream. :)

", "rawMarkdown": "Congrats and thanks for sharing! Absolutely god-like performance. Have a nice dream. :)", "votes": 2 }, { "id": 2477691, "postDate": "2023-10-11T13:52:33.107Z", "content": "

Excited to see the code! Thanks for sharing it.

", "rawMarkdown": "Excited to see the code! Thanks for sharing it. " }, { "id": 517655, "postDate": "2019-04-16T10:19:37.500Z", "content": "

Great ! following you

", "rawMarkdown": "Great ! following you" }, { "id": 433060, "postDate": "2018-12-04T16:06:55.147Z", "content": "

nice

", "rawMarkdown": "nice\n" }, { "id": 397325, "postDate": "2018-10-02T10:33:42.113Z", "content": "

Thanks for sharing your approach!! This is really cool!

", "rawMarkdown": "Thanks for sharing your approach!! This is really cool!" }, { "id": 334056, "postDate": "2018-05-26T12:20:48.673Z", "content": "

Thank you for sharing! It's amazing.\nHere are quick questions about features.

\n\n
    \n
  1. What is a difference between \" ip_app_os_device_day_click_time_next_1 \" and \" ip_app_os_device_day_click_time_next_2 \" ?
    2. \n
  2. How did you make \" ip_app_os_device_day_click_time_lag1 \" ? Is it a kind of previous click time feature?\nI guess it is a stored feature of previous click time features. right?
    3. \n
\n\n

I fount clues in what you commented below:
\nI've created additional features that would store prev1_app, prev2_app, prev_click_time, prev1_click_time for (user = ip, device, os) where prev1 is the previous app/click_time, prev2 is previous previous ... This way I'm able to have a mini-history for the \"user\".

\n\n

Thanks again! I've learned from you a lot.

", "rawMarkdown": "Thank you for sharing! It's amazing.\nHere are quick questions about features.\n\n1. What is a difference between \" ip_app_os_device_day_click_time_next_1 \" and \" ip_app_os_device_day_click_time_next_2 \" ?\n2. How did you make \" ip_app_os_device_day_click_time_lag1 \" ? Is it a kind of previous click time feature?\nI guess it is a stored feature of previous click time features. right?\n\nI fount clues in what you commented below: \nI've created additional features that would store prev1_app, prev2_app, prev_click_time, prev1_click_time for (user = ip, device, os) where prev1 is the previous app/click_time, prev2 is previous previous ... This way I'm able to have a mini-history for the \"user\". \n\nThanks again! I've learned from you a lot.\n", "replies": [ { "id": 334162, "postDate": "2018-05-26T17:01:35.233Z", "content": "

Yes,mini-history of a virtual user,if he click too often,he have a trend of not downloading a app.

", "rawMarkdown": "Yes,mini-history of a virtual user,if he click too often,he have a trend of not downloading a app." } ] }, { "id": 325774, "postDate": "2018-05-08T21:01:49.110Z", "content": "

@bestfitting my only question to you is what loss function you used.

\n\n

I've tried both cross entropy and mse and found out that better ROC AUC doesn't properly reflect into lower loss. My best AUC scores were not generated by lowest loss. I've even experimented with a blend between mse and cross entropy but couldn't managed to find a good balance. I had an epoch with a very very low mse (0.0004) loss that had the best AUC one would dream on ... 0.5 :-)

", "rawMarkdown": "@bestfitting my only question to you is what loss function you used. \n\nI've tried both cross entropy and mse and found out that better ROC AUC doesn't properly reflect into lower loss. My best AUC scores were not generated by lowest loss. I've even experimented with a blend between mse and cross entropy but couldn't managed to find a good balance. I had an epoch with a very very low mse (0.0004) loss that had the best AUC one would dream on ... 0.5 :-) ", "replies": [ { "id": 325782, "postDate": "2018-05-08T21:08:47.383Z", "content": "

loss = 'binary_crossentropy'
\nI didn't care the gap between the loss and auc,just choose higest auc epoch, as the best auc means almost lowest loss in my training process.

", "rawMarkdown": "loss = 'binary_crossentropy'
\nI didn't care the gap between the loss and auc,just choose higest auc epoch, as the best auc means almost lowest loss in my training process.\n", "votes": 3 } ] }, { "id": 325280, "postDate": "2018-05-08T09:18:38.563Z", "content": "

Congratulations and thanks for sharing!\nThe nn model is amazing, could you shame more detail for us?\nthanks!

", "rawMarkdown": "Congratulations and thanks for sharing!\nThe nn model is amazing, could you shame more detail for us?\nthanks!" }, { "id": 1210673, "postDate": "2021-02-19T15:47:07.387Z", "rawMarkdown": "", "isDeleted": true }, { "id": 329243, "postDate": "2018-05-16T04:37:49.910Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 327800, "postDate": "2018-05-12T14:23:10.870Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1495840, "postDate": "2021-08-29T20:37:54.023Z", "content": "

Thanks for sharing!! very helpful

", "rawMarkdown": "Thanks for sharing!! very helpful\n\n", "votes": 1 }, { "id": 332869, "postDate": "2018-05-24T00:15:13.547Z", "content": "

thanks for sharing

", "rawMarkdown": "thanks for sharing", "votes": 1 }, { "id": 329694, "postDate": "2018-05-17T02:12:14.787Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing", "votes": 1 }, { "id": 328329, "postDate": "2018-05-14T01:44:30.880Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing", "votes": 1 }, { "id": 328140, "postDate": "2018-05-13T13:17:30.450Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": 1 }, { "id": 327554, "postDate": "2018-05-11T21:11:09.980Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": 1 }, { "id": 326811, "postDate": "2018-05-10T11:24:28.040Z", "content": "

congrats and thanks! :D

", "rawMarkdown": "congrats and thanks! :D", "votes": 1 }, { "id": 326628, "postDate": "2018-05-10T04:39:40.997Z", "content": "

very thanks for sharing your approach:)

", "rawMarkdown": "very thanks for sharing your approach:)", "votes": 1 }, { "id": 325789, "postDate": "2018-05-08T21:24:03.147Z", "content": "

Thank you! Learn a lot from you!

", "rawMarkdown": "Thank you! Learn a lot from you!", "votes": 1 }, { "id": 325145, "postDate": "2018-05-08T07:04:46.867Z", "content": "

Congratulations and thanks for sharing!

", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 }, { "id": 325110, "postDate": "2018-05-08T06:09:05.527Z", "content": "

Congratulations and thanks for sharing!

", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 }, { "id": 1145479, "postDate": "2021-01-09T07:09:12.300Z", "content": "

thanks for sharing!

", "rawMarkdown": "thanks for sharing!" }, { "id": 334502, "postDate": "2018-05-27T16:18:22.217Z", "content": "

thanks for sharing

", "rawMarkdown": "thanks for sharing" } ], "comments": [ { "id": 325195, "author_name": "Michael Jahrer", "author_url": "", "post_date": "2018-05-08T08:02:58.430000", "content": "

wow brilliant, never see RNN performed so well

", "votes": 6, "replies": [] }, { "id": 325193, "author_name": "Feiyang Pan", "author_url": "", "post_date": "2018-05-08T08:01:41.843000", "content": "

Congrats! I am very impressed by the performances of your NN models. I wonder what are the detailed structures of your NNs? You have mentioned RNN and the res-links so I am very curious about the details.

", "votes": 6, "replies": [] }, { "id": 325317, "author_name": "Pranav Pandya", "author_url": "", "post_date": "2018-05-08T09:50:36.780000", "content": "

Congrats @bestfitting and thanks for sharing the approach.

", "votes": 3, "replies": [] }, { "id": 328716, "author_name": "Kevin Liao", "author_url": "", "post_date": "2018-05-15T00:13:01.303000", "content": "

Truly good stuff!!

", "votes": 1, "replies": [] }, { "id": 328404, "author_name": "Bhavika", "author_url": "", "post_date": "2018-05-14T08:16:33.053000", "content": "

Thanks for sharing.\nIt's too great resource to learn.

", "votes": 1, "replies": [] }, { "id": 327957, "author_name": "Bowen He", "author_url": "", "post_date": "2018-05-13T03:09:34.453000", "content": "

Congratulations and thank you so much for sharing! \nI have a little question when you talk about preprocessing for NN, what make you decide to use \n'Fill-NA:Fill max value for delta feature.Fill mean for other feature'\ninstead of filling mean for all features?

", "votes": 1, "replies": [ { "id": 327961, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-13T03:15:14.510000", "content": "

When we use NN models in such kind of problem,the data preprocess is very important.
\nIf there is no click from a clicktime,the next_click_feature will be NA,we can not fill mean or zero to NA,max delta is more reasonable.
\nOn the other side,for other features fill mean to NA item is more reasonable.

", "votes": 2, "replies": [] }, { "id": 327967, "author_name": "Bowen He", "author_url": "", "post_date": "2018-05-13T03:19:06.113000", "content": "

Thanks! Learn a lot from you.

", "votes": 0, "replies": [] } ] }, { "id": 327272, "author_name": "Adhi Narayanan Y R", "author_url": "", "post_date": "2018-05-11T06:49:42.290000", "content": "

cool

", "votes": 1, "replies": [] }, { "id": 326626, "author_name": "mamas", "author_url": "", "post_date": "2018-05-10T04:26:12.007000", "content": "

Thanks for sharing your solution, I finally lost to you, but really enjoyed competing with you :)
\nbtw, I couldn't understand \"I trained second level NN models using predictions from the whole train and test dataset\".\nDoes it mean you did pseudo-labelling, or is it just Stacking? could you please tell me what you use for training data and target?

", "votes": 1, "replies": [ { "id": 326742, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-10T08:59:13.683000", "content": "

@mamasinkgs,you've a great job,although we can not get good position only by luck,we do need it sometimes.As there are so many competitions on kaggle,you can get better rank for sure.
\nIt's stacking,I tried to avoid use terms like OOF but I found that I should use them :)
\nplease refer to here

", "votes": 1, "replies": [] }, { "id": 326823, "author_name": "mamas", "author_url": "", "post_date": "2018-05-10T11:41:50.530000", "content": "

I got what you mean.
\nThanks @bestfitting, I'll try harder and get better rank :)

", "votes": 2, "replies": [] }, { "id": 328913, "author_name": "mamas", "author_url": "", "post_date": "2018-05-15T10:59:10.600000", "content": "

btw, I'm curious about how much the score will become better if we combine the best submission. I think your unique methods will give a big score up when ensembling. could you upload your best submission if possible? 1st & 5th & 6th has already uploaded. https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/56423

", "votes": 0, "replies": [] } ] }, { "id": 326087, "author_name": "twoone", "author_url": "", "post_date": "2018-05-09T08:52:22.687000", "content": "

Very thanks for your sharing。I have some small questions and look forward to your rely。How do you decide your NN models hyperparameters and your NN architecture？

", "votes": 1, "replies": [ { "id": 326567, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-09T23:37:57.753000", "content": "

As I said in an interview by kaggle recently,I always try to find related solutions and papers and learn from them and then modify the setting and parameters based on experiments,by doing so, I can save a lot of time,we can try to stand on the shoulders of gaints.I read papers everyday and implement some of them as a practice and get better understanding of them , when a competition started I will select some of them to use and find more related.

", "votes": 17, "replies": [] }, { "id": 326622, "author_name": "", "author_url": "", "post_date": "2018-05-10T04:00:59.150000", "content": "", "votes": 0, "replies": [] }, { "id": 326624, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-10T04:16:04.523000", "content": "

@the1own not sure what repo is bestfitting using but I believe https://arxiv.org/list/cs.AI/recent to be the most comprehensive one. There's also this one https://towardsdatascience.com/ but that's not necessary a papers repo but more like \"trying to understand\" the papers repo ;-)

", "votes": 1, "replies": [] }, { "id": 326625, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-10T04:17:15.520000", "content": "

＠the1owl,do you like this one?\nWhen we find a good papers,we can follow the papers it refered,and the papers list as best papers of a conference is a good place to find the good papers.:)

", "votes": 6, "replies": [] }, { "id": 326660, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-10T05:37:53.783000", "content": "

@bestfitting Thanks a lot for this share! I think that it solves the main problem of arxiv ... too many papers published everyday :D

", "votes": 1, "replies": [] }, { "id": 326809, "author_name": "", "author_url": "", "post_date": "2018-05-10T11:21:30.550000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 326015, "author_name": "juniorcompressor", "author_url": "", "post_date": "2018-05-09T06:59:13.603000", "content": "

Congratulations

", "votes": 1, "replies": [] }, { "id": 325962, "author_name": "Tuhin Saha", "author_url": "", "post_date": "2018-05-09T05:58:39.393000", "content": "

Congratulations on your 3rd place. \nI had a feeling that NN will not do so well in this problem. But it is nice to see it doing so well. Thanks for sharing your approach.

", "votes": 1, "replies": [] }, { "id": 325938, "author_name": "yyqing", "author_url": "", "post_date": "2018-05-09T05:03:14.467000", "content": "

Thank you for your sharing. I did my last submission on the train to Changsha. Just happened to find that you work at the city.

", "votes": 1, "replies": [] }, { "id": 325906, "author_name": "Yuan", "author_url": "", "post_date": "2018-05-09T03:21:48.950000", "content": "

Congratulations. I made a plan to use RNN to realise it at first, but failed. Could you opensource your program and share some more experience? Thanks.

", "votes": 1, "replies": [ { "id": 326564, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-09T23:29:15.743000", "content": "

Thanks,you can refer to the network structure I posted and used the setting I mentioned　here,as I must am very busy in next two weeks I can not provide full set of my codes. sorry.

", "votes": 0, "replies": [] }, { "id": 326803, "author_name": "Yuan", "author_url": "", "post_date": "2018-05-10T11:12:17.053000", "content": "

OK，thanks for your reply. I will try again. If needed, look forward to your help. Thank you.

", "votes": 0, "replies": [] } ] }, { "id": 325878, "author_name": "PW", "author_url": "", "post_date": "2018-05-09T02:12:36.480000", "content": "

wow brilliant. I have a question about target encoder: how do you calculate the features like \"ip_app_mean_is_attributed\"? \n some methods like calculate target mean of yesterday or some other skills?? thanks

", "votes": 1, "replies": [ { "id": 326562, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-09T23:24:23.837000", "content": "

I used this kind of setting:7 8->9 and 7 9 ->8 and 8,9->7 in trainset and 7 8 9 for 10.

", "votes": 0, "replies": [] }, { "id": 326578, "author_name": "PW", "author_url": "", "post_date": "2018-05-10T01:19:34.603000", "content": "

thanks for your reply(duo xie)

", "votes": 0, "replies": [] } ] }, { "id": 325825, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-05-08T23:14:27.970000", "content": "

Congratulations @bestfitting and thank you very much for sharing your solution. I could barely beat @Andy Harless NN's performance so I gave up on my NN model. You have shown me how best to use it with this data. That is a lesson learnt for life :-) I will certainly look into the activation function described, I if i have any ideas, I will pm you.

\n\n

Happy Kaggling and see u in the next one!!!

", "votes": 1, "replies": [ { "id": 326561, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-09T23:22:41.310000", "content": "

I hope you can verify my finding in this competition,my setting:batchsize:100000,Adam,LR=0.01 for first 10 epoch.　

", "votes": 2, "replies": [] }, { "id": 326617, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-05-10T03:18:04.410000", "content": "

I will look into it when I have a breather perhaps in a couple of weeks. Will keep you posted if I find useful insights.

", "votes": 0, "replies": [] } ] }, { "id": 325542, "author_name": "Shawn Xiao", "author_url": "", "post_date": "2018-05-08T14:23:40.643000", "content": "

Congrats and thanks for sharing! I have 2 small questions.

\n\n
    \n
  1. What is res-links? (I'm not good at deep learning, so maybe it is a stupy question...
    2. \n
  2. How big is your video memory? Intuitively, video memory should be not able to load this data set. So it is hard to use GPU in my opinion. Could you explain it?
    3. \n
\n\n

Thanks~

", "votes": 1, "replies": [ { "id": 325555, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-08T14:42:12.387000", "content": "

@Shawn Xiao

\n\n
    \n
  1. You kind of need to understand how a NN works in order to understand residuals - it's basically a short-cut between layers where the input to a set of layers is added after the processing done by the layers
    2. \n
  2. GPU memory is only relevant from the point of view of how much data can you use for a batch - the more video memory you have the bigger the batch that goes through the network. In NN is not necessary to use all the data in a pass - there are papers suggesting various batch sizing methods to optimize learning process. It might be counter intuitive, but sometimes, not passing all the data through the NN in one go generates better results.
    3. \n
", "votes": 1, "replies": [] }, { "id": 325601, "author_name": "Shawn Xiao", "author_url": "", "post_date": "2018-05-08T16:08:28.777000", "content": "

@Mihai Cvasnievschi

\n\n

Thanks for answering. I still have something confused.

\n\n
    \n
  1. You mean that res-link is residuals? Maybe I would learn more about deep learning.
    2. \n
  2. My bad. I thougth bestfitting trains lgb model using GPU, which is strange for me. If he trains NN model using GPU, I understand that it could using GPU. Your explanation is very well.
    3. \n
", "votes": 0, "replies": [] }, { "id": 325609, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-08T16:17:49.700000", "content": "

@Shawn Xiao

\n\n
    \n
  1. Think about it like this
    2. \n
\n\n

B1 = I1 -> L1 -> L2 -> L3 => L3 + I1 = I2

\n\n

B2 = I2 -> L4 -> L5 => L5 + I2 = I3

\n\n

where B res block, I = input, L = NN layer\nYou can stack as many blocks as you want, having the shortcuts would simplify the back-propagation of gradients through the network and would force each block to learn a residual value. I think this article details the idea better than me - https://towardsdatascience.com/an-overview-of-resnet-and-its-variants-5281e2f56035?gi=cd366098c148

\n\n

Just keep in mind it doesn't have to apply to image processing and it can be applied for any kind of NN

\n\n

2 LightGBM can be used with GPU without having to load everything into GPU ram - see here https://lightgbm.readthedocs.io/en/latest/GPU-Performance.html

", "votes": 3, "replies": [] }, { "id": 325714, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-08T19:21:37.437000", "content": "

@Mihai Cvasnievschi,your comment is just want I want to write,thank you.

", "votes": 2, "replies": [] }, { "id": 325759, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-08T20:39:04.497000", "content": "

My pleasure, my intuition told me an NN can perform well and I've followed it. My approach is a bit different. Yet I didn't wanted to use any features based on data not know from the point of view of a row.

\n\n

So my NN variants would only relay on cumcounts, previous values and variations. One of this days I would spend some time to implement all the features used by top performers and would be curious to see the result.

\n\n

Edited ... tried to show you my code but I'm too new to this forum and can't manage to get the markdown to work :D

", "votes": 0, "replies": [] }, { "id": 325769, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-08T20:56:28.800000", "content": "

I have recieved the code in my email.

", "votes": 0, "replies": [] }, { "id": 325779, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-08T21:07:47.970000", "content": "

Err sorry for spam then :-) I kind of hate the spam nature of Kaggle.

", "votes": 0, "replies": [] }, { "id": 325881, "author_name": "Shawn Xiao", "author_url": "", "post_date": "2018-05-09T02:16:23.960000", "content": "

@bestfitting @Mihai Cvasnievschi\nThank you all very much. Learn a lot from you!

", "votes": 1, "replies": [] } ] }, { "id": 325349, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-08T10:23:09.520000", "content": "

Congrats! I'm happy to see NN performing so well, gives me hope I can improve my NN model ;-)

", "votes": 1, "replies": [] }, { "id": 325296, "author_name": "Darragh", "author_url": "", "post_date": "2018-05-08T09:30:47.667000", "content": "

Thanks for the write up bestfitting - I was wondering if anyone would try RNN on this dataset :)

", "votes": 1, "replies": [] }, { "id": 325225, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-08T08:37:18.043000", "content": "

Congratulations on the approach and the result. Using RNN on click series is brillant, and using oof on all data is also a great approach.

", "votes": 1, "replies": [ { "id": 325718, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-08T19:31:47.007000", "content": "

As I can train an epoch in 120s-150s also,25 epoches is enough although I trained 30-35 epoches,OOF is not unacceptable,if we use binary format file instead of pandas, it is very quick.

", "votes": 1, "replies": [] } ] }, { "id": 325185, "author_name": "rocuku", "author_url": "", "post_date": "2018-05-08T07:47:30.667000", "content": "

Thanks for sharing and congratulations! It's nice to see a NN model act so well.
\nI wonder how to use RNN in this problem, is there any sequence data? Can't wait to see more details about this!

", "votes": 1, "replies": [ { "id": 325340, "author_name": "Mihai Cvasnievschi", "author_url": "", "post_date": "2018-05-08T10:16:08.410000", "content": "

@rocuku - data is actually a sequence of clicks and you can consider (ip, device, os) to be a \"user\". You can use an RNN to predict the next click for a given (ip, device, os) ... or to predict if the next click is_attributed ...

\n\n

My NN uses a pseudo RNN model, meaning I've created additional features that would store prev1_app, prev2_app, prev_click_time, prev1_click_time for (user = ip, device, os) where prev1 is the previous app/click_time, prev2 is previous previous ... This way I'm able to have a mini-history for the \"user\". I've created a sigmoid based attention model to act like gating for the vectors and it actually improved my score a lot.

\n\n

Initially I've considered running a full RNN but in the end I've considered having the data within the dataset and do a naive approach that would achieve similar results.

", "votes": 3, "replies": [] }, { "id": 325844, "author_name": "rocuku", "author_url": "", "post_date": "2018-05-09T00:37:52.273000", "content": "

Thanks for sharing, very inspiring!
\nI have never thought about RNN and attention model can use like this.

", "votes": 1, "replies": [] } ] }, { "id": 325180, "author_name": "Meyk", "author_url": "", "post_date": "2018-05-08T07:41:46.663000", "content": "

That's simply amazing! \nIt will be very enlightening to see your solution. As I am beggining with NN that would be great amount of knowledge ;)

\n\n

Congratulations!

", "votes": 1, "replies": [] }, { "id": 325175, "author_name": "Danijel Kivaranovic", "author_url": "", "post_date": "2018-05-08T07:32:46.617000", "content": "

Wow simply amazing what you did! You are truely by far the best Kaggler at the moment mastering any kind of competition! I need to read your solution carefully and try to catch up a bit :) Congrats!!!

", "votes": 1, "replies": [] }, { "id": 325170, "author_name": "KenMa", "author_url": "", "post_date": "2018-05-08T07:27:24.737000", "content": "

Congrats for the 3rd place @bestfitting. Just curious how a RNN could be implemented on this scenario. Chaining a series clicks of a specific ip? Would you mind sharing something more about the structure of the NN/RNN model?A short code snippet would be helpful :)

", "votes": 1, "replies": [] }, { "id": 325139, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-05-08T06:54:43.343000", "content": "

Congratulations..I was eagerly waiting for NN basee solutions to see.. Glad you mentioned it properly to some extend.. Thank you very much..

", "votes": 1, "replies": [] }, { "id": 325135, "author_name": "Eric", "author_url": "", "post_date": "2018-05-08T06:50:54.097000", "content": "

@bestfitting, thanks for sharing your solution. I am noob on NN and would like to gain more knowledge. Could you share more details about your CNN solutions and give some piece of code? Thanks in advance and well done for your 3rd position even though I imagine that droping from first to 3rd could be somehow frustrating. But for new comers like me, being 3rd is already a BIG ACHIEVEMENT and you should be proud of yourself!

", "votes": 1, "replies": [ { "id": 326559, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-09T23:17:51.217000", "content": "

@eric I edited the post fo answer you question,I hope it will be helpful.Yes,I am quite happy with the top 3 result.Thank you!

", "votes": 1, "replies": [] }, { "id": 327097, "author_name": "Eric", "author_url": "", "post_date": "2018-05-10T20:08:40.130000", "content": "

Thanks Bestfitting. You are an amazing competitor and I admire your fairness. You are my hero!

", "votes": 1, "replies": [] } ] }, { "id": 325114, "author_name": "kyowill", "author_url": "", "post_date": "2018-05-08T06:15:12.650000", "content": "

Can u share the feature engineering in this competition?

", "votes": 1, "replies": [ { "id": 325732, "author_name": "bestfitting", "author_url": "", "post_date": "2018-05-08T19:49:23.137000", "content": "

I suggest you read the great thread of @CPMP,I hope I can use his features in my model.

", "votes": 1, "replies": [] }, { "id": 326091, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-09T09:03:24.103000", "content": "

@bestfitting, thanks for referring to me. Let us know if you can improve your model with more features.

", "votes": 1, "replies": [] } ] }, { "id": 325103, "author_name": "olivier", "author_url": "", "post_date": "2018-05-08T05:58:38.453000", "content": "

@bestfitting, thanks for sharing and congratulations for your 3rd place !

", "votes": 1, "replies": [] }, { "id": 325096, "author_name": "Wesam Elshamy", "author_url": "", "post_date": "2018-05-08T05:45:02.033000", "content": "

Congrats @bestfitting and thanks for sharing. How can you view the private LB score before the competition ends?

", "votes": 1, "replies": [] }, { "id": 325091, "author_name": "Pengyue Wang", "author_url": "", "post_date": "2018-05-08T05:36:54.567000", "content": "

Thanks for sharing! I'll definitely spend more time on NN models.

", "votes": 1, "replies": [] }, { "id": 325073, "author_name": "", "author_url": "", "post_date": "2018-05-08T05:05:07.040000", "content": "", "votes": 1, "replies": [] }, { "id": 325068, "author_name": "", "author_url": "", "post_date": "2018-05-08T04:54:32.403000", "content": "", "votes": 1, "replies": [] }, { "id": 325062, "author_name": "", "author_url": "", "post_date": "2018-05-08T04:39:23.817000", "content": "", "votes": 1, "replies": [] }, { "id": 325058, "author_name": "", "author_url": "", "post_date": "2018-05-08T04:33:31.240000", "content": "", "votes": 1, "replies": [] }, { "id": 325612, "author_name": "", "author_url": "", "post_date": "2018-05-08T16:20:44.777000", "content": "", "votes": 2, "replies": [] }, { "id": 325202, "author_name": "", "author_url": "", "post_date": "2018-05-08T08:11:10.280000", "content": "", "votes": 2, "replies": [ { "id": 325724, "author_name": "", "author_url": "", "post_date": "2018-05-08T19:38:25.583000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 325130, "author_name": "", "author_url": "", "post_date": "2018-05-08T06:43:51.717000", "content": "", "votes": 2, "replies": [ { "id": 325731, "author_name": "", "author_url": "", "post_date": "2018-05-08T19:48:03.113000", "content": "", "votes": 3, "replies": [] }, { "id": 325758, "author_name": "", "author_url": "", "post_date": "2018-05-08T20:36:25.073000", "content": "", "votes": 0, "replies": [] }, { "id": 325766, "author_name": "", "author_url": "", "post_date": "2018-05-08T20:47:33.723000", "content": "", "votes": 2, "replies": [] }, { "id": 325781, "author_name": "", "author_url": "", "post_date": "2018-05-08T21:08:23.850000", "content": "", "votes": 0, "replies": [] }, { "id": 325796, "author_name": "", "author_url": "", "post_date": "2018-05-08T21:45:13.613000", "content": "", "votes": 1, "replies": [] }, { "id": 325953, "author_name": "", "author_url": "", "post_date": "2018-05-09T05:32:07.107000", "content": "", "votes": 0, "replies": [] }, { "id": 326558, "author_name": "", "author_url": "", "post_date": "2018-05-09T23:15:10.423000", "content": "", "votes": 0, "replies": [] }, { "id": 326665, "author_name": "", "author_url": "", "post_date": "2018-05-10T05:57:33.300000", "content": "", "votes": 2, "replies": [] }, { "id": 326739, "author_name": "", "author_url": "", "post_date": "2018-05-10T08:49:57.340000", "content": "", "votes": 1, "replies": [] }, { "id": 327096, "author_name": "", "author_url": "", "post_date": "2018-05-10T20:06:18.247000", "content": "", "votes": 0, "replies": [] }, { "id": 327703, "author_name": "", "author_url": "", "post_date": "2018-05-12T08:08:33.490000", "content": "", "votes": 2, "replies": [] }, { "id": 327821, "author_name": "", "author_url": "", "post_date": "2018-05-12T15:08:30.790000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 325095, "author_name": "", "author_url": "", "post_date": "2018-05-08T05:44:45.107000", "content": "", "votes": 2, "replies": [] }, { "id": 325057, "author_name": "", "author_url": "", "post_date": "2018-05-08T04:31:37.683000", "content": "", "votes": 2, "replies": [] }, { "id": 2477691, "author_name": "", "author_url": "", "post_date": "2023-10-11T13:52:33.107000", "content": "", "votes": 0, "replies": [] }, { "id": 517655, "author_name": "", "author_url": "", "post_date": "2019-04-16T10:19:37.500000", "content": "", "votes": 0, "replies": [] }, { "id": 433060, "author_name": "", "author_url": "", "post_date": "2018-12-04T16:06:55.147000", "content": "", "votes": 0, "replies": [] }, { "id": 397325, "author_name": "", "author_url": "", "post_date": "2018-10-02T10:33:42.113000", "content": "", "votes": 0, "replies": [] }, { "id": 334056, "author_name": "", "author_url": "", "post_date": "2018-05-26T12:20:48.673000", "content": "", "votes": 0, "replies": [ { "id": 334162, "author_name": "", "author_url": "", "post_date": "2018-05-26T17:01:35.233000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 325774, "author_name": "", "author_url": "", "post_date": "2018-05-08T21:01:49.110000", "content": "", "votes": 0, "replies": [ { "id": 325782, "author_name": "", "author_url": "", "post_date": "2018-05-08T21:08:47.383000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 325280, "author_name": "", "author_url": "", "post_date": "2018-05-08T09:18:38.563000", "content": "", "votes": 0, "replies": [] }, { "id": 1210673, "author_name": "", "author_url": "", "post_date": "2021-02-19T15:47:07.387000", "content": "", "votes": 0, "replies": [] }, { "id": 329243, "author_name": "", "author_url": "", "post_date": "2018-05-16T04:37:49.910000", "content": "", "votes": 1, "replies": [] }, { "id": 327800, "author_name": "", "author_url": "", "post_date": "2018-05-12T14:23:10.870000", "content": "", "votes": 1, "replies": [] }, { "id": 1495840, "author_name": "", "author_url": "", "post_date": "2021-08-29T20:37:54.023000", "content": "", "votes": 1, "replies": [] }, { "id": 332869, "author_name": "", "author_url": "", "post_date": "2018-05-24T00:15:13.547000", "content": "", "votes": 1, "replies": [] }, { "id": 329694, "author_name": "", "author_url": "", "post_date": "2018-05-17T02:12:14.787000", "content": "", "votes": 1, "replies": [] }, { "id": 328329, "author_name": "", "author_url": "", "post_date": "2018-05-14T01:44:30.880000", "content": "", "votes": 1, "replies": [] }, { "id": 328140, "author_name": "", "author_url": "", "post_date": "2018-05-13T13:17:30.450000", "content": "", "votes": 1, "replies": [] }, { "id": 327554, "author_name": "", "author_url": "", "post_date": "2018-05-11T21:11:09.980000", "content": "", "votes": 1, "replies": [] }, { "id": 326811, "author_name": "", "author_url": "", "post_date": "2018-05-10T11:24:28.040000", "content": "", "votes": 1, "replies": [] }, { "id": 326628, "author_name": "", "author_url": "", "post_date": "2018-05-10T04:39:40.997000", "content": "", "votes": 1, "replies": [] }, { "id": 325789, "author_name": "", "author_url": "", "post_date": "2018-05-08T21:24:03.147000", "content": "", "votes": 1, "replies": [] }, { "id": 325145, "author_name": "", "author_url": "", "post_date": "2018-05-08T07:04:46.867000", "content": "", "votes": 1, "replies": [] }, { "id": 325110, "author_name": "", "author_url": "", "post_date": "2018-05-08T06:09:05.527000", "content": "", "votes": 1, "replies": [] }, { "id": 1145479, "author_name": "", "author_url": "", "post_date": "2021-01-09T07:09:12.300000", "content": "", "votes": 0, "replies": [] }, { "id": 334502, "author_name": "", "author_url": "", "post_date": "2018-05-27T16:18:22.217000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "325054": "Congrats to all the winners(['flowlight', 'komaki'].shuffle(),PPP is already in use) and all the kagglers who have worked hard and have learned a lot of from this competition.
\n\nThanks to TalkingData and Kaggle for such an interesting competition.
\n\nHere is the summary of my solution,it is mainly based on NN models.
\n\nIt was very hard for me to choose among Landmark Competitions and TalkingData's,so I started them at the same time,when I found I can get decent NN models, I focus on this one,it is always interesting to solve a problem other than CV/NLP using NN models.
\n\nMy models mainly based on 23 features,by using these features,my single LGBM model scored 0.9817 on public LB,it is not a good one compared to other kagglers,this is the first time that I used LGBM in kaggle competition indeed,so there are a lot to learn from you!
\n\nI designed NN models based on those 23 features,and prepared the features to fed them into network carefully[NA,out of vocabulary,log,scale],my NN model can reach 0.9820 on public LB.As we all know, the click delta is important,so I fed deltas of last 5 and next 5 click_times to the network and designed a model with RNN cell to find the patterns of the click series,my model can reach 0.9821 on public LB and 0.9830 on private LB.
\n\nThen I designed different NN models to add diversities,they are very simple,for example,adding some res-links to dense layers.I don’t know the single model performance of these 4 models because I judged them only by diversities.
\n\nAfter have ensembled my NN models and LGBM models by weighted average,I can get 0.9827 on public LB and 0.9835 on private LB.
\n\nThen,I predicted full set of train data on my n-fold models,it’s a little time consuming,but it’s a relatively small dataset for me when compared to other datasets I have met,I can train and predict a fold of my model in 2.x hours on a 1080i GPU.
\n\nI trained second level NN models using predictions from the whole train and test dataset,and added some group by features based on IP,app-os-channel,my ensemble score improved to 0.9833 on public LB and 0.9840 on private LB,which is a huge improvement.
\n\nThis improvement happened at 30 hours before the competition end,I hope I could get such an improvement two days earlier,because I had no time to solve some limitations/weaknesses of my NN models,as I simulated on part of the data,I found NN models may led to a small drop(0.0003 also) on private dataset in some situation,which can explain my minor drop on private LB.So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :) I had thought I will be very happy if I am still in top 5 when the private LB revealed
\n\nSorry for not so detailed,I feel quite sleepy now.
\n\n------------------------------------Some Details------------------------------
\nThe features:
\n
channel                                  1011\nos                                        544\nhour                                      472\napp                                       468\nip_app_os_device_day_click_time_next_1     320\napp_channel_os_mean_is_attributed         189\nip_app_mean_is_attributed                 124\nip_app_os_device_day_click_time_next_2     120\nip_os_device_count_click_id               113\nip_var_hour                                94\nip_day_hour_count_click_id                 91\nip_mean_is_attributed                      74\nip_count_click_id                          73\nip_app_os_device_day_click_time_lag1       67\napp_mean_is_attributed                     67\nip_nunique_os_device                       65\nip_nunique_app                             63\nip_nunique_os                              51\nip_nunique_app_channel                     49\nip_os_device_mean_is_attributed            46\ndevice                                     41\napp_channel_os_count_click_id              37\nip_hour_mean_is_attributed                 21\n
\n\na simple GRU network:
\n
class GRU_V0a():\n    def __init__(self, **kw):\n        super(GRU_V0a, self).__init__(**kw)\n\tself.categorical=['app', 'device', 'os', 'channel', 'hour']\n\tself.continous=[col for col in features if col not in self.categorical]\n        self.categorical_num = {\n            'app': (769, 16),\n            'device': (4228, 16),\n            'os': (957, 16),\n            'channel': (501, 8),\n            'hour': (24, 8),\n        }\n    def build_model(self):\n        categorial_inp = Input(shape=(len(self.categorical),))\n        cat_embeds = []\n        for idx, col in enumerate(self.categorical):\n            x = Lambda(lambda x: x[:, idx,None])(categorial_inp)\n            x = Embedding(self.categorical_num[col][0], self.categorical_num[col][1],input_length=1)(x)\n            cat_embeds.append(x)\n        embeds = concatenate(cat_embeds, axis=2)\n        embeds = GaussianDropout(0.2)(embeds)\n        continous_inp = Input(shape=(len(self.continous),))\n        cx = Reshape([1,len(self.continous)])(continous_inp)\n        x = concatenate([embeds, cx], axis=2)\n        x = CuDNNGRU(128)(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(64)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.20)(x)\n        x = Dense(32)(x)\n        x = PReLU()(x)\n        x = BatchNormalization()(x)\n        x = Dropout(0.05)(x)\n        outp = Dense(1, activation='sigmoid')(x)\n        model = Model(inputs=[categorial_inp, continous_inp], output=outp)\n        print(model.summary())\n        return model\n
\nThanks [@aharless][1] publish the code which I used as a reference,such as GaussianDropout.
\nIt's really simple? :)And the devil is in detail:
\nTo make NN model work,preprocess is very important.
\nFill-NA:Fill max value for delta feature.Fill mean for other feature.
\nLog of delta features,part of count features when the values are large and all the nunique features.
\nStandardScale
\n\nIt is not a tranditional RNN,the input_lenght=1 indeed,so I just used the function of CudnnGRU to get pattern of clicks which can be expressed like the following:\n
zx=sigmoid(K.dot(Wz,X)\nhx=tanh(K.dot(W,X)\nh=zx*hx\n
\n\nAs we want to know the theory behind the stucture,and I have no time to prove it or write a paper,\nwe can have a look at the paper by Google:[Searching for Activation Functions][2] Swish:x · σ(βx), where σ(z) = (1 + exp(−z))−1,we can get some ideas to explain my setting.\n\nAs to private score estimation,it's always a interesting part of my competition :).
\nThere is not certain method to do so,I just bear in mind:
\nDistribution variances lead to score variances.
\nFor example,in this competition,some category values in test-set are not in trainset,so I changed the same ratio of category value of validation set to values unseen in trainset.And the App19 is a very important app with high ratios of download and is imbalance in train and test-set.What's more,the ratio is differenct between the public and private set,so I tried to keep the ratio of my validatition as test set...We can also use a submission which I believe it is stable as True label,and caculate AUC based on it,if the public and private score as close,then we can believe it too. \n
\nThanks for all the congrats to me ,I will upvote your comment and will not reply to everyone to save space of this page.\n
\n\n\n [1]: https://www.kaggle.com/aharless/gpu-nn-validation-more-features/code\n [2]: http://Searching%20for%20Activation%20Functions", "325195": "wow brilliant, never see RNN performed so well", "325193": "Congrats! I am very impressed by the performances of your NN models. I wonder what are the detailed structures of your NNs? You have mentioned RNN and the res-links so I am very curious about the details. ", "325317": "Congrats @bestfitting and thanks for sharing the approach. ", "328716": "Truly good stuff!!", "328404": "Thanks for sharing.\nIt's too great resource to learn.", "327957": "Congratulations and thank you so much for sharing! \nI have a little question when you talk about preprocessing for NN, what make you decide to use \n'Fill-NA:Fill max value for delta feature.Fill mean for other feature'\ninstead of filling mean for all features?", "327272": "cool", "326626": "Thanks for sharing your solution, I finally lost to you, but really enjoyed competing with you :)
\nbtw, I couldn't understand \"I trained second level NN models using predictions from the whole train and test dataset\".\nDoes it mean you did pseudo-labelling, or is it just Stacking? could you please tell me what you use for training data and target?", "326087": "Very thanks for your sharing。I have some small questions and look forward to your rely。How do you decide your NN models hyperparameters and your NN architecture？", "326015": "Congratulations", "325962": "Congratulations on your 3rd place. \nI had a feeling that NN will not do so well in this problem. But it is nice to see it doing so well. Thanks for sharing your approach. ", "325938": "Thank you for your sharing. I did my last submission on the train to Changsha. Just happened to find that you work at the city.", "325906": "Congratulations. I made a plan to use RNN to realise it at first, but failed. Could you opensource your program and share some more experience? Thanks.", "325878": " wow brilliant. I have a question about target encoder: how do you calculate the features like \"ip_app_mean_is_attributed\"? \n some methods like calculate target mean of yesterday or some other skills?? thanks ", "325825": "Congratulations @bestfitting and thank you very much for sharing your solution. I could barely beat @Andy Harless NN's performance so I gave up on my NN model. You have shown me how best to use it with this data. That is a lesson learnt for life :-) I will certainly look into the activation function described, I if i have any ideas, I will pm you. \n\nHappy Kaggling and see u in the next one!!!", "325542": "Congrats and thanks for sharing! I have 2 small questions.\n\n1. What is res-links? (I'm not good at deep learning, so maybe it is a stupy question...\n2. How big is your video memory? Intuitively, video memory should be not able to load this data set. So it is hard to use GPU in my opinion. Could you explain it?\n\nThanks~", "325349": "Congrats! I'm happy to see NN performing so well, gives me hope I can improve my NN model ;-)", "325296": "Thanks for the write up bestfitting - I was wondering if anyone would try RNN on this dataset :) ", "325225": "Congratulations on the approach and the result. Using RNN on click series is brillant, and using oof on all data is also a great approach.", "325185": "Thanks for sharing and congratulations! It's nice to see a NN model act so well. \nI wonder how to use RNN in this problem, is there any sequence data? Can't wait to see more details about this!", "325180": "That's simply amazing! \nIt will be very enlightening to see your solution. As I am beggining with NN that would be great amount of knowledge ;)\n\nCongratulations!", "325175": "Wow simply amazing what you did! You are truely by far the best Kaggler at the moment mastering any kind of competition! I need to read your solution carefully and try to catch up a bit :) Congrats!!!", "325170": "Congrats for the 3rd place @bestfitting. Just curious how a RNN could be implemented on this scenario. Chaining a series clicks of a specific ip? Would you mind sharing something more about the structure of the NN/RNN model?A short code snippet would be helpful :)", "325139": "Congratulations..I was eagerly waiting for NN basee solutions to see.. Glad you mentioned it properly to some extend.. Thank you very much..", "325135": "[@bestfitting](https://www.kaggle.com/bestfitting), thanks for sharing your solution. I am noob on NN and would like to gain more knowledge. Could you share more details about your CNN solutions and give some piece of code? Thanks in advance and well done for your 3rd position even though I imagine that droping from first to 3rd could be somehow frustrating. But for new comers like me, being 3rd is already a BIG ACHIEVEMENT and you should be proud of yourself!", "325114": "Can u share the feature engineering in this competition?", "325103": "@bestfitting, thanks for sharing and congratulations for your 3rd place !", "325096": "Congrats @bestfitting and thanks for sharing. How can you view the private LB score before the competition ends?", "325091": "Thanks for sharing! I'll definitely spend more time on NN models.", "325073": "Thanks for sharing and congratulations! It's great to hear from the winners.", "325068": "Encouraging result for deep learning enthusiast!", "325062": "Congrats and thanks for sharing! \ndoes nn model is better than lgb for this competition？", "325058": "god-like performance.", "325612": "Awesome to know an NN solution could be used to solve a problem that people mostly used LGB for. Thanks for your write up.", "325202": "Congrats bestfitting ...Truly impressed by your talent.\n\n\n> So when you were talking about 0.9835 solution,I wanted to tell you don’t expect too much on my solution :) \n\nIf you managed to achieve it...you wouldn't have no excuse to do not share it on Bojan's thread ^^", "325130": "Congratulations and nice work!\n\nI have some questions:\n\n 1. How did you choice features? \n 2. NN? You mean full connection neuron network with multi layers?\n 3. RNN? I do not know how to apply RNN(Recurrent Neural Network) on this conpetition.How to organize the features to apply RNN.\n 4. You do not mention ensemble, you do not ensemble your results?", "325095": "Congratulations and good work!", "325057": "Congrats and thanks for sharing! Absolutely god-like performance. Have a nice dream. :)", "2477691": "Excited to see the code! Thanks for sharing it. ", "517655": "Great ! following you", "433060": "nice\n", "397325": "Thanks for sharing your approach!! This is really cool!", "334056": "Thank you for sharing! It's amazing.\nHere are quick questions about features.\n\n1. What is a difference between \" ip_app_os_device_day_click_time_next_1 \" and \" ip_app_os_device_day_click_time_next_2 \" ?\n2. How did you make \" ip_app_os_device_day_click_time_lag1 \" ? Is it a kind of previous click time feature?\nI guess it is a stored feature of previous click time features. right?\n\nI fount clues in what you commented below: \nI've created additional features that would store prev1_app, prev2_app, prev_click_time, prev1_click_time for (user = ip, device, os) where prev1 is the previous app/click_time, prev2 is previous previous ... This way I'm able to have a mini-history for the \"user\". \n\nThanks again! I've learned from you a lot.\n", "325774": "@bestfitting my only question to you is what loss function you used. \n\nI've tried both cross entropy and mse and found out that better ROC AUC doesn't properly reflect into lower loss. My best AUC scores were not generated by lowest loss. I've even experimented with a blend between mse and cross entropy but couldn't managed to find a good balance. I had an epoch with a very very low mse (0.0004) loss that had the best AUC one would dream on ... 0.5 :-) ", "325280": "Congratulations and thanks for sharing!\nThe nn model is amazing, could you shame more detail for us?\nthanks!", "1210673": "", "329243": "", "327800": "", "1495840": "Thanks for sharing!! very helpful\n\n", "332869": "thanks for sharing", "329694": "Thanks for sharing", "328329": "Thanks for sharing", "328140": "Thanks for sharing!", "327554": "Thanks for sharing!", "326811": "congrats and thanks! :D", "326628": "very thanks for sharing your approach:)", "325789": "Thank you! Learn a lot from you!", "325145": "Congratulations and thanks for sharing!", "325110": "Congratulations and thanks for sharing!", "1145479": "thanks for sharing!", "334502": "thanks for sharing" } }