I referred kernel . Kindly help me to understand
\n\n\n", "messages": [ { "id": "323406", "postDate": "05/05/2018 03:45:30", "content": "\nD = 2 ** 26\ntrain_df['category'] = (train_df['ip'].astype(str) + \"_\" + train_df['app'].astype(str) + \"_\" + \n train_df['device'].astype( str) + \"_\" + train_df['os'].astype(str)).apply(hash) % D\nclick_buffer = np.full(D, 3000000000, dtype=np.uint32)\ntrain_df['epochtime'] = train_df['click_time'].astype(np.int64) // 10 ** 9\nnext_clicks = []\nfor category, t in zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values)):\n next_clicks.append(click_buffer[category] - t)\n click_buffer[category] = t\n
I referred kernel . Kindly help me to understand
\n\n\n", "rawMarkdown": "I referred [kernel][1] . Kindly help me to understand \n\n - Why should we need to use \"3000000000\" here? \n - Why should we divide cilck_time with 10 ** 9? \n\n> D = 2 ** 26\n> train_df['category'] = (train_df['ip'].astype(str) + \"_\" + train_df['app'].astype(str) + \"_\" + \n> train_df['device'].astype( str) + \"_\" + train_df['os'].astype(str)).apply(hash) % D\n> click_buffer = np.full(D, 3000000000, dtype=np.uint32)\n> train_df['epochtime'] = train_df['click_time'].astype(np.int64) // 10 ** 9\n> next_clicks = []\n> for category, t in zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values)):\n> next_clicks.append(click_buffer[category] - t)\n> click_buffer[category] = t\n\n \n\n\n [1]: https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977", "votes": null }, { "id": "323420", "postDate": "05/05/2018 05:23:46", "content": "\nD = 2 ** 26\ntrain_df['category'] = (train_df['ip'].astype(str) + \"_\" + train_df['app'].astype(str) + \"_\" + \n train_df['device'].astype( str) + \"_\" + train_df['os'].astype(str)).apply(hash) % D\nclick_buffer = np.full(D, 3000000000, dtype=np.uint32)\ntrain_df['epochtime'] = train_df['click_time'].astype(np.int64) // 10 ** 9\nnext_clicks = []\nfor category, t in zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values)):\n next_clicks.append(click_buffer[category] - t)\n click_buffer[category] = t\n
Use this kernel by Andrii Sydorchuk for FE'ing nextclick instead. It's a lot faster at building this feature and you don't have to worry about the remotely minute potential for hash conflicts. But to answer your Qs, the 300xxx number is just a large placeholder (think of it has fillna placeholder that fits in uint32 but doesn't occur in our samples). And the 10 ** 9 is to get you down to seconds, which is the highest resolution we're provided in the dataset anyway.
", "rawMarkdown": "Use [this kernel][1] by Andrii Sydorchuk for FE'ing nextclick instead. It's a lot faster at building this feature and you don't have to worry about the remotely minute potential for hash conflicts. But to answer your Qs, the 300xxx number is just a large placeholder (think of it has fillna placeholder that fits in uint32 but doesn't occur in our samples). And the 10 ** 9 is to get you down to seconds, which is the highest resolution we're provided in the dataset anyway.\n\n\n [1]: https://www.kaggle.com/asydorchuk", "votes": null }, { "id": "323424", "postDate": "05/05/2018 05:49:39", "content": "thank you
", "rawMarkdown": "thank you", "votes": null }, { "id": "323436", "postDate": "05/05/2018 06:35:51", "content": "little curious. Kindly help me.
\n\npython\n zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values))\n python\n next_clicks.append(click_buffer[category] - t)\n@ Pallavi
\n\nI'l answer your second question first and that leads into the first one
\n\nThis click time can be calculated without the hashing trick as you may know but will create a NAN value for the last record in every group since there is no record you can compare it against. A default value is therefore used which is the 3 billion from which the click_time of the last record is subtracted
\n\nWhy Reversed - Records are in chronological order of click time. So you get the last click of every category and calculate the time delta using the 3B value and then go on to the clicks earlier in time.
\n\nHope this helps \nRegards\nShanth
", "rawMarkdown": "Pallavi \n\nI'l answer your second question first and that leads into the first one \n\nThis click time can be calculated without the hashing trick as you may know but will create a NAN value for the last record in every group since there is no record you can compare it against. A default value is therefore used which is the 3 billion from which the click_time of the last record is subtracted \n\nWhy Reversed - Records are in chronological order of click time. So you get the last click of every category and calculate the time delta using the 3B value and then go on to the clicks earlier in time. \n\nHope this helps \nRegards\nShanth", "votes": null }, { "id": "323727", "postDate": "05/06/2018 02:26:14", "content": "Thanks for response. Its useful.\nI agree with your answer for 2nd question. But, Why only 3B ? Why not any other number?
", "rawMarkdown": "Thanks for response. Its useful.\nI agree with your answer for 2nd question. But, Why only 3B ? Why not any other number?", "votes": null }, { "id": "323758", "postDate": "05/06/2018 05:27:28", "content": "When you notice the value generated for the column epoch time, its a very large INT. Choosing 3 Billion could have to do with the max value of the epoch time field.
", "rawMarkdown": "When you notice the value generated for the column epoch time, its a very large INT. Choosing 3 Billion could have to do with the max value of the epoch time field.", "votes": null } ], "comments": [ { "id": 323420, "author_name": "authman", "author_url": "", "post_date": "05/05/2018 05:23:46", "content": "Use this kernel by Andrii Sydorchuk for FE'ing nextclick instead. It's a lot faster at building this feature and you don't have to worry about the remotely minute potential for hash conflicts. But to answer your Qs, the 300xxx number is just a large placeholder (think of it has fillna placeholder that fits in uint32 but doesn't occur in our samples). And the 10 ** 9 is to get you down to seconds, which is the highest resolution we're provided in the dataset anyway.
", "votes": null, "replies": [ { "id": 323424, "author_name": "pallaviroyal", "author_url": "", "post_date": "05/05/2018 05:49:39", "content": "thank you
", "votes": null, "replies": [] } ] }, { "id": 323436, "author_name": "pallaviroyal", "author_url": "", "post_date": "05/05/2018 06:35:51", "content": "little curious. Kindly help me.
\n\npython\n zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values))\n python\n next_clicks.append(click_buffer[category] - t)\n@ Pallavi
\n\nI'l answer your second question first and that leads into the first one
\n\nThis click time can be calculated without the hashing trick as you may know but will create a NAN value for the last record in every group since there is no record you can compare it against. A default value is therefore used which is the 3 billion from which the click_time of the last record is subtracted
\n\nWhy Reversed - Records are in chronological order of click time. So you get the last click of every category and calculate the time delta using the 3B value and then go on to the clicks earlier in time.
\n\nHope this helps \nRegards\nShanth
", "votes": null, "replies": [] }, { "id": 323727, "author_name": "pallaviroyal", "author_url": "", "post_date": "05/06/2018 02:26:14", "content": "Thanks for response. Its useful.\nI agree with your answer for 2nd question. But, Why only 3B ? Why not any other number?
", "votes": null, "replies": [] }, { "id": 323758, "author_name": "shanth84", "author_url": "", "post_date": "05/06/2018 05:27:28", "content": "When you notice the value generated for the column epoch time, its a very large INT. Choosing 3 Billion could have to do with the max value of the epoch time field.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "323406": "I referred [kernel][1] . Kindly help me to understand \n\n - Why should we need to use \"3000000000\" here? \n - Why should we divide cilck_time with 10 ** 9? \n\n> D = 2 ** 26\n> train_df['category'] = (train_df['ip'].astype(str) + \"_\" + train_df['app'].astype(str) + \"_\" + \n> train_df['device'].astype( str) + \"_\" + train_df['os'].astype(str)).apply(hash) % D\n> click_buffer = np.full(D, 3000000000, dtype=np.uint32)\n> train_df['epochtime'] = train_df['click_time'].astype(np.int64) // 10 ** 9\n> next_clicks = []\n> for category, t in zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values)):\n> next_clicks.append(click_buffer[category] - t)\n> click_buffer[category] = t\n\n \n\n\n [1]: https://www.kaggle.com/bk0000/non-blending-lightgbm-model-lb-0-977", "323420": "Use [this kernel][1] by Andrii Sydorchuk for FE'ing nextclick instead. It's a lot faster at building this feature and you don't have to worry about the remotely minute potential for hash conflicts. But to answer your Qs, the 300xxx number is just a large placeholder (think of it has fillna placeholder that fits in uint32 but doesn't occur in our samples). And the 10 ** 9 is to get you down to seconds, which is the highest resolution we're provided in the dataset anyway.\n\n\n [1]: https://www.kaggle.com/asydorchuk", "323424": "thank you", "323436": "little curious. Kindly help me.\n\n - Why should we do reverse? \n ```python\n zip(reversed(train_df['category'].values), reversed(train_df['epochtime'].values))\n ``` \n - Why should we minus click_time from click_buffer placeholder value(3000xxx)? \n ```python\n next_clicks.append(click_buffer[category] - t)\n ```", "323665": "Pallavi \n\nI'l answer your second question first and that leads into the first one \n\nThis click time can be calculated without the hashing trick as you may know but will create a NAN value for the last record in every group since there is no record you can compare it against. A default value is therefore used which is the 3 billion from which the click_time of the last record is subtracted \n\nWhy Reversed - Records are in chronological order of click time. So you get the last click of every category and calculate the time delta using the 3B value and then go on to the clicks earlier in time. \n\nHope this helps \nRegards\nShanth", "323727": "Thanks for response. Its useful.\nI agree with your answer for 2nd question. But, Why only 3B ? Why not any other number?", "323758": "When you notice the value generated for the column epoch time, its a very large INT. Choosing 3 Billion could have to do with the max value of the epoch time field." }, "source": "meta" }