{ "id": 55304, "title": "open solution (late start) [0.9567]", "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/55304", "author_name": "", "post_date": "2018-04-25T01:08:47.082866Z", "votes": 5, "comment_count": 7, "views": 0, "content": "

Hi,

\n\n

Jakub Czakon and me would like to make some (quite) late start of the open solution to this competition. We are opening our code, available on this repository. If you would like to use it just go ahead and do it.

\n\n

We are open for your questions and suggestions what we should develop next :)

\n\n

We will be improving this solution for the next two weeks -> we are very curious how much work we will be able to contribute within this time period.

\n\n

Happy Training :)

", "messages": [ { "id": "318985", "postDate": "04/25/2018 01:08:47", "content": "

Hi,

\n\n

Jakub Czakon and me would like to make some (quite) late start of the open solution to this competition. We are opening our code, available on this repository. If you would like to use it just go ahead and do it.

\n\n

We are open for your questions and suggestions what we should develop next :)

\n\n

We will be improving this solution for the next two weeks -> we are very curious how much work we will be able to contribute within this time period.

\n\n

Happy Training :)

", "rawMarkdown": "Hi,\n\nJakub Czakon and me would like to make some (quite) late start of the open solution to this competition. We are opening our code, available on this [repository](https://github.com/neptune-ml/open-solution-talking-data). If you would like to use it just go ahead and do it.\n\nWe are open for your questions and suggestions what we should develop next :)\n\nWe will be improving this solution for the next two weeks -> we are very curious how much work we will be able to contribute within this time period.\n\nHappy Training :)", "votes": null }, { "id": "319006", "postDate": "04/25/2018 02:53:49", "content": "

Hey, guys! Welcome to the game.

\n\n

Thanks for sharing your results. And thanks for sharing information on the Neptune platform. I'll have to check it out.

", "rawMarkdown": "Hey, guys! Welcome to the game.\n\nThanks for sharing your results. And thanks for sharing information on the Neptune platform. I'll have to check it out.", "votes": null }, { "id": "319556", "postDate": "04/26/2018 09:42:20", "content": "

Hi kamil:

\n\n

got KeyError: 'Column not found: is_attributed' as shown below with command line : python main.py -- evaluate --pipeline_name solution_1, any lead to solve this? command line: python main.py -- train --pipeline_name solution_1 sh appeared without error.

\n\n
\n

(neptune) Haos-iMac:neptune-talking-data-solution1 haotienlee$ python main.py -- evaluate --pipeline_name solution_1\n neptune: Executing in Offline Mode.\n neptune: Executing in Offline Mode.\n 2018-04-26 17-24-59 talking-data >>> reading data in\n 0%| | 0/1 [00:00>> read in chunk ./input/talking_data/train_day9_hour4.csv of shape (4032691, 7)\n 100%|█████████████████████████████████████████████| 1/1 [00:02<00:00, 2.66s/it]\n 2018-04-26 17-25-03 talking-data >>> combined dataset shape: (4032691, 7)\n 2018-04-26 17-25-06 talking-data >>> Target distribution in valid: 0.002396910648497492\n 2018-04-26 17-25-06 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 17-25-06 steps >>> step feature_by_type_split loading transformer...\n 2018-04-26 17-25-06 steps >>> step feature_by_type_split transforming...\n 2018-04-26 17-25-06 steps >>> step time_delta adapting inputs\n 2018-04-26 17-25-06 steps >>> step time_delta loading transformer...\n 2018-04-26 17-25-06 steps >>> step time_delta transforming...\n 2018-04-26 17-29-02 steps >>> step time_delta caching outputs...\n 2018-04-26 17-29-03 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 17-29-03 steps >>> step feature_by_type_split loading transformer...\n 2018-04-26 17-29-03 steps >>> step feature_by_type_split transforming...\n 2018-04-26 17-29-04 steps >>> step confidence_rates adapting inputs\n 2018-04-26 17-29-04 steps >>> step confidence_rates loading transformer...\n 2018-04-26 17-29-04 steps >>> step confidence_rates transforming...\n Traceback (most recent call last):\n File \"main.py\", line 258, in \n action()\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 722, in call\n return self.main(*args, **kwargs)\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 697, in main\n rv = self.invoke(ctx)\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 1066, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 895, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 535, in invoke\n return callback(*args, **kwargs)\n File \"main.py\", line 97, in evaluate\n _evaluate(pipeline_name, dev_mode)\n File \"main.py\", line 129, in _evaluate\n output = pipeline.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 151, in transform\n step_inputs[input_step.name] = input_step.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 151, in transform\n step_inputs[input_step.name] = input_step.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 151, in transform\n step_inputs[input_step.name] = input_step.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 157, in transform\n step_output_data = self._cached_transform(step_inputs)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 165, in _cached_transform\n step_output_data = self.transformer.transform(**step_inputs)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/feature_extraction.py\", line 189, in transform\n group_object['is_attributed'].apply(self._rate_calculation).reset_index().rename(\n File \"//anaconda/envs/neptune/lib/python3.6/site-packages/pandas/core/base.py\", line 259, in getitem\n raise KeyError(\"Column not found: {key}\".format(key=key))\n KeyError: 'Column not found: is_attributed'

\n \n

(neptune) Haos-iMac:neptune-talking-data-solution1 haotienlee$ python main.py -- train --pipeline_name solution_1\n neptune: Executing in Offline Mode.\n neptune: Executing in Offline Mode.\n 2018-04-26 14-51-05 talking-data >>> reading data in\n 0%| | 0/48 [00:00>> read in chunk ./input/talking_data/train_day7_hour0.csv of shape (3604365, 7)\n 2%|▉ | 1/48 [00:02<02:19, 2.97s/it]2018-04-26 14-51-10 talking-data >>> read in chunk ./input/talking_data/train_day7_hour1.csv of shape (3308150, 7)\n 4%|█▊ | 2/48 [00:05<02:07, 2.78s/it]2018-04-26 14-51-12 talking-data >>> read in chunk ./input/talking_data/train_day7_hour2.csv of shape (3095633, 7)\n 6%|██▊ | 3/48 [00:07<01:56, 2.59s/it]2018-04-26 14-51-15 talking-data >>> read in chunk ./input/talking_data/train_day7_hour3.csv of shape (3220271, 7)\n 8%|███▋ | 4/48 [00:09<01:50, 2.50s/it]2018-04-26 14-51-18 talking-data >>> read in chunk ./input/talking_data/train_day7_hour4.csv of shape (3645493, 7)\n 10%|████▌ | 5/48 [00:13<02:02, 2.85s/it]2018-04-26 14-51-21 talking-data >>> read in chunk ./input/talking_data/train_day7_hour5.csv of shape (3227349, 7)\n 12%|█████▌ | 6/48 [00:15<01:51, 2.66s/it]2018-04-26 14-51-23 talking-data >>> read in chunk ./input/talking_data/train_day7_hour6.csv of shape (2917284, 7)\n 15%|██████▍ | 7/48 [00:17<01:40, 2.46s/it]2018-04-26 14-51-25 talking-data >>> read in chunk ./input/talking_data/train_day7_hour7.csv of shape (2924033, 7)\n 17%|███████▎ | 8/48 [00:19<01:33, 2.35s/it]2018-04-26 14-51-27 talking-data >>> read in chunk ./input/talking_data/train_day7_hour8.csv of shape (2801293, 7)\n 19%|████████▎ | 9/48 [00:21<01:27, 2.24s/it]2018-04-26 14-51-29 talking-data >>> read in chunk ./input/talking_data/train_day7_hour9.csv of shape (2961319, 7)\n 21%|████████▉ | 10/48 [00:23<01:22, 2.17s/it]2018-04-26 14-51-31 talking-data >>> read in chunk ./input/talking_data/train_day7_hour10.csv of shape (3300746, 7)\n 23%|█████████▊ | 11/48 [00:25<01:21, 2.20s/it]2018-04-26 14-51-33 talking-data >>> read in chunk ./input/talking_data/train_day7_hour11.csv of shape (3134200, 7)\n 25%|██████████▊ | 12/48 [00:28<01:18, 2.19s/it]2018-04-26 14-51-35 talking-data >>> read in chunk ./input/talking_data/train_day7_hour12.csv of shape (3000091, 7)\n 27%|███████████▋ | 13/48 [00:30<01:15, 2.17s/it]2018-04-26 14-51-38 talking-data >>> read in chunk ./input/talking_data/train_day7_hour13.csv of shape (3235239, 7)\n 29%|████████████▌ | 14/48 [00:33<01:19, 2.35s/it]2018-04-26 14-51-40 talking-data >>> read in chunk ./input/talking_data/train_day7_hour14.csv of shape (3161797, 7)\n 31%|█████████████▍ | 15/48 [00:35<01:17, 2.35s/it]2018-04-26 14-51-42 talking-data >>> read in chunk ./input/talking_data/train_day7_hour15.csv of shape (2864021, 7)\n 33%|██████████████▎ | 16/48 [00:37<01:12, 2.26s/it]2018-04-26 14-51-44 talking-data >>> read in chunk ./input/talking_data/train_day7_hour16.csv of shape (2314780, 7)\n 35%|███████████████▏ | 17/48 [00:39<01:03, 2.06s/it]2018-04-26 14-51-45 talking-data >>> read in chunk ./input/talking_data/train_day7_hour17.csv of shape (1263077, 7)\n 38%|████████████████▏ | 18/48 [00:39<00:51, 1.71s/it]2018-04-26 14-51-45 talking-data >>> read in chunk ./input/talking_data/train_day7_hour18.csv of shape (726684, 7)\n 40%|█████████████████ | 19/48 [00:40<00:39, 1.35s/it]2018-04-26 14-51-46 talking-data >>> read in chunk ./input/talking_data/train_day7_hour19.csv of shape (493056, 7)\n 42%|█████████████████▉ | 20/48 [00:40<00:29, 1.06s/it]2018-04-26 14-51-46 talking-data >>> read in chunk ./input/talking_data/train_day7_hour20.csv of shape (409255, 7)\n 44%|██████████████████▊ | 21/48 [00:41<00:22, 1.20it/s]2018-04-26 14-51-46 talking-data >>> read in chunk ./input/talking_data/train_day7_hour21.csv of shape (518866, 7)\n 46%|███████████████████▋ | 22/48 [00:41<00:18, 1.43it/s]2018-04-26 14-51-48 talking-data >>> read in chunk ./input/talking_data/train_day7_hour22.csv of shape (1172535, 7)\n 48%|████████████████████▌ | 23/48 [00:42<00:20, 1.24it/s]2018-04-26 14-51-49 talking-data >>> read in chunk ./input/talking_data/train_day7_hour23.csv of shape (2333773, 7)\n 50%|█████████████████████▌ | 24/48 [00:44<00:25, 1.05s/it]2018-04-26 14-51-52 talking-data >>> read in chunk ./input/talking_data/train_day8_hour0.csv of shape (3493769, 7)\n 52%|██████████████████████▍ | 25/48 [00:46<00:33, 1.47s/it]2018-04-26 14-51-54 talking-data >>> read in chunk ./input/talking_data/train_day8_hour1.csv of shape (3065649, 7)\n 54%|███████████████████████▎ | 26/48 [00:48<00:36, 1.66s/it]2018-04-26 14-51-56 talking-data >>> read in chunk ./input/talking_data/train_day8_hour2.csv of shape (3585843, 7)\n 56%|████████████████████████▏ | 27/48 [00:51<00:39, 1.90s/it]2018-04-26 14-51-59 talking-data >>> read in chunk ./input/talking_data/train_day8_hour3.csv of shape (3172056, 7)\n 58%|█████████████████████████ | 28/48 [00:53<00:41, 2.06s/it]2018-04-26 14-52-01 talking-data >>> read in chunk ./input/talking_data/train_day8_hour4.csv of shape (3545132, 7)\n 60%|█████████████████████████▉ | 29/48 [00:56<00:42, 2.22s/it]2018-04-26 14-52-03 talking-data >>> read in chunk ./input/talking_data/train_day8_hour5.csv of shape (3160269, 7)\n 62%|██████████████████████████▉ | 30/48 [00:58<00:40, 2.23s/it]2018-04-26 14-52-06 talking-data >>> read in chunk ./input/talking_data/train_day8_hour6.csv of shape (2983655, 7)\n 65%|███████████████████████████▊ | 31/48 [01:00<00:37, 2.20s/it]2018-04-26 14-52-08 talking-data >>> read in chunk ./input/talking_data/train_day8_hour7.csv of shape (3155262, 7)\n 67%|████████████████████████████▋ | 32/48 [01:02<00:35, 2.21s/it]2018-04-26 14-52-10 talking-data >>> read in chunk ./input/talking_data/train_day8_hour8.csv of shape (2976057, 7)\n 69%|█████████████████████████████▌ | 33/48 [01:04<00:32, 2.17s/it]2018-04-26 14-52-12 talking-data >>> read in chunk ./input/talking_data/train_day8_hour9.csv of shape (3068314, 7)\n 71%|██████████████████████████████▍ | 34/48 [01:07<00:30, 2.17s/it]2018-04-26 14-52-14 talking-data >>> read in chunk ./input/talking_data/train_day8_hour10.csv of shape (3377086, 7)\n 73%|███████████████████████████████▎ | 35/48 [01:09<00:29, 2.24s/it]2018-04-26 14-52-17 talking-data >>> read in chunk ./input/talking_data/train_day8_hour11.csv of shape (3430977, 7)\n 75%|████████████████████████████████▎ | 36/48 [01:12<00:27, 2.33s/it]2018-04-26 14-52-19 talking-data >>> read in chunk ./input/talking_data/train_day8_hour12.csv of shape (3485357, 7)\n 77%|█████████████████████████████████▏ | 37/48 [01:14<00:25, 2.36s/it]2018-04-26 14-52-22 talking-data >>> read in chunk ./input/talking_data/train_day8_hour13.csv of shape (3616634, 7)\n 79%|██████████████████████████████████ | 38/48 [01:17<00:24, 2.42s/it]2018-04-26 14-52-25 talking-data >>> read in chunk ./input/talking_data/train_day8_hour14.csv of shape (3676695, 7)\n 81%|██████████████████████████████████▉ | 39/48 [01:19<00:22, 2.55s/it]2018-04-26 14-52-28 talking-data >>> read in chunk ./input/talking_data/train_day8_hour15.csv of shape (3336168, 7)\n 83%|███████████████████████████████████▊ | 40/48 [01:23<00:22, 2.77s/it]2018-04-26 14-52-32 talking-data >>> read in chunk ./input/talking_data/train_day8_hour16.csv of shape (2455567, 7)\n 85%|████████████████████████████████████▋ | 41/48 [01:27<00:21, 3.11s/it]2018-04-26 14-52-34 talking-data >>> read in chunk ./input/talking_data/train_day8_hour17.csv of shape (1387383, 7)\n 88%|█████████████████████████████████████▋ | 42/48 [01:28<00:16, 2.68s/it]2018-04-26 14-52-35 talking-data >>> read in chunk ./input/talking_data/train_day8_hour18.csv of shape (794088, 7)\n 90%|██████████████████████████████████████▌ | 43/48 [01:29<00:10, 2.13s/it]2018-04-26 14-52-35 talking-data >>> read in chunk ./input/talking_data/train_day8_hour19.csv of shape (554053, 7)\n 92%|███████████████████████████████████████▍ | 44/48 [01:30<00:06, 1.69s/it]2018-04-26 14-52-36 talking-data >>> read in chunk ./input/talking_data/train_day8_hour20.csv of shape (447324, 7)\n 94%|████████████████████████████████████████▎ | 45/48 [01:30<00:03, 1.29s/it]2018-04-26 14-52-36 talking-data >>> read in chunk ./input/talking_data/train_day8_hour21.csv of shape (550518, 7)\n 96%|█████████████████████████████████████████▏ | 46/48 [01:31<00:02, 1.04s/it]2018-04-26 14-52-37 talking-data >>> read in chunk ./input/talking_data/train_day8_hour22.csv of shape (1260525, 7)\n 98%|██████████████████████████████████████████ | 47/48 [01:32<00:01, 1.03s/it]2018-04-26 14-52-39 talking-data >>> read in chunk ./input/talking_data/train_day8_hour23.csv of shape (2366694, 7)\n 100%|███████████████████████████████████████████| 48/48 [01:33<00:00, 1.26s/it]\n 2018-04-26 14-53-12 talking-data >>> combined dataset shape: (122578385, 7)\n 0%| | 0/1 [00:00>> read in chunk ./input/talking_data/train_day9_hour4.csv of shape (4032691, 7)\n 100%|█████████████████████████████████████████████| 1/1 [00:03<00:00, 3.22s/it]\n 2018-04-26 14-53-16 talking-data >>> combined dataset shape: (4032691, 7)\n 2018-04-26 14-57-45 talking-data >>> Target distribution in train: 0.002492160424531617\n 2018-04-26 14-57-45 talking-data >>> Target distribution in valid: 0.002396910648497492\n 2018-04-26 14-57-45 talking-data >>> shuffling data\n 2018-04-26 15-00-12 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 15-00-12 steps >>> step feature_by_type_split fitting and transforming...\n 2018-04-26 15-00-15 steps >>> step feature_by_type_split saving transformer...\n 2018-04-26 15-00-15 steps >>> step time_delta adapting inputs\n 2018-04-26 15-00-15 steps >>> step time_delta fitting and transforming...\n 2018-04-26 15-58-41 steps >>> step time_delta saving transformer...\n 2018-04-26 15-58-41 steps >>> step time_delta caching outputs...\n 2018-04-26 16-00-38 steps >>> step time_delta saving outputs...\n 2018-04-26 16-02-13 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 16-02-13 steps >>> step feature_by_type_split loading transformer...\n 2018-04-26 16-02-13 steps >>> step feature_by_type_split transforming...\n 2018-04-26 16-02-18 steps >>> step confidence_rates adapting inputs\n 2018-04-26 16-02-18 steps >>> step confidence_rates fitting and transforming... 2018-04-26 16-13-04 steps >>> step confidence_rates saving transformer...\n 2018-04-26 16-13-04 steps >>> step confidence_rates caching outputs...\n 2018-04-26 16-14-05 steps >>> step confidence_rates saving outputs...\n 2018-04-26 16-16-01 steps >>> step time_delta loading output...\n 2018-04-26 16-17-19 steps >>> step feature_joiner adapting inputs\n 2018-04-26 16-17-19 steps >>> step feature_joiner fitting and transforming...\n 2018-04-26 16-17-29 steps >>> step feature_joiner saving transformer...\n 2018-04-26 16-17-29 steps >>> step feature_joiner caching outputs...\n 2018-04-26 16-24-29 steps >>> step feature_joiner saving outputs...\n 2018-04-26 16-37-06 steps >>> step feature_by_type_split_valid adapting inputs\n 2018-04-26 16-37-07 steps >>> copying transformer from ./output/baseline/transformers/feature_by_type_split to ./output/baseline/transformers/feature_by_type_split_valid\n 2018-04-26 16-37-07 steps >>> step feature_by_type_split_valid loading transformer...\n 2018-04-26 16-37-07 steps >>> step feature_by_type_split_valid transforming...\n 2018-04-26 16-37-10 steps >>> step time_delta_valid adapting inputs\n 2018-04-26 16-37-10 steps >>> copying transformer from ./output/baseline/transformers/time_delta to ./output/baseline/transformers/time_delta_valid\n 2018-04-26 16-37-10 steps >>> step time_delta_valid loading transformer...\n 2018-04-26 16-37-10 steps >>> step time_delta_valid transforming...\n 2018-04-26 16-41-15 steps >>> step time_delta_valid caching outputs...\n 2018-04-26 16-41-19 steps >>> step time_delta_valid saving outputs...\n 2018-04-26 16-41-23 steps >>> step feature_by_type_split_valid adapting inputs\n 2018-04-26 16-41-23 steps >>> step feature_by_type_split_valid loading transformer...\n 2018-04-26 16-41-23 steps >>> step feature_by_type_split_valid transforming...\n 2018-04-26 16-41-23 steps >>> step confidence_rates_valid adapting inputs\n 2018-04-26 16-41-23 steps >>> copying transformer from ./output/baseline/transformers/confidence_rates to ./output/baseline/transformers/confidence_rates_valid\n 2018-04-26 16-41-23 steps >>> step confidence_rates_valid loading transformer...\n 2018-04-26 16-41-23 steps >>> step confidence_rates_valid transforming...\n 2018-04-26 16-42-41 steps >>> step confidence_rates_valid caching outputs...\n 2018-04-26 16-42-43 steps >>> step confidence_rates_valid saving outputs...\n 2018-04-26 16-42-45 steps >>> step time_delta_valid loading output...\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid adapting inputs\n 2018-04-26 16-42-45 steps >>> copying transformer from ./output/baseline/transformers/feature_joiner to ./output/baseline/transformers/feature_joiner_valid\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid loading transformer...\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid transforming...\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid caching outputs...\n 2018-04-26 16-42-50 steps >>> step feature_joiner_valid saving outputs...\n 2018-04-26 16-42-59 steps >>> step light_gbm adapting inputs\n 2018-04-26 16-42-59 steps >>> step light_gbm fitting and transforming...\n //anaconda/envs/neptune/lib/python3.6/site-packages/lightgbm/basic.py:1036: UserWarning: Using categorical_feature in Dataset.\n warnings.warn('Using categorical_feature in Dataset.')\n [LightGBM] [Info] Number of positive: 305485, number of negative: 122272900\n [LightGBM] [Info] Total Bins 804\n [LightGBM] [Info] Number of data: 122578385, number of used features: 6\n //anaconda/envs/neptune/lib/python3.6/site-packages/lightgbm/basic.py:681: UserWarning: categorical_feature in param dict is overrided.\n warnings.warn('categorical_feature in param dict is overrided.')\n [1] train's auc: 0.942571 valid's auc: 0.92844\n Training until validation scores don't improve for 10 rounds.\n [2] train's auc: 0.960287 valid's auc: 0.977513\n [3] train's auc: 0.955238 valid's auc: 0.974518\n [4] train's auc: 0.967606 valid's auc: 0.984576\n [5] train's auc: 0.966103 valid's auc: 0.983874\n [6] train's auc: 0.96741 valid's auc: 0.984465\n [7] train's auc: 0.968365 valid's auc: 0.988336\n [8] train's auc: 0.968876 valid's auc: 0.988653\n [9] train's auc: 0.968986 valid's auc: 0.988622\n [10] train's auc: 0.968818 valid's auc: 0.988786\n [11] train's auc: 0.969468 valid's auc: 0.990106\n [12] train's auc: 0.97054 valid's auc: 0.990744\n [13] train's auc: 0.971427 valid's auc: 0.990874\n [LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n [14] train's auc: 0.971635 valid's auc: 0.990463\n [15] train's auc: 0.971087 valid's auc: 0.990451\n [16] train's auc: 0.968315 valid's auc: 0.990033\n [17] train's auc: 0.968475 valid's auc: 0.990112\n [LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n [18] train's auc: 0.968403 valid's auc: 0.987794\n [19] train's auc: 0.966061 valid's auc: 0.98744\n [20] train's auc: 0.966158 valid's auc: 0.9874\n [21] train's auc: 0.966187 valid's auc: 0.987395\n [22] train's auc: 0.963534 valid's auc: 0.986903\n [23] train's auc: 0.963573 valid's auc: 0.98683\n Early stopping, best iteration is:\n [13] train's auc: 0.971427 valid's auc: 0.990874\n 2018-04-26 16-49-00 steps >>> step light_gbm saving transformer...\n 2018-04-26 16-49-00 steps >>> step output adapting inputs\n 2018-04-26 16-49-00 steps >>> step output fitting and transforming...\n 2018-04-26 16-49-00 steps >>> step output saving transformer...

\n
", "rawMarkdown": "Hi kamil:\n\ngot KeyError: 'Column not found: is_attributed' as shown below with command line : python main.py -- evaluate --pipeline_name solution_1, any lead to solve this? command line: python main.py -- train --pipeline_name solution_1 sh appeared without error.\n\n> (neptune) Haos-iMac:neptune-talking-data-solution1 haotienlee$ python main.py -- evaluate --pipeline_name solution_1\nneptune: Executing in Offline Mode.\nneptune: Executing in Offline Mode.\n2018-04-26 17-24-59 talking-data >>> reading data in\n 0%| | 0/1 [00:00>> read in chunk ./input/talking_data/train_day9_hour4.csv of shape (4032691, 7)\n100%|█████████████████████████████████████████████| 1/1 [00:02<00:00, 2.66s/it]\n2018-04-26 17-25-03 talking-data >>> combined dataset shape: (4032691, 7)\n2018-04-26 17-25-06 talking-data >>> Target distribution in valid: 0.002396910648497492\n2018-04-26 17-25-06 steps >>> step feature_by_type_split adapting inputs\n2018-04-26 17-25-06 steps >>> step feature_by_type_split loading transformer...\n2018-04-26 17-25-06 steps >>> step feature_by_type_split transforming...\n2018-04-26 17-25-06 steps >>> step time_delta adapting inputs\n2018-04-26 17-25-06 steps >>> step time_delta loading transformer...\n2018-04-26 17-25-06 steps >>> step time_delta transforming...\n2018-04-26 17-29-02 steps >>> step time_delta caching outputs...\n2018-04-26 17-29-03 steps >>> step feature_by_type_split adapting inputs\n2018-04-26 17-29-03 steps >>> step feature_by_type_split loading transformer...\n2018-04-26 17-29-03 steps >>> step feature_by_type_split transforming...\n2018-04-26 17-29-04 steps >>> step confidence_rates adapting inputs\n2018-04-26 17-29-04 steps >>> step confidence_rates loading transformer...\n2018-04-26 17-29-04 steps >>> step confidence_rates transforming...\nTraceback (most recent call last):\n File \"main.py\", line 258, in", "votes": null }, { "id": "319600", "postDate": "04/26/2018 12:18:43", "content": "

Is there any GPU or memory requirements for my machine ?

", "rawMarkdown": "Is there any GPU or memory requirements for my machine ?", "votes": null }, { "id": "319618", "postDate": "04/26/2018 13:01:35", "content": "

for offline mode using OSX CPU 3.4GHz 4 cores, 28GB RAM would be ok, but Ubuntu with 16GB RAM would suffer memory error issue.

", "rawMarkdown": "for offline mode using OSX CPU 3.4GHz 4 cores, 28GB RAM would be ok, but Ubuntu with 16GB RAM would suffer memory error issue.", "votes": null }, { "id": "319973", "postDate": "04/27/2018 07:56:27", "content": "

Check the newest master. There was a bug but it's fixed now. Also notice that solution-1 branch was created. Current (simple) solution with confidence_rates and time_deltas will be freezed on that branch.

", "rawMarkdown": "Check the newest master. There was a bug but it's fixed now. Also notice that solution-1 branch was created. Current (simple) solution with confidence_rates and time_deltas will be freezed on that branch.", "votes": null }, { "id": "320801", "postDate": "04/29/2018 22:29:04", "content": "

Hi yananchen,

\n\n

At this point we do not have hardware requirements, however, it would be good to have more than 16GB memory (thanks @danieleewww) and as many cores as possible. You can always try to play with parameters to make your experiments more memory or CPU efficient.

\n\n

Best,

\n\n

Kamil

", "rawMarkdown": "Hi yananchen,\n\nAt this point we do not have hardware requirements, however, it would be good to have more than 16GB memory (thanks @danieleewww) and as many cores as possible. You can always try to play with [parameters](https://github.com/minerva-ml/open-solution-talking-data/blob/master/neptune.yaml) to make your experiments more memory or CPU efficient.\n\nBest,\n\nKamil", "votes": null }, { "id": "320802", "postDate": "04/29/2018 22:30:39", "content": "

Hi @Bryan Arnold,

\n\n

Thanks! Let us know if you found our solution useful :)

", "rawMarkdown": "Hi @Bryan Arnold,\n\nThanks! Let us know if you found our solution useful :)", "votes": null } ], "comments": [ { "id": 319006, "author_name": "puremath86", "author_url": "", "post_date": "04/25/2018 02:53:49", "content": "

Hey, guys! Welcome to the game.

\n\n

Thanks for sharing your results. And thanks for sharing information on the Neptune platform. I'll have to check it out.

", "votes": null, "replies": [ { "id": 320802, "author_name": "kkaczmarek", "author_url": "", "post_date": "04/29/2018 22:30:39", "content": "

Hi @Bryan Arnold,

\n\n

Thanks! Let us know if you found our solution useful :)

", "votes": null, "replies": [] } ] }, { "id": 319556, "author_name": "danieleewww", "author_url": "", "post_date": "04/26/2018 09:42:20", "content": "

Hi kamil:

\n\n

got KeyError: 'Column not found: is_attributed' as shown below with command line : python main.py -- evaluate --pipeline_name solution_1, any lead to solve this? command line: python main.py -- train --pipeline_name solution_1 sh appeared without error.

\n\n
\n

(neptune) Haos-iMac:neptune-talking-data-solution1 haotienlee$ python main.py -- evaluate --pipeline_name solution_1\n neptune: Executing in Offline Mode.\n neptune: Executing in Offline Mode.\n 2018-04-26 17-24-59 talking-data >>> reading data in\n 0%| | 0/1 [00:00>> read in chunk ./input/talking_data/train_day9_hour4.csv of shape (4032691, 7)\n 100%|█████████████████████████████████████████████| 1/1 [00:02<00:00, 2.66s/it]\n 2018-04-26 17-25-03 talking-data >>> combined dataset shape: (4032691, 7)\n 2018-04-26 17-25-06 talking-data >>> Target distribution in valid: 0.002396910648497492\n 2018-04-26 17-25-06 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 17-25-06 steps >>> step feature_by_type_split loading transformer...\n 2018-04-26 17-25-06 steps >>> step feature_by_type_split transforming...\n 2018-04-26 17-25-06 steps >>> step time_delta adapting inputs\n 2018-04-26 17-25-06 steps >>> step time_delta loading transformer...\n 2018-04-26 17-25-06 steps >>> step time_delta transforming...\n 2018-04-26 17-29-02 steps >>> step time_delta caching outputs...\n 2018-04-26 17-29-03 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 17-29-03 steps >>> step feature_by_type_split loading transformer...\n 2018-04-26 17-29-03 steps >>> step feature_by_type_split transforming...\n 2018-04-26 17-29-04 steps >>> step confidence_rates adapting inputs\n 2018-04-26 17-29-04 steps >>> step confidence_rates loading transformer...\n 2018-04-26 17-29-04 steps >>> step confidence_rates transforming...\n Traceback (most recent call last):\n File \"main.py\", line 258, in \n action()\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 722, in call\n return self.main(*args, **kwargs)\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 697, in main\n rv = self.invoke(ctx)\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 1066, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 895, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File \"/Users/haotienlee/.local/lib/python3.6/site-packages/click/core.py\", line 535, in invoke\n return callback(*args, **kwargs)\n File \"main.py\", line 97, in evaluate\n _evaluate(pipeline_name, dev_mode)\n File \"main.py\", line 129, in _evaluate\n output = pipeline.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 151, in transform\n step_inputs[input_step.name] = input_step.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 151, in transform\n step_inputs[input_step.name] = input_step.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 151, in transform\n step_inputs[input_step.name] = input_step.transform(data)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 157, in transform\n step_output_data = self._cached_transform(step_inputs)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/steps/base.py\", line 165, in _cached_transform\n step_output_data = self.transformer.transform(**step_inputs)\n File \"/Users/haotienlee/Desktop/MLDL_Project/open-solution-talking-data/neptune-talking-data-solution1/feature_extraction.py\", line 189, in transform\n group_object['is_attributed'].apply(self._rate_calculation).reset_index().rename(\n File \"//anaconda/envs/neptune/lib/python3.6/site-packages/pandas/core/base.py\", line 259, in getitem\n raise KeyError(\"Column not found: {key}\".format(key=key))\n KeyError: 'Column not found: is_attributed'

\n \n

(neptune) Haos-iMac:neptune-talking-data-solution1 haotienlee$ python main.py -- train --pipeline_name solution_1\n neptune: Executing in Offline Mode.\n neptune: Executing in Offline Mode.\n 2018-04-26 14-51-05 talking-data >>> reading data in\n 0%| | 0/48 [00:00>> read in chunk ./input/talking_data/train_day7_hour0.csv of shape (3604365, 7)\n 2%|▉ | 1/48 [00:02<02:19, 2.97s/it]2018-04-26 14-51-10 talking-data >>> read in chunk ./input/talking_data/train_day7_hour1.csv of shape (3308150, 7)\n 4%|█▊ | 2/48 [00:05<02:07, 2.78s/it]2018-04-26 14-51-12 talking-data >>> read in chunk ./input/talking_data/train_day7_hour2.csv of shape (3095633, 7)\n 6%|██▊ | 3/48 [00:07<01:56, 2.59s/it]2018-04-26 14-51-15 talking-data >>> read in chunk ./input/talking_data/train_day7_hour3.csv of shape (3220271, 7)\n 8%|███▋ | 4/48 [00:09<01:50, 2.50s/it]2018-04-26 14-51-18 talking-data >>> read in chunk ./input/talking_data/train_day7_hour4.csv of shape (3645493, 7)\n 10%|████▌ | 5/48 [00:13<02:02, 2.85s/it]2018-04-26 14-51-21 talking-data >>> read in chunk ./input/talking_data/train_day7_hour5.csv of shape (3227349, 7)\n 12%|█████▌ | 6/48 [00:15<01:51, 2.66s/it]2018-04-26 14-51-23 talking-data >>> read in chunk ./input/talking_data/train_day7_hour6.csv of shape (2917284, 7)\n 15%|██████▍ | 7/48 [00:17<01:40, 2.46s/it]2018-04-26 14-51-25 talking-data >>> read in chunk ./input/talking_data/train_day7_hour7.csv of shape (2924033, 7)\n 17%|███████▎ | 8/48 [00:19<01:33, 2.35s/it]2018-04-26 14-51-27 talking-data >>> read in chunk ./input/talking_data/train_day7_hour8.csv of shape (2801293, 7)\n 19%|████████▎ | 9/48 [00:21<01:27, 2.24s/it]2018-04-26 14-51-29 talking-data >>> read in chunk ./input/talking_data/train_day7_hour9.csv of shape (2961319, 7)\n 21%|████████▉ | 10/48 [00:23<01:22, 2.17s/it]2018-04-26 14-51-31 talking-data >>> read in chunk ./input/talking_data/train_day7_hour10.csv of shape (3300746, 7)\n 23%|█████████▊ | 11/48 [00:25<01:21, 2.20s/it]2018-04-26 14-51-33 talking-data >>> read in chunk ./input/talking_data/train_day7_hour11.csv of shape (3134200, 7)\n 25%|██████████▊ | 12/48 [00:28<01:18, 2.19s/it]2018-04-26 14-51-35 talking-data >>> read in chunk ./input/talking_data/train_day7_hour12.csv of shape (3000091, 7)\n 27%|███████████▋ | 13/48 [00:30<01:15, 2.17s/it]2018-04-26 14-51-38 talking-data >>> read in chunk ./input/talking_data/train_day7_hour13.csv of shape (3235239, 7)\n 29%|████████████▌ | 14/48 [00:33<01:19, 2.35s/it]2018-04-26 14-51-40 talking-data >>> read in chunk ./input/talking_data/train_day7_hour14.csv of shape (3161797, 7)\n 31%|█████████████▍ | 15/48 [00:35<01:17, 2.35s/it]2018-04-26 14-51-42 talking-data >>> read in chunk ./input/talking_data/train_day7_hour15.csv of shape (2864021, 7)\n 33%|██████████████▎ | 16/48 [00:37<01:12, 2.26s/it]2018-04-26 14-51-44 talking-data >>> read in chunk ./input/talking_data/train_day7_hour16.csv of shape (2314780, 7)\n 35%|███████████████▏ | 17/48 [00:39<01:03, 2.06s/it]2018-04-26 14-51-45 talking-data >>> read in chunk ./input/talking_data/train_day7_hour17.csv of shape (1263077, 7)\n 38%|████████████████▏ | 18/48 [00:39<00:51, 1.71s/it]2018-04-26 14-51-45 talking-data >>> read in chunk ./input/talking_data/train_day7_hour18.csv of shape (726684, 7)\n 40%|█████████████████ | 19/48 [00:40<00:39, 1.35s/it]2018-04-26 14-51-46 talking-data >>> read in chunk ./input/talking_data/train_day7_hour19.csv of shape (493056, 7)\n 42%|█████████████████▉ | 20/48 [00:40<00:29, 1.06s/it]2018-04-26 14-51-46 talking-data >>> read in chunk ./input/talking_data/train_day7_hour20.csv of shape (409255, 7)\n 44%|██████████████████▊ | 21/48 [00:41<00:22, 1.20it/s]2018-04-26 14-51-46 talking-data >>> read in chunk ./input/talking_data/train_day7_hour21.csv of shape (518866, 7)\n 46%|███████████████████▋ | 22/48 [00:41<00:18, 1.43it/s]2018-04-26 14-51-48 talking-data >>> read in chunk ./input/talking_data/train_day7_hour22.csv of shape (1172535, 7)\n 48%|████████████████████▌ | 23/48 [00:42<00:20, 1.24it/s]2018-04-26 14-51-49 talking-data >>> read in chunk ./input/talking_data/train_day7_hour23.csv of shape (2333773, 7)\n 50%|█████████████████████▌ | 24/48 [00:44<00:25, 1.05s/it]2018-04-26 14-51-52 talking-data >>> read in chunk ./input/talking_data/train_day8_hour0.csv of shape (3493769, 7)\n 52%|██████████████████████▍ | 25/48 [00:46<00:33, 1.47s/it]2018-04-26 14-51-54 talking-data >>> read in chunk ./input/talking_data/train_day8_hour1.csv of shape (3065649, 7)\n 54%|███████████████████████▎ | 26/48 [00:48<00:36, 1.66s/it]2018-04-26 14-51-56 talking-data >>> read in chunk ./input/talking_data/train_day8_hour2.csv of shape (3585843, 7)\n 56%|████████████████████████▏ | 27/48 [00:51<00:39, 1.90s/it]2018-04-26 14-51-59 talking-data >>> read in chunk ./input/talking_data/train_day8_hour3.csv of shape (3172056, 7)\n 58%|█████████████████████████ | 28/48 [00:53<00:41, 2.06s/it]2018-04-26 14-52-01 talking-data >>> read in chunk ./input/talking_data/train_day8_hour4.csv of shape (3545132, 7)\n 60%|█████████████████████████▉ | 29/48 [00:56<00:42, 2.22s/it]2018-04-26 14-52-03 talking-data >>> read in chunk ./input/talking_data/train_day8_hour5.csv of shape (3160269, 7)\n 62%|██████████████████████████▉ | 30/48 [00:58<00:40, 2.23s/it]2018-04-26 14-52-06 talking-data >>> read in chunk ./input/talking_data/train_day8_hour6.csv of shape (2983655, 7)\n 65%|███████████████████████████▊ | 31/48 [01:00<00:37, 2.20s/it]2018-04-26 14-52-08 talking-data >>> read in chunk ./input/talking_data/train_day8_hour7.csv of shape (3155262, 7)\n 67%|████████████████████████████▋ | 32/48 [01:02<00:35, 2.21s/it]2018-04-26 14-52-10 talking-data >>> read in chunk ./input/talking_data/train_day8_hour8.csv of shape (2976057, 7)\n 69%|█████████████████████████████▌ | 33/48 [01:04<00:32, 2.17s/it]2018-04-26 14-52-12 talking-data >>> read in chunk ./input/talking_data/train_day8_hour9.csv of shape (3068314, 7)\n 71%|██████████████████████████████▍ | 34/48 [01:07<00:30, 2.17s/it]2018-04-26 14-52-14 talking-data >>> read in chunk ./input/talking_data/train_day8_hour10.csv of shape (3377086, 7)\n 73%|███████████████████████████████▎ | 35/48 [01:09<00:29, 2.24s/it]2018-04-26 14-52-17 talking-data >>> read in chunk ./input/talking_data/train_day8_hour11.csv of shape (3430977, 7)\n 75%|████████████████████████████████▎ | 36/48 [01:12<00:27, 2.33s/it]2018-04-26 14-52-19 talking-data >>> read in chunk ./input/talking_data/train_day8_hour12.csv of shape (3485357, 7)\n 77%|█████████████████████████████████▏ | 37/48 [01:14<00:25, 2.36s/it]2018-04-26 14-52-22 talking-data >>> read in chunk ./input/talking_data/train_day8_hour13.csv of shape (3616634, 7)\n 79%|██████████████████████████████████ | 38/48 [01:17<00:24, 2.42s/it]2018-04-26 14-52-25 talking-data >>> read in chunk ./input/talking_data/train_day8_hour14.csv of shape (3676695, 7)\n 81%|██████████████████████████████████▉ | 39/48 [01:19<00:22, 2.55s/it]2018-04-26 14-52-28 talking-data >>> read in chunk ./input/talking_data/train_day8_hour15.csv of shape (3336168, 7)\n 83%|███████████████████████████████████▊ | 40/48 [01:23<00:22, 2.77s/it]2018-04-26 14-52-32 talking-data >>> read in chunk ./input/talking_data/train_day8_hour16.csv of shape (2455567, 7)\n 85%|████████████████████████████████████▋ | 41/48 [01:27<00:21, 3.11s/it]2018-04-26 14-52-34 talking-data >>> read in chunk ./input/talking_data/train_day8_hour17.csv of shape (1387383, 7)\n 88%|█████████████████████████████████████▋ | 42/48 [01:28<00:16, 2.68s/it]2018-04-26 14-52-35 talking-data >>> read in chunk ./input/talking_data/train_day8_hour18.csv of shape (794088, 7)\n 90%|██████████████████████████████████████▌ | 43/48 [01:29<00:10, 2.13s/it]2018-04-26 14-52-35 talking-data >>> read in chunk ./input/talking_data/train_day8_hour19.csv of shape (554053, 7)\n 92%|███████████████████████████████████████▍ | 44/48 [01:30<00:06, 1.69s/it]2018-04-26 14-52-36 talking-data >>> read in chunk ./input/talking_data/train_day8_hour20.csv of shape (447324, 7)\n 94%|████████████████████████████████████████▎ | 45/48 [01:30<00:03, 1.29s/it]2018-04-26 14-52-36 talking-data >>> read in chunk ./input/talking_data/train_day8_hour21.csv of shape (550518, 7)\n 96%|█████████████████████████████████████████▏ | 46/48 [01:31<00:02, 1.04s/it]2018-04-26 14-52-37 talking-data >>> read in chunk ./input/talking_data/train_day8_hour22.csv of shape (1260525, 7)\n 98%|██████████████████████████████████████████ | 47/48 [01:32<00:01, 1.03s/it]2018-04-26 14-52-39 talking-data >>> read in chunk ./input/talking_data/train_day8_hour23.csv of shape (2366694, 7)\n 100%|███████████████████████████████████████████| 48/48 [01:33<00:00, 1.26s/it]\n 2018-04-26 14-53-12 talking-data >>> combined dataset shape: (122578385, 7)\n 0%| | 0/1 [00:00>> read in chunk ./input/talking_data/train_day9_hour4.csv of shape (4032691, 7)\n 100%|█████████████████████████████████████████████| 1/1 [00:03<00:00, 3.22s/it]\n 2018-04-26 14-53-16 talking-data >>> combined dataset shape: (4032691, 7)\n 2018-04-26 14-57-45 talking-data >>> Target distribution in train: 0.002492160424531617\n 2018-04-26 14-57-45 talking-data >>> Target distribution in valid: 0.002396910648497492\n 2018-04-26 14-57-45 talking-data >>> shuffling data\n 2018-04-26 15-00-12 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 15-00-12 steps >>> step feature_by_type_split fitting and transforming...\n 2018-04-26 15-00-15 steps >>> step feature_by_type_split saving transformer...\n 2018-04-26 15-00-15 steps >>> step time_delta adapting inputs\n 2018-04-26 15-00-15 steps >>> step time_delta fitting and transforming...\n 2018-04-26 15-58-41 steps >>> step time_delta saving transformer...\n 2018-04-26 15-58-41 steps >>> step time_delta caching outputs...\n 2018-04-26 16-00-38 steps >>> step time_delta saving outputs...\n 2018-04-26 16-02-13 steps >>> step feature_by_type_split adapting inputs\n 2018-04-26 16-02-13 steps >>> step feature_by_type_split loading transformer...\n 2018-04-26 16-02-13 steps >>> step feature_by_type_split transforming...\n 2018-04-26 16-02-18 steps >>> step confidence_rates adapting inputs\n 2018-04-26 16-02-18 steps >>> step confidence_rates fitting and transforming... 2018-04-26 16-13-04 steps >>> step confidence_rates saving transformer...\n 2018-04-26 16-13-04 steps >>> step confidence_rates caching outputs...\n 2018-04-26 16-14-05 steps >>> step confidence_rates saving outputs...\n 2018-04-26 16-16-01 steps >>> step time_delta loading output...\n 2018-04-26 16-17-19 steps >>> step feature_joiner adapting inputs\n 2018-04-26 16-17-19 steps >>> step feature_joiner fitting and transforming...\n 2018-04-26 16-17-29 steps >>> step feature_joiner saving transformer...\n 2018-04-26 16-17-29 steps >>> step feature_joiner caching outputs...\n 2018-04-26 16-24-29 steps >>> step feature_joiner saving outputs...\n 2018-04-26 16-37-06 steps >>> step feature_by_type_split_valid adapting inputs\n 2018-04-26 16-37-07 steps >>> copying transformer from ./output/baseline/transformers/feature_by_type_split to ./output/baseline/transformers/feature_by_type_split_valid\n 2018-04-26 16-37-07 steps >>> step feature_by_type_split_valid loading transformer...\n 2018-04-26 16-37-07 steps >>> step feature_by_type_split_valid transforming...\n 2018-04-26 16-37-10 steps >>> step time_delta_valid adapting inputs\n 2018-04-26 16-37-10 steps >>> copying transformer from ./output/baseline/transformers/time_delta to ./output/baseline/transformers/time_delta_valid\n 2018-04-26 16-37-10 steps >>> step time_delta_valid loading transformer...\n 2018-04-26 16-37-10 steps >>> step time_delta_valid transforming...\n 2018-04-26 16-41-15 steps >>> step time_delta_valid caching outputs...\n 2018-04-26 16-41-19 steps >>> step time_delta_valid saving outputs...\n 2018-04-26 16-41-23 steps >>> step feature_by_type_split_valid adapting inputs\n 2018-04-26 16-41-23 steps >>> step feature_by_type_split_valid loading transformer...\n 2018-04-26 16-41-23 steps >>> step feature_by_type_split_valid transforming...\n 2018-04-26 16-41-23 steps >>> step confidence_rates_valid adapting inputs\n 2018-04-26 16-41-23 steps >>> copying transformer from ./output/baseline/transformers/confidence_rates to ./output/baseline/transformers/confidence_rates_valid\n 2018-04-26 16-41-23 steps >>> step confidence_rates_valid loading transformer...\n 2018-04-26 16-41-23 steps >>> step confidence_rates_valid transforming...\n 2018-04-26 16-42-41 steps >>> step confidence_rates_valid caching outputs...\n 2018-04-26 16-42-43 steps >>> step confidence_rates_valid saving outputs...\n 2018-04-26 16-42-45 steps >>> step time_delta_valid loading output...\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid adapting inputs\n 2018-04-26 16-42-45 steps >>> copying transformer from ./output/baseline/transformers/feature_joiner to ./output/baseline/transformers/feature_joiner_valid\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid loading transformer...\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid transforming...\n 2018-04-26 16-42-45 steps >>> step feature_joiner_valid caching outputs...\n 2018-04-26 16-42-50 steps >>> step feature_joiner_valid saving outputs...\n 2018-04-26 16-42-59 steps >>> step light_gbm adapting inputs\n 2018-04-26 16-42-59 steps >>> step light_gbm fitting and transforming...\n //anaconda/envs/neptune/lib/python3.6/site-packages/lightgbm/basic.py:1036: UserWarning: Using categorical_feature in Dataset.\n warnings.warn('Using categorical_feature in Dataset.')\n [LightGBM] [Info] Number of positive: 305485, number of negative: 122272900\n [LightGBM] [Info] Total Bins 804\n [LightGBM] [Info] Number of data: 122578385, number of used features: 6\n //anaconda/envs/neptune/lib/python3.6/site-packages/lightgbm/basic.py:681: UserWarning: categorical_feature in param dict is overrided.\n warnings.warn('categorical_feature in param dict is overrided.')\n [1] train's auc: 0.942571 valid's auc: 0.92844\n Training until validation scores don't improve for 10 rounds.\n [2] train's auc: 0.960287 valid's auc: 0.977513\n [3] train's auc: 0.955238 valid's auc: 0.974518\n [4] train's auc: 0.967606 valid's auc: 0.984576\n [5] train's auc: 0.966103 valid's auc: 0.983874\n [6] train's auc: 0.96741 valid's auc: 0.984465\n [7] train's auc: 0.968365 valid's auc: 0.988336\n [8] train's auc: 0.968876 valid's auc: 0.988653\n [9] train's auc: 0.968986 valid's auc: 0.988622\n [10] train's auc: 0.968818 valid's auc: 0.988786\n [11] train's auc: 0.969468 valid's auc: 0.990106\n [12] train's auc: 0.97054 valid's auc: 0.990744\n [13] train's auc: 0.971427 valid's auc: 0.990874\n [LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n [14] train's auc: 0.971635 valid's auc: 0.990463\n [15] train's auc: 0.971087 valid's auc: 0.990451\n [16] train's auc: 0.968315 valid's auc: 0.990033\n [17] train's auc: 0.968475 valid's auc: 0.990112\n [LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n [18] train's auc: 0.968403 valid's auc: 0.987794\n [19] train's auc: 0.966061 valid's auc: 0.98744\n [20] train's auc: 0.966158 valid's auc: 0.9874\n [21] train's auc: 0.966187 valid's auc: 0.987395\n [22] train's auc: 0.963534 valid's auc: 0.986903\n [23] train's auc: 0.963573 valid's auc: 0.98683\n Early stopping, best iteration is:\n [13] train's auc: 0.971427 valid's auc: 0.990874\n 2018-04-26 16-49-00 steps >>> step light_gbm saving transformer...\n 2018-04-26 16-49-00 steps >>> step output adapting inputs\n 2018-04-26 16-49-00 steps >>> step output fitting and transforming...\n 2018-04-26 16-49-00 steps >>> step output saving transformer...

\n
", "votes": null, "replies": [ { "id": 319973, "author_name": "jakubczakon", "author_url": "", "post_date": "04/27/2018 07:56:27", "content": "

Check the newest master. There was a bug but it's fixed now. Also notice that solution-1 branch was created. Current (simple) solution with confidence_rates and time_deltas will be freezed on that branch.

", "votes": null, "replies": [] } ] }, { "id": 319600, "author_name": "", "author_url": "", "post_date": "04/26/2018 12:18:43", "content": "

Is there any GPU or memory requirements for my machine ?

", "votes": null, "replies": [ { "id": 319618, "author_name": "danieleewww", "author_url": "", "post_date": "04/26/2018 13:01:35", "content": "

for offline mode using OSX CPU 3.4GHz 4 cores, 28GB RAM would be ok, but Ubuntu with 16GB RAM would suffer memory error issue.

", "votes": null, "replies": [] }, { "id": 320801, "author_name": "kkaczmarek", "author_url": "", "post_date": "04/29/2018 22:29:04", "content": "

Hi yananchen,

\n\n

At this point we do not have hardware requirements, however, it would be good to have more than 16GB memory (thanks @danieleewww) and as many cores as possible. You can always try to play with parameters to make your experiments more memory or CPU efficient.

\n\n

Best,

\n\n

Kamil

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "318985": "Hi,\n\nJakub Czakon and me would like to make some (quite) late start of the open solution to this competition. We are opening our code, available on this [repository](https://github.com/neptune-ml/open-solution-talking-data). If you would like to use it just go ahead and do it.\n\nWe are open for your questions and suggestions what we should develop next :)\n\nWe will be improving this solution for the next two weeks -> we are very curious how much work we will be able to contribute within this time period.\n\nHappy Training :)", "319006": "Hey, guys! Welcome to the game.\n\nThanks for sharing your results. And thanks for sharing information on the Neptune platform. I'll have to check it out.", "319556": "Hi kamil:\n\ngot KeyError: 'Column not found: is_attributed' as shown below with command line : python main.py -- evaluate --pipeline_name solution_1, any lead to solve this? command line: python main.py -- train --pipeline_name solution_1 sh appeared without error.\n\n> (neptune) Haos-iMac:neptune-talking-data-solution1 haotienlee$ python main.py -- evaluate --pipeline_name solution_1\nneptune: Executing in Offline Mode.\nneptune: Executing in Offline Mode.\n2018-04-26 17-24-59 talking-data >>> reading data in\n 0%| | 0/1 [00:00>> read in chunk ./input/talking_data/train_day9_hour4.csv of shape (4032691, 7)\n100%|█████████████████████████████████████████████| 1/1 [00:02<00:00, 2.66s/it]\n2018-04-26 17-25-03 talking-data >>> combined dataset shape: (4032691, 7)\n2018-04-26 17-25-06 talking-data >>> Target distribution in valid: 0.002396910648497492\n2018-04-26 17-25-06 steps >>> step feature_by_type_split adapting inputs\n2018-04-26 17-25-06 steps >>> step feature_by_type_split loading transformer...\n2018-04-26 17-25-06 steps >>> step feature_by_type_split transforming...\n2018-04-26 17-25-06 steps >>> step time_delta adapting inputs\n2018-04-26 17-25-06 steps >>> step time_delta loading transformer...\n2018-04-26 17-25-06 steps >>> step time_delta transforming...\n2018-04-26 17-29-02 steps >>> step time_delta caching outputs...\n2018-04-26 17-29-03 steps >>> step feature_by_type_split adapting inputs\n2018-04-26 17-29-03 steps >>> step feature_by_type_split loading transformer...\n2018-04-26 17-29-03 steps >>> step feature_by_type_split transforming...\n2018-04-26 17-29-04 steps >>> step confidence_rates adapting inputs\n2018-04-26 17-29-04 steps >>> step confidence_rates loading transformer...\n2018-04-26 17-29-04 steps >>> step confidence_rates transforming...\nTraceback (most recent call last):\n File \"main.py\", line 258, in", "319600": "Is there any GPU or memory requirements for my machine ?", "319618": "for offline mode using OSX CPU 3.4GHz 4 cores, 28GB RAM would be ok, but Ubuntu with 16GB RAM would suffer memory error issue.", "319973": "Check the newest master. There was a bug but it's fixed now. Also notice that solution-1 branch was created. Current (simple) solution with confidence_rates and time_deltas will be freezed on that branch.", "320801": "Hi yananchen,\n\nAt this point we do not have hardware requirements, however, it would be good to have more than 16GB memory (thanks @danieleewww) and as many cores as possible. You can always try to play with [parameters](https://github.com/minerva-ml/open-solution-talking-data/blob/master/neptune.yaml) to make your experiments more memory or CPU efficient.\n\nBest,\n\nKamil", "320802": "Hi @Bryan Arnold,\n\nThanks! Let us know if you found our solution useful :)" }, "source": "meta" }