From the advice given under this topic, our group selected AWS EC2 instance to have a trial play with the whole data. We choose r4.8xlarge windows instance which as 32 CPU and 244GB memory with $3.6/hour.\nWe extract hour and minute from click_time and remove click_time and attributed_time and we noticed from spyder IDE that the memory usage is around 26%(almost 64GB) when we run 5-fold cv using lightgbm. I think this would be a kind of benchmark of memory usage for people who wants to run algorithms on whole data set. Using whole data set may not be a good idea but if you want, then 32GB or even 64GB is not enough for some complicated algorithms.
", "messages": [ { "id": "293320", "postDate": "03/09/2018 16:24:13", "content": "From the advice given under this topic, our group selected AWS EC2 instance to have a trial play with the whole data. We choose r4.8xlarge windows instance which as 32 CPU and 244GB memory with $3.6/hour.\nWe extract hour and minute from click_time and remove click_time and attributed_time and we noticed from spyder IDE that the memory usage is around 26%(almost 64GB) when we run 5-fold cv using lightgbm. I think this would be a kind of benchmark of memory usage for people who wants to run algorithms on whole data set. Using whole data set may not be a good idea but if you want, then 32GB or even 64GB is not enough for some complicated algorithms.
", "rawMarkdown": "From the advice given under this topic, our group selected AWS EC2 instance to have a trial play with the whole data. We choose r4.8xlarge windows instance which as 32 CPU and 244GB memory with $3.6/hour.\nWe extract hour and minute from click_time and remove click_time and attributed_time and we noticed from spyder IDE that the memory usage is around 26%(almost 64GB) when we run 5-fold cv using lightgbm. I think this would be a kind of benchmark of memory usage for people who wants to run algorithms on whole data set. Using whole data set may not be a good idea but if you want, then 32GB or even 64GB is not enough for some complicated algorithms.", "votes": null }, { "id": "293527", "postDate": "03/10/2018 03:51:50", "content": "Hi, Is there any other option $3.6/hr is costly for me and i want to use whole dataset.
", "rawMarkdown": "Hi, Is there any other option $3.6/hr is costly for me and i want to use whole dataset.", "votes": null }, { "id": "293530", "postDate": "03/10/2018 04:08:47", "content": "I'm using a 16GB RAM Mac doing some feature engineering and training a lightgbm model using 50% of the trainset.
", "rawMarkdown": "I'm using a 16GB RAM Mac doing some feature engineering and training a lightgbm model using 50% of the trainset.", "votes": null }, { "id": "293533", "postDate": "03/10/2018 04:13:27", "content": "It means you are on top using lightgbm and only 50% trainset. AMAZING
Well, that's interesting and could I ask how is the memory usage when you run lightgbm? Did it use almost 100% of your RAM? Actually I forget the memory usage when I train lightgbm using whole dataset. But when I did 5 fold cv lightgbm on whole data set, the memory usage is around 64GB.
", "rawMarkdown": "Well, that's interesting and could I ask how is the memory usage when you run lightgbm? Did it use almost 100% of your RAM? Actually I forget the memory usage when I train lightgbm using whole dataset. But when I did 5 fold cv lightgbm on whole data set, the memory usage is around 64GB.", "votes": null }, { "id": "294046", "postDate": "03/11/2018 06:39:50", "content": "Amazing! However, for further FE or more complicated algorithms, 16GB RAM could be enough?
", "rawMarkdown": "Amazing! However, for further FE or more complicated algorithms, 16GB RAM could be enough?", "votes": null }, { "id": "294211", "postDate": "03/11/2018 15:17:09", "content": "You could try google cloud engine, they always have $300 coupons for starters.
", "rawMarkdown": "You could try google cloud engine, they always have $300 coupons for starters.", "votes": null }, { "id": "294297", "postDate": "03/11/2018 17:46:34", "content": "I trained using only 1 oof. Remember this is a time series challenge you can't do random kFold. I'm using all my RAM available and a huge SWAP file (most of my dataset goes to swap file). Since Mac SSDs are really fast I can train a model in about 1 hour
", "rawMarkdown": "I trained using only 1 oof. Remember this is a time series challenge you can't do random kFold. I'm using all my RAM available and a huge SWAP file (most of my dataset goes to swap file). Since Mac SSDs are really fast I can train a model in about 1 hour", "votes": null }, { "id": "296776", "postDate": "03/15/2018 18:40:49", "content": "using 50% of the trainset.\nMay I get a hint on how did you select that 50% of train set? Thanks
", "rawMarkdown": "using 50% of the trainset.\n\nMay I get a hint on how did you select that 50% of train set? Thanks", "votes": null }, { "id": "296972", "postDate": "03/16/2018 03:24:03", "content": "I think ec2 spot instance is a better choice, much cheaper ... and I built my environment and download data on an external EBS to keep it long time availabel.
", "rawMarkdown": "I think ec2 spot instance is a better choice, much cheaper ... and I built my environment and download data on an external EBS to keep it long time availabel.", "votes": null }, { "id": "298020", "postDate": "03/18/2018 14:54:26", "content": "Amazing!!
", "rawMarkdown": "Amazing!!", "votes": null }, { "id": "298287", "postDate": "03/19/2018 09:11:54", "content": "Are you using R or Python? And most important: who have any hint about a tutorial/guide to use SWAP file in R using windows 10? Very interesting discovery!
", "rawMarkdown": "Are you using R or Python? And most important: who have any hint about a tutorial/guide to use SWAP file in R using windows 10? Very interesting discovery!", "votes": null }, { "id": "298848", "postDate": "03/20/2018 04:44:59", "content": "I am using a 32GB RAM and 64G virtual memory from my SSD (WIN10, python). Training 50%-60% of the train dataset may take around 1 hour.
", "rawMarkdown": "I am using a 32GB RAM and 64G virtual memory from my SSD (WIN10, python). Training 50%-60% of the train dataset may take around 1 hour.", "votes": null }, { "id": "300415", "postDate": "03/21/2018 15:46:43", "content": "I'm thinking if we can change the data type from int64 to int32 (or int16), I mean except the time column. Then we could get a somehow smaller data set.
", "rawMarkdown": "I'm thinking if we can change the data type from int64 to int32 (or int16), I mean except the time column. Then we could get a somehow smaller data set.", "votes": null }, { "id": "300421", "postDate": "03/21/2018 15:51:09", "content": "actually you can drop the time column, only left day, hour in int
", "rawMarkdown": "actually you can drop the time column, only left day, hour in int", "votes": null }, { "id": "300432", "postDate": "03/21/2018 15:54:50", "content": "Good point, these are two days' data.
", "rawMarkdown": "Good point, these are two days' data.", "votes": null }, { "id": "300492", "postDate": "03/21/2018 16:31:10", "content": "Have you heard about this`?\n=)
", "rawMarkdown": "Have you heard about [this][1]`?\n=)\n\n\n [1]: https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d", "votes": null } ], "comments": [ { "id": 293527, "author_name": "blasteraj", "author_url": "", "post_date": "03/10/2018 03:51:50", "content": "Hi, Is there any other option $3.6/hr is costly for me and i want to use whole dataset.
", "votes": null, "replies": [ { "id": 294211, "author_name": "wythhh", "author_url": "", "post_date": "03/11/2018 15:17:09", "content": "You could try google cloud engine, they always have $300 coupons for starters.
", "votes": null, "replies": [] }, { "id": 296972, "author_name": "zyczyc", "author_url": "", "post_date": "03/16/2018 03:24:03", "content": "I think ec2 spot instance is a better choice, much cheaper ... and I built my environment and download data on an external EBS to keep it long time availabel.
", "votes": null, "replies": [] } ] }, { "id": 293530, "author_name": "titericz", "author_url": "", "post_date": "03/10/2018 04:08:47", "content": "I'm using a 16GB RAM Mac doing some feature engineering and training a lightgbm model using 50% of the trainset.
", "votes": null, "replies": [ { "id": 293533, "author_name": "sohaibomar", "author_url": "", "post_date": "03/10/2018 04:13:27", "content": "It means you are on top using lightgbm and only 50% trainset. AMAZING
Well, that's interesting and could I ask how is the memory usage when you run lightgbm? Did it use almost 100% of your RAM? Actually I forget the memory usage when I train lightgbm using whole dataset. But when I did 5 fold cv lightgbm on whole data set, the memory usage is around 64GB.
", "votes": null, "replies": [] }, { "id": 294046, "author_name": "xiumugengmu", "author_url": "", "post_date": "03/11/2018 06:39:50", "content": "Amazing! However, for further FE or more complicated algorithms, 16GB RAM could be enough?
", "votes": null, "replies": [] }, { "id": 294297, "author_name": "titericz", "author_url": "", "post_date": "03/11/2018 17:46:34", "content": "I trained using only 1 oof. Remember this is a time series challenge you can't do random kFold. I'm using all my RAM available and a huge SWAP file (most of my dataset goes to swap file). Since Mac SSDs are really fast I can train a model in about 1 hour
", "votes": null, "replies": [] }, { "id": 296776, "author_name": "sohaibomar", "author_url": "", "post_date": "03/15/2018 18:40:49", "content": "using 50% of the trainset.\nMay I get a hint on how did you select that 50% of train set? Thanks
", "votes": null, "replies": [] }, { "id": 298020, "author_name": "aakashnain", "author_url": "", "post_date": "03/18/2018 14:54:26", "content": "Amazing!!
", "votes": null, "replies": [] }, { "id": 298287, "author_name": "enricospada", "author_url": "", "post_date": "03/19/2018 09:11:54", "content": "Are you using R or Python? And most important: who have any hint about a tutorial/guide to use SWAP file in R using windows 10? Very interesting discovery!
", "votes": null, "replies": [] }, { "id": 298848, "author_name": "hfsong", "author_url": "", "post_date": "03/20/2018 04:44:59", "content": "I am using a 32GB RAM and 64G virtual memory from my SSD (WIN10, python). Training 50%-60% of the train dataset may take around 1 hour.
", "votes": null, "replies": [] } ] }, { "id": 300415, "author_name": "taojiang", "author_url": "", "post_date": "03/21/2018 15:46:43", "content": "I'm thinking if we can change the data type from int64 to int32 (or int16), I mean except the time column. Then we could get a somehow smaller data set.
", "votes": null, "replies": [ { "id": 300421, "author_name": "hfsong", "author_url": "", "post_date": "03/21/2018 15:51:09", "content": "actually you can drop the time column, only left day, hour in int
", "votes": null, "replies": [] }, { "id": 300432, "author_name": "taojiang", "author_url": "", "post_date": "03/21/2018 15:54:50", "content": "Good point, these are two days' data.
", "votes": null, "replies": [] } ] }, { "id": 300492, "author_name": "drhabib", "author_url": "", "post_date": "03/21/2018 16:31:10", "content": "Have you heard about this`?\n=)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "293320": "From the advice given under this topic, our group selected AWS EC2 instance to have a trial play with the whole data. We choose r4.8xlarge windows instance which as 32 CPU and 244GB memory with $3.6/hour.\nWe extract hour and minute from click_time and remove click_time and attributed_time and we noticed from spyder IDE that the memory usage is around 26%(almost 64GB) when we run 5-fold cv using lightgbm. I think this would be a kind of benchmark of memory usage for people who wants to run algorithms on whole data set. Using whole data set may not be a good idea but if you want, then 32GB or even 64GB is not enough for some complicated algorithms.", "293527": "Hi, Is there any other option $3.6/hr is costly for me and i want to use whole dataset.", "293530": "I'm using a 16GB RAM Mac doing some feature engineering and training a lightgbm model using 50% of the trainset.", "293533": "It means you are on top using lightgbm and only 50% trainset. AMAZING