Because my computer RAM limited（only 16G）,Now my use 5M training data(60 features),it seems that my features didn't import my model perference,just curious if use more data can import the performance so that i need to rent a server.
", "messages": [ { "id": "1083996", "postDate": "11/19/2020 15:58:23", "content": "Because my computer RAM limited（only 16G）,Now my use 5M training data(60 features),it seems that my features didn't import my model perference,just curious if use more data can import the performance so that i need to rent a server.
", "rawMarkdown": "Because my computer RAM limited（only 16G）,Now my use 5M training data(60 features),it seems that my features didn't import my model perference,just curious if use more data can import the performance so that i need to rent a server.", "votes": null }, { "id": "1084024", "postDate": "11/19/2020 16:21:29", "content": "In my case it helps ~0.01 moving from 10M to 80M rows.
", "rawMarkdown": "In my case it helps ~0.01 moving from 10M to 80M rows.", "votes": null }, { "id": "1084029", "postDate": "11/19/2020 16:28:49", "content": "Amazing,thanks for your information
", "rawMarkdown": "Amazing,thanks for your information", "votes": null }, { "id": "1084030", "postDate": "11/19/2020 16:29:09", "content": "@bacicnikola you mean you add more training data and your score boost from 0.787 to 0.797?👀
", "rawMarkdown": "bacicnikola you mean you add more training data and your score boost from 0.787 to 0.797?👀", "votes": null }, { "id": "1084033", "postDate": "11/19/2020 16:34:33", "content": "5M rows and #12 is amazing! You can use free 300$ of GCP.
", "rawMarkdown": "5M rows and #12 is amazing! You can use free 300$ of GCP.", "votes": null }, { "id": "1084053", "postDate": "11/19/2020 17:10:20", "content": "Well actually it boost my validation score. I'm always making a submission on full dataset, so I don't actually know the LB difference…
", "rawMarkdown": "Well actually it boost my validation score. I'm always making a submission on full dataset, so I don't actually know the LB difference...", "votes": null }, { "id": "1084190", "postDate": "11/19/2020 20:25:23", "content": "My CV is correlated with LB.
\n20M rows to 80M rows also boost by +0.01 like @bacicnikola
by \"making a submission on full dataset\" do you mean you use the most recent historical records from full dataset in test time (e.g. to do all the merge etc.), or do you mean retraining a classifier on the full dataset (number if boosting rounds selected from CV)?
", "rawMarkdown": "by \"making a submission on full dataset\" do you mean you use the most recent historical records from full dataset in test time (e.g. to do all the merge etc.), or do you mean retraining a classifier on the full dataset (number if boosting rounds selected from CV)?", "votes": null }, { "id": "1084447", "postDate": "11/20/2020 03:32:38", "content": "Do you submit 1 model to predict?
\ni submit 10 models, which training 10m row. data is spilted by mod(user_id, 10)
yes,only one lgb
", "rawMarkdown": "yes,only one lgb", "votes": null }, { "id": "1084453", "postDate": "11/20/2020 03:43:22", "content": "Great job,thanks
", "rawMarkdown": "Great job,thanks", "votes": null }, { "id": "1084537", "postDate": "11/20/2020 06:04:39", "content": "Amazing, 791 with only 5M data!!!
", "rawMarkdown": "Amazing, 791 with only 5M data!!!", "votes": null }, { "id": "1084667", "postDate": "11/20/2020 09:07:56", "content": "I mean retraining a classifier on the full dataset.
", "rawMarkdown": "I mean retraining a classifier on the full dataset.", "votes": null }, { "id": "1084755", "postDate": "11/20/2020 11:02:58", "content": "Good to hear that! I'm currently using even less than 10M rows for training, so hopefully that big improvement is awaiting me, too :)
", "rawMarkdown": "Good to hear that! I'm currently using even less than 10M rows for training, so hopefully that big improvement is awaiting me, too :)", "votes": null }, { "id": "1085560", "postDate": "11/21/2020 01:20:33", "content": "As the data grows, the size of the state variable increases rapidly. As a result, when using the full amount of data, there is an out-of-memory situation for online infering. This is what happened to me. I don't know anyone else.
", "rawMarkdown": "As the data grows, the size of the state variable increases rapidly. As a result, when using the full amount of data, there is an out-of-memory situation for online infering. This is what happened to me. I don't know anyone else.", "votes": null }, { "id": "1085857", "postDate": "11/21/2020 08:43:38", "content": "Yes, me too, I'm fighting to avoid OOM. The more features the more memory used.
", "rawMarkdown": "Yes, me too, I'm fighting to avoid OOM. The more features the more memory used.", "votes": null }, { "id": "1086924", "postDate": "11/22/2020 07:19:04", "content": "When you say \"my computer RAM\" - if you are talking about your local machine than it's not a real hard chore to increase the effective RAM if you have some space on an SSD in the system.
\nOn Ubuntu Linux you can use SSD memory as swapfile - I add an SSD to my system and use 200GB of that as swap.
\nOn that same machine when I boot up in Windows 10 I use 200GB for virtual memory. (if you guess the SSD I add is a half GB than you win the prize):
The Linux is a bit tricky - email me and I can send you the code snippets to get the swapfile raised from the default (think it's 2GB) to a larger value.
\nThis could also work if you have standard hard drive rather than SSD's but the speed performance on a hard drive really sucks. With SSD's on Ubuntu I can feel the bump in performance when the swap is needed but it's something you can live with.
\nI have this setup running on 4 different local machines - you can still have kernels die with what are likely memory issues but for the most part it works.
", "rawMarkdown": "When you say \"my computer RAM\" - if you are talking about your local machine than it's not a real hard chore to increase the effective RAM if you have some space on an SSD in the system.\n\nOn Ubuntu Linux you can use SSD memory as swapfile - I add an SSD to my system and use 200GB of that as swap. \nOn that same machine when I boot up in Windows 10 I use 200GB for virtual memory. (if you guess the SSD I add is a half GB than you win the prize):\n\nThe Linux is a bit tricky - email me and I can send you the code snippets to get the swapfile raised from the default (think it's 2GB) to a larger value.\n\nThis could also work if you have standard hard drive rather than SSD's but the speed performance on a hard drive really sucks. With SSD's on Ubuntu I can feel the bump in performance when the swap is needed but it's something you can live with.\n\nI have this setup running on 4 different local machines - you can still have kernels die with what are likely memory issues but for the most part it works.", "votes": null }, { "id": "1091433", "postDate": "11/26/2020 02:46:34", "content": "thanks for you help
", "rawMarkdown": "thanks for you help", "votes": null }, { "id": "1100722", "postDate": "12/03/2020 09:21:08", "content": "Amazing! I can use my 32G RAM joining!
", "rawMarkdown": "Amazing! I can use my 32G RAM joining!", "votes": null }, { "id": "1101545", "postDate": "12/04/2020 02:18:28", "content": "You mean use 80M to training or just generating features？
", "rawMarkdown": "You mean use 80M to training or just generating features？", "votes": null }, { "id": "1108680", "postDate": "12/10/2020 22:33:53", "content": "My CV used to be correlated with LB really well, but not in this case of training more data…Training on the full dataset improved my CV by ~0.01 (like others have mentioned) but didn't improve my LB as much.
\nNot sure if anyone else is facing the same situation.
In my case it helps ~0.01 moving from 10M to 80M rows.
", "votes": null, "replies": [ { "id": 1084029, "author_name": "mahluo", "author_url": "", "post_date": "11/19/2020 16:28:49", "content": "Amazing,thanks for your information
", "votes": null, "replies": [] }, { "id": 1084030, "author_name": "chenxin1991", "author_url": "", "post_date": "11/19/2020 16:29:09", "content": "@bacicnikola you mean you add more training data and your score boost from 0.787 to 0.797?👀
", "votes": null, "replies": [] }, { "id": 1084053, "author_name": "bacicnikola", "author_url": "", "post_date": "11/19/2020 17:10:20", "content": "Well actually it boost my validation score. I'm always making a submission on full dataset, so I don't actually know the LB difference…
", "votes": null, "replies": [] }, { "id": 1084193, "author_name": "samshipengs", "author_url": "", "post_date": "11/19/2020 20:28:41", "content": "by \"making a submission on full dataset\" do you mean you use the most recent historical records from full dataset in test time (e.g. to do all the merge etc.), or do you mean retraining a classifier on the full dataset (number if boosting rounds selected from CV)?
", "votes": null, "replies": [] }, { "id": 1084667, "author_name": "bacicnikola", "author_url": "", "post_date": "11/20/2020 09:07:56", "content": "I mean retraining a classifier on the full dataset.
", "votes": null, "replies": [] }, { "id": 1084755, "author_name": "alijs1", "author_url": "", "post_date": "11/20/2020 11:02:58", "content": "Good to hear that! I'm currently using even less than 10M rows for training, so hopefully that big improvement is awaiting me, too :)
", "votes": null, "replies": [] } ] }, { "id": 1084033, "author_name": "adityaecdrid", "author_url": "", "post_date": "11/19/2020 16:34:33", "content": "5M rows and #12 is amazing! You can use free 300$ of GCP.
", "votes": null, "replies": [] }, { "id": 1084190, "author_name": "mpware", "author_url": "", "post_date": "11/19/2020 20:25:23", "content": "My CV is correlated with LB.
\n20M rows to 80M rows also boost by +0.01 like @bacicnikola
Great job,thanks
", "votes": null, "replies": [] } ] }, { "id": 1084447, "author_name": "kurupical", "author_url": "", "post_date": "11/20/2020 03:32:38", "content": "Do you submit 1 model to predict?
\ni submit 10 models, which training 10m row. data is spilted by mod(user_id, 10)
yes,only one lgb
", "votes": null, "replies": [] } ] }, { "id": 1084537, "author_name": "shichenyang", "author_url": "", "post_date": "11/20/2020 06:04:39", "content": "Amazing, 791 with only 5M data!!!
", "votes": null, "replies": [] }, { "id": 1085560, "author_name": "shichenyang", "author_url": "", "post_date": "11/21/2020 01:20:33", "content": "As the data grows, the size of the state variable increases rapidly. As a result, when using the full amount of data, there is an out-of-memory situation for online infering. This is what happened to me. I don't know anyone else.
", "votes": null, "replies": [ { "id": 1085857, "author_name": "mpware", "author_url": "", "post_date": "11/21/2020 08:43:38", "content": "Yes, me too, I'm fighting to avoid OOM. The more features the more memory used.
", "votes": null, "replies": [] } ] }, { "id": 1086924, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/22/2020 07:19:04", "content": "When you say \"my computer RAM\" - if you are talking about your local machine than it's not a real hard chore to increase the effective RAM if you have some space on an SSD in the system.
\nOn Ubuntu Linux you can use SSD memory as swapfile - I add an SSD to my system and use 200GB of that as swap.
\nOn that same machine when I boot up in Windows 10 I use 200GB for virtual memory. (if you guess the SSD I add is a half GB than you win the prize):
The Linux is a bit tricky - email me and I can send you the code snippets to get the swapfile raised from the default (think it's 2GB) to a larger value.
\nThis could also work if you have standard hard drive rather than SSD's but the speed performance on a hard drive really sucks. With SSD's on Ubuntu I can feel the bump in performance when the swap is needed but it's something you can live with.
\nI have this setup running on 4 different local machines - you can still have kernels die with what are likely memory issues but for the most part it works.
", "votes": null, "replies": [ { "id": 1091433, "author_name": "mahluo", "author_url": "", "post_date": "11/26/2020 02:46:34", "content": "thanks for you help
", "votes": null, "replies": [] } ] }, { "id": 1100722, "author_name": "daishu", "author_url": "", "post_date": "12/03/2020 09:21:08", "content": "Amazing! I can use my 32G RAM joining!
", "votes": null, "replies": [] }, { "id": 1101545, "author_name": "juzqyxs", "author_url": "", "post_date": "12/04/2020 02:18:28", "content": "You mean use 80M to training or just generating features？
", "votes": null, "replies": [] }, { "id": 1108680, "author_name": "raphael1123", "author_url": "", "post_date": "12/10/2020 22:33:53", "content": "My CV used to be correlated with LB really well, but not in this case of training more data…Training on the full dataset improved my CV by ~0.01 (like others have mentioned) but didn't improve my LB as much.
\nNot sure if anyone else is facing the same situation.