Hi everyone! At the beginning I will say only that this is my first competition on Kaggle, I'm still learning and I'm very sorry if my question seems obvious to anyone. I am very curious which validation set do you use? This topic is currently sparing my dreams. I searched the forum for information about what fragments of the training set people use for validation. Of course, I have found a lot of helpful information here and here.
\n\nI know it is usefull to take data from November 9th between 4 and 5 am because they are probably part of the test set on Kaggle. I still wonder if this is a good practice. I also noticed that many people simply take the last 5% of the training set.
\n\nDoes any of you use cross validation? If so, how do you propose it?
\n\nThank you in advance for your answer.
", "messages": [ { "id": "306695", "postDate": "03/30/2018 20:44:42", "content": "Hi everyone! At the beginning I will say only that this is my first competition on Kaggle, I'm still learning and I'm very sorry if my question seems obvious to anyone. I am very curious which validation set do you use? This topic is currently sparing my dreams. I searched the forum for information about what fragments of the training set people use for validation. Of course, I have found a lot of helpful information here and here.
\n\nI know it is usefull to take data from November 9th between 4 and 5 am because they are probably part of the test set on Kaggle. I still wonder if this is a good practice. I also noticed that many people simply take the last 5% of the training set.
\n\nDoes any of you use cross validation? If so, how do you propose it?
\n\nThank you in advance for your answer.
", "rawMarkdown": "Hi everyone! At the beginning I will say only that this is my first competition on Kaggle, I'm still learning and I'm very sorry if my question seems obvious to anyone. I am very curious which validation set do you use? This topic is currently sparing my dreams. I searched the forum for information about what fragments of the training set people use for validation. Of course, I have found a lot of helpful information [here][1] and [here][2]. \n\nI know it is usefull to take data from November 9th between 4 and 5 am because they are probably part of the test set on Kaggle. I still wonder if this is a good practice. I also noticed that many people simply take the last 5% of the training set. \n\nDoes any of you use cross validation? If so, how do you propose it?\n\nThank you in advance for your answer.\n\n [1]: https://www.kaggle.com/konradb/validation-set/code\n [2]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51492", "votes": null }, { "id": "306710", "postDate": "03/30/2018 21:18:02", "content": "I use whole day 9 as a validation set.
", "rawMarkdown": "I use whole day 9 as a validation set.", "votes": null }, { "id": "306717", "postDate": "03/30/2018 21:30:12", "content": "Do you notice a big difference between the score of your validation and public score on Kaggle on submit? I am also curious if you train the model on the entire training set?
", "rawMarkdown": "Do you notice a big difference between the score of your validation and public score on Kaggle on submit? I am also curious if you train the model on the entire training set?", "votes": null }, { "id": "306724", "postDate": "03/30/2018 21:50:47", "content": "I use a validation set from day 9 that matches the times for the test data (6 full hours in separate 2 hour segments plus 1 additional second at the end of each of the segments). Sometimes I divide it into \"public\" and \"private\" subsets corresponding to the public and private test data. (The public data are the first hour, which corresponds to the first 4032690 records of my validation data for day 9.)
", "rawMarkdown": "I use a [validation set from day 9][1] that matches the [times for the test data][2] (6 full hours in separate 2 hour segments plus 1 additional second at the end of each of the segments). Sometimes I divide it into \"public\" and \"private\" [subsets][3] corresponding to the public and private test data. (The public data are the first hour, which corresponds to the first 4032690 records of my validation data for day 9.)\n\n\n [1]: https://www.kaggle.com/aharless/training-and-validation-data-pickle\n [2]: https://www.kaggle.com/aharless/critical-times\n [3]: https://www.kaggle.com/aharless/validation-of-kireev-style-nn/log", "votes": null }, { "id": "306872", "postDate": "03/31/2018 07:55:06", "content": "Thank you very much for your answer. A lot of valuable information has appeared here too.
", "rawMarkdown": "Thank you very much for your answer. A lot of valuable information has appeared [here][1] too.\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53325", "votes": null }, { "id": "306873", "postDate": "03/31/2018 07:56:08", "content": "Does anyone use some kind of cross validation?
", "rawMarkdown": "Does anyone use some kind of cross validation?", "votes": null }, { "id": "306879", "postDate": "03/31/2018 08:19:32", "content": "Difference between my CV and LB is consistent, in 0.002 - 0.003 range. And I do not use all training set. I train on day 8, validate on 9, if validation score increases, the retrain on day 8+9 and predict on test.
", "rawMarkdown": "Difference between my CV and LB is consistent, in 0.002 - 0.003 range. And I do not use all training set. I train on day 8, validate on 9, if validation score increases, the retrain on day 8+9 and predict on test.", "votes": null }, { "id": "306943", "postDate": "03/31/2018 11:38:01", "content": "For me the difference at this point is around 0.015 ... I certainly do something wrong and I try to understand what the problem is. I will try to modify my current script, bringing it closer to your validation. Maybe it's just my impression, but from what I read the forum people try to deliberately overfit...
\n\nOut of curiosity, your result 0.9704 - is it one model or stacking?
", "rawMarkdown": "For me the difference at this point is around 0.015 ... I certainly do something wrong and I try to understand what the problem is. I will try to modify my current script, bringing it closer to your validation. Maybe it's just my impression, but from what I read the forum people try to deliberately overfit...\n\nOut of curiosity, your result 0.9704 - is it one model or stacking?", "votes": null }, { "id": "306948", "postDate": "03/31/2018 11:57:05", "content": "It's a single model.
", "rawMarkdown": "It's a single model.", "votes": null }, { "id": "307080", "postDate": "03/31/2018 19:09:13", "content": "I am using the kaggle kernel to do this competition because my local machine only has 4gb ram. could someone tell me which subset of the train data should I pick?
", "rawMarkdown": "I am using the kaggle kernel to do this competition because my local machine only has 4gb ram. could someone tell me which subset of the train data should I pick?", "votes": null } ], "comments": [ { "id": 306710, "author_name": "sohaibomar", "author_url": "", "post_date": "03/30/2018 21:18:02", "content": "I use whole day 9 as a validation set.
", "votes": null, "replies": [ { "id": 306717, "author_name": "skalskip", "author_url": "", "post_date": "03/30/2018 21:30:12", "content": "Do you notice a big difference between the score of your validation and public score on Kaggle on submit? I am also curious if you train the model on the entire training set?
", "votes": null, "replies": [] }, { "id": 306879, "author_name": "sohaibomar", "author_url": "", "post_date": "03/31/2018 08:19:32", "content": "Difference between my CV and LB is consistent, in 0.002 - 0.003 range. And I do not use all training set. I train on day 8, validate on 9, if validation score increases, the retrain on day 8+9 and predict on test.
", "votes": null, "replies": [] }, { "id": 306943, "author_name": "skalskip", "author_url": "", "post_date": "03/31/2018 11:38:01", "content": "For me the difference at this point is around 0.015 ... I certainly do something wrong and I try to understand what the problem is. I will try to modify my current script, bringing it closer to your validation. Maybe it's just my impression, but from what I read the forum people try to deliberately overfit...
\n\nOut of curiosity, your result 0.9704 - is it one model or stacking?
", "votes": null, "replies": [] }, { "id": 306948, "author_name": "sohaibomar", "author_url": "", "post_date": "03/31/2018 11:57:05", "content": "It's a single model.
", "votes": null, "replies": [] } ] }, { "id": 306724, "author_name": "aharless", "author_url": "", "post_date": "03/30/2018 21:50:47", "content": "I use a validation set from day 9 that matches the times for the test data (6 full hours in separate 2 hour segments plus 1 additional second at the end of each of the segments). Sometimes I divide it into \"public\" and \"private\" subsets corresponding to the public and private test data. (The public data are the first hour, which corresponds to the first 4032690 records of my validation data for day 9.)
", "votes": null, "replies": [ { "id": 306872, "author_name": "skalskip", "author_url": "", "post_date": "03/31/2018 07:55:06", "content": "Thank you very much for your answer. A lot of valuable information has appeared here too.
", "votes": null, "replies": [] } ] }, { "id": 306873, "author_name": "skalskip", "author_url": "", "post_date": "03/31/2018 07:56:08", "content": "Does anyone use some kind of cross validation?
", "votes": null, "replies": [] }, { "id": 307080, "author_name": "konohayui", "author_url": "", "post_date": "03/31/2018 19:09:13", "content": "I am using the kaggle kernel to do this competition because my local machine only has 4gb ram. could someone tell me which subset of the train data should I pick?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "306695": "Hi everyone! At the beginning I will say only that this is my first competition on Kaggle, I'm still learning and I'm very sorry if my question seems obvious to anyone. I am very curious which validation set do you use? This topic is currently sparing my dreams. I searched the forum for information about what fragments of the training set people use for validation. Of course, I have found a lot of helpful information [here][1] and [here][2]. \n\nI know it is usefull to take data from November 9th between 4 and 5 am because they are probably part of the test set on Kaggle. I still wonder if this is a good practice. I also noticed that many people simply take the last 5% of the training set. \n\nDoes any of you use cross validation? If so, how do you propose it?\n\nThank you in advance for your answer.\n\n [1]: https://www.kaggle.com/konradb/validation-set/code\n [2]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51492", "306710": "I use whole day 9 as a validation set.", "306717": "Do you notice a big difference between the score of your validation and public score on Kaggle on submit? I am also curious if you train the model on the entire training set?", "306724": "I use a [validation set from day 9][1] that matches the [times for the test data][2] (6 full hours in separate 2 hour segments plus 1 additional second at the end of each of the segments). Sometimes I divide it into \"public\" and \"private\" [subsets][3] corresponding to the public and private test data. (The public data are the first hour, which corresponds to the first 4032690 records of my validation data for day 9.)\n\n\n [1]: https://www.kaggle.com/aharless/training-and-validation-data-pickle\n [2]: https://www.kaggle.com/aharless/critical-times\n [3]: https://www.kaggle.com/aharless/validation-of-kireev-style-nn/log", "306872": "Thank you very much for your answer. A lot of valuable information has appeared [here][1] too.\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53325", "306873": "Does anyone use some kind of cross validation?", "306879": "Difference between my CV and LB is consistent, in 0.002 - 0.003 range. And I do not use all training set. I train on day 8, validate on 9, if validation score increases, the retrain on day 8+9 and predict on test.", "306943": "For me the difference at this point is around 0.015 ... I certainly do something wrong and I try to understand what the problem is. I will try to modify my current script, bringing it closer to your validation. Maybe it's just my impression, but from what I read the forum people try to deliberately overfit...\n\nOut of curiosity, your result 0.9704 - is it one model or stacking?", "306948": "It's a single model.", "307080": "I am using the kaggle kernel to do this competition because my local machine only has 4gb ram. could someone tell me which subset of the train data should I pick?" }, "source": "meta" }