![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F991320%2F69f6324e3cee76b4a665b17ddb7765fa%2F49141_0.jpg?generation=1565292608513763&alt=media\")
So far, we have been doing kfolds in a stratified manner. However, it seems CV and LB vary a lot. With a CV of 0.92, we get 0.79. I would, thus, like to know what your CV approach is and if we are doing something wrong in the cross validation?
", "messages": [ { "id": 592484, "postDate": "2019-08-05T11:19:25.797Z", "content": "So far, we have been doing kfolds in a stratified manner. However, it seems CV and LB vary a lot. With a CV of 0.92, we get 0.79. I would, thus, like to know what your CV approach is and if we are doing something wrong in the cross validation?
", "rawMarkdown": "So far, we have been doing kfolds in a stratified manner. However, it seems CV and LB vary a lot. With a CV of 0.92, we get 0.79. I would, thus, like to know what your CV approach is and if we are doing something wrong in the cross validation? ", "votes": 31 }, { "id": 593276, "postDate": "2019-08-06T11:52:12.697Z", "content": "Not sure how reliable is LB as well. In my opinion it is not too indicative. So we can (probably) expect quite serious shakeup :D
", "rawMarkdown": "Not sure how reliable is LB as well. In my opinion it is not too indicative. So we can (probably) expect quite serious shakeup :D ", "votes": 4 }, { "id": 596394, "postDate": "2019-08-10T15:29:27.783Z", "content": "It seems like the test set has a very different distribution than the training set. Could it be the case that the test set has more worse cases (3's and 4's) than the training set, which consists mostly of 0's and 2's?
\n\nFor me the score on LB is consistently ± 0.15 lower than local validation score. 😬
", "rawMarkdown": "It seems like the test set has a very different distribution than the training set. Could it be the case that the test set has more worse cases (3's and 4's) than the training set, which consists mostly of 0's and 2's? \n\nFor me the score on LB is consistently ± 0.15 lower than local validation score. 😬 ", "votes": 1, "replies": [ { "id": 596433, "postDate": "2019-08-10T16:24:19.603Z", "content": "You could check to see ur models predicted distribution by changing ur lb score based on conditions
", "rawMarkdown": "You could check to see ur models predicted distribution by changing ur lb score based on conditions", "votes": 2 } ] }, { "id": 595171, "postDate": "2019-08-08T23:42:32.527Z", "content": "My hypothesis:\n- Our models do really well classifying class 0\n- Train data set has a higher proportion of class 0 data than test data set
\n\nTherefore CV is consistently better than LB
", "rawMarkdown": "My hypothesis:\n- Our models do really well classifying class 0\n- Train data set has a higher proportion of class 0 data than test data set\n\nTherefore CV is consistently better than LB", "votes": 1 }, { "id": 594703, "postDate": "2019-08-08T10:03:33.013Z", "content": "Talk at kaggle days on Building the K-fold cross validation strategy by Dmytro https://www.youtube.com/watch?v=HfQjDfJ44Fs&t=34s . Might trigger some ideas.
https://www.kaggle.com/dmytropoplavskiy
", "rawMarkdown": "Talk at kaggle days on `Building the K-fold cross validation strategy` by Dmytro https://www.youtube.com/watch?v=HfQjDfJ44Fs&t=34s . Might trigger some ideas.\n\nhttps://www.kaggle.com/dmytropoplavskiy", "votes": 1 }, { "id": 593002, "postDate": "2019-08-06T05:14:47.210Z", "content": "I've just been using a specific seed(for val split) that has an ok correlation between cv and lb. No kfold cv yet
", "rawMarkdown": "I've just been using a specific seed(for val split) that has an ok correlation between cv and lb. No kfold cv yet", "votes": 1, "replies": [ { "id": 593134, "postDate": "2019-08-06T07:58:50.683Z", "content": "This is it, a true kaggler always tunes his random seed.
", "rawMarkdown": "This is it, a true kaggler always tunes his random seed.", "votes": 18 }, { "id": 594595, "postDate": "2019-08-08T07:42:13.537Z", "content": "@sidhanthholalkere hehe okay, do you mind sharing your seed number?
use 42. 42 is the answer to all your cv and lb problems =)
\n\n
Ah thank you! That is probably the problem. I've been using 1234. 😁
", "rawMarkdown": "Ah thank you! That is probably the problem. I've been using 1234. 😁 ", "votes": 1 }, { "id": 596965, "postDate": "2019-08-11T15:36:36.253Z", "content": "thanks
", "rawMarkdown": "thanks", "votes": 1 } ] }, { "id": 592581, "postDate": "2019-08-05T14:02:33.367Z", "content": "same with you.
", "rawMarkdown": "same with you.", "votes": 1 }, { "id": 594047, "postDate": "2019-08-07T13:45:42.170Z", "content": "Same here, sometimes the scores are anti-correlated. I think this will be a battle of who can make their model generalize best.
", "rawMarkdown": "Same here, sometimes the scores are anti-correlated. I think this will be a battle of who can make their model generalize best." }, { "id": 593271, "postDate": "2019-08-06T11:48:04.563Z", "content": "CV doesn't work for me. Looks like properly built validation set shows better correlation.
", "rawMarkdown": "CV doesn't work for me. Looks like properly built validation set shows better correlation." }, { "id": 593175, "postDate": "2019-08-06T08:51:09.847Z", "content": "From what i understand, it seems to be correlated to the test set sizes images too. A usual preprocessing seems to be insufficient. I didn't pay too much attention on it tho
", "rawMarkdown": "From what i understand, it seems to be correlated to the test set sizes images too. A usual preprocessing seems to be insufficient. I didn't pay too much attention on it tho" }, { "id": 592569, "postDate": "2019-08-05T13:42:32.197Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 593276, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-06T11:52:12.697000", "content": "Not sure how reliable is LB as well. In my opinion it is not too indicative. So we can (probably) expect quite serious shakeup :D
", "votes": 4, "replies": [] }, { "id": 596394, "author_name": "Carlo", "author_url": "", "post_date": "2019-08-10T15:29:27.783000", "content": "It seems like the test set has a very different distribution than the training set. Could it be the case that the test set has more worse cases (3's and 4's) than the training set, which consists mostly of 0's and 2's?
\n\nFor me the score on LB is consistently ± 0.15 lower than local validation score. 😬
", "votes": 1, "replies": [ { "id": 596433, "author_name": "sh", "author_url": "", "post_date": "2019-08-10T16:24:19.603000", "content": "You could check to see ur models predicted distribution by changing ur lb score based on conditions
", "votes": 2, "replies": [] } ] }, { "id": 595171, "author_name": "S D", "author_url": "", "post_date": "2019-08-08T23:42:32.527000", "content": "My hypothesis:\n- Our models do really well classifying class 0\n- Train data set has a higher proportion of class 0 data than test data set
\n\nTherefore CV is consistently better than LB
", "votes": 1, "replies": [] }, { "id": 594703, "author_name": "cyberia", "author_url": "", "post_date": "2019-08-08T10:03:33.013000", "content": "Talk at kaggle days on Building the K-fold cross validation strategy by Dmytro https://www.youtube.com/watch?v=HfQjDfJ44Fs&t=34s . Might trigger some ideas.
https://www.kaggle.com/dmytropoplavskiy
", "votes": 1, "replies": [] }, { "id": 593002, "author_name": "sh", "author_url": "", "post_date": "2019-08-06T05:14:47.210000", "content": "I've just been using a specific seed(for val split) that has an ok correlation between cv and lb. No kfold cv yet
", "votes": 1, "replies": [ { "id": 593134, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2019-08-06T07:58:50.683000", "content": "This is it, a true kaggler always tunes his random seed.
", "votes": 18, "replies": [] }, { "id": 594595, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-08-08T07:42:13.537000", "content": "@sidhanthholalkere hehe okay, do you mind sharing your seed number?
use 42. 42 is the answer to all your cv and lb problems =)
\n\n
Ah thank you! That is probably the problem. I've been using 1234. 😁
", "votes": 1, "replies": [] }, { "id": 596965, "author_name": "Deepak Dhaka", "author_url": "", "post_date": "2019-08-11T15:36:36.253000", "content": "thanks
", "votes": 1, "replies": [] } ] }, { "id": 592581, "author_name": "Gary", "author_url": "", "post_date": "2019-08-05T14:02:33.367000", "content": "same with you.
", "votes": 1, "replies": [] }, { "id": 594047, "author_name": "shatz", "author_url": "", "post_date": "2019-08-07T13:45:42.170000", "content": "Same here, sometimes the scores are anti-correlated. I think this will be a battle of who can make their model generalize best.
", "votes": 0, "replies": [] }, { "id": 593271, "author_name": "Georgi Pamukov", "author_url": "", "post_date": "2019-08-06T11:48:04.563000", "content": "CV doesn't work for me. Looks like properly built validation set shows better correlation.
", "votes": 0, "replies": [] }, { "id": 593175, "author_name": "Samuel Amoyal", "author_url": "", "post_date": "2019-08-06T08:51:09.847000", "content": "From what i understand, it seems to be correlated to the test set sizes images too. A usual preprocessing seems to be insufficient. I didn't pay too much attention on it tho
", "votes": 0, "replies": [] }, { "id": 592569, "author_name": "", "author_url": "", "post_date": "2019-08-05T13:42:32.197000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "592484": "So far, we have been doing kfolds in a stratified manner. However, it seems CV and LB vary a lot. With a CV of 0.92, we get 0.79. I would, thus, like to know what your CV approach is and if we are doing something wrong in the cross validation? ", "593276": "Not sure how reliable is LB as well. In my opinion it is not too indicative. So we can (probably) expect quite serious shakeup :D ", "596394": "It seems like the test set has a very different distribution than the training set. Could it be the case that the test set has more worse cases (3's and 4's) than the training set, which consists mostly of 0's and 2's? \n\nFor me the score on LB is consistently ± 0.15 lower than local validation score. 😬 ", "595171": "My hypothesis:\n- Our models do really well classifying class 0\n- Train data set has a higher proportion of class 0 data than test data set\n\nTherefore CV is consistently better than LB", "594703": "Talk at kaggle days on `Building the K-fold cross validation strategy` by Dmytro https://www.youtube.com/watch?v=HfQjDfJ44Fs&t=34s . Might trigger some ideas.\n\nhttps://www.kaggle.com/dmytropoplavskiy", "593002": "I've just been using a specific seed(for val split) that has an ok correlation between cv and lb. No kfold cv yet", "592581": "same with you.", "594047": "Same here, sometimes the scores are anti-correlated. I think this will be a battle of who can make their model generalize best.", "593271": "CV doesn't work for me. Looks like properly built validation set shows better correlation.", "593175": "From what i understand, it seems to be correlated to the test set sizes images too. A usual preprocessing seems to be insufficient. I didn't pay too much attention on it tho", "592569": "" } }