![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2F0abe1fc1bc8ba81186d59e1e45c2bfcb%2Ff0.png?generation=1595303057355592&alt=media\")
I have trained the @cdeotte triple stratified 192 data on effnet b2\nBut while training im getting cv score like this\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2F5f32881ed281a3c78f8b2b5d7fe9b252%2Ff1.png?generation=1595303055728834&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2Ff6ad53cc4256f19ecd41840e7e57d32b%2Ff2.png?generation=1595303053724869&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2F22ffba73ac25c56e76e67f7625f872ca%2Ff3.png?generation=1595303060996695&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2F82f3f7b980618cf86b4c6a5555a72f2b%2Ff4.png?generation=1595303053554457&alt=media\")
And the overall oof score is \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2Fcd087916f7b71ab2851a5cd73d456d30%2Foverall.png?generation=1595303170356915&alt=media\")
Validation score is getting boosted to 0.972\nLooking at some public notebook,Is there any chance that the cv score will get high uto 0.972 in chriss 192x192 data??
\n\nThis is my fold data report\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2Ffc38485a387bd2982e8ed1238b40f6be%2Ffault.png?generation=1595305705556165&alt=media\")
Submitting this file got me an LB Score of : 0.9227
", "messages": [ { "id": 937485, "postDate": "2020-07-21T03:46:38.697Z", "content": "I have trained the @cdeotte triple stratified 192 data on effnet b2\nBut while training im getting cv score like this\n
And the overall oof score is \n
Validation score is getting boosted to 0.972\nLooking at some public notebook,Is there any chance that the cv score will get high uto 0.972 in chriss 192x192 data??
\n\nThis is my fold data report\n
Submitting this file got me an LB Score of : 0.9227
", "rawMarkdown": "I have trained the @cdeotte triple stratified 192 data on effnet b2\nBut while training im getting cv score like this\n\n\n\n\n\n\n\n\n\n\n\nAnd the overall oof score is \n\n\nValidation score is getting boosted to 0.972\nLooking at some public notebook,Is there any chance that the cv score will get high uto 0.972 in chriss 192x192 data??\n\nThis is my fold data report\n\n\n\n\n\nSubmitting this file got me an LB Score of : 0.9227\n\n\n" }, { "id": 937521, "postDate": "2020-07-21T04:18:24.553Z", "content": "What does \"Number Train file at fold 1 : 40\" mean? If I fork my TFRecord notebook and use your setttings of 192x129, EffNetB2, Inc2019=1, and Inc2018=1, i get the following result (Note this fold 2 is your fold 1)
\n![\"image\"](\"http://playagricola.com/Kaggle/pic-7-20.png\")
Are you writing your code using the JPEGs?
", "rawMarkdown": "What does \"Number Train file at fold 1 : 40\" mean? If I fork my TFRecord notebook and use your setttings of 192x129, EffNetB2, Inc2019=1, and Inc2018=1, i get the following result (Note this fold 2 is your fold 1)\n\n\n\nAre you writing your code using the JPEGs?", "replies": [ { "id": 937530, "postDate": "2020-07-21T04:24:14.853Z", "content": "@cdeotte I just took the number of tfrecords that have at fold {x} while training\nSorry for the typo\nI have updated it
", "rawMarkdown": "@cdeotte I just took the number of tfrecords that have at fold {x} while training\nSorry for the typo\nI have updated it \n" }, { "id": 937537, "postDate": "2020-07-21T04:28:36.263Z", "content": "Ok that is the mistake. Every fold should only have 3 validation TFRecords. The external data never goes into validation. Only 3 of the 15 train TFRecords should be in validation each fold.
\nThen your fold 0 should have 12+12 = 24 TFRecords, your fold 1 should have 12+12+12=36, your fold 2 should have 24, your fold 3 should have 36 and your fold 4 should have 24 in train
", "rawMarkdown": "Ok that is the mistake. Every fold should only have 3 validation TFRecords. The external data never goes into validation. Only 3 of the 15 train TFRecords should be in validation each fold.\n\nThen your fold 0 should have 12+12 = 24 TFRecords, your fold 1 should have 12+12+12=36, your fold 2 should have 24, your fold 3 should have 36 and your fold 4 should have 24 in train", "votes": 1 }, { "id": 937539, "postDate": "2020-07-21T04:30:30.443Z", "content": "Ok i will redo the strategy.But why only 3 valid sets and why external data should not go to validation?
", "rawMarkdown": "Ok i will redo the strategy.But why only 3 valid sets and why external data should not go to validation?" }, { "id": 937541, "postDate": "2020-07-21T04:31:06.820Z", "content": "The train data is 15 TFRecords, divide them into 5 groups of 3 records. The external data has 15 even numbered TFRecords and 15 odd numbered records. Divide the even numbers (0, 2, 4, …, 28) into 5 groups of 3. And divide the odd numbers into (1, 3, 5, …, 29) into 5 groups of 3.
\nThe validation for fold 1 is one group from train, fold 2 is one group from train, etc.
\nThe fold 1 trains with the other 4 groups of train and 4 groups of even and 4 groups of odd, etc etc
", "rawMarkdown": "The train data is 15 TFRecords, divide them into 5 groups of 3 records. The external data has 15 even numbered TFRecords and 15 odd numbered records. Divide the even numbers (0, 2, 4, ..., 28) into 5 groups of 3. And divide the odd numbers into (1, 3, 5, ..., 29) into 5 groups of 3.\n\nThe validation for fold 1 is one group from train, fold 2 is one group from train, etc.\n\nThe fold 1 trains with the other 4 groups of train and 4 groups of even and 4 groups of odd, etc etc" }, { "id": 937542, "postDate": "2020-07-21T04:31:59.910Z", "content": "If you do an experiment with external data and want to compare it with an experiment without external data, both experiments need the same validation sets. Therefore you always use validation without external data
", "rawMarkdown": "If you do an experiment **with** external data and want to compare it with an experiment **without** external data, both experiments need the same validation sets. Therefore you always use validation without external data", "votes": 1 }, { "id": 937545, "postDate": "2020-07-21T04:32:49.233Z", "content": "Thanks for the help @cdeotte 😊
", "rawMarkdown": "Thanks for the help @cdeotte 😊 ", "votes": 1 }, { "id": 937546, "postDate": "2020-07-21T04:32:57.890Z", "content": "If you add some random validation data that is very easy to classify, it will not help LB. If you add this easy external data to your validation set, it will give the illusion that your CV increased but it doesn't really.
", "rawMarkdown": "If you add some random validation data that is very easy to classify, it will not help LB. If you add this **easy** external data to your validation set, it will give the illusion that your CV increased but it doesn't really." }, { "id": 937555, "postDate": "2020-07-21T04:40:16.900Z", "content": "Ha, didnt realised that. Will change my strategy,By the thanks for the information too
", "rawMarkdown": "Ha, didnt realised that. Will change my strategy,By the thanks for the information too" }, { "id": 937570, "postDate": "2020-07-21T04:52:21.100Z", "content": "Experimenting with using external data is good. I was just giving a second reason why we can't add external data to validation folds. Reason 1 is because we want to compare different experiments so all experiments need the same validation set. Reason 2 is if we add external data that is easy to classify to the validation fold, then the overall CV score will artificially increase.
\nBut there is not problem with adding easy external data to the train fold. You can add anything to the train fold (as long as it isn't duplicate images that leak validation information). And it is a good strategy to try adding lots of different things to the training fold.
", "rawMarkdown": "Experimenting with using external data is good. I was just giving a second reason why we can't add external data to validation folds. Reason 1 is because we want to compare different experiments so all experiments need the same validation set. Reason 2 is if we add external data that is easy to classify to the validation fold, then the overall CV score will artificially increase.\n\nBut there is not problem with adding easy external data to the train fold. You can add anything to the train fold (as long as it isn't duplicate images that leak validation information). And it is a good strategy to try adding lots of different things to the training fold.", "votes": 3 }, { "id": 937732, "postDate": "2020-07-21T06:24:38.160Z", "content": "@cdeotte like you said there is cv and lb gap while using external data since there isnt any images in external data that have similar ones in our competition files.So what i propose is removing those images that doesnt correlate with test images or posting in which external tf-files are there these images are in so that we wont use that while training
", "rawMarkdown": "@cdeotte like you said there is cv and lb gap while using external data since there isnt any images in external data that have similar ones in our competition files.So what i propose is removing those images that doesnt correlate with test images or posting in which external tf-files are there these images are in so that we wont use that while training" } ] }, { "id": 937492, "postDate": "2020-07-21T03:52:29.193Z", "content": "Hmm…. this doesn't look right. Did you add more data to the validation folds? Or add validation data to the training folds?
\nWhich data are you training with and which data are you validating with?
", "rawMarkdown": "Hmm.... this doesn't look right. Did you add more data to the validation folds? Or add validation data to the training folds?\n\nWhich data are you training with and which data are you validating with?", "replies": [ { "id": 937494, "postDate": "2020-07-21T03:53:51.863Z", "content": "@cdeotte No i can assure that there isnt any validation data in train set\nLike your notebook I added INC2019 = [0,1,0,1,0] and INC2018 = [1,1,1,1,1]
", "rawMarkdown": "@cdeotte No i can assure that there isnt any validation data in train set\nLike your notebook I added INC2019 = [0,1,0,1,0] and INC2018 = [1,1,1,1,1]" }, { "id": 937497, "postDate": "2020-07-21T03:55:41.463Z", "content": "Ok, i'll look into it. Something isn't right
", "rawMarkdown": "Ok, i'll look into it. Something isn't right" }, { "id": 937502, "postDate": "2020-07-21T03:59:00.147Z", "content": "Yes i even thought that the triple stratified data doesnt have any leaks.But this oof report is misleading it.Anyways we really appreciate your hardwork in making this data @cdeotte
", "rawMarkdown": "Yes i even thought that the triple stratified data doesnt have any leaks.But this oof report is misleading it.Anyways we really appreciate your hardwork in making this data @cdeotte " }, { "id": 937527, "postDate": "2020-07-21T04:22:05.613Z", "rawMarkdown": "", "isDeleted": true }, { "id": 937558, "postDate": "2020-07-21T04:42:44.487Z", "content": "I'm still investigating this. I believe it is caused by 2 things. (1) The test images are different than train images and (2) there are only 78 malignant out of 3295 public test images.
\nThis will cause weird difference between CV and LB. For example, let's say your model learns to classify all green images as malignant. Now imagine that there are no green images in train data. Therefore this new behavior will not help CV. What if the test data had 13 green images (out of its 78 malignant images). Then your LB would increase by 0.09 AUC!! ( = 0.50 * 13 / 78)
So your CV could be 0.86 and your LB could be 0.95!!
\nThis isn't so far fetched because 15% of the test images are different than the train images. 15% of test images come from a source of size 1920x1080 and there are not train images that come from 1920x1080. And these images have different colors than rest of train as explained here
Thanks. 21 days to merger and 28 days to deadline. Plenty of time to figure out the CV LB problem.
", "rawMarkdown": "Thanks. 21 days to merger and 28 days to deadline. Plenty of time to figure out the CV LB problem." }, { "id": 937592, "postDate": "2020-07-21T05:10:35.663Z", "content": "Thanks to people like @cdeotte hope cv lb problem will be resolved by that time😄
", "rawMarkdown": "Thanks to people like @cdeotte hope cv lb problem will be resolved by that time😄 ", "votes": 1 } ] }, { "id": 937488, "postDate": "2020-07-21T03:48:56.033Z", "content": "Did you try submitting it?
", "rawMarkdown": "Did you try submitting it?", "replies": [ { "id": 937489, "postDate": "2020-07-21T03:49:40.533Z", "content": "Yes but the LB Score is 0.9227
", "rawMarkdown": "Yes but the LB Score is 0.9227" } ] } ], "comments": [ { "id": 937521, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:18:24.553000", "content": "What does \"Number Train file at fold 1 : 40\" mean? If I fork my TFRecord notebook and use your setttings of 192x129, EffNetB2, Inc2019=1, and Inc2018=1, i get the following result (Note this fold 2 is your fold 1)
\n
Are you writing your code using the JPEGs?
", "votes": 0, "replies": [ { "id": 937530, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T04:24:14.853000", "content": "@cdeotte I just took the number of tfrecords that have at fold {x} while training\nSorry for the typo\nI have updated it
", "votes": 0, "replies": [] }, { "id": 937537, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:28:36.263000", "content": "Ok that is the mistake. Every fold should only have 3 validation TFRecords. The external data never goes into validation. Only 3 of the 15 train TFRecords should be in validation each fold.
\nThen your fold 0 should have 12+12 = 24 TFRecords, your fold 1 should have 12+12+12=36, your fold 2 should have 24, your fold 3 should have 36 and your fold 4 should have 24 in train
", "votes": 1, "replies": [] }, { "id": 937539, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T04:30:30.443000", "content": "Ok i will redo the strategy.But why only 3 valid sets and why external data should not go to validation?
", "votes": 0, "replies": [] }, { "id": 937541, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:31:06.820000", "content": "The train data is 15 TFRecords, divide them into 5 groups of 3 records. The external data has 15 even numbered TFRecords and 15 odd numbered records. Divide the even numbers (0, 2, 4, …, 28) into 5 groups of 3. And divide the odd numbers into (1, 3, 5, …, 29) into 5 groups of 3.
\nThe validation for fold 1 is one group from train, fold 2 is one group from train, etc.
\nThe fold 1 trains with the other 4 groups of train and 4 groups of even and 4 groups of odd, etc etc
", "votes": 0, "replies": [] }, { "id": 937542, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:31:59.910000", "content": "If you do an experiment with external data and want to compare it with an experiment without external data, both experiments need the same validation sets. Therefore you always use validation without external data
", "votes": 1, "replies": [] }, { "id": 937545, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T04:32:49.233000", "content": "Thanks for the help @cdeotte 😊
", "votes": 1, "replies": [] }, { "id": 937546, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:32:57.890000", "content": "If you add some random validation data that is very easy to classify, it will not help LB. If you add this easy external data to your validation set, it will give the illusion that your CV increased but it doesn't really.
", "votes": 0, "replies": [] }, { "id": 937555, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T04:40:16.900000", "content": "Ha, didnt realised that. Will change my strategy,By the thanks for the information too
", "votes": 0, "replies": [] }, { "id": 937570, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:52:21.100000", "content": "Experimenting with using external data is good. I was just giving a second reason why we can't add external data to validation folds. Reason 1 is because we want to compare different experiments so all experiments need the same validation set. Reason 2 is if we add external data that is easy to classify to the validation fold, then the overall CV score will artificially increase.
\nBut there is not problem with adding easy external data to the train fold. You can add anything to the train fold (as long as it isn't duplicate images that leak validation information). And it is a good strategy to try adding lots of different things to the training fold.
", "votes": 3, "replies": [] }, { "id": 937732, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T06:24:38.160000", "content": "@cdeotte like you said there is cv and lb gap while using external data since there isnt any images in external data that have similar ones in our competition files.So what i propose is removing those images that doesnt correlate with test images or posting in which external tf-files are there these images are in so that we wont use that while training
", "votes": 0, "replies": [] } ] }, { "id": 937492, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T03:52:29.193000", "content": "Hmm…. this doesn't look right. Did you add more data to the validation folds? Or add validation data to the training folds?
\nWhich data are you training with and which data are you validating with?
", "votes": 0, "replies": [ { "id": 937494, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T03:53:51.863000", "content": "@cdeotte No i can assure that there isnt any validation data in train set\nLike your notebook I added INC2019 = [0,1,0,1,0] and INC2018 = [1,1,1,1,1]
", "votes": 0, "replies": [] }, { "id": 937497, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T03:55:41.463000", "content": "Ok, i'll look into it. Something isn't right
", "votes": 0, "replies": [] }, { "id": 937502, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T03:59:00.147000", "content": "Yes i even thought that the triple stratified data doesnt have any leaks.But this oof report is misleading it.Anyways we really appreciate your hardwork in making this data @cdeotte
", "votes": 0, "replies": [] }, { "id": 937527, "author_name": "", "author_url": "", "post_date": "2020-07-21T04:22:05.613000", "content": "", "votes": 0, "replies": [] }, { "id": 937558, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-21T04:42:44.487000", "content": "I'm still investigating this. I believe it is caused by 2 things. (1) The test images are different than train images and (2) there are only 78 malignant out of 3295 public test images.
\nThis will cause weird difference between CV and LB. For example, let's say your model learns to classify all green images as malignant. Now imagine that there are no green images in train data. Therefore this new behavior will not help CV. What if the test data had 13 green images (out of its 78 malignant images). Then your LB would increase by 0.09 AUC!! ( = 0.50 * 13 / 78)
So your CV could be 0.86 and your LB could be 0.95!!
\nThis isn't so far fetched because 15% of the test images are different than the train images. 15% of test images come from a source of size 1920x1080 and there are not train images that come from 1920x1080. And these images have different colors than rest of train as explained here
Thanks. 21 days to merger and 28 days to deadline. Plenty of time to figure out the CV LB problem.
", "votes": 0, "replies": [] }, { "id": 937592, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T05:10:35.663000", "content": "Thanks to people like @cdeotte hope cv lb problem will be resolved by that time😄
", "votes": 1, "replies": [] } ] }, { "id": 937488, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-07-21T03:48:56.033000", "content": "Did you try submitting it?
", "votes": 0, "replies": [ { "id": 937489, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-21T03:49:40.533000", "content": "Yes but the LB Score is 0.9227
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "937485": "I have trained the @cdeotte triple stratified 192 data on effnet b2\nBut while training im getting cv score like this\n\n\n\n\n\n\n\n\n\n\n\nAnd the overall oof score is \n\n\nValidation score is getting boosted to 0.972\nLooking at some public notebook,Is there any chance that the cv score will get high uto 0.972 in chriss 192x192 data??\n\nThis is my fold data report\n\n\n\n\n\nSubmitting this file got me an LB Score of : 0.9227\n\n\n", "937521": "What does \"Number Train file at fold 1 : 40\" mean? If I fork my TFRecord notebook and use your setttings of 192x129, EffNetB2, Inc2019=1, and Inc2018=1, i get the following result (Note this fold 2 is your fold 1)\n\n\n\nAre you writing your code using the JPEGs?", "937492": "Hmm.... this doesn't look right. Did you add more data to the validation folds? Or add validation data to the training folds?\n\nWhich data are you training with and which data are you validating with?", "937488": "Did you try submitting it?" } }