I have seen a few notebooks not doing transforms during validation, and just using them during train, oof, and test. What do you find is working best using the same transforms all across the board or do you do something different for some of the phases?
\nMy personal workflow is I augment train, oof, and test but not validation. I do put validation through Normalization though, and I am wondering if people use things like color_constancy on validation as well, since that is sort of normalization as opposed to an augmentation.
I have seen a few notebooks not doing transforms during validation, and just using them during train, oof, and test. What do you find is working best using the same transforms all across the board or do you do something different for some of the phases?
\nMy personal workflow is I augment train, oof, and test but not validation. I do put validation through Normalization though, and I am wondering if people use things like color_constancy on validation as well, since that is sort of normalization as opposed to an augmentation.
No. when making OOF or predicting TEST, I don't apply transforms
\ntransforms are applied only for training
@deepkim so you don't do TTA. TTA seems to help at least for me.
", "rawMarkdown": "deepkim so you don't do TTA. TTA seems to help at least for me.", "votes": null }, { "id": "970095", "postDate": "08/14/2020 07:37:33", "content": "No, TTA really is helping, I am using train_transforms on train, test and oof while validation_transforms on validation.
@sarques how do your validation transforms differ? just normalization of some sort?
", "rawMarkdown": "sarques how do your validation transforms differ? just normalization of some sort?", "votes": null }, { "id": "970108", "postDate": "08/14/2020 07:59:22", "content": "I added a bit of Flip in validation with normalization, it gave a little bit better results than without it. In train, I am using RandomResize, Flip, Hair Augmentation etc.
", "rawMarkdown": "I added a bit of Flip in validation with normalization, it gave a little bit better results than without it. In train, I am using RandomResize, Flip, Hair Augmentation etc.", "votes": null }, { "id": "970611", "postDate": "08/14/2020 15:55:04", "content": "No. I use less transforms in TTA than in training.
", "rawMarkdown": "No. I use less transforms in TTA than in training.", "votes": null }, { "id": "970852", "postDate": "08/14/2020 22:04:02", "content": "@sarques So I assume you have tried to use your train transforms on validation? My own experience is that using the same transforms on validation as on train gives the best result. Noticeably, as in you can see in the first few epochs how it's working. I tried to just do normalization on validation, and some other things like color constancy, but it was never as good as using the same transform set.
", "rawMarkdown": "sarques So I assume you have tried to use your train transforms on validation? My own experience is that using the same transforms on validation as on train gives the best result. Noticeably, as in you can see in the first few epochs how it's working. I tried to just do normalization on validation, and some other things like color constancy, but it was never as good as using the same transform set.", "votes": null }, { "id": "970866", "postDate": "08/14/2020 22:49:12", "content": "what's the difference between your oof and your validation set?
", "rawMarkdown": "what's the difference between your oof and your validation set?", "votes": null }, { "id": "970870", "postDate": "08/14/2020 22:59:02", "content": "@optimo not sure who you are asking. My OOF uses same transforms as train and validation. Only difference is OOF is using TTA.
", "rawMarkdown": "optimo not sure who you are asking. My OOF uses same transforms as train and validation. Only difference is OOF is using TTA.", "votes": null }, { "id": "970882", "postDate": "08/15/2020 00:03:12", "content": "I have the same question as Optimo. When we say CV score we means the score obtained on OOF predictions. You make a difference between CV and OOF where no one else does AFAIK. Please enlighten us.
", "rawMarkdown": "I have the same question as Optimo. When we say CV score we means the score obtained on OOF predictions. You make a difference between CV and OOF where no one else does AFAIK. Please enlighten us.", "votes": null }, { "id": "970886", "postDate": "08/15/2020 00:28:35", "content": "I think topic author has 3 different predictions
\nIn this case oof predictions metric differ from validation part (as validation part during training doesn't have augmentations - to make training/validation faster and less memory consuming, and doesn't have \"internal\" TTA blending)
\n\n\nYou make a difference between CV and OOF where no one else does AFAIK
\n
CV should \"mimic\" test set part - and in this case, yes CV score should be considered with TTA (if you use it for Test part predictions).
", "rawMarkdown": "I think topic author has 3 different predictions\n- OOF with TTA \n- Validation set (by verbose option active)\n- Test set (public part)\n\nIn this case oof predictions metric differ from validation part (as validation part during training doesn't have augmentations - to make training/validation faster and less memory consuming, and doesn't have \"internal\" TTA blending)\n\n---\n\n> You make a difference between CV and OOF where no one else does AFAIK\n\nCV should \"mimic\" test set part - and in this case, yes CV score should be considered with TTA (if you use it for Test part predictions).", "votes": null }, { "id": "970896", "postDate": "08/15/2020 00:48:14", "content": "I guess I would call this validation my \"early stopping metric\", but I do understand why you make such a distinction
", "rawMarkdown": "I guess I would call this validation my \"early stopping metric\", but I do understand why you make such a distinction", "votes": null }, { "id": "970910", "postDate": "08/15/2020 02:02:59", "content": "I do train, validation, OOF+TTA, test+TTA
", "rawMarkdown": "I do train, validation, OOF+TTA, test+TTA", "votes": null }, { "id": "970912", "postDate": "08/15/2020 02:03:35", "content": "Yes I use it for finding my best model/early stopping
", "rawMarkdown": "Yes I use it for finding my best model/early stopping", "votes": null }, { "id": "970913", "postDate": "08/15/2020 02:04:30", "content": "Same, when I refer to CV I mean OOF
", "rawMarkdown": "Same, when I refer to CV I mean OOF", "votes": null }, { "id": "970984", "postDate": "08/15/2020 04:23:59", "content": "Ohh, yes, sorry for the bad explanation. So you are doing normalization and color constancy on validation and it somehow is not giving better results, right? Since you said using the same transform set for both train and validation is giving better results, I guess we need same transforms for all three, i.e., train, oof and test.
", "rawMarkdown": "Ohh, yes, sorry for the bad explanation. So you are doing normalization and color constancy on validation and it somehow is not giving better results, right? Since you said using the same transform set for both train and validation is giving better results, I guess we need same transforms for all three, i.e., train, oof and test.", "votes": null } ], "comments": [ { "id": 969930, "author_name": "deepkim", "author_url": "", "post_date": "08/14/2020 04:28:28", "content": "No. when making OOF or predicting TEST, I don't apply transforms
\ntransforms are applied only for training
@deepkim so you don't do TTA. TTA seems to help at least for me.
", "votes": null, "replies": [] }, { "id": 970095, "author_name": "sarques", "author_url": "", "post_date": "08/14/2020 07:37:33", "content": "No, TTA really is helping, I am using train_transforms on train, test and oof while validation_transforms on validation.
@sarques how do your validation transforms differ? just normalization of some sort?
", "votes": null, "replies": [] }, { "id": 970108, "author_name": "sarques", "author_url": "", "post_date": "08/14/2020 07:59:22", "content": "I added a bit of Flip in validation with normalization, it gave a little bit better results than without it. In train, I am using RandomResize, Flip, Hair Augmentation etc.
", "votes": null, "replies": [] }, { "id": 970852, "author_name": "brianfeeny", "author_url": "", "post_date": "08/14/2020 22:04:02", "content": "@sarques So I assume you have tried to use your train transforms on validation? My own experience is that using the same transforms on validation as on train gives the best result. Noticeably, as in you can see in the first few epochs how it's working. I tried to just do normalization on validation, and some other things like color constancy, but it was never as good as using the same transform set.
", "votes": null, "replies": [] }, { "id": 970984, "author_name": "sarques", "author_url": "", "post_date": "08/15/2020 04:23:59", "content": "Ohh, yes, sorry for the bad explanation. So you are doing normalization and color constancy on validation and it somehow is not giving better results, right? Since you said using the same transform set for both train and validation is giving better results, I guess we need same transforms for all three, i.e., train, oof and test.
", "votes": null, "replies": [] } ] }, { "id": 970611, "author_name": "bfishh", "author_url": "", "post_date": "08/14/2020 15:55:04", "content": "No. I use less transforms in TTA than in training.
", "votes": null, "replies": [] }, { "id": 970866, "author_name": "optimo", "author_url": "", "post_date": "08/14/2020 22:49:12", "content": "what's the difference between your oof and your validation set?
", "votes": null, "replies": [ { "id": 970870, "author_name": "brianfeeny", "author_url": "", "post_date": "08/14/2020 22:59:02", "content": "@optimo not sure who you are asking. My OOF uses same transforms as train and validation. Only difference is OOF is using TTA.
", "votes": null, "replies": [] }, { "id": 970882, "author_name": "cpmpml", "author_url": "", "post_date": "08/15/2020 00:03:12", "content": "I have the same question as Optimo. When we say CV score we means the score obtained on OOF predictions. You make a difference between CV and OOF where no one else does AFAIK. Please enlighten us.
", "votes": null, "replies": [ { "id": 970913, "author_name": "brianfeeny", "author_url": "", "post_date": "08/15/2020 02:04:30", "content": "Same, when I refer to CV I mean OOF
", "votes": null, "replies": [] } ] }, { "id": 970886, "author_name": "kyakovlev", "author_url": "", "post_date": "08/15/2020 00:28:35", "content": "I think topic author has 3 different predictions
\nIn this case oof predictions metric differ from validation part (as validation part during training doesn't have augmentations - to make training/validation faster and less memory consuming, and doesn't have \"internal\" TTA blending)
\n\n\nYou make a difference between CV and OOF where no one else does AFAIK
\n
CV should \"mimic\" test set part - and in this case, yes CV score should be considered with TTA (if you use it for Test part predictions).
", "votes": null, "replies": [ { "id": 970910, "author_name": "brianfeeny", "author_url": "", "post_date": "08/15/2020 02:02:59", "content": "I do train, validation, OOF+TTA, test+TTA
", "votes": null, "replies": [] } ] }, { "id": 970896, "author_name": "optimo", "author_url": "", "post_date": "08/15/2020 00:48:14", "content": "I guess I would call this validation my \"early stopping metric\", but I do understand why you make such a distinction
", "votes": null, "replies": [ { "id": 970912, "author_name": "brianfeeny", "author_url": "", "post_date": "08/15/2020 02:03:35", "content": "Yes I use it for finding my best model/early stopping
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "969790": "I have seen a few notebooks not doing transforms during validation, and just using them during train, oof, and test. What do you find is working best using the same transforms all across the board or do you do something different for some of the phases?\n\nMy personal workflow is I augment train, oof, and test but not validation. I do put validation through Normalization though, and I am wondering if people use things like `color_constancy` on validation as well, since that is sort of normalization as opposed to an augmentation.", "969930": "No. when making OOF or predicting TEST, I don't apply transforms\ntransforms are applied only for training", "969935": "deepkim so you don't do TTA. TTA seems to help at least for me.", "970095": "No, TTA really is helping, I am using `train_transforms` on train, test and oof while `validation_transforms` on validation.", "970103": "sarques how do your validation transforms differ? just normalization of some sort?", "970108": "I added a bit of Flip in validation with normalization, it gave a little bit better results than without it. In train, I am using RandomResize, Flip, Hair Augmentation etc.", "970611": "No. I use less transforms in TTA than in training.", "970852": "sarques So I assume you have tried to use your train transforms on validation? My own experience is that using the same transforms on validation as on train gives the best result. Noticeably, as in you can see in the first few epochs how it's working. I tried to just do normalization on validation, and some other things like color constancy, but it was never as good as using the same transform set.", "970866": "what's the difference between your oof and your validation set?", "970870": "optimo not sure who you are asking. My OOF uses same transforms as train and validation. Only difference is OOF is using TTA.", "970882": "I have the same question as Optimo. When we say CV score we means the score obtained on OOF predictions. You make a difference between CV and OOF where no one else does AFAIK. Please enlighten us.", "970886": "I think topic author has 3 different predictions\n- OOF with TTA \n- Validation set (by verbose option active)\n- Test set (public part)\n\nIn this case oof predictions metric differ from validation part (as validation part during training doesn't have augmentations - to make training/validation faster and less memory consuming, and doesn't have \"internal\" TTA blending)\n\n---\n\n> You make a difference between CV and OOF where no one else does AFAIK\n\nCV should \"mimic\" test set part - and in this case, yes CV score should be considered with TTA (if you use it for Test part predictions).", "970896": "I guess I would call this validation my \"early stopping metric\", but I do understand why you make such a distinction", "970910": "I do train, validation, OOF+TTA, test+TTA", "970912": "Yes I use it for finding my best model/early stopping", "970913": "Same, when I refer to CV I mean OOF", "970984": "Ohh, yes, sorry for the bad explanation. So you are doing normalization and color constancy on validation and it somehow is not giving better results, right? Since you said using the same transform set for both train and validation is giving better results, I guess we need same transforms for all three, i.e., train, oof and test." }, "source": "meta" }