![\"\"](\"https://i.imgur.com/hdof2bl.png\")
Like with most image problems augmentations are quite important to aid in generalization of the model. Especially given that we only have 30k images it is beneficial to help the model learn general characteristics rather than memorizing the inputs.
\nSome reasonable options have been laid out in the starter kernels, but I thought it would be beneficial to measure the effect of applying them by benchmarking against the validation set. What I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time.
\nThis can accomplish 2 things. It can tell us which augmentations are destroying important information and it can also possibly give us candidates for test time augmentation. We want predictions that are still of high quality but have lower correlation with the non-TTA predictions.
\nHere is the list of the augmentations explored. Taken from public kernels.
\n
And here is their respective performance in terms of AUC
\n![\"\"](\"https://i.imgur.com/PDTlf2z.png\")
And loss
\n![\"\"](\"https://i.imgur.com/0NArcgy.png\")
A few of them seem to be particularly destructive. Grid distortion, gaussian noise, median blur, gaussian blur and to a lesser extent several others seem to really hurt validation performance. This may have to do with these augmentations not being seen often enough during training to make the model durable to these kinds of alterations or it could be that these augmentations are too aggressive and simply remove the required information to make this judgement.
", "messages": [ { "id": "1181995", "postDate": "02/02/2021 08:42:48", "content": "Like with most image problems augmentations are quite important to aid in generalization of the model. Especially given that we only have 30k images it is beneficial to help the model learn general characteristics rather than memorizing the inputs.
\nSome reasonable options have been laid out in the starter kernels, but I thought it would be beneficial to measure the effect of applying them by benchmarking against the validation set. What I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time.
\nThis can accomplish 2 things. It can tell us which augmentations are destroying important information and it can also possibly give us candidates for test time augmentation. We want predictions that are still of high quality but have lower correlation with the non-TTA predictions.
\nHere is the list of the augmentations explored. Taken from public kernels.
\n
And here is their respective performance in terms of AUC
\n
And loss
\n
A few of them seem to be particularly destructive. Grid distortion, gaussian noise, median blur, gaussian blur and to a lesser extent several others seem to really hurt validation performance. This may have to do with these augmentations not being seen often enough during training to make the model durable to these kinds of alterations or it could be that these augmentations are too aggressive and simply remove the required information to make this judgement.
", "rawMarkdown": "Like with most image problems augmentations are quite important to aid in generalization of the model. Especially given that we only have 30k images it is beneficial to help the model learn general characteristics rather than memorizing the inputs. \n\nSome reasonable options have been laid out in the starter kernels, but I thought it would be beneficial to measure the effect of applying them by benchmarking against the validation set. What I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time. \n\nThis can accomplish 2 things. It can tell us which augmentations are destroying important information and it can also possibly give us candidates for test time augmentation. We want predictions that are still of high quality but have lower correlation with the non-TTA predictions. \n\nHere is the list of the augmentations explored. Taken from public kernels. \n\n\n\nAnd here is their respective performance in terms of AUC\n\n\n\nAnd loss\n\n\n\nA few of them seem to be particularly destructive. Grid distortion, gaussian noise, median blur, gaussian blur and to a lesser extent several others seem to really hurt validation performance. This may have to do with these augmentations not being seen often enough during training to make the model durable to these kinds of alterations or it could be that these augmentations are too aggressive and simply remove the required information to make this judgement.", "votes": null }, { "id": "1182021", "postDate": "02/02/2021 09:00:29", "content": "Great work!Thanks!
", "rawMarkdown": "Great work!Thanks!", "votes": null }, { "id": "1182098", "postDate": "02/02/2021 09:51:42", "content": "Great Work.
\nWhy have you ignored RandomResizeCrop? Which is a popular augmentation in many of the public notebooks i have seen.
That was just an oversight when copying over the augmentations. I ran that one and it performed very well 0.939 auc. I wouldn't expect it to be particularly problematic because it is one of the operations that is applied 100% of the time during training
", "rawMarkdown": "That was just an oversight when copying over the augmentations. I ran that one and it performed very well 0.939 auc. I wouldn't expect it to be particularly problematic because it is one of the operations that is applied 100% of the time during training", "votes": null }, { "id": "1182306", "postDate": "02/02/2021 12:04:55", "content": "Cool. Thanks for the reply
", "rawMarkdown": "Cool. Thanks for the reply", "votes": null }, { "id": "1184971", "postDate": "02/03/2021 23:18:03", "content": "\n\nWhat I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time.
\n
When you say 'take a single model from a single fold', you mean to take a model trained on 4/5th of the data, and use the remaining 1/5th of data as the validation set?
\nWas the original model trained on augmented data? I would assume that if the model is trained on the same type of augmentations it will see during validation, that will help avoid 'destroying important information'.
", "rawMarkdown": "> What I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time.\n\nWhen you say 'take a single model from a single fold', you mean to take a model trained on 4/5th of the data, and use the remaining 1/5th of data as the validation set?\n\nWas the original model trained on augmented data? I would assume that if the model is trained on the same type of augmentations it will see during validation, that will help avoid 'destroying important information'.", "votes": null }, { "id": "1185067", "postDate": "02/04/2021 00:29:17", "content": "Yes, that is what I did. And yes the model was trained on these augmentations.
\nThat is what is so telling that information has been destroyed or at least the augmented version is teaching something at odds with the reality of the images.
", "rawMarkdown": "Yes, that is what I did. And yes the model was trained on these augmentations.\n\nThat is what is so telling that information has been destroyed or at least the augmented version is teaching something at odds with the reality of the images.", "votes": null }, { "id": "1194893", "postDate": "02/10/2021 12:51:21", "content": "@ryches Cool viz. It can be clearly seen that GridDistortion and MedianBlur have the lowest AUC score relative to the other augmentations. And that effect is magnified in the loss graph. What loss function did you use here?
", "rawMarkdown": "ryches Cool viz. It can be clearly seen that GridDistortion and MedianBlur have the lowest AUC score relative to the other augmentations. And that effect is magnified in the loss graph. What loss function did you use here?", "votes": null }, { "id": "1195237", "postDate": "02/10/2021 16:19:43", "content": "Binary cross entropy was the loss I used
", "rawMarkdown": "Binary cross entropy was the loss I used", "votes": null }, { "id": "1197079", "postDate": "02/11/2021 23:21:23", "content": "Thanks for sharing. Really neat idea for quickly testing augmentations. Will definitely be adding this to my workflows :)
", "rawMarkdown": "Thanks for sharing. Really neat idea for quickly testing augmentations. Will definitely be adding this to my workflows :)", "votes": null }, { "id": "1210884", "postDate": "02/19/2021 19:39:30", "content": "Thanks @ryches !! I am new in Kaggle. Can you provide me code how you calculated AUC for each augmentations.
\nThanks.
Haaa😄 Cool
", "rawMarkdown": "Haaa😄 Cool", "votes": null }, { "id": "1211640", "postDate": "02/20/2021 12:00:37", "content": "i add to add that sometimes destructive augmentation is useful too.
\ni read a paper (but forget the title for the time being) that you can treat extreme (and bad) augmentation as unlabelled or noisy labelled. Then you can use pseudo labels on them.
follow up on this:
\nhttps://www.youtube.com/watch?v=K-1mN2mz66k
\nNegative Data Augmentation
\" Negative Data Augmentation, a strategy for using label-corrupting, rather than label-preserving transformations in Deep Learning.\"
\nthis uses destructive augmentation
", "rawMarkdown": "follow up on this:\nhttps://www.youtube.com/watch?v=K-1mN2mz66k\nNegative Data Augmentation\n\n\" Negative Data Augmentation, a strategy for using label-corrupting, rather than label-preserving transformations in Deep Learning.\"\n\nthis uses destructive augmentation", "votes": null } ], "comments": [ { "id": 1182021, "author_name": "bcwang", "author_url": "", "post_date": "02/02/2021 09:00:29", "content": "Great work!Thanks!
", "votes": null, "replies": [] }, { "id": 1182098, "author_name": "aravindpadman", "author_url": "", "post_date": "02/02/2021 09:51:42", "content": "Great Work.
\nWhy have you ignored RandomResizeCrop? Which is a popular augmentation in many of the public notebooks i have seen.
That was just an oversight when copying over the augmentations. I ran that one and it performed very well 0.939 auc. I wouldn't expect it to be particularly problematic because it is one of the operations that is applied 100% of the time during training
", "votes": null, "replies": [] }, { "id": 1182306, "author_name": "aravindpadman", "author_url": "", "post_date": "02/02/2021 12:04:55", "content": "Cool. Thanks for the reply
", "votes": null, "replies": [] }, { "id": 1210885, "author_name": "adg1822", "author_url": "", "post_date": "02/19/2021 19:41:07", "content": "Haaa😄 Cool
", "votes": null, "replies": [] } ] }, { "id": 1184971, "author_name": "gianlucarossi", "author_url": "", "post_date": "02/03/2021 23:18:03", "content": "\n\nWhat I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time.
\n
When you say 'take a single model from a single fold', you mean to take a model trained on 4/5th of the data, and use the remaining 1/5th of data as the validation set?
\nWas the original model trained on augmented data? I would assume that if the model is trained on the same type of augmentations it will see during validation, that will help avoid 'destroying important information'.
", "votes": null, "replies": [ { "id": 1185067, "author_name": "ryches", "author_url": "", "post_date": "02/04/2021 00:29:17", "content": "Yes, that is what I did. And yes the model was trained on these augmentations.
\nThat is what is so telling that information has been destroyed or at least the augmented version is teaching something at odds with the reality of the images.
", "votes": null, "replies": [] } ] }, { "id": 1194893, "author_name": "prvnkmr", "author_url": "", "post_date": "02/10/2021 12:51:21", "content": "@ryches Cool viz. It can be clearly seen that GridDistortion and MedianBlur have the lowest AUC score relative to the other augmentations. And that effect is magnified in the loss graph. What loss function did you use here?
", "votes": null, "replies": [ { "id": 1195237, "author_name": "ryches", "author_url": "", "post_date": "02/10/2021 16:19:43", "content": "Binary cross entropy was the loss I used
", "votes": null, "replies": [] } ] }, { "id": 1197079, "author_name": "anjum48", "author_url": "", "post_date": "02/11/2021 23:21:23", "content": "Thanks for sharing. Really neat idea for quickly testing augmentations. Will definitely be adding this to my workflows :)
", "votes": null, "replies": [] }, { "id": 1210884, "author_name": "adg1822", "author_url": "", "post_date": "02/19/2021 19:39:30", "content": "Thanks @ryches !! I am new in Kaggle. Can you provide me code how you calculated AUC for each augmentations.
\nThanks.
i add to add that sometimes destructive augmentation is useful too.
\ni read a paper (but forget the title for the time being) that you can treat extreme (and bad) augmentation as unlabelled or noisy labelled. Then you can use pseudo labels on them.
follow up on this:
\nhttps://www.youtube.com/watch?v=K-1mN2mz66k
\nNegative Data Augmentation
\" Negative Data Augmentation, a strategy for using label-corrupting, rather than label-preserving transformations in Deep Learning.\"
\nthis uses destructive augmentation
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1181995": "Like with most image problems augmentations are quite important to aid in generalization of the model. Especially given that we only have 30k images it is beneficial to help the model learn general characteristics rather than memorizing the inputs. \n\nSome reasonable options have been laid out in the starter kernels, but I thought it would be beneficial to measure the effect of applying them by benchmarking against the validation set. What I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time. \n\nThis can accomplish 2 things. It can tell us which augmentations are destroying important information and it can also possibly give us candidates for test time augmentation. We want predictions that are still of high quality but have lower correlation with the non-TTA predictions. \n\nHere is the list of the augmentations explored. Taken from public kernels. \n\n\n\nAnd here is their respective performance in terms of AUC\n\n\n\nAnd loss\n\n\n\nA few of them seem to be particularly destructive. Grid distortion, gaussian noise, median blur, gaussian blur and to a lesser extent several others seem to really hurt validation performance. This may have to do with these augmentations not being seen often enough during training to make the model durable to these kinds of alterations or it could be that these augmentations are too aggressive and simply remove the required information to make this judgement.", "1182021": "Great work!Thanks!", "1182098": "Great Work.\nWhy have you ignored RandomResizeCrop? Which is a popular augmentation in many of the public notebooks i have seen.", "1182116": "That was just an oversight when copying over the augmentations. I ran that one and it performed very well 0.939 auc. I wouldn't expect it to be particularly problematic because it is one of the operations that is applied 100% of the time during training", "1182306": "Cool. Thanks for the reply", "1184971": "> What I did was take a single model from a single fold and do inference on its validation set with each of the augmentations individually turned on and applied 100% of the time.\n\nWhen you say 'take a single model from a single fold', you mean to take a model trained on 4/5th of the data, and use the remaining 1/5th of data as the validation set?\n\nWas the original model trained on augmented data? I would assume that if the model is trained on the same type of augmentations it will see during validation, that will help avoid 'destroying important information'.", "1185067": "Yes, that is what I did. And yes the model was trained on these augmentations.\n\nThat is what is so telling that information has been destroyed or at least the augmented version is teaching something at odds with the reality of the images.", "1194893": "ryches Cool viz. It can be clearly seen that GridDistortion and MedianBlur have the lowest AUC score relative to the other augmentations. And that effect is magnified in the loss graph. What loss function did you use here?", "1195237": "Binary cross entropy was the loss I used", "1197079": "Thanks for sharing. Really neat idea for quickly testing augmentations. Will definitely be adding this to my workflows :)", "1210884": "Thanks @ryches !! I am new in Kaggle. Can you provide me code how you calculated AUC for each augmentations.\nThanks.", "1210885": "Haaa😄 Cool", "1211640": "i add to add that sometimes destructive augmentation is useful too.\ni read a paper (but forget the title for the time being) that you can treat extreme (and bad) augmentation as unlabelled or noisy labelled. Then you can use pseudo labels on them.", "1235746": "follow up on this:\nhttps://www.youtube.com/watch?v=K-1mN2mz66k\nNegative Data Augmentation\n\n\" Negative Data Augmentation, a strategy for using label-corrupting, rather than label-preserving transformations in Deep Learning.\"\n\nthis uses destructive augmentation" }, "source": "meta" }