I was gathering information about ML/DL on Twitter and found an interesting topic.
\nThe summary is this.
Train a model using intense data augmentation.\nFine tune the model with clean data (i.e. no DA) to improve accuracy.\nThe topic reminded me of the great notebook by @underwearfitting.
\nThis training notebook has just been used with intense augmentation.
\nAnd weights are also available.
\nI added a few tricks to this notebook and fine-tune some epochs without DA.
\nAs a result, the AUC increased from 0.9528 to 0.9548 (in fold0).
\nBut I haven't tried to see if it works for all folds.
\nYou can Try if you have a model that is trained with strong augmentation.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2294613%2Ffd5be348106d7a7b361adbc790e01fc8%2F2021-01-19.png?generation=1611046617592541&alt=media\")
This paper seems to be reference.
\nData Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
[1/21 updated]
\nI experimented with all the folds.
\nThe results are shown below.
| fold | \nbase weight | \nRefined with no DA | \n
|---|---|---|
| fold 0 | \n0.9528 | \n0.9548 | \n
| fold 1 | \n0.9551 | \n0.9545 | \n
| fold 2 | \n0.9552 | \n0.9557 | \n
| fold 3 | \n0.9572 | \n0.9572 | \n
| fold 4 | \n0.9543 | \n0.9548 | \n
If I have more time and left GPU quota, I would like to change the optimizer, lr, and others for further experiments.
", "messages": [ { "id": "1159448", "postDate": "01/19/2021 09:06:05", "content": "I was gathering information about ML/DL on Twitter and found an interesting topic.
\nThe summary is this.
Train a model using intense data augmentation.\nFine tune the model with clean data (i.e. no DA) to improve accuracy.\nThe topic reminded me of the great notebook by @underwearfitting.
\nThis training notebook has just been used with intense augmentation.
\nAnd weights are also available.
\nI added a few tricks to this notebook and fine-tune some epochs without DA.
\nAs a result, the AUC increased from 0.9528 to 0.9548 (in fold0).
\nBut I haven't tried to see if it works for all folds.
\nYou can Try if you have a model that is trained with strong augmentation.
\n
This paper seems to be reference.
\nData Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
[1/21 updated]
\nI experimented with all the folds.
\nThe results are shown below.
| fold | \nbase weight | \nRefined with no DA | \n
|---|---|---|
| fold 0 | \n0.9528 | \n0.9548 | \n
| fold 1 | \n0.9551 | \n0.9545 | \n
| fold 2 | \n0.9552 | \n0.9557 | \n
| fold 3 | \n0.9572 | \n0.9572 | \n
| fold 4 | \n0.9543 | \n0.9548 | \n
If I have more time and left GPU quota, I would like to change the optimizer, lr, and others for further experiments.
", "rawMarkdown": "I was gathering information about ML/DL on Twitter and found an interesting topic.\nThe summary is this.\n```\nTrain a model using intense data augmentation.\nFine tune the model with clean data (i.e. no DA) to improve accuracy.\n```\nThe topic reminded me of the [great notebook](https://www.kaggle.com/underwearfitting/single-fold-training-of-resnet200d-lb0-965) by @underwearfitting.\nThis training notebook has just been used with intense augmentation.\nAnd [weights are also available](https://www.kaggle.com/underwearfitting/resnet200d-baseline-benchmark-public).\nI added a few tricks to this notebook and fine-tune some epochs without DA.\nAs a result, the AUC increased from 0.9528 to 0.9548 (in fold0).\nBut I haven't tried to see if it works for all folds.\nYou can Try if you have a model that is trained with strong augmentation.\n\n\nThis paper seems to be reference.\n[Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data](https://arxiv.org/abs/1909.09148)\n\n[1/21 updated]\nI experimented with all the folds.\nThe results are shown below.\n| fold | [base weight](https://www.kaggle.com/underwearfitting/resnet200d-baseline-benchmark-public) | Refined with no DA |\n| --- | --- |\n| fold 0 | 0.9528 | **0.9548** |\n| fold 1 | 0.9551 | 0.9545 |\n| fold 2 | 0.9552 | **0.9557** |\n| fold 3 | 0.9572 | 0.9572 |\n| fold 4 | 0.9543 | **0.9548** | \n\nIf I have more time and left GPU quota, I would like to change the optimizer, lr, and others for further experiments.", "votes": null }, { "id": "1159507", "postDate": "01/19/2021 10:15:06", "content": "Very interesting idea. Thanks for sharing! Will def try it.
", "rawMarkdown": "Very interesting idea. Thanks for sharing! Will def try it.", "votes": null }, { "id": "1159682", "postDate": "01/19/2021 11:50:30", "content": "Hi,great idea,you mean is to load the trained model using augumentation and train again without augumentation?By the way,did you retrain the model use the lr is not change?Thanks!
", "rawMarkdown": "Hi,great idea,you mean is to load the trained model using augumentation and train again without augumentation?By the way,did you retrain the model use the lr is not change?Thanks!", "votes": null }, { "id": "1159753", "postDate": "01/19/2021 12:54:03", "content": "Your perception is correct.
\nBut lr should be a little smaller.
Thanks for your reply! I see you from 0.952 to 0.954,you use the origin notebook can get 0.952 score?Thanks!
", "rawMarkdown": "Thanks for your reply! I see you from 0.952 to 0.954,you use the origin notebook can get 0.952 score?Thanks!", "votes": null }, { "id": "1161682", "postDate": "01/20/2021 17:21:20", "content": "thank you @itsuki9180 for this interesting idea but when i try to retrain the model with the trained weight with augmentation it gives me out of memory error however i decrease batch size to just 1
", "rawMarkdown": "thank you @itsuki9180 for this interesting idea but when i try to retrain the model with the trained weight with augmentation it gives me out of memory error however i decrease batch size to just 1", "votes": null }, { "id": "1174057", "postDate": "01/28/2021 08:29:12", "content": "One more tip, When performing TTA, recommend to use the same augmentation as it in train.
\ne.g. if Horizontal flip in TTA, leave the H-flip augmentation in the training.
Great work! Did you use the annotation dataset or the normal image dataset? Thanks for the great analysis!
", "rawMarkdown": "Great work! Did you use the annotation dataset or the normal image dataset? Thanks for the great analysis!", "votes": null }, { "id": "1220222", "postDate": "02/27/2021 18:33:33", "content": "I can confirm that this increases CV for 10 of my models. Best increase was 96.17 -> 96.34. Thank you for sharing.
Very interesting idea. Thanks for sharing! Will def try it.
", "votes": null, "replies": [] }, { "id": 1159682, "author_name": "bcwang", "author_url": "", "post_date": "01/19/2021 11:50:30", "content": "Hi,great idea,you mean is to load the trained model using augumentation and train again without augumentation?By the way,did you retrain the model use the lr is not change?Thanks!
", "votes": null, "replies": [ { "id": 1159753, "author_name": "itsuki9180", "author_url": "", "post_date": "01/19/2021 12:54:03", "content": "Your perception is correct.
\nBut lr should be a little smaller.
Thanks for your reply! I see you from 0.952 to 0.954,you use the origin notebook can get 0.952 score?Thanks!
", "votes": null, "replies": [] } ] }, { "id": 1161682, "author_name": "mohamed3abdelrazik", "author_url": "", "post_date": "01/20/2021 17:21:20", "content": "thank you @itsuki9180 for this interesting idea but when i try to retrain the model with the trained weight with augmentation it gives me out of memory error however i decrease batch size to just 1
", "votes": null, "replies": [] }, { "id": 1174057, "author_name": "itsuki9180", "author_url": "", "post_date": "01/28/2021 08:29:12", "content": "One more tip, When performing TTA, recommend to use the same augmentation as it in train.
\ne.g. if Horizontal flip in TTA, leave the H-flip augmentation in the training.
Great work! Did you use the annotation dataset or the normal image dataset? Thanks for the great analysis!
", "votes": null, "replies": [] }, { "id": 1220222, "author_name": "tuckerarrants", "author_url": "", "post_date": "02/27/2021 18:33:33", "content": "I can confirm that this increases CV for 10 of my models. Best increase was 96.17 -> 96.34. Thank you for sharing.