When applying custom loss function, fancy Data augmentation (mixup, fmix etc.) or other custom training stuffs, I think it's better to do not use them in the first few epochs and one, instead, should let the vanilla loss function, augmentation, etc. learn general features and basic representations from the data.
\nThen, you can use your custom stuffs upon the epoch threshold
\nfor epoch in range(num_epochs):\n Do basic stuffs\n if epoch > epoch_threshold:\n Do Custom stuffs\nThis improved both CV and LB of my model
", "messages": [ { "id": "1158403", "postDate": "01/18/2021 14:53:41", "content": "When applying custom loss function, fancy Data augmentation (mixup, fmix etc.) or other custom training stuffs, I think it's better to do not use them in the first few epochs and one, instead, should let the vanilla loss function, augmentation, etc. learn general features and basic representations from the data.
\nThen, you can use your custom stuffs upon the epoch threshold
\nfor epoch in range(num_epochs):\n Do basic stuffs\n if epoch > epoch_threshold:\n Do Custom stuffs\nThis improved both CV and LB of my model
", "rawMarkdown": "When applying custom loss function, fancy Data augmentation (mixup, fmix etc.) or other custom training stuffs, I think it's better to do not use them in the first few epochs and one, instead, should let the vanilla loss function, augmentation, etc. learn general features and basic representations from the data. \n\nThen, you can use your custom stuffs upon the epoch threshold\n\n```\nfor epoch in range(num_epochs):\n Do basic stuffs\n if epoch > epoch_threshold:\n Do Custom stuffs\n```\n\n**This improved both CV and LB of my model**", "votes": null }, { "id": "1158454", "postDate": "01/18/2021 15:16:15", "content": "Good morning Serigne.
\nDid it change dramatically your CV or leaderboar score ?
\nAnd if so, how did you come up with this intuition ?
\nSorry in advance for the noob questions
great idea
\nyou are great @serigne
\ncan't wait to give it a try, thank you
This competition is good because ideas come out constantly. @serigne
", "rawMarkdown": "This competition is good because ideas come out constantly. @serigne", "votes": null }, { "id": "1158491", "postDate": "01/18/2021 15:37:39", "content": "I didn't try for augmentation (I use only torchvision heavy augs so far), but for loss and other custom training stuffs, it improved both CV and LB.
\nHowever I didn't apply it yet to my best model .
\nI do most of my recent experiments in a lighter model (seresneXt50) which is now scoring 0.903 on LB and which will probably beat soon my best model :)
", "rawMarkdown": "I didn't try for augmentation (I use only torchvision heavy augs so far), but for loss and other custom training stuffs, it improved both CV and LB. \n\nHowever I didn't apply it yet to my best model . \n\nI do most of my recent experiments in a lighter model (seresneXt50) which is now scoring 0.903 on LB and which will probably beat soon my best model :)", "votes": null }, { "id": "1158499", "postDate": "01/18/2021 15:43:05", "content": "Do you know if there is an seresneXt model in Tensorflow?
", "rawMarkdown": "Do you know if there is an seresneXt model in Tensorflow?", "votes": null }, { "id": "1158525", "postDate": "01/18/2021 16:04:07", "content": "i found this @ayu055
\nhttps://github.com/HiKapok/TF-SENet
why not do stuff with a linearly decreasing/increasing probability ;)
\nUse a linear scheduler:
\nclass LinearScheduler():\n def __init__(self, start_value, stop_value, nr_steps):\n super(LinearScheduler, self).__init__()\n self.prob = 1\n self.i = 0\n self.drop_values = np.linspace(start=start_value, stop=stop_value, num=int(nr_steps))\n\n def step(self):\n if self.i < len(self.drop_values):\n self.prob = self.drop_values[self.i]\n self.i += 1\nThen set up the sceduler with starting and stoping values:
\nprob_sche=LinearScheduler(1,0,num_of_epochs*len(train_set_loader))
This way the values will decrease from 1 to 0 from the start of the training to the end
\nThen in the training loop:
\nfor batch_idx, (images, target) in enumerate(trainloader):\n th=random.random()\n if prob_sche.prob>=th:\n do_stuff=True \n else:\n do_stuff=False\n\n if do_stuff:\n #do stuff\n else:\n #do other stuff\n\n prob_sche.step()\nThis way it blends your custom stuff during training for example mixup/cutmix! This is just an idea, never tried it :)
", "rawMarkdown": "why not do stuff with a linearly decreasing/increasing probability ;)\n\nUse a linear scheduler: \n\n```\nclass LinearScheduler():\n def __init__(self, start_value, stop_value, nr_steps):\n super(LinearScheduler, self).__init__()\n self.prob = 1\n self.i = 0\n self.drop_values = np.linspace(start=start_value, stop=stop_value, num=int(nr_steps))\n\n def step(self):\n if self.i < len(self.drop_values):\n self.prob = self.drop_values[self.i]\n self.i += 1\n```\nThen set up the sceduler with starting and stoping values:\n`prob_sche=LinearScheduler(1,0,num_of_epochs*len(train_set_loader))`\n\nThis way the values will decrease from 1 to 0 from the start of the training to the end\n\nThen in the training loop:\n\n```\nfor batch_idx, (images, target) in enumerate(trainloader):\n th=random.random()\n if prob_sche.prob>=th:\n do_stuff=True \n else:\n do_stuff=False\n \n if do_stuff:\n #do stuff\n else:\n #do other stuff\n\n prob_sche.step()\n```\nThis way it blends your custom stuff during training for example mixup/cutmix! This is just an idea, never tried it :)", "votes": null }, { "id": "1158747", "postDate": "01/18/2021 18:59:33", "content": "Nice tip @serigne. What was the chosen epoch_threshold for your experiment or any random numbers e.g. 1, 3.. can make the model more generalized and improve the LB?
", "rawMarkdown": "Nice tip @serigne. What was the chosen epoch_threshold for your experiment or any random numbers e.g. 1, 3.. can make the model more generalized and improve the LB?", "votes": null }, { "id": "1158755", "postDate": "01/18/2021 19:04:05", "content": "Thanks. I saw this previously but it is in Tensorflow 1.16. I was looking for a Tensorflow 2 implementation.
", "rawMarkdown": "Thanks. I saw this previously but it is in Tensorflow 1.16. I was looking for a Tensorflow 2 implementation.", "votes": null }, { "id": "1159298", "postDate": "01/19/2021 07:17:07", "content": "Thanks for sharing your idea @serigne
", "rawMarkdown": "Thanks for sharing your idea @serigne", "votes": null }, { "id": "1159418", "postDate": "01/19/2021 08:38:13", "content": "@ayu055 https://github.com/osmr/imgclsmob/tree/master/tensorflow2
\nI will try some models data from this URL
I thought about this as well! It should theoretically help a lot with overfitting in the late stage training of a fold. Gonna test it this week and share the results.
", "rawMarkdown": "I thought about this as well! It should theoretically help a lot with overfitting in the late stage training of a fold. Gonna test it this week and share the results.", "votes": null }, { "id": "1161314", "postDate": "01/20/2021 13:42:34", "content": "Thank you for the contributions towards this comp :) @serigne Learning a lot lmao
", "rawMarkdown": "Thank you for the contributions towards this comp :) @serigne Learning a lot lmao", "votes": null }, { "id": "1162993", "postDate": "01/21/2021 12:46:45", "content": "Ever Since I have started my Kaggle journey , your contributions have been a integral part of my learning
\n@serigne Thanks again
Excellent Idea. Thanks, @serigne for sharing with us.
", "rawMarkdown": "Excellent Idea. Thanks, @serigne for sharing with us.", "votes": null }, { "id": "1163902", "postDate": "01/22/2021 03:27:33", "content": "@serigne this was valuable, thanks. Not only did score improve, got a stable model as well.
\nI would also like to add to this:
\nAfter epoch thresholding, I linearly incremented with each epoch, the probability of applying heavy augs. Little tweak if model is unable to learn further due to this threshold step.
This is an interesting idea. It seems like the opposite of what's suggested by the Mixup Without Hesitation paper. Have there been any formal experiments exploring this?
", "rawMarkdown": "This is an interesting idea. It seems like the opposite of what's suggested by the Mixup Without Hesitation paper. Have there been any formal experiments exploring this?", "votes": null }, { "id": "1167755", "postDate": "01/24/2021 13:16:31", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "1169466", "postDate": "01/25/2021 14:09:21", "content": "Can you please tell how many epochs your models normally took for converging? @serigne
\nWhile working with B4, my model converged in 10-15 epochs.
It depends on your training scheduler.
\nI train with larger number of epochs.
", "rawMarkdown": "It depends on your training scheduler. \n\nI train with larger number of epochs.", "votes": null } ], "comments": [ { "id": 1158454, "author_name": "rizialdi", "author_url": "", "post_date": "01/18/2021 15:16:15", "content": "Good morning Serigne.
\nDid it change dramatically your CV or leaderboar score ?
\nAnd if so, how did you come up with this intuition ?
\nSorry in advance for the noob questions
I didn't try for augmentation (I use only torchvision heavy augs so far), but for loss and other custom training stuffs, it improved both CV and LB.
\nHowever I didn't apply it yet to my best model .
\nI do most of my recent experiments in a lighter model (seresneXt50) which is now scoring 0.903 on LB and which will probably beat soon my best model :)
", "votes": null, "replies": [] }, { "id": 1158499, "author_name": "ayu055", "author_url": "", "post_date": "01/18/2021 15:43:05", "content": "Do you know if there is an seresneXt model in Tensorflow?
", "votes": null, "replies": [] }, { "id": 1158525, "author_name": "mobassir", "author_url": "", "post_date": "01/18/2021 16:04:07", "content": "i found this @ayu055
\nhttps://github.com/HiKapok/TF-SENet
Thanks. I saw this previously but it is in Tensorflow 1.16. I was looking for a Tensorflow 2 implementation.
", "votes": null, "replies": [] }, { "id": 1159418, "author_name": "zhangeng", "author_url": "", "post_date": "01/19/2021 08:38:13", "content": "@ayu055 https://github.com/osmr/imgclsmob/tree/master/tensorflow2
\nI will try some models data from this URL
great idea
\nyou are great @serigne
\ncan't wait to give it a try, thank you
This competition is good because ideas come out constantly. @serigne
", "votes": null, "replies": [] }, { "id": 1158621, "author_name": "yannmajewski", "author_url": "", "post_date": "01/18/2021 17:28:10", "content": "why not do stuff with a linearly decreasing/increasing probability ;)
\nUse a linear scheduler:
\nclass LinearScheduler():\n def __init__(self, start_value, stop_value, nr_steps):\n super(LinearScheduler, self).__init__()\n self.prob = 1\n self.i = 0\n self.drop_values = np.linspace(start=start_value, stop=stop_value, num=int(nr_steps))\n\n def step(self):\n if self.i < len(self.drop_values):\n self.prob = self.drop_values[self.i]\n self.i += 1\nThen set up the sceduler with starting and stoping values:
\nprob_sche=LinearScheduler(1,0,num_of_epochs*len(train_set_loader))
This way the values will decrease from 1 to 0 from the start of the training to the end
\nThen in the training loop:
\nfor batch_idx, (images, target) in enumerate(trainloader):\n th=random.random()\n if prob_sche.prob>=th:\n do_stuff=True \n else:\n do_stuff=False\n\n if do_stuff:\n #do stuff\n else:\n #do other stuff\n\n prob_sche.step()\nThis way it blends your custom stuff during training for example mixup/cutmix! This is just an idea, never tried it :)
", "votes": null, "replies": [ { "id": 1160165, "author_name": "capiru", "author_url": "", "post_date": "01/19/2021 17:55:14", "content": "I thought about this as well! It should theoretically help a lot with overfitting in the late stage training of a fold. Gonna test it this week and share the results.
", "votes": null, "replies": [] } ] }, { "id": 1158747, "author_name": "saurabh2mishra", "author_url": "", "post_date": "01/18/2021 18:59:33", "content": "Nice tip @serigne. What was the chosen epoch_threshold for your experiment or any random numbers e.g. 1, 3.. can make the model more generalized and improve the LB?
", "votes": null, "replies": [] }, { "id": 1159298, "author_name": "saurabhshahane", "author_url": "", "post_date": "01/19/2021 07:17:07", "content": "Thanks for sharing your idea @serigne
", "votes": null, "replies": [] }, { "id": 1161314, "author_name": "reighns", "author_url": "", "post_date": "01/20/2021 13:42:34", "content": "Thank you for the contributions towards this comp :) @serigne Learning a lot lmao
", "votes": null, "replies": [] }, { "id": 1162993, "author_name": "tanulsingh077", "author_url": "", "post_date": "01/21/2021 12:46:45", "content": "Ever Since I have started my Kaggle journey , your contributions have been a integral part of my learning
\n@serigne Thanks again
Excellent Idea. Thanks, @serigne for sharing with us.
", "votes": null, "replies": [] }, { "id": 1163902, "author_name": "albernard", "author_url": "", "post_date": "01/22/2021 03:27:33", "content": "@serigne this was valuable, thanks. Not only did score improve, got a stable model as well.
\nI would also like to add to this:
\nAfter epoch thresholding, I linearly incremented with each epoch, the probability of applying heavy augs. Little tweak if model is unable to learn further due to this threshold step.
This is an interesting idea. It seems like the opposite of what's suggested by the Mixup Without Hesitation paper. Have there been any formal experiments exploring this?
", "votes": null, "replies": [] }, { "id": 1167755, "author_name": "mcarujo", "author_url": "", "post_date": "01/24/2021 13:16:31", "content": "Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 1169466, "author_name": "zaber666", "author_url": "", "post_date": "01/25/2021 14:09:21", "content": "Can you please tell how many epochs your models normally took for converging? @serigne
\nWhile working with B4, my model converged in 10-15 epochs.
It depends on your training scheduler.
\nI train with larger number of epochs.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1158403": "When applying custom loss function, fancy Data augmentation (mixup, fmix etc.) or other custom training stuffs, I think it's better to do not use them in the first few epochs and one, instead, should let the vanilla loss function, augmentation, etc. learn general features and basic representations from the data. \n\nThen, you can use your custom stuffs upon the epoch threshold\n\n```\nfor epoch in range(num_epochs):\n Do basic stuffs\n if epoch > epoch_threshold:\n Do Custom stuffs\n```\n\n**This improved both CV and LB of my model**", "1158454": "Good morning Serigne.\nDid it change dramatically your CV or leaderboar score ?\nAnd if so, how did you come up with this intuition ? \nSorry in advance for the noob questions", "1158463": "great idea\nyou are great @serigne \ncan't wait to give it a try, thank you", "1158485": "This competition is good because ideas come out constantly. @serigne", "1158491": "I didn't try for augmentation (I use only torchvision heavy augs so far), but for loss and other custom training stuffs, it improved both CV and LB. \n\nHowever I didn't apply it yet to my best model . \n\nI do most of my recent experiments in a lighter model (seresneXt50) which is now scoring 0.903 on LB and which will probably beat soon my best model :)", "1158499": "Do you know if there is an seresneXt model in Tensorflow?", "1158525": "i found this @ayu055 \nhttps://github.com/HiKapok/TF-SENet", "1158621": "why not do stuff with a linearly decreasing/increasing probability ;)\n\nUse a linear scheduler: \n\n```\nclass LinearScheduler():\n def __init__(self, start_value, stop_value, nr_steps):\n super(LinearScheduler, self).__init__()\n self.prob = 1\n self.i = 0\n self.drop_values = np.linspace(start=start_value, stop=stop_value, num=int(nr_steps))\n\n def step(self):\n if self.i < len(self.drop_values):\n self.prob = self.drop_values[self.i]\n self.i += 1\n```\nThen set up the sceduler with starting and stoping values:\n`prob_sche=LinearScheduler(1,0,num_of_epochs*len(train_set_loader))`\n\nThis way the values will decrease from 1 to 0 from the start of the training to the end\n\nThen in the training loop:\n\n```\nfor batch_idx, (images, target) in enumerate(trainloader):\n th=random.random()\n if prob_sche.prob>=th:\n do_stuff=True \n else:\n do_stuff=False\n \n if do_stuff:\n #do stuff\n else:\n #do other stuff\n\n prob_sche.step()\n```\nThis way it blends your custom stuff during training for example mixup/cutmix! This is just an idea, never tried it :)", "1158747": "Nice tip @serigne. What was the chosen epoch_threshold for your experiment or any random numbers e.g. 1, 3.. can make the model more generalized and improve the LB?", "1158755": "Thanks. I saw this previously but it is in Tensorflow 1.16. I was looking for a Tensorflow 2 implementation.", "1159298": "Thanks for sharing your idea @serigne", "1159418": "ayu055 [https://github.com/osmr/imgclsmob/tree/master/tensorflow2](url) \nI will try some models data from this URL", "1160165": "I thought about this as well! It should theoretically help a lot with overfitting in the late stage training of a fold. Gonna test it this week and share the results.", "1161314": "Thank you for the contributions towards this comp :) @serigne Learning a lot lmao", "1162993": "Ever Since I have started my Kaggle journey , your contributions have been a integral part of my learning \n@serigne Thanks again", "1163204": "Excellent Idea. Thanks, @serigne for sharing with us.", "1163902": "serigne this was valuable, thanks. Not only did score improve, got a stable model as well.\nI would also like to add to this:\nAfter epoch thresholding, I linearly incremented with each epoch, the probability of applying heavy augs. Little tweak if model is unable to learn further due to this threshold step.", "1164999": "This is an interesting idea. It seems like the opposite of what's suggested by the Mixup Without Hesitation paper. Have there been any formal experiments exploring this?", "1167755": "Thanks for sharing.", "1169466": "Can you please tell how many epochs your models normally took for converging? @serigne \nWhile working with B4, my model converged in 10-15 epochs.", "1169886": "It depends on your training scheduler. \n\nI train with larger number of epochs." }, "source": "meta" }